Mutable.ai

Here is my detailed explanation based on the code provided:

The configuration files for this codebase can be found under:

• /dolly/config/v100_config.json
• /dolly/config/a10_config.json
• /dolly/config/a100_config.json

These JSON files contain various hyperparameters and configuration options for training the model, such as:

• Whether to use fp16 or bf16 precision
• The optimizer type and parameters
• Scheduler type and parameters
• Zero optimization settings
• Gradient accumulation steps
• Etc.

The training code itself is located in trainer.py under /dolly/training/. This file contains the main train() function that executes model training along with various helper functions for loading datasets, tokenizers, models, preprocessing data, etc.

The hyperparameters for training a particular run can be specified when calling the train() function. For example:

train( input_model ="gpt2", local_output_dir="output_dir", epochs=10, per_device_train_batch_size=8, lr=0.001 )

So in summary, the configuration files under /dolly/config/ contain default hyperparameters that can be overridden when calling train(), and the actual training code lives in trainer.py under /dolly/training/.

Hope this helps! Let me know if you have any other questions.

The full file path for the pipeline.py example file is:

/dolly/examples/pipeline.py

As you can see, this file imports and uses the InstructionTextGenerationPipeline class from the training/generate.py file at:

/dolly/training/generate.py

It also references the train_dolly.py file for details on how to train the model, located at:

/dolly/train_dolly.py

So in summary:

• /dolly/examples/pipeline.py - Executes generation using the pipeline
• /dolly/training/generate.py - Defines the pipeline classes used for generation
• /dolly/train_dolly.py - Details on how to train the model

The pipeline.py file also loads a pretrained model from either HuggingFace or a locally trained model, as described in the README.md file at:

/dolly/README.md

Hope this provides more clarity and context around the file paths and relationships! Let me know if you have any other questions.

The configuration file for the "a100" model is located at:

/dolly/deepspeed/a100_config.json

This configuration file is customized for training the model on Nvidia A100 GPUs, as mentioned in the Getting Started with Training section of the README.