--- library_name: peft license: apache-2.0 base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T tags: - axolotl - generated_from_trainer model-index: - name: tinyllama-1.1B_alpaca_2k_lora results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.4.1` ```yaml # Upload the final model to Huggingface hub_model_id: byvuong/tinyllama-1.1B_alpaca_2k_lora # Store the training logs in weights and biases wandb_entity: byvuong-org wandb_project: tinyllama-1.1B_alpaca_2k_lora # The rest of this config stays the same: base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T model_type: LlamaForCausalLM tokenizer_type: LlamaTokenizer load_in_8bit: true load_in_4bit: false strict: false datasets: - path: mhenrichsen/alpaca_2k_test type: alpaca dataset_prepared_path: val_set_size: 0.05 output_dir: ./outputs/lora-out sequence_len: 4096 sample_packing: true eval_sample_packing: false pad_to_sequence_len: true adapter: lora lora_model_dir: lora_r: 32 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true lora_fan_in_fan_out: gradient_accumulation_steps: 4 micro_batch_size: 2 num_epochs: 4 optimizer: adamw_bnb_8bit lr_scheduler: cosine learning_rate: 0.0002 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true warmup_steps: 10 evals_per_epoch: 4 saves_per_epoch: 1 debug: deepspeed: weight_decay: 0.0 fsdp: fsdp_config: special_tokens: ```

# tinyllama-1.1B_alpaca_2k_lora This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) on the None dataset. It achieves the following results on the evaluation set: - Loss: 1.2133 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 4 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 1.4613 | 0.0784 | 1 | 1.4899 | | 1.385 | 0.2353 | 3 | 1.4862 | | 1.3665 | 0.4706 | 6 | 1.4402 | | 1.2691 | 0.7059 | 9 | 1.3409 | | 1.2269 | 0.9412 | 12 | 1.2944 | | 1.2531 | 1.1569 | 15 | 1.2793 | | 1.2266 | 1.3922 | 18 | 1.2552 | | 1.136 | 1.6275 | 21 | 1.2342 | | 1.2704 | 1.8627 | 24 | 1.2297 | | 1.1491 | 2.0784 | 27 | 1.2232 | | 1.1515 | 2.3137 | 30 | 1.2230 | | 1.195 | 2.5490 | 33 | 1.2190 | | 1.1126 | 2.7843 | 36 | 1.2178 | | 1.1511 | 3.0196 | 39 | 1.2138 | | 1.1888 | 3.2353 | 42 | 1.2105 | | 1.1008 | 3.4706 | 45 | 1.2118 | | 1.1896 | 3.7059 | 48 | 1.2133 | ### Framework versions - PEFT 0.13.2 - Transformers 4.45.2 - Pytorch 2.3.1+cu121 - Datasets 3.0.1 - Tokenizers 0.20.1