README.md

metadata

license: apache-2.0
base_model: teknium/OpenHermes-2.5-Mistral-7B
tags:
  - fine-tuning
  - dpo
  - arena-dataset
  - peft
  - lora
  - rlhf
datasets:
  - lmarena-ai/arena-human-preference-55k

Mistral-7B DPO Model

This model is a Direct Preference Optimization (DPO) version of teknium/OpenHermes-2.5-Mistral-7B using LoRA on the Arena Human Preference dataset.

Training Details

• Base Model: teknium/OpenHermes-2.5-Mistral-7B
• Dataset: lmarena-ai/arena-human-preference-55k (1000 samples)
• Method: Direct Preference Optimization with LoRA (r=16, alpha=32)
• Training Steps: 100
• Learning Rate: 5e-5
• Beta: 0.1

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B")
tokenizer = AutoTokenizer.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B")

# Load fine-tuned model
model = PeftModel.from_pretrained(base_model, "gCao/mistral-7b-dpo-arena")

# Generate
prompt = "### Instruction:\nExplain machine learning\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)