eric-mitchell/direct-preference-optimization

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2907 stars · 235 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A reference implementation of Direct Preference Optimization (DPO) for training language models from human preference data, built on HuggingFace models. It supports the two-stage pipeline of supervised fine-tuning followed by preference learning, including variants like conservative DPO and IPO.

Use cases

• train a language model from human preference data
• run DPO fine-tuning on a HuggingFace causal LM
• align an LLM without training a separate reward model
• fine-tune Pythia or Llama on Anthropic-HH preference datasets
• compare DPO, conservative DPO, and IPO training losses
• run supervised fine-tuning before preference optimization

When to choose

• you want the canonical reference implementation of the DPO paper
• you need to fine-tune any HuggingFace causal LM with preference data
• you want to experiment with DPO variants like cDPO and IPO
• you need multi-GPU FSDP training for preference optimization

When to avoid

• you need a production training framework with broad ecosystem support like TRL
• you only want to run inference on an already-aligned model
• you need RLHF with an explicit reward model or PPO
• you are not working with language models

Facets

library · maturity stable

llm-training machine-learning deep-learning large-language-models machine-learning reinforcement-learning python dpo preference-optimization rlhf alignment fine-tuning huggingface reference-implementation gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="eric-mitchell/direct-preference-optimization")