PKU-Alignment/safe-rlhf

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1611 stars · 133 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Beaver is a modular open-source framework from Peking University for training large language models with SFT, RLHF, and Safe RLHF (constrained reward maximization). It includes a large human-labeled preference dataset (up to 1M pairs) and pre-trained reward/cost model checkpoints.

Use cases

• train an LLM with RLHF
• align a language model with safety constraints
• run supervised fine-tuning on LLaMA or OPT
• train a reward model from human preferences
• train a cost model for harmlessness
• research safe reinforcement learning from human feedback
• evaluate LLM safety with BIG-bench or GPT-4 evaluation

When to choose

• you need a reproducible RLHF or Safe RLHF training pipeline
• you want human-labeled helpfulness and harmlessness preference data
• you are doing alignment research with reward and cost models
• you want pre-trained aligned model checkpoints like Beaver-7B

When to avoid

• you only need to run inference on an existing LLM
• you lack multi-GPU resources for large model training
• you need a general-purpose RL library unrelated to LLM alignment

Facets

framework · maturity active

llm-training machine-learning deep-learning rag large-language-models machine-learning artificial-intelligence deep-learning python rlhf safe-rlhf alignment ai-safety reward-model cost-model sft deepspeed preference-learning llm-alignment gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="PKU-Alignment/safe-rlhf")