# OpenLMLab/MOSS-RLHF

Secrets of RLHF in Large Language Models Part I: PPO

Repository: https://github.com/OpenLMLab/MOSS-RLHF
Canonical: https://ross.abutalabs.com/products/moss-rlhf
Language: Python
License: Apache-2.0
License Family: permissive
Topics: rlhf, alignment, ai-safety
Last push: 2024-03-03T04:56:49+00:00

## Health v2 (maintenance only)
Score: 29/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 35, longevity 82
- inputs: {"age_days": 1155, "days_push": 913, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1429, forks 103 (observed 2026-08-28T04:04:42.160985+00:00)

## What it is
MOSS-RLHF is the open-source companion code for the paper 'Secrets of RLHF in Large Language Models Part I: PPO', providing implementations of PPO-based RLHF training and reward model training for large language models. It also releases 7B English and Chinese reward models, SFT and policy models, and a preference-strength-annotated HH-RLHF dataset.

## Use cases
- train an LLM with PPO-based RLHF
- train a reward model from human preference data
- reproduce RLHF alignment experiments from the paper
- download pretrained 7B reward and policy models
- use a cleaned HH-RLHF dataset with preference strength labels
- study reward model strength measurement for alignment research

## When to choose
- you want a research-grade reference implementation of PPO for LLM alignment
- you need reward model training code with an annotated preference dataset
- you are studying RLHF mechanics or writing an alignment paper
- you want 7B English/Chinese reward models to score LLM outputs

## When to avoid
- you need a production-ready, actively maintained RLHF training framework
- you want scalable multi-node training with the latest algorithm variants like GRPO or DPO out of the box
- you need commercial use of the released models (model weights are AGPL-3.0, data is CC BY-NC 4.0)
- you expect frequent updates - the last release was March 2024

## Facets
- artifact type: library
- maturity: maintenance
- function: llm-training, machine-learning, deep-learning, rag
- domain: large-language-models, machine-learning, deep-learning, artificial-intelligence
- platform: python
- tags: rlhf, ppo, reward-model, alignment, ai-safety, research-code, llm-alignment, gpu, linux

## Member repositories
- OpenLMLab/MOSS-RLHF (main) score 29

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:42.160985+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:37:14.033122+00:00, confidence not recorded.
  - readme: https://github.com/OpenLMLab/MOSS-RLHF (fetched 2026-08-28T04:04:42.160985+00:00, sha ddd40fafaf8c)
- Data as of 2026-08-30T08:39:29.467469+00:00.