# policy-gradient/GRPO-Zero

Implementing DeepSeek R1's GRPO algorithm from scratch

Repository: https://github.com/policy-gradient/GRPO-Zero
Canonical: https://ross.abutalabs.com/products/grpo-zero
Language: Python
License: Apache-2.0
License Family: permissive
Last push: 2025-04-18T14:13:35+00:00

## Health v2 (maintenance only)
Score: 27/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 17, release rhythm 35, longevity 36
- inputs: {"age_days": 512, "days_push": 502, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1897, forks 99 (observed 2026-08-28T04:05:51.082982+00:00)

## What it is
A minimal from-scratch Python implementation of DeepSeek's GRPO (Group Relative Policy Optimization) algorithm for reinforcement learning training of large language models, depending only on PyTorch and tokenizers. It includes DAPO improvements like token-level policy gradient loss and KL divergence removal, and runs on a single 24-48GB GPU.

## Use cases
- train an LLM with GRPO reinforcement learning from scratch
- run RLHF-style policy gradient training on a single consumer GPU
- learn how GRPO works by reading a minimal implementation
- fine-tune Qwen2.5 models on reasoning tasks with reinforcement learning
- train LLMs without transformers or vLLM dependencies
- experiment with DAPO improvements to GRPO

## When to choose
- you want a minimal, dependency-light GRPO implementation that fits in 24-48GB VRAM
- you want to understand or modify the GRPO algorithm internals
- you're training small models on verifiable-reward tasks like CountDown

## When to avoid
- you need a production-scale RLHF pipeline for large models
- you want built-in support for many model architectures or distributed training
- you need a turnkey fine-tuning framework with extensive configs

## Facets
- artifact type: library
- maturity: active
- function: llm-training, reinforcement-learning, machine-learning
- domain: large-language-models, reinforcement-learning, deep-learning, machine-learning
- platform: python
- tags: grpo, rlhf, policy-gradient, deepseek-r1, pytorch, from-scratch, low-vram, dapo, gpu, linux

## Member repositories
- policy-gradient/GRPO-Zero (main) score 27

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:51.082982+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:12:48.690969+00:00, confidence not recorded.
  - readme: https://github.com/policy-gradient/GRPO-Zero (fetched 2026-08-28T04:05:51.082982+00:00, sha 8b2c5250a90a)
- Data as of 2026-08-30T08:39:29.467469+00:00.