# EleutherAI/gpt-neox

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

Repository: https://github.com/EleutherAI/gpt-neox
Canonical: https://ross.abutalabs.com/products/gpt-neox
Homepage: https://www.eleuther.ai/
Language: Python
License: Apache-2.0
License Family: permissive
Topics: deepspeed-library, gpt-3, transformers, language-model
Last push: 2026-06-11T19:25:44+00:00

## Health v2 (maintenance only)
Score: 62/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 87, release rhythm 8, longevity 100
- inputs: {"age_days": 2080, "days_push": 83, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 7459, forks 1119 (observed 2026-08-28T04:09:59.996916+00:00)

## What it is
GPT-NeoX is EleutherAI's library for training large-scale autoregressive transformer language models on GPUs, built on NVIDIA's Megatron and augmented with DeepSpeed techniques. It supports distributed training with ZeRO and 3D parallelism across diverse hardware and cluster schedulers like Slurm, MPI, and IBM Job Step Manager.

## Use cases
- pretrain a multi-billion parameter GPT-style language model from scratch
- run distributed LLM training with ZeRO and 3D parallelism on a GPU cluster
- train a transformer model on a Slurm or MPI supercomputer
- reproduce or extend the Pythia model training setup
- experiment with rotary and alibi positional embeddings at scale
- launch large-scale training runs on AWS, CoreWeave, or national lab supercomputers

## When to choose
- you need to train language models with billions of parameters from scratch
- you want model/data/pipeline parallelism with DeepSpeed ZeRO optimizations
- you need to run training across heterogeneous HPC systems and schedulers
- you want predefined configs for architectures like Pythia, PaLM, Falcon, or LLaMA

## When to avoid
- you only need inference - use Hugging Face transformers instead
- you are fine-tuning small models or working with limited GPU resources
- you want a simple high-level training API rather than large-scale distributed training

## Facets
- artifact type: library
- maturity: active
- function: llm-training, machine-learning, deep-learning, gpu-computing, transformers
- domain: large-language-models, deep-learning, machine-learning, artificial-intelligence, gpu-computing
- platform: python, cloud
- tags: megatron, deepspeed, distributed-training, autoregressive-transformers, hpc, model-parallelism, zero-optimizer, 3d-parallelism, pythia, pretraining, gpu, linux, docker

## Member repositories
- EleutherAI/gpt-neox (main) score 62

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:09:59.996916+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:37:38.783319+00:00, confidence not recorded.
  - readme: https://github.com/EleutherAI/gpt-neox (fetched 2026-08-28T04:09:59.996916+00:00, sha 6606636bdf50)
  - homepage: https://www.eleuther.ai/ (fetched 2026-08-29T08:33:05.911810+00:00, sha 46c7f1a08777)
  - site_page: https://www.eleuther.ai/about (fetched 2026-08-29T08:33:05.924832+00:00, sha 8a099f02f9a1)
  - site_page: https://www.eleuther.ai/releases (fetched 2026-08-29T08:33:05.921225+00:00, sha 6b771c6c6df0)
- Data as of 2026-08-30T08:39:29.467469+00:00.