EleutherAI/gpt-neox

Health v2 · maintenance only

Adoption not part of the score

7459 stars · 1119 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

GPT-NeoX is EleutherAI's library for training large-scale autoregressive transformer language models on GPUs, built on NVIDIA's Megatron and augmented with DeepSpeed techniques. It supports distributed training with ZeRO and 3D parallelism across diverse hardware and cluster schedulers like Slurm, MPI, and IBM Job Step Manager.

Use cases

• pretrain a multi-billion parameter GPT-style language model from scratch
• run distributed LLM training with ZeRO and 3D parallelism on a GPU cluster
• train a transformer model on a Slurm or MPI supercomputer
• reproduce or extend the Pythia model training setup
• experiment with rotary and alibi positional embeddings at scale
• launch large-scale training runs on AWS, CoreWeave, or national lab supercomputers

When to choose

• you need to train language models with billions of parameters from scratch
• you want model/data/pipeline parallelism with DeepSpeed ZeRO optimizations
• you need to run training across heterogeneous HPC systems and schedulers
• you want predefined configs for architectures like Pythia, PaLM, Falcon, or LLaMA

When to avoid

• you only need inference - use Hugging Face transformers instead
• you are fine-tuning small models or working with limited GPU resources
• you want a simple high-level training API rather than large-scale distributed training

Facets

library · maturity active

llm-training machine-learning deep-learning gpu-computing transformers large-language-models deep-learning machine-learning artificial-intelligence gpu-computing python cloud megatron deepspeed distributed-training autoregressive-transformers hpc model-parallelism zero-optimizer 3d-parallelism pythia pretraining gpu linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="EleutherAI/gpt-neox")