NVIDIA/Megatron-LM

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

17615 stars · 4417 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

NVIDIA's GPU-optimized library for training large transformer models at scale, comprising Megatron-LM (reference training scripts) and Megatron Core (composable building blocks with advanced parallelism). It supports tensor, pipeline, data, expert, and context parallelism plus mixed precision including FP8.

Use cases

• pretrain large language models on multi-GPU clusters
• train LLaMA or DeepSeek style models with tensor and pipeline parallelism
• train mixture-of-experts models at scale
• build a custom LLM training framework from composable blocks
• shard optimizer states to fit large models in GPU memory
• convert Hugging Face checkpoints to Megatron format

When to choose

• you need to pretrain or fine-tune multi-billion-parameter transformers on NVIDIA GPUs
• you want state-of-the-art parallelism strategies like TP, PP, EP, CP, and FSDP
• you are a research team or framework developer building custom training pipelines

When to avoid

• you only need inference or serving rather than training
• you lack NVIDIA GPUs or multi-node infrastructure
• you want a simple high-level fine-tuning API with minimal configuration

Facets

library · maturity active

llm-training deep-learning machine-learning gpu-computing large-language-models deep-learning machine-learning gpu-computing python distributed-training transformer tensor-parallelism pipeline-parallelism mixture-of-experts fp8 nvidia linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="NVIDIA/Megatron-LM")