# NVIDIA/Megatron-LM

Ongoing research training transformer models at scale

Repository: https://github.com/NVIDIA/Megatron-LM
Canonical: https://ross.abutalabs.com/products/megatron-lm
Homepage: https://docs.nvidia.com/megatron-core/developer-guide/latest/get-started/quickstart.html
Language: Python
License: NOASSERTION
License Family: other
Topics: large-language-models, model-para, transformers
Last push: 2026-08-27T00:26:38+00:00

## Health v2 (maintenance only)
Score: 99/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 98, longevity 100
- inputs: {"age_days": 2722, "days_push": 7, "days_rel": 14, "gap_med": 22, "n_releases_24m": 24}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 17615, forks 4417 (observed 2026-08-28T04:11:20.070452+00:00)

## What it is
NVIDIA's GPU-optimized library for training large transformer models at scale, comprising Megatron-LM (reference training scripts) and Megatron Core (composable building blocks with advanced parallelism). It supports tensor, pipeline, data, expert, and context parallelism plus mixed precision including FP8.

## Use cases
- pretrain large language models on multi-GPU clusters
- train LLaMA or DeepSeek style models with tensor and pipeline parallelism
- train mixture-of-experts models at scale
- build a custom LLM training framework from composable blocks
- shard optimizer states to fit large models in GPU memory
- convert Hugging Face checkpoints to Megatron format

## When to choose
- you need to pretrain or fine-tune multi-billion-parameter transformers on NVIDIA GPUs
- you want state-of-the-art parallelism strategies like TP, PP, EP, CP, and FSDP
- you are a research team or framework developer building custom training pipelines

## When to avoid
- you only need inference or serving rather than training
- you lack NVIDIA GPUs or multi-node infrastructure
- you want a simple high-level fine-tuning API with minimal configuration

## Facets
- artifact type: library
- maturity: active
- function: llm-training, deep-learning, machine-learning, gpu-computing
- domain: large-language-models, deep-learning, machine-learning, gpu-computing
- platform: python
- tags: distributed-training, transformer, tensor-parallelism, pipeline-parallelism, mixture-of-experts, fp8, nvidia, linux, gpu, docker

## Member repositories
- NVIDIA/Megatron-LM (main) score 99

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:20.070452+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:02:35.924218+00:00, confidence not recorded.
  - readme: https://github.com/NVIDIA/Megatron-LM (fetched 2026-08-28T04:11:20.070452+00:00, sha 0af414bbe1d2)
  - homepage: https://docs.nvidia.com/megatron-core/developer-guide/latest/get-started/quickstart.html (fetched 2026-08-29T08:00:56.170184+00:00, sha 5c9573c3650f)
  - site_page: https://docs.nvidia.com/megatron-core/developer-guide/latest/get-started/install.html (fetched 2026-08-29T08:00:56.187255+00:00, sha 81a056cf3d49)
  - site_page: https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/moe.html (fetched 2026-08-29T08:00:56.190308+00:00, sha b42f263cae78)
  - site_page: https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/multi_token_prediction.html (fetched 2026-08-29T08:00:56.194236+00:00, sha e438d95423a0)
  - site_page: https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/multi_latent_attention.html (fetched 2026-08-29T08:00:56.196555+00:00, sha a0f116aee748)
  - site_page: https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/context_parallel.html (fetched 2026-08-29T08:00:56.199226+00:00, sha 31506a7ab598)
  - site_page: https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/megatron_fsdp.html (fetched 2026-08-29T08:00:56.201683+00:00, sha f9d63adf0f62)
  - site_page: https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/dist_optimizer.html (fetched 2026-08-29T08:00:56.205234+00:00, sha d48b8f13bfbb)
  - site_page: https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/optimizer_cpu_offload.html (fetched 2026-08-29T08:00:56.207242+00:00, sha df85a4f6130f)
- Data as of 2026-08-30T08:39:29.467469+00:00.