# BigScience

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Repository: https://github.com/bigscience-workshop/Megatron-DeepSpeed
Canonical: https://ross.abutalabs.com/products/bigscience
Language: Python
License: NOASSERTION
License Family: other
Last push: 2024-03-20T16:10:21+00:00

## Health v2 (maintenance only)
Score: 32/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 1888, "days_push": 896, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1448, forks 226 (observed 2026-08-28T04:04:45.483113+00:00)

## What it is
BigScience is a research project training large transformer language models (BERT, GPT-style) at scale, built on a fork of Megatron-LM integrated with DeepSpeed. The repository provides training scripts, documentation, and tooling for distributed multi-GPU pretraining and fine-tuning of large language models.

## Use cases
- pretrain large GPT-style language models on GPU clusters
- train BERT models with distributed DeepSpeed training
- fine-tune pretrained transformer language models
- run zero-shot evaluation of large language models
- generate text with pretrained transformer checkpoints
- set up multi-node distributed LLM training pipelines

## When to choose
- you need to pretrain or fine-tune large transformer LMs across many GPUs or nodes
- you want Megatron-LM's efficiency combined with DeepSpeed's memory optimizations
- you're reproducing BigScience-style open collaborative LLM training experiments

## When to avoid
- you just want to run inference with an existing LLM rather than train one
- you need a maintained, actively developed training framework
- you lack access to NVIDIA GPU clusters or multi-node HPC infrastructure
- you want a simple high-level API like Hugging Face Trainer

## Facets
- artifact type: library
- maturity: maintenance
- function: llm-training, deep-learning, machine-learning, benchmarking
- domain: large-language-models, deep-learning, machine-learning, gpu-computing
- platform: python
- tags: megatron-lm, deepspeed, distributed-training, transformer-models, research-project, gpt, bert, hpc, linux, gpu, docker

## Member repositories
- bigscience-workshop/Megatron-DeepSpeed (main) score 32
- bigscience-workshop/bigscience (docs) score 32

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:45.483113+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:36:02.237951+00:00, confidence not recorded.
  - readme: https://github.com/bigscience-workshop/Megatron-DeepSpeed (fetched 2026-08-28T04:04:45.483113+00:00, sha 4e6019354113)
- Data as of 2026-08-30T08:39:29.467469+00:00.