BigScience

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1448 stars · 226 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

BigScience is a research project training large transformer language models (BERT, GPT-style) at scale, built on a fork of Megatron-LM integrated with DeepSpeed. The repository provides training scripts, documentation, and tooling for distributed multi-GPU pretraining and fine-tuning of large language models.

Use cases

• pretrain large GPT-style language models on GPU clusters
• train BERT models with distributed DeepSpeed training
• fine-tune pretrained transformer language models
• run zero-shot evaluation of large language models
• generate text with pretrained transformer checkpoints
• set up multi-node distributed LLM training pipelines

When to choose

• you need to pretrain or fine-tune large transformer LMs across many GPUs or nodes
• you want Megatron-LM's efficiency combined with DeepSpeed's memory optimizations
• you're reproducing BigScience-style open collaborative LLM training experiments

When to avoid

• you just want to run inference with an existing LLM rather than train one
• you need a maintained, actively developed training framework
• you lack access to NVIDIA GPU clusters or multi-node HPC infrastructure
• you want a simple high-level API like Hugging Face Trainer

Facets

library · maturity maintenance

llm-training deep-learning machine-learning benchmarking large-language-models deep-learning machine-learning gpu-computing python megatron-lm deepspeed distributed-training transformer-models research-project gpt bert hpc linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="bigscience-workshop/Megatron-DeepSpeed")