microsoft/torchscale

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

3138 stars · 226 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A PyTorch library from Microsoft implementing foundation Transformer architectures such as DeepNet, Magneto, RetNet, LongNet, BitNet, and X-MoE for scaling (multimodal) large language models. It provides encoder, decoder, and encoder-decoder building blocks with a focus on training stability, efficiency, and length extrapolation.

Use cases

• train large transformer models stably at extreme depth
• implement RetNet or LongNet architectures for LLM research
• build multimodal foundation models across language, vision, and speech
• experiment with sparse mixture-of-experts scaling
• create BERT-like encoders or GPT-like decoders in a few lines of code

When to choose

• you need research-grade Transformer architecture variants like RetNet, BitNet, or X-MoE
• you want stable training of very deep or very long-context Transformers
• you are building foundation models spanning multiple modalities

When to avoid

• you just need a production LLM inference server or fine-tuning framework
• you want a high-level API like Hugging Face Transformers with pretrained checkpoints
• you need a non-PyTorch framework such as JAX or TensorFlow

Facets

library · maturity maintenance

deep-learning machine-learning llm-training deep-learning large-language-models computer-vision speech-processing python transformer-architectures foundation-models mixture-of-experts retnet longnet bitnet pytorch research natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoft/torchscale")