THUDM/SwissArmyTransformer

Health v2 · maintenance only

Adoption not part of the score

1121 stars · 99 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

SwissArmyTransformer (sat) is a PyTorch library for developing custom Transformer model variants where models like BERT, GPT, T5, GLM, and ViT share a common backbone code extended via lightweight mixins. It integrates DeepSpeed ZeRO and model parallelism to support pretraining and finetuning of large models from 100M to 20B parameters.

Use cases

• develop custom transformer model variants
• finetune large pretrained language models
• add prefix-tuning or p-tuning to existing models
• pretrain large models up to 20B parameters
• build autoregressive text generation with beam search
• add classification heads to pretrained transformers

When to choose

• you want to experiment with transformer architecture modifications without rewriting backbone code
• you need efficient pretraining or finetuning of large models with deepspeed and model parallelism
• you want to apply parameter-efficient techniques like prefix-tuning across different model families
• you need cached autoregressive inference for generation tasks

When to avoid

• you need a general-purpose deep learning framework rather than a transformer-specific library
• your models are not transformer-based
• you prefer higher-level abstractions like huggingface transformers for standard finetuning workflows
• you work without gpu resources since the library targets large-scale training

Facets

library · maturity active

llm-training machine-learning deep-learning llm-inference deep-learning large-language-models machine-learning python transformer pytorch pretrained-models model-parallelism deepspeed finetuning mixin-architecture large-models natural-language-processing gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="THUDM/SwissArmyTransformer")