kingoflolz/mesh-transformer-jax

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

6380 stars · 877 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A JAX/Haiku library implementing model-parallel training and inference of transformer models using xmap/pjit operators, similar to Megatron-LM's parallelism scheme, optimized for TPUs. It is the codebase behind GPT-J-6B, a 6 billion parameter autoregressive language model trained on The Pile.

Use cases

• train large transformer models on TPUs with model parallelism
• run inference with GPT-J-6B
• fine-tune GPT-J-6B on custom datasets
• generate text with a 6B parameter open-source language model
• experiment with Megatron-style tensor parallelism in JAX

When to choose

• you need to train or fine-tune transformers up to ~40B parameters on TPU hardware
• you want to run or fine-tune GPT-J-6B
• you prefer JAX/Haiku over PyTorch for large-scale model parallelism

When to avoid

• you need to scale beyond ~40B parameters (use GPT-NeoX or DeepSpeed instead)
• you are working on GPUs rather than TPUs
• you need a actively developed or general-purpose LLM training framework

Facets

library · maturity maintenance

llm-training llm-inference deep-learning machine-learning large-language-models deep-learning machine-learning gpu-computing python cloud jax haiku model-parallelism tpu gpt-j transformers megatron-style-parallelism gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="kingoflolz/mesh-transformer-jax")