AnswerDotAI/ModernBERT

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1758 stars · 147 forks observed · 2026-10-07

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

ModernBERT is the research repository for a modernized BERT-family bidirectional encoder trained on 2 trillion tokens with an 8192-token context, featuring FlexBERT modular encoder building blocks and Flash Attention. It contains pre-training and GLUE evaluation code built on MosaicBERT and the Composer framework, with model checkpoints released on HuggingFace.

Use cases

• pretrain a modern BERT-style encoder from scratch
• fine-tune an encoder for text classification
• build single or multi-vector retrieval embeddings
• reproduce ModernBERT GLUE benchmark results
• experiment with modular encoder architecture blocks via YAML configs
• run long-context (8192 token) encoder inference on consumer GPUs

When to choose

• you need a fast, memory-efficient encoder for classification or retrieval
• you want to pre-train or fine-tune a BERT-style model with modern optimizations like Flash Attention
• you need long-context encoder support up to 8192 tokens
• you want reproducible research training code for encoder models

When to avoid

• you just want to use ModernBERT checkpoints in a pipeline - use the HuggingFace integration instead
• you need decoder-only generative LLMs rather than encoder models
• you lack a GPU or cannot install Flash Attention dependencies
• you need a production-hardened, well-documented training framework

Facets

library · maturity active

machine-learning nlp llm-training deep-learning machine-learning deep-learning python bert encoder embeddings transformer flash-attention pretraining research-code natural-language-processing gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="AnswerDotAI/ModernBERT")