allenai/longformer

Health v2 · maintenance only

Adoption not part of the score

2205 stars · 286 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Longformer is a pretrained transformer model family (including the LongformerEncoderDecoder/LED variant) that processes long documents up to 16K tokens using a linear-scaling attention mechanism combining local windowed and global attention. It is integrated with Hugging Face Transformers and released under Apache-2.0 by the Allen Institute for AI.

Use cases

• process long documents with a transformer model
• summarize long scientific papers or reports
• question answering over long documents
• fine-tune a transformer on documents longer than 512 tokens
• convert an existing pretrained model into a long-context version
• character-level language modeling on long text

When to choose

• you need transformer models that handle documents of thousands of tokens
• you want a drop-in Hugging Face-compatible long-context encoder or encoder-decoder
• you need pretrained weights for long-document QA, classification, or summarization

When to avoid

• your inputs fit within standard 512-token limits and a regular BERT/RoBERTa suffices
• you need actively developed long-context models with the latest architectures
• you lack GPU resources, since long sequences require significant memory

Facets

library · maturity maintenance

machine-learning deep-learning nlp llm-training deep-learning machine-learning python transformer long-context attention-mechanism huggingface pretrained-models summarization question-answering natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="allenai/longformer")