MoonshotAI/Attention-Residuals

Health v2 · maintenance only

Flags: no_releases young no_license

Adoption not part of the score

3487 stars · 205 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Official implementation of Attention Residuals (AttnRes), a drop-in replacement for standard residual connections in Transformers that lets each layer selectively aggregate earlier representations via learned attention over depth. Includes a Block AttnRes variant that reduces memory overhead from O(Ld) to O(Nd) while retaining most of the gains.

Use cases

• replace residual connections in transformer models with learned depth attention
• mitigate PreNorm hidden-state magnitude growth in deep transformers
• train deeper LLMs with improved layer-wise information flow
• reproduce results from the AttnRes paper
• experiment with block-wise residual aggregation in PyTorch

When to choose

• you are training or fine-tuning deep transformer models and want a better residual scheme
• you are doing architecture research on residual connections and depth-wise attention
• you want to reproduce or extend the AttnRes paper's results

When to avoid

• you need a production-ready, battle-tested component for an existing model stack
• you cannot tolerate any per-layer parameter or memory overhead
• you need a framework-agnostic solution outside PyTorch

Facets

library · maturity experimental

deep-learning machine-learning llm-training deep-learning machine-learning large-language-models python transformers residual-connections attention-over-depth research-code pytorch architecture

Member repositories

For agents

markdown · JSON · MCP: product_card(name="MoonshotAI/Attention-Residuals")