# MoonshotAI/Attention-Residuals

Repository: https://github.com/MoonshotAI/Attention-Residuals
Canonical: https://ross.abutalabs.com/products/attention-residuals
License Family: other
Last push: 2026-03-17T06:23:11+00:00

## Health v2 (maintenance only)
Score: 47/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 72, release rhythm 35, longevity 12
- inputs: {"age_days": 171, "days_push": 169, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, young, no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 3487, forks 205 (observed 2026-08-28T04:08:07.059380+00:00)

## What it is
Official implementation of Attention Residuals (AttnRes), a drop-in replacement for standard residual connections in Transformers that lets each layer selectively aggregate earlier representations via learned attention over depth. Includes a Block AttnRes variant that reduces memory overhead from O(Ld) to O(Nd) while retaining most of the gains.

## Use cases
- replace residual connections in transformer models with learned depth attention
- mitigate PreNorm hidden-state magnitude growth in deep transformers
- train deeper LLMs with improved layer-wise information flow
- reproduce results from the AttnRes paper
- experiment with block-wise residual aggregation in PyTorch

## When to choose
- you are training or fine-tuning deep transformer models and want a better residual scheme
- you are doing architecture research on residual connections and depth-wise attention
- you want to reproduce or extend the AttnRes paper's results

## When to avoid
- you need a production-ready, battle-tested component for an existing model stack
- you cannot tolerate any per-layer parameter or memory overhead
- you need a framework-agnostic solution outside PyTorch

## Facets
- artifact type: library
- maturity: experimental
- function: deep-learning, machine-learning, llm-training
- domain: deep-learning, machine-learning, large-language-models
- platform: python
- tags: transformers, residual-connections, attention-over-depth, research-code, pytorch, architecture

## Member repositories
- MoonshotAI/Attention-Residuals (main) score 47

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:07.059380+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:36:06.895546+00:00, confidence not recorded.
  - readme: https://github.com/MoonshotAI/Attention-Residuals (fetched 2026-08-28T04:08:07.059380+00:00, sha aa2c5d551614)
- Data as of 2026-08-30T08:39:29.467469+00:00.