# szagoruyko/attention-transfer

Improving Convolutional Networks via Attention Transfer (ICLR 2017)

Repository: https://github.com/szagoruyko/attention-transfer
Canonical: https://ross.abutalabs.com/products/attention-transfer
Homepage: https://arxiv.org/abs/1612.03928
Language: Jupyter Notebook
License Family: other
Topics: pytorch, knowledge-distillation, attention, deep-learning
Last push: 2018-07-11T11:49:59+00:00

## Health v2 (maintenance only)
Score: 32/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 3515, "days_push": 2975, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1463, forks 273 (observed 2026-08-28T04:04:47.773169+00:00)

## What it is
PyTorch reference implementation of the ICLR 2017 paper 'Paying More Attention to Attention', which improves convolutional neural networks by transferring spatial attention maps from a teacher network to a student network. It includes CIFAR-10 and ImageNet experiment code, pretrained ResNet-18 models, and a Jupyter notebook for visualizing attention maps.

## Use cases
- implement attention transfer for knowledge distillation in pytorch
- reproduce cifar-10 and imagenet results from the attention transfer paper
- train a smaller student resnet using a larger teacher network
- visualize attention maps of a resnet-34 in a jupyter notebook
- download a pretrained resnet-18 model trained with attention transfer
- compare attention transfer against classic knowledge distillation

## When to choose
- you want the official reference code for the attention transfer distillation method
- you are doing research on knowledge distillation or attention-based transfer in CNNs
- you need reproducible CIFAR-10 baselines for teacher-student training in PyTorch

## When to avoid
- you need a maintained, production-ready training framework
- you require a permissively licensed dependency - the repo has no explicit license
- you want transformer or NLP attention mechanisms rather than CNN spatial attention
- you need grad-based attention transfer, which was never added to the repo

## Facets
- artifact type: library
- maturity: maintenance
- function: machine-learning, deep-learning, image-processing, computer-vision
- domain: deep-learning, computer-vision, machine-learning
- platform: python
- tags: pytorch, knowledge-distillation, attention-maps, student-teacher-networks, cifar-10, imagenet, resnet, research-code, iclr-2017, jupyter-notebook, research, gpu

## Member repositories
- szagoruyko/attention-transfer (main) score 32

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:47.773169+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:35:14.835908+00:00, confidence not recorded.
  - readme: https://github.com/szagoruyko/attention-transfer (fetched 2026-08-28T04:04:47.773169+00:00, sha 12dbdfc68ffc)
  - homepage: https://arxiv.org/abs/1612.03928 (fetched 2026-08-29T11:43:45.804909+00:00, sha b21bffa74ebd)
  - site_page: https://info.arxiv.org/about/donate.html (fetched 2026-08-29T11:43:45.814168+00:00, sha cca9c3a11c56)
  - site_page: https://info.arxiv.org/about/ourmembers.html (fetched 2026-08-29T11:43:45.818014+00:00, sha 47cbc55ff1de)
  - site_page: https://info.arxiv.org/about (fetched 2026-08-29T11:43:45.819909+00:00, sha a1f16f915a9a)
  - site_page: https://info.arxiv.org/labs/index.html (fetched 2026-08-29T11:43:45.816400+00:00, sha b14a8d05a0ec)
- Data as of 2026-08-30T08:39:29.467469+00:00.