# keyu-tian/SparK

[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

Repository: https://github.com/keyu-tian/SparK
Canonical: https://ross.abutalabs.com/products/keyu-tian-spark
Homepage: https://arxiv.org/abs/2301.03580
Language: Python
License: MIT
License Family: permissive
Topics: bert, convnet, convolutional-neural-networks, masked-image-modeling, pre-trained-model, self-supervised-learning, sparse-convolution, ssl, cnn, iclr, iclr2023, deep-learning, object-detection, pytorch, instance-segmentation, mask-rcnn, mae, masked-autoencoder, pretrain, pretraining
Last push: 2024-01-23T06:15:47+00:00

## Health v2 (maintenance only)
Score: 22/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 8, longevity 95
- inputs: {"age_days": 1336, "days_push": 953, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1376, forks 85 (observed 2026-08-28T04:04:33.256844+00:00)

## What it is
SparK is the official PyTorch implementation of an ICLR 2023 Spotlight paper that applies BERT/MAE-style masked image modeling to convolutional networks using sparse convolution and a hierarchical decoder. It can pretrain any CNN (e.g., ResNet, ConvNeXt) in a self-supervised manner without backbone modifications.

## Use cases
- pretrain a ResNet or ConvNeXt with masked image modeling
- self-supervised pretraining for CNN backbones
- improve object detection and instance segmentation with pretrained CNN features
- reproduce BERT-style pretraining on convolutional networks
- compare sparse masked modeling against contrastive learning baselines
- fine-tune pretrained CNN weights on ImageNet classification

## When to choose
- you want MAE/BERT-style self-supervised pretraining but must use a CNN backbone
- you need better transfer features for detection or segmentation with convolutional models
- you are researching masked image modeling on hierarchical architectures

## When to avoid
- you are pretraining vision transformers, where standard MAE applies directly
- you need a maintained production library rather than a research codebase
- you lack GPU resources for large-scale pretraining

## Facets
- artifact type: library
- maturity: stable
- function: machine-learning, deep-learning, llm-training
- domain: computer-vision, deep-learning, image-processing
- platform: python
- tags: masked-image-modeling, self-supervised-learning, sparse-convolution, cnn-pretraining, bert-style, pytorch, iclr2023, resnet, convnext, object-detection, gpu, linux

## Member repositories
- keyu-tian/SparK (main) score 22

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:33.256844+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:40:30.405856+00:00, confidence not recorded.
  - readme: https://github.com/keyu-tian/SparK (fetched 2026-08-28T04:04:33.256844+00:00, sha 378f2b637281)
  - homepage: https://arxiv.org/abs/2301.03580 (fetched 2026-08-29T11:56:43.294624+00:00, sha 7b1f2162c48d)
  - site_page: https://info.arxiv.org/about/donate.html (fetched 2026-08-29T11:56:43.303975+00:00, sha cca9c3a11c56)
  - site_page: https://info.arxiv.org/about/ourmembers.html (fetched 2026-08-29T11:56:43.307484+00:00, sha 47cbc55ff1de)
  - site_page: https://info.arxiv.org/about (fetched 2026-08-29T11:56:43.309281+00:00, sha a1f16f915a9a)
  - site_page: https://info.arxiv.org/labs/index.html (fetched 2026-08-29T11:56:43.305833+00:00, sha b14a8d05a0ec)
- Data as of 2026-08-30T08:39:29.467469+00:00.