Ma-Lab-Berkeley/CRATE

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1275 stars · 99 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

CRATE is the official PyTorch implementation of the Coding RAte reduction TransformEr, a family of 'white-box' transformer architectures derived mathematically as unrolled optimization of a sparse rate reduction objective. It accompanies published NeurIPS, CPAL, and ICLR papers and provides models and training code for vision tasks such as segmentation and masked autoencoding.

Use cases

• implement white-box transformer architectures in pytorch
• train an interpretable vision transformer
• reproduce the NeurIPS 2023 CRATE paper results
• study transformers as optimization of a compression objective
• experiment with sparse rate reduction losses
• pretrain masked autoencoding models with structured diffusion

When to choose

• you are researching or teaching principled, mathematically derived transformer architectures
• you want to reproduce or extend the CRATE papers' vision experiments
• you need an interpretable alternative to black-box transformer blocks in PyTorch

When to avoid

• you need a production-ready, plug-and-play transformer with state-of-the-art benchmarks
• you want a maintained general-purpose model zoo rather than research code
• you are not working in PyTorch or need heavy ecosystem tooling

Facets

library · maturity active

deep-learning machine-learning transformers deep-learning machine-learning computer-vision python pytorch white-box-transformer sparse-rate-reduction transformer-architecture interpretability vision-transformer masked-autoencoder self-supervised-learning image-segmentation research-code natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Ma-Lab-Berkeley/CRATE")