Smerity/sha-rnn

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1180 stars · 131 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A PyTorch implementation of the Single Headed Attention RNN (SHA-LSTM) from Stephen Merity's paper, combining an LSTM with a single attention head for byte-level language modeling. It achieves strong enwik8 results while training in under 24 hours on a single 12GB GPU.

Use cases

• train a byte-level language model on a single GPU
• reproduce SHA-LSTM results on enwik8
• model long-range dependencies without transformer memory overhead
• avoid fragile transformer warmup and training instability
• build a small language model exportable to ONNX with standard components

When to choose

• you need an efficient, low-resource language model trainable on one GPU
• you want LSTM-based architecture with simple attention for long contexts
• you need a model composed of standard, productionizable components

When to avoid

• you need state-of-the-art transformer performance
• you want a polished library with CLI options and stable API
• you need an actively maintained project - the code is research-grade and the author notes it is not kind

Facets

library · maturity maintenance

machine-learning deep-learning deep-learning machine-learning python rnn lstm attention language-modeling pytorch research-code sha-lstm natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Smerity/sha-rnn")