jiachenzhu/DyT

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1043 stars · 87 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Official PyTorch implementation of DynamicTanh (DyT), a learnable element-wise tanh operation that replaces normalization layers in Transformers. Released as research code accompanying the CVPR 2025 paper 'Transformers without Normalization', with training scripts for ViT and ConvNeXt on ImageNet-1K.

Use cases

• replace layer normalization in transformers with dynamic tanh
• train ViT models without normalization layers
• reproduce CVPR 2025 DyT paper results on ImageNet
• experiment with normalization-free deep learning architectures
• apply DyT to ConvNeXt models
• compare LayerNorm vs DyT performance in vision transformers

When to choose

• you want to replace normalization layers in a Transformer with a learnable tanh operation
• you are reproducing or extending the 'Transformers without Normalization' paper
• you are researching normalization-free deep learning architectures

When to avoid

• you need a production-ready, pip-installable library with broad model support
• you are not working in PyTorch
• you need stable, well-maintained software rather than research code

Facets

library · maturity active

machine-learning deep-learning deep-learning machine-learning computer-vision python pytorch transformers normalization research-code cvpr-2025 dynamic-tanh gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="jiachenzhu/DyT")