ZhuiyiTechnology/roformer

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1146 stars · 64 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

RoFormer is an MLM pre-trained language model built on rotary position embeddings (RoPE), a relative position encoding method with strong theoretical properties. The repository provides the model implementation, pseudo code for adding RoPE to self-attention, and downloadable pre-trained Chinese RoFormer model checkpoints.

Use cases

• pretrain or fine-tune a transformer with rotary position embeddings
• use RoPE in self-attention for relative position encoding
• download pretrained Chinese RoFormer checkpoints
• apply RoPE to linear attention models
• build Chinese text similarity models with RoFormer-sim
• train GPT-style character-level models with RoFormer

When to choose

• you need relative position encoding that works with linear attention
• you want pretrained Chinese transformer checkpoints with RoPE
• you use bert4keras/Keras and want to add rotary embeddings with minimal code changes

When to avoid

• you need PyTorch or Hugging Face ecosystem tooling rather than bert4keras
• you need actively maintained code or recent releases
• you need multilingual or English-only pretrained models

Facets

library · maturity maintenance

machine-learning nlp deep-learning machine-learning deep-learning python rope rotary-position-embedding pretrained-language-model transformer chinese-nlp bert4keras natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ZhuiyiTechnology/roformer")