wdndev/tiny-llm-zh resource

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1078 stars · 125 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

An educational project that implements a small-parameter Chinese large language model from scratch, covering the full pipeline: tokenizer training, pretraining, SFT, RLHF/DPO alignment, evaluation, quantization, and deployment. It uses a Llama-style architecture (RMSNorm, RoPE, MHA) with Transformers and DeepSpeed, and supports MoE, vLLM, and llama.cpp inference.

Use cases

• learn how to train a large language model from scratch
• understand the full LLM pipeline from tokenizer to deployment
• train a small Chinese language model
• study RLHF and DPO alignment implementation
• experiment with MoE architecture in transformers
• practice pretraining and SFT with deepspeed
• learn LLM quantization and deployment with vLLM or llama.cpp

When to choose

• you want a hands-on, end-to-end walkthrough of building and training an LLM
• you need a small, resource-friendly Chinese model for learning purposes
• you want to study modern LLM techniques like RoPE, MoE, DPO, and quantization in one codebase

When to avoid

• you need a production-quality model with strong benchmark results
• you require a commercially licensed model (no license is specified)
• you need a large multilingual or English-focused model

Facets

learning-resource · maturity active

llm-training llm-inference machine-learning deep-learning rag large-language-models deep-learning tutorials machine-learning python cloud chinese-llm tokenizer-training sft rlhf dpo moe deepspeed vllm llama-cpp model-quantization pretraining educational-project natural-language-processing gpu linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="wdndev/tiny-llm-zh")