JIA-Lab-research/LongLoRA

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2686 stars · 281 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

LongLoRA is an efficient fine-tuning approach and codebase that extends the context length of pre-trained LLMs (Llama2 7B/13B/70B) using shifted sparse attention and an improved LoRA method. It also ships LongAlpaca, a set of long-context instruction-following models and the LongAlpaca-12k training dataset.

Use cases

• extend llama2 context length beyond 8k tokens
• fine-tune an llm on long documents cheaply
• train a long-context chat model with lora
• reduce gpu memory when fine-tuning long-context models
• get a model that answers questions about long pdfs
• evaluate long-context models on LongBench and L-Eval

When to choose

• you need to extend an open LLM's context window with limited GPU budget
• you want LoRA/QLoRA fine-tuning for long-context instruction data
• you want ready-made long-context models and datasets as a starting point

When to avoid

• you need production long-context inference rather than training
• your model is not a Llama-family architecture
• you need actively maintained tooling - the repo's last release was mid-2024

Facets

library · maturity maintenance

llm-training machine-learning deep-learning large-language-models machine-learning deep-learning python lora fine-tuning long-context shifted-sparse-attention llama2 qlora iclr-2024 gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="JIA-Lab-research/LongLoRA")