xiaowu0162/LongMemEval resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1035 stars · 81 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

LongMemEval is a benchmark of 500 high-quality questions for evaluating the long-term memory abilities of chat assistants across five skills including information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention. The repository provides the dataset, evaluation code, and tooling for running chat systems against timestamped multi-session chat histories.

Use cases

• evaluate long-term memory of chat assistants
• benchmark llm memory across multiple sessions
• test temporal reasoning in conversational ai
• measure knowledge update handling in chatbots
• compare rag memory systems on long chat histories
• evaluate abstention when information is missing

When to choose

• you need a standardized benchmark for long-term conversational memory
• you are building or comparing memory-augmented chat assistants
• you want needle-in-a-haystack style evaluation over multi-session chat histories

When to avoid

• you need a general-purpose LLM benchmark unrelated to memory
• you want a training dataset rather than an evaluation benchmark
• you need agentic long-term memory evaluation, for which LongMemEval-V2 is more appropriate

Facets

dataset · maturity active

benchmarking rag llm-inference chatbot large-language-models artificial-intelligence chatbots testing python cli long-term-memory chat-assistants iclr-2025 needle-in-a-haystack memory-benchmark multi-session-dialogue evaluation retrieval-augmented-generation natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="xiaowu0162/LongMemEval")