gkamradt/needle-in-a-haystack

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

2375 stars · 246 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A benchmarking tool (niah / needlehaystack) that pressure-tests LLM long-context retrieval by sweeping context length x needle depth cells against configured models and scoring each response. It supports single-fact lookup, multi-fact recall, and UUID-chain multi-step reasoning tasks, writing one JSONL result row per cell.

Use cases

• test how well an LLM retrieves a fact from a long context
• measure retrieval accuracy at different context lengths and depths
• compare long-context performance across OpenAI, Anthropic, and Cohere models
• run a needle-in-a-haystack evaluation sweep
• probe whether models miss information buried in the middle of long prompts
• evaluate multi-step reasoning over long context with UUID chains

When to choose

• you need a reproducible, configurable sweep of context-length vs depth accuracy for LLMs
• you want to benchmark raw long-context retrieval across multiple providers
• you want scored JSONL results you can analyze, including reconstructing the exact context each cell saw

When to avoid

• you need a general LLM evaluation harness for coding, math, or reasoning benchmarks beyond context retrieval
• you want to test a RAG pipeline's retriever rather than a model's raw context handling
• you need a production inference client - this is an evaluation tool, not a serving layer

Facets

cli-tool · maturity active

benchmarking llm-inference testing large-language-models artificial-intelligence testing machine-learning python cli llm-evaluation long-context context-window needle-in-a-haystack lost-in-the-middle jsonl-results yaml-config model-comparison openai anthropic cohere uuid-chain-reasoning

Member repositories

For agents

markdown · JSON · MCP: product_card(name="gkamradt/needle-in-a-haystack")