Open-source benchmarking software

817 open-source libraries, tools and applications tagged benchmarking, ranked by how closely benchmarking is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.

817 products, primary matches first, then adoption-weighted; health v2 shown.

Product	Health v2	Stars	Maturity
meta-llama/PurpleLlama Purple Llama is Meta's umbrella project of tools and benchmarks for assessing and improving the security of large language models. It inclu…	68	4401	active
HttpRunner HttpRunner (hrp) is an open-source, Go-based all-in-one testing framework for API testing (HTTP/HTTP2/WebSocket/RPC), load testing, and mul…	44	4302	active
deepflowio/deepflow DeepFlow is an open-source eBPF-based observability platform providing zero-code distributed tracing, metrics, request logs, and continuous…	97	4294	active
richzhang/PerceptualSimilarity A PyTorch library implementing the LPIPS (Learned Perceptual Image Patch Similarity) metric, which measures perceptual distance between ima…	23	4280	stable
idealvin/coost Coost is a lightweight, cross-platform C++11 base library similar to Boost but much smaller (~1MB static library). It bundles coroutines, a…	86	4220	active
marcelroed/gigatoken Gigatoken is a Rust-based tokenizer library for language modeling that tokenizes text at GB/s throughput, claiming ~1000x speedups over Hug…	58	4134	active
uxlfoundation/oneDNN oneDNN is an open-source cross-platform performance library providing optimized building blocks (primitives) for deep learning applications…	99	4067	stable
deepchecks/deepchecks Deepchecks is an open-source Python library for continuous validation of machine learning models and data, covering testing, CI integration…	57	4060	active
Marin Marin is an open-source Python framework and research program for training foundation models, covering the full pipeline from data curation…	79	3988	active
LLaDA Official PyTorch implementation of LLaDA, a family of large language diffusion models (8B base/instruct, MoE, and iLLaDA variants) with pre…	59	3986	active
IREE IREE is an MLIR-based end-to-end machine learning compiler and runtime that lowers models from frameworks like PyTorch, TensorFlow, JAX, an…	95	3980	active
thomhurst/TUnit TUnit is a modern, fast .NET testing framework built on Microsoft.Testing.Platform that discovers tests at compile time via Roslyn source g…	90	3979	active
open-mmlab/mmpretrain MMPretrain is OpenMMLab's PyTorch-based toolbox and benchmark for image classification model pre-training, covering supervised, self-superv…	23	3855	active
google-research/scenic Scenic is a JAX-based library from Google Research focused on attention-based models for computer vision, providing shared lightweight libr…	76	3842	active
mit-han-lab/llm-awq AWQ is a Python library implementing Activation-aware Weight Quantization for compressing large language models to low-bit (INT3/4) weights…	41	3638	active
nv-tlabs/kimodo Kimodo is NVIDIA's official implementation of a kinematic motion diffusion model trained on 700 hours of motion capture data to generate hi…	58	3615	active
truera/trulens TruLens is a Python library for evaluating and tracking LLM-based applications and AI agents. It provides OpenTelemetry-native tracing of e…	99	3601	active
ashvardanian/StringZilla StringZilla is a high-performance string processing library for C, C++, Python, Rust, Swift, JS, and Go that uses SIMD, SWAR, and GPU instr…	94	3570	active
visionml/pytracking PyTracking is a PyTorch-based framework for visual object tracking and video object segmentation, providing official implementations of tra…	23	3516	active
POSTECH-CVLab/PyTorch-StudioGAN PyTorch-StudioGAN is a PyTorch library providing unified implementations of representative GAN architectures (BigGAN, StyleGAN2/3, etc.) fo…	23	3483	stable
mani-skill/ManiSkill ManiSkill is an open-source GPU-parallelized robotics simulation framework and benchmark built on SAPIEN, focused on manipulation skills. I…	85	3442	active
crystaldba/postgres-mcp Postgres MCP Pro is an open-source Model Context Protocol (MCP) server that gives AI agents and developers safe, configurable read/write ac…	63	3388	active
pester/Pester Pester is the ubiquitous test and mock framework for PowerShell, providing a Describe/Context/It DSL with Should assertions and powerful co…	96	3346	active
samber/cc-skills-golang A curated collection of 46 Go-specific agent skills (reusable instruction sets) for AI coding assistants like Claude Code, Codex, Gemini CL…	77	3337	active
nextest-rs/nextest cargo-nextest is a next-generation test runner for Rust, running tests up to 3x faster than cargo test with per-test process isolation. It …	94	3323	active
LLM-as-a-Verifier A Python framework that uses LLM token logprobs to produce fine-grained verification scores for agent trajectories without additional train…	59	3322	active
modelscope/3D-Speaker 3D-Speaker is an open-source Python toolkit for single- and multi-modal speaker verification, speaker recognition, and speaker diarization,…	53	3148	active
dexmal/dexbotic Dexbotic is an open-source PyTorch-based toolbox for developing Vision-Language-Action (VLA) models for embodied intelligence. It unifies p…	70	3144	active
MiroFlow MiroFlow is a performance-first open-source agent framework for building, orchestrating, and benchmarking LLM-powered agents, with MiroThin…	56	3114	active
lammps/lammps LAMMPS (Large-scale Atomic/Molecular Massively Parallel Simulator) is an open-source classical molecular dynamics simulation code focused o…	95	3104	stable
UKGovernmentBEIS/inspect_ai Inspect is an open-source Python framework from the UK AI Security Institute for running large language model evaluations, with composable …	95	3054	active
ulab-uiuc/LLMRouter LLMRouter is an open-source Python library that intelligently routes queries to the most suitable LLM based on task complexity, cost, and p…	62	2995	active
hello245m/free-stockdb A local-first quantitative data engine for China A-share daily/minute K-lines and ETF minute/tick data, combining incremental sync, cleanin…	77	2912	active
mll-lab-nu/RAGEN RAGEN is a Python framework for training reasoning LLM agents with multi-turn reinforcement learning using the StarPO algorithm. It also pr…	62	2803	active
HiLab-git/SSL4MIS A benchmark and code collection of semi-supervised learning methods for medical image segmentation, re-implementing approaches like Mean Te…	40	2699	active
itayinbarr/little-coder little-coder is a terminal coding agent built on the pi agent loop, tuned for small local language models like Qwen via Ollama or llama.cpp…	79	2691	active
vllm-project/production-stack vLLM Production Stack is a reference implementation for deploying vLLM LLM inference at scale on Kubernetes, distributed via Helm charts. I…	88	2667	active
robosuite robosuite is a modular simulation framework powered by the MuJoCo physics engine for robot learning, offering standardized benchmark enviro…	66	2643	active
alexfernandez/loadtest A Node.js CLI tool and library that runs load tests against HTTP or WebSocket URLs, with an Apache ab-compatible interface plus a --rps opt…	57	2632	active
tezc/sc A collection of portable, stand-alone C99 libraries and data structures such as hashmap, heap, linked list, logger, and ini parser. Each mo…	61	2573	active
Unakar/Logic-RL Logic-RL is a research framework that reproduces R1-Zero-style LLM reasoning via rule-based reinforcement learning (GRPO) on logic puzzles,…	23	2452	active
learnhouse/learnhouse LearnHouse is a next-generation open-source learning management system (LMS) for creating, sharing, and selling educational content. It com…	100	2404	active
akfamily/akquant AKQuant is a high-performance quantitative research and backtesting framework with a Rust core engine and Python interface for strategy dev…	76	2392	active
1517005260/graph-rag-agent A Python framework that integrates GraphRAG, LightRAG, and Neo4j-based graph building to construct knowledge graphs and perform explainable…	40	2343	active
chrisworsey55/atlas-gic ATLAS is a Python framework of self-improving AI trading agents that optimize their own prompts against market outcomes, using multi-agent …	58	2312	active
MCPJam/inspector MCPJam is an open-source testing and evaluation platform for MCP server developers, providing an inspector playground, OAuth/EMA debugger, …	83	2270	active
galilai-group/stable-worldmodel A Python library providing a unified platform for reproducible world model research, covering data collection, training, and evaluation via…	78	2260	active
google-deepmind/tapnet Google DeepMind's official repository for Tracking Any Point (TAP), containing the TAP-Vid and TAPVid-3D benchmarks, the TAPIR and TAPNext …	75	2217	active
dfeneyrou/palanteer Palanteer is a visual nanosecond profiler and logger for C++ and Python programs, with a single-header C++ instrumentation library and most…	29	2215	active
samchon/nestia Nestia is a set of helper libraries for NestJS that turns plain TypeScript types into runtime validation, a typed client SDK, OpenAPI/Swagg…	93	2179	active
scouter-project/scouter Scouter is an open-source Application Performance Management (APM) tool similar to New Relic or AppDynamics, using a Java agent to instrume…	66	2178	active
tdunning/t-digest A Java library implementing the t-digest data structure for accurate online accumulation of rank-based statistics such as quantiles and tri…	32	2173	stable
PrimeIntellect-ai/prime-rl prime-rl is a Python framework for large-scale, fully asynchronous reinforcement learning training of language models, built on FSDP2 for t…	88	2164	active
hexo-ai/sia SIA is a Python framework implementing a self-improving AI loop in which a Meta-Agent generates a task-specific Target Agent and a Feedback…	56	2161	active
geatpy-dev/geatpy Geatpy is a high-performance genetic and evolutionary algorithm toolbox for Python supporting single-objective, multi-objective, and many-o…	49	2122	active
Blazemeter/taurus Taurus (bzt) is an automation-friendly framework for continuous performance and functional testing that wraps tools like JMeter, Gatling, L…	88	2116	active
HRM-Text HRM-Text is a 1B-parameter text generation model based on the hierarchical recurrent HRM architecture, released with a complete pretraining…	58	2093	active
NanoNets/docext docext is an on-premises document intelligence toolkit powered by vision-language models, offering OCR-free structured data extraction, PDF…	47	2086	active
Carpet Mod Fabric Carpet is a Minecraft mod for the Fabric mod loader that gives technical players and server operators fine-grained control over vani…	100	2082	active
NVIDIA-NeMo/RL NeMo RL is NVIDIA's open-source post-training library for scaling reinforcement learning methods (GRPO, PPO, DPO, SFT, distillation) on LLM…	80	2073	active
juanmc2005/diart Diart is a Python framework for building AI-powered real-time audio applications, best known for state-of-the-art streaming speaker diariza…	59	2041	active
uncertainty-toolbox/uncertainty-toolbox A Python library for predictive uncertainty quantification, providing metrics, visualizations, and recalibration procedures for regression …	23	2015	active
meta-recsys/generative-recommenders Meta's research library implementing HSTU and M-FALCON from the ICML'24 paper 'Actions Speak Louder than Words: Trillion-Parameter Sequenti…	70	2004	active
logpai/logparser Logparser is a Python machine learning toolkit and benchmark suite for automated log parsing. It extracts event templates from unstructured…	31	1994	active
MatrAIx-ai/MatrAIx-Persona-8B MatrAIx is a population-scale, persona-driven evaluation framework that instantiates sampled persona records as LLM agents to simulate hete…	56	1955	active
alibaba/EasyCV EasyCV is an all-in-one PyTorch-based computer vision toolkit from Alibaba covering self-supervised learning, vision transformers, and majo…	28	1955	active
deepseek-ai/TileKernels TileKernels is a Python library of optimized GPU kernels for LLM operations, written in the TileLang DSL. It provides kernels for MoE routi…	58	1949	active
AIStore AIStore (AIS) is a lightweight, distributed object storage stack built by NVIDIA specifically for AI workloads. It provides linear scalabil…	95	1946	active
BytedTsinghua-SIA/DAPO DAPO is an open-source reinforcement learning system for large-scale LLM training, released by ByteDance Seed and Tsinghua AIR. It implemen…	26	1946	active
wuyoscar/Internal-Safety-Collapse ISC-Bench/TVD is a research framework for studying 'Internal Safety Collapse' in frontier LLMs, where agents placed in adversarial codespac…	60	1935	active
IPADS-SAI/MobiAgent MobiAgent is a systematic framework for building customizable GUI agents that operate mobile phones, comprising the MobiMind agent model fa…	54	1920	active
LeapLabTHU/Absolute-Zero-Reasoner Official implementation of Absolute Zero Reasoner (AZR), a system that trains LLM reasoning via reinforced self-play with zero external dat…	33	1916	active
stepci/stepci Step CI is an open-source, language-agnostic API quality assurance framework configured via YAML, JSON, or JavaScript. It tests REST, Graph…	23	1868	active
RoboVerseOrg/RoboVerse RoboVerse is a unified platform, dataset, and benchmark for scalable and generalizable robot learning, providing tasks, robots, scenes, ass…	65	1863	active
openreasoner/openr OpenR is an open-source Python framework that integrates search, reinforcement learning, and process supervision to improve chain-of-though…	23	1859	active
ArviZ ArviZ is a Python package for exploratory analysis of Bayesian models, providing posterior analysis, diagnostics, model comparison, plottin…	90	1858	active
DTolm/VkFFT VkFFT is an open-source, GPU-accelerated multidimensional Fast Fourier Transform library supporting Vulkan, CUDA, HIP, OpenCL, Level Zero, …	53	1784	active
johnbillion/query-monitor Query Monitor is a WordPress and WooCommerce plugin that adds a developer tools panel for debugging database queries, PHP errors, hooks, bl…	93	1779	stable
CalculatedContent/WeightWatcher WeightWatcher is an open-source Python diagnostic tool for analyzing pre-trained deep neural networks without access to training or test da…	56	1777	active
SoapUI SoapUI is a free, open-source, cross-platform desktop application for functional testing of SOAP and REST APIs and web services. It support…	72	1710	active
shekyan/slowhttptest SlowHTTPTest is a highly configurable command-line tool that simulates Application Layer Denial of Service attacks by prolonging HTTP conne…	84	1650	active
InterDigitalInc/CompressAI CompressAI is a PyTorch library and evaluation platform for end-to-end learned data compression research. It provides custom layers, entrop…	76	1648	active
opendilab/LightZero LightZero is a lightweight, open-source Python toolkit that unifies Monte Carlo Tree Search (MCTS) with deep reinforcement learning, implem…	66	1647	active
Meta-Harness Meta-Harness is a Python framework for automated end-to-end search over task-specific model harnesses — the code surrounding a fixed base L…	57	1625	active
pytorch/FBGEMM FBGEMM is a collection of highly optimized low-precision matrix multiplication and convolution kernels for server-side deep learning infere…	91	1595	active
DeepGraphLearning/torchdrug TorchDrug is a PyTorch-based machine learning library for drug discovery, covering graph neural networks, deep generative models, and reinf…	23	1589	active
ARISE-Initiative/robomimic robomimic is a modular Python framework for robot learning from demonstration, providing standardized demonstration datasets and offline le…	65	1572	active
ByteDance-Seed/Triton-distributed Triton-distributed is a distributed compiler built on OpenAI Triton for computation-communication overlapping on multi-GPU systems. It lets…	61	1559	active
nubank/fklearn fklearn is a Python machine learning library from Nubank that applies functional programming principles to build, validate, and deploy mode…	83	1550	active
facebookresearch/mmf MMF is a modular PyTorch framework for vision and language multimodal research from Facebook AI Research. It ships reference implementation…	67	5637	maintenance
KruxAI/ragbuilder RagBuilder is a Python toolkit that automatically builds an optimal, production-ready Retrieval-Augmented Generation (RAG) pipeline for you…	65	1539	active
open-thought/reasoning-gym Reasoning Gym is a Python library of procedural dataset generators and algorithmically verifiable reasoning environments for training LLMs …	60	1512	active
mbzuai-oryx/Video-ChatGPT Video-ChatGPT is a video conversation model that combines large language models with a pretrained visual encoder adapted for spatiotemporal…	72	1508	active
Lightning-AI/lightning-thunder Lightning Thunder is a source-to-source deep learning compiler for PyTorch that optimizes models for training and inference. It provides a …	70	1475	active
LocoMuJoCo LocoMuJoCo is an imitation learning benchmark for whole-body locomotion control built on MuJoCo, featuring humanoid, quadruped, and musculo…	70	1468	active
ScottZt/jin-ce-zhi-suan A self-hosted quantitative trading research and backtesting system for Chinese A-share markets, built with Python and FastAPI with a web da…	70	1467	active
tensorflow/tpu A collection of reference models and tools for training machine learning models on Google Cloud TPUs, maintained as a public mirror by the …	68	5278	maintenance
aeon-toolkit/aeon aeon is a scikit-learn compatible Python toolkit for machine learning on time series, covering classification, regression, clustering, fore…	93	1457	active
google/oss-fuzz-gen A Google framework that uses large language models to automatically generate fuzz targets for real-world C/C++, Java, and Python projects, …	55	1443	active
JuliaGPU/CUDA.jl CUDA.jl is the main Julia package for programming NVIDIA CUDA GPUs, offering a high-level CuArray abstraction, a compiler for writing CUDA …	99	1433	stable