Open-source benchmarking software
817 open-source libraries, tools and applications tagged benchmarking, ranked by how closely benchmarking is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
817 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| linkedin/kafka-monitor Xinfra Monitor (formerly Kafka Monitor) is a Java framework for running long-running Kafka system tests and monitoring Kafka cluster health… | 24 | 2065 | active |
| test-prof/test-prof TestProf is a Ruby gem providing a toolbox of profilers and helpers for analyzing and speeding up test suites, with integrations for ruby-p… | 74 | 2050 | active |
| andrewrk/poop A Linux command-line benchmarking tool that compares the performance of multiple commands using perf_event_open hardware counters, with a c… | 53 | 2046 | active |
| EvalAI EvalAI is an open-source platform for evaluating and comparing machine learning and AI algorithms at scale. It provides a central leaderboa… | 67 | 2045 | active |
| ruby-prof/ruby-prof ruby-prof is a fast code profiler for Ruby that measures method call times and generates call graphs and reports. It helps developers ident… | 68 | 2033 | active |
| elastic/rally Rally is a macrobenchmarking framework for Elasticsearch, developed in Python by Elastic. It automates cluster setup and teardown, benchmar… | 81 | 2031 | active |
| phpbench/phpbench PHPBench is a benchmarking framework for PHP, analogous to PHPUnit but for measuring performance rather than correctness. It runs code repe… | 84 | 2024 | active |
| leinlin/Miku-LuaProfiler A Lua profiler plugin for Unity that instruments xLua-based Lua code to report per-function CPU time and GC memory usage. It works in the U… | 46 | 2021 | active |
| tatsu-lab/alpaca_eval AlpacaEval is an LLM-based automatic evaluation framework for instruction-following language models, producing win rates against a GPT-4 ba… | 33 | 2018 | active |
| GoogleCloudPlatform/PerfKitBenchmarker PerfKit Benchmarker is a Python tool that automates running a canonical set of benchmarks across cloud providers like GCP, AWS, and Azure. … | 68 | 2014 | active |
| clementchadebec/benchmark_VAE Pythae is a PyTorch library that unifies implementations of many Variational Autoencoder (VAE) variants under a common interface, enabling … | 23 | 2000 | active |
| Strix Halo AI Toolboxes A collection of pre-built Toolbx/Docker containers ('toolboxes') for running llama.cpp LLM inference on AMD Ryzen AI Max 'Strix Halo' integ… | 64 | 1998 | active |
| Hyperopt Hyperopt is a Python library for distributed asynchronous hyperparameter optimization over search spaces with real-valued, discrete, and co… | 84 | 7595 | maintenance |
| luigirizzo/netmap Netmap is a framework for extremely fast packet I/O from userspace, implemented as a kernel module for FreeBSD, Linux, and Windows. It incl… | 62 | 1973 | stable |
| ar51an/iperf3-win-builds A repository providing pre-compiled iperf3 Windows 64-bit executables, built via Cygwin without modifying the original source. It fills the… | 89 | 1960 | active |
| twitter/pelikan Pelikan is Twitter's framework for building high-performance cache services, written in Rust with a legacy C codebase. It includes products… | 23 | 1957 | active |
| RRZE-HPC/likwid LIKWID is a command-line tool suite and library for performance-oriented programmers on Linux, supporting Intel, AMD, ARMv8, and POWER9 CPU… | 87 | 1947 | active |
| mlco2/codecarbon CodeCarbon is a lightweight Python library and CLI that estimates and tracks CO2 emissions from local computing hardware (CPU, GPU, RAM) by… | 96 | 1921 | active |
| charkost/prosopite Prosopite is a Ruby gem that auto-detects N+1 queries in Rails applications with zero false positives or false negatives. It monitors SQL q… | 72 | 1915 | active |
| spiritLHLS/ecs A shell-based 'Fusion Monster' VPS server testing script that benchmarks CPU, disk, network speed, and IP quality across many Linux distrib… | 74 | 7221 | maintenance |
| ashvardanian/NumKong NumKong is a mixed-precision numeric and linear-algebra library with ~2,000 SIMD-accelerated kernels for dot products, distances, matrix mu… | 93 | 1899 | active |
| max0x7ba/atomic_queue A C++14 header-only library providing multiple-producer multiple-consumer lock-free queues built on circular buffers and std::atomic. It is… | 98 | 1897 | stable |
| hcoles/pitest Pitest (PIT) is a state-of-the-art mutation testing system for Java and the JVM. It mutates bytecode to verify the effectiveness of unit te… | 99 | 1875 | active |
| linkedin/greykite Greykite is a Python time series forecasting library from LinkedIn built around its Silverkite algorithm, with support for trend, seasonali… | 23 | 1856 | active |
| trevin-creator/autoresearch-mlx An Apple Silicon (MLX) port of Karpathy's autoresearch that runs autonomous AI research loops on a Mac without PyTorch or CUDA. A coding ag… | 52 | 1852 | active |
| GestaltCogTeam/BasicTS BasicTS is a Python benchmark library and toolkit for fair and scalable time series analysis, built on PyTorch. It supports forecasting, cl… | 61 | 1826 | active |
| evalplus/evalplus EvalPlus is a Python framework and benchmark suite for rigorously evaluating LLM-synthesized code, providing HumanEval+ and MBPP+ datasets … | 38 | 1826 | active |
| linyimin0812/spring-startup-analyzer A Java agent/library that instruments Spring and Spring Boot applications to collect startup data and generate an interactive HTML startup … | 31 | 1824 | active |
| Vchitect/VBench VBench is a comprehensive benchmark suite for evaluating video generative models across hierarchical evaluation dimensions, with prompt sui… | 58 | 1802 | active |
| wtf wtf (what the fuzz) is a distributed, code-coverage guided, snapshot-based fuzzer for attacking user- and kernel-mode targets on Windows an… | 82 | 1800 | active |
| memtest86plus/memtest86plus Memtest86+ is a free, open-source, stand-alone memory tester for x86, x86-64, ARM AArch64, and LoongArch64 computers. It boots directly fro… | 78 | 1788 | active |
| evanphx/benchmark-ips A Ruby gem that enhances the standard Benchmark library with iterations-per-second (IPS) benchmarking. It automatically determines appropri… | 59 | 1776 | stable |
| AngusKit/AngusTester AngusTester is a self-hosted agile development and testing platform combining project management, functional/API/scenario testing, service … | 65 | 1772 | active |
| SamSaffron/memory_profiler memory_profiler is a Ruby gem that reports memory usage and object allocations (allocated vs retained) by gem, file, location, and class. I… | 62 | 1767 | active |
| robocasa/robocasa RoboCasa is a large-scale simulation framework for training and benchmarking generally capable robots on everyday tasks, focused on kitchen… | 78 | 1759 | active |
| decisionintelligence/TFB TFB is a comprehensive and fair benchmarking framework for time series forecasting methods, covering deep learning, machine learning, and s… | 70 | 1753 | active |
| martinus/nanobench ankerl::nanobench is a single-header, platform-independent microbenchmarking library for C++11/14/17/20. It measures ns/op, op/s, instructi… | 93 | 1738 | stable |
| cleverhans-lab/cleverhans CleverHans is a Python library for benchmarking machine learning systems' vulnerability to adversarial examples, providing reference implem… | 23 | 6455 | maintenance |
| pawurb/hotpath-rs hotpath-rs is an easy-to-configure Rust performance profiler that pinpoints where code spends time, burns CPU, and allocates memory. It ins… | 78 | 1731 | active |
| HypoPG/hypopg HypoPG is a PostgreSQL extension written in C that adds support for hypothetical (virtual) indexes. It lets you test whether an index would… | 82 | 1730 | stable |
| sumerc/yappi Yappi is a fast tracing profiler for Python written in C that is aware of multithreading, asyncio, and gevent/greenlet concurrency. It supp… | 85 | 1729 | stable |
| JonMagon/KDiskMark KDiskMark is an open-source HDD and SSD benchmark tool for Linux with a friendly Qt-based GUI, acting as a frontend to the Flexible I/O Tes… | 78 | 1716 | active |
| zai-org/ImageReward ImageReward is the first general-purpose human preference reward model for text-to-image generation, trained on 137k expert comparison pair… | 49 | 1705 | stable |
| vllm-project/guidellm GuideLLM is an SLO-aware benchmarking and evaluation platform for LLM inference deployments, simulating real-world workloads against OpenAI… | 92 | 1698 | active |
| mattpocock/evalite Evalite is a TypeScript-native eval runner for testing LLM-powered applications, built on Vitest with a local dev server UI for exploring o… | 55 | 1692 | active |
| XHGui XHGui is a self-hosted web application that provides a graphical interface for viewing and analyzing XHProf profiling data collected from P… | 76 | 1689 | active |
| google/vizier Open Source Vizier is a Python-based service and library for black-box and hyperparameter optimization, based on Google's internal Vizier t… | 79 | 1680 | active |
| NCCL NVIDIA's official test suite for NCCL that measures the performance and correctness of collective communication operations across GPUs. It … | 74 | 1664 | active |
| mourner/flamebearer A command-line tool that analyzes JavaScript CPU profiles from Chrome DevTools Performance recordings and Node .cpuprofile files, printing … | 86 | 1661 | active |
| tevador/RandomX RandomX is a proof-of-work algorithm optimized for general-purpose CPUs that uses random code execution and memory-hard techniques to resis… | 89 | 1655 | stable |
| tikv/pprof-rs pprof-rs is a CPU profiler library for Rust programs, built on top of backtrace-rs. It samples running threads and produces stack-trace rep… | 75 | 1654 | active |
| facebookresearch/DomainBed DomainBed is a PyTorch suite containing benchmark datasets and algorithms for domain generalization research. It implements many algorithms… | 10 | 1632 | active |
| mlcommons/inference Reference implementations of the MLPerf Inference benchmark suite, the industry-standard benchmarks for measuring how fast ML systems run m… | 79 | 1631 | active |
| web-arena-x/webarena WebArena is a standalone, self-hostable web environment and benchmark for building and evaluating autonomous web agents. It provides realis… | 40 | 1620 | active |
| NVIDIA/RULER RULER is a benchmark framework from NVIDIA that generates synthetic test examples to measure the true effective context size of long-contex… | 64 | 1619 | active |
| uber/ADR ADR (Agentic AI Detection and Response) is an open-source enterprise security framework for AI agents, providing endpoint discovery of AI t… | 66 | 1618 | active |
| Clinic.js Clinic.js is an open-source Node.js performance profiling suite that diagnoses issues like event-loop delays, I/O bottlenecks, and memory l… | 23 | 5953 | maintenance |
| bamlab/flashlight Flashlight is a CLI tool that audits Android apps (native, React Native, Flutter) and generates a performance score from aggregated metrics… | 67 | 1606 | active |
| google/uncertainty-baselines A library of high-quality, minimal-dependency implementations of standard and state-of-the-art uncertainty and robustness methods for deep … | 77 | 1601 | active |
| facebook/CacheLib CacheLib is a C++ library providing a pluggable, thread-safe in-process caching engine with transparent DRAM and SSD tiering. It includes C… | 100 | 1599 | active |
| nschloe/tuna Tuna is a lightweight Python profile viewer that visualizes runtime (cProfile) and import profiles in the browser using d3. It is a modern,… | 82 | 1599 | active |
| microprofile Microprofile is an embeddable CPU and GPU profiler written in a few C++ files that you drop into your project. It provides scoped timers, c… | 64 | 1592 | active |
| photosynthesis-team/piq PyTorch Image Quality (PIQ) is a collection of measures and metrics for image quality assessment in image-to-image tasks, written in pure P… | 23 | 1576 | stable |
| RightNow-AI/autokernel AutoKernel is an open-source autoresearch pipeline that takes any PyTorch model, profiles it to find GPU kernel bottlenecks, extracts them … | 44 | 1571 | active |
| google-deepmind/bsuite bsuite (Behaviour Suite for Reinforcement Learning) is a collection of carefully-designed experiments from DeepMind that investigate core c… | 79 | 1564 | stable |
| alibaba/FederatedScope FederatedScope is a comprehensive federated learning platform built on PyTorch with an event-driven architecture. It provides convenient us… | 23 | 1545 | active |
| skforecast Skforecast is a Python library for time series forecasting that turns any scikit-learn compatible estimator (LightGBM, XGBoost, CatBoost, K… | 92 | 1536 | active |
| gradle/gradle-profiler A command-line tool from Gradle that automates gathering profiling and benchmarking data for Gradle builds. It measures build execution tim… | 91 | 1533 | active |
| Criterion.rs Criterion.rs is a statistics-driven microbenchmarking library for Rust that measures and analyzes code performance with rigorous statistica… | 64 | 5535 | maintenance |
| Benchee Benchee is an easy-to-use and extensible microbenchmarking library for Elixir. It runs functions with warmup, measures run time and memory … | 81 | 1519 | stable |
| bestiejs/benchmark.js Benchmark.js is a JavaScript benchmarking library that supports high-resolution timers and returns statistically significant results. It po… | 10 | 5453 | maintenance |
| adsr/phpspy phpspy is a low-overhead sampling profiler for PHP 7+ written in C. It can attach to running PHP processes (CLI, Apache, FPM) without code … | 65 | 1500 | active |
| scientistproject/Scientist.net Scientist.net is a .NET library for carefully refactoring critical code paths by running old and new implementations side by side in produc… | 54 | 1490 | active |
| cheind/py-motmetrics py-motmetrics is a Python library for evaluating multiple object tracking (MOT) results with MOTChallenge-aligned CLEAR MOT, Identity, and … | 61 | 1488 | active |
| callstack/reassure Reassure is a performance testing library for React and React Native that automates performance regression testing. It measures render dura… | 84 | 1484 | active |
| OpenGenerativeAI/llm-colosseum LLM Colosseum is a benchmarking tool that pits large language models against each other in real-time Street Fighter III matches to evaluate… | 18 | 1482 | active |
| reczoo/FuxiCTR FuxiCTR is an open-source Python library for click-through rate (CTR) prediction built on PyTorch and TensorFlow. It offers a configurable,… | 75 | 1481 | active |
| evo-hq/evo Evo is a plugin for agentic coding frameworks (Claude Code, Codex, Cursor, etc.) that turns a codebase into an autoresearch loop: it discov… | 77 | 1467 | active |
| ionelmc/pytest-benchmark A pytest plugin that provides a benchmark fixture for measuring code performance within tests. It automatically calibrates test rounds to t… | 82 | 1457 | stable |
| timescale/tsbs TSBS is a Go-based benchmark suite for generating time series datasets and measuring read/write performance across many time series databas… | 66 | 1454 | active |
| nvzqz/divan Divan is a fast and simple benchmarking library for Rust projects, used via a #[divan::bench] attribute and cargo bench. It offers allocati… | 68 | 1449 | active |
| brianfrankcooper/YCSB The Yahoo! Cloud Serving Benchmark (YCSB) is a Java framework and CLI tool for benchmarking NoSQL and cloud data stores against standardize… | 63 | 5233 | maintenance |
| nucliweb/webperf-snippets A curated collection of JavaScript snippets for measuring and debugging web performance (Core Web Vitals, loading, and interaction metrics)… | 83 | 1448 | active |
| boost-ext/ut UT (μt) is a C++20 micro unit testing framework distributed as a single header or C++20 module. It provides a macro-free, DSL-style API for… | 74 | 1445 | active |
| ferd/recon Recon is an Erlang library providing safe functions and scripts for diagnosing and debugging Erlang systems in production. It allows inspec… | 64 | 1430 | stable |
| NVIDIA/gdrcopy GDRCopy is a low-latency GPU memory copy library built on NVIDIA GPUDirect RDMA technology, allowing CPU-driven copies between host and GPU… | 83 | 1416 | active |
| Lyken17/pytorch-OpCounter THOP (PyTorch-OpCounter) is a Python library that counts the MACs/FLOPs and parameters of PyTorch models. It profiles arbitrary nn.Modules … | 32 | 5083 | maintenance |
| microsoft/waza Waza is a Go CLI from Microsoft for evaluating AI agent skills through structured benchmarks. Users define test cases and validation rules … | 82 | 1413 | active |
| microsoft/diskspd DiskSpd is a command-line storage load generator and performance testing tool from Microsoft's Windows and Cloud Server Infrastructure engi… | 83 | 1409 | active |
| LouisBrunner/valgrind-macos A patched mirror of Valgrind that adds support for recent macOS versions, distributed via a Homebrew tap. It lets developers run memory err… | 77 | 1398 | active |
| nschloe/perfplot A Python library that extends timeit by benchmarking code snippets across varying input parameters and plotting the timing results. It is u… | 37 | 1386 | active |
| Zefan-Cai/KVCache-Factory KVCache-Factory is a unified Python framework implementing many KV cache compression, retrieval, merging, and quantization methods for long… | 64 | 1384 | active |
| kengz/SLM-Lab SLM Lab is a modular deep reinforcement learning framework built in PyTorch, offering ready-to-use algorithms like PPO, SAC, DQN, and A2C v… | 91 | 1365 | active |
| nviennot/core-to-core-latency A Rust CLI tool that measures the latency of inter-core communication on a CPU by pinning threads to different cores and timing compare-exc… | 52 | 1364 | active |
| uxlfoundation/scikit-learn-intelex Intel's Extension for Scikit-learn is a free AI accelerator that speeds up existing scikit-learn workflows on CPUs and GPUs, claiming up to… | 87 | 1356 | active |
| appneta/tcpreplay Tcpreplay is a suite of GPLv3 command-line utilities for editing and replaying captured network traffic (pcap files) back onto the network … | 92 | 1352 | active |
| fenrus75/powertop PowerTOP is a Linux command-line tool for diagnosing power consumption and power management issues, with an interactive top-like TUI for ex… | 94 | 1345 | active |
| Spark spark is a performance profiler for Minecraft clients, servers, and proxies, combining a lightweight CPU profiler, memory inspection tools … | 76 | 1344 | active |
| awslabs/dgl-ke DGL-KE is a high-performance Python package built on Deep Graph Library (DGL) for training, evaluating, and inferring knowledge graph embed… | 60 | 1335 | active |
| guonaihong/gout gout is a Go HTTP client library designed as a 'Swiss Army Knife' for making HTTP requests with a fluent API. It supports multiple body enc… | 43 | 1327 | active |