Open-source benchmarking software
817 open-source libraries, tools and applications tagged benchmarking, ranked by how closely benchmarking is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
817 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| fortio/fortio Fortio is a Go-based load testing library, CLI tool, and web UI originally built for Istio. It generates load at a specified queries-per-se… | 99 | 3734 | stable |
| edtechre/pybroker PyBroker is a Python framework for developing algorithmic trading strategies with a focus on machine learning. It provides a fast NumPy/Num… | 92 | 3611 | active |
| RUC-NLPIR/FlashRAG FlashRAG is a modular Python toolkit for reproducing and developing Retrieval-Augmented Generation (RAG) research, bundling 36 pre-processe… | 88 | 3590 | active |
| LinShunKang/MyPerf4J MyPerf4J is a high-performance Java application performance monitoring (APM) tool implemented as a non-intrusive JavaAgent using ASM byteco… | 77 | 3569 | active |
| davidmarkclements/0x 0x is a single-command Node.js profiling tool that generates interactive flamegraphs from stack traces. It wraps any Node process, captures… | 39 | 3551 | active |
| namhyung/uftrace uftrace is a function call graph tracer for C, C++, Rust, and Python programs that records function entry/exit with timestamps, arguments, … | 83 | 3474 | active |
| badafans/better-cloudflare-ip A Go CLI tool that finds the best-performing Cloudflare Anycast IP addresses for a user's specific network environment. It randomly samples… | 70 | 3460 | active |
| MTEB MTEB (Massive Text Embedding Benchmark) is a Python library and CLI for benchmarking embedding models across 1000+ tasks, languages, and mo… | 95 | 3455 | active |
| modelscope/evalscope EvalScope is a Python framework from the ModelScope community for evaluating large language models, vision-language models, embedding model… | 91 | 3454 | active |
| yandex/perforator Perforator is a cluster-wide continuous profiling tool by Yandex that collects CPU profiles from production servers using eBPF with negligi… | 77 | 3437 | active |
| AltraMayor/f3 F3 (Fight Flash Fraud) is a command-line tool that tests flash storage devices like SD cards and USB drives to detect counterfeit capacity … | 79 | 3418 | stable |
| microsoft/nni NNI (Neural Network Intelligence) is an open-source AutoML toolkit from Microsoft that automates hyperparameter tuning, neural architecture… | 10 | 14360 | maintenance |
| Xdebug Xdebug is a PHP extension providing step debugging, profiling, tracing, code coverage, and improved error reporting for PHP applications. I… | 92 | 3416 | active |
| minitest Minitest is a small, fast, and readable testing suite for Ruby supporting TDD, BDD, spec-style expectations, and benchmarking. It ships as … | 70 | 3413 | stable |
| IQA-PyTorch A pure Python/PyTorch toolbox for image quality assessment (IQA) providing GPU-accelerated reimplementations of many full-reference and no-… | 83 | 3408 | active |
| NovaSky-AI/SkyThought SkyThought is the open-source repository behind Sky-T1, a family of reasoning language models trained for under $450, including training sc… | 22 | 3401 | active |
| bojand/ghz ghz is a command-line utility and Go package for benchmarking and load testing gRPC services. It supports unary, streaming, and duplex call… | 71 | 3358 | active |
| shankarpandala/lazypredict Lazy Predict is a Python library that trains dozens of machine learning models with minimal code to quickly identify which algorithms perfo… | 86 | 3352 | active |
| intel/pcm Intel Performance Counter Monitor (PCM) is an API and a set of tools for monitoring performance and energy metrics of Intel Core, Xeon, Ato… | 79 | 3345 | active |
| Celtoys/Remotery Remotery is a realtime CPU/GPU profiler implemented in a single C file, with a remote web-browser viewer for inspecting sampled traces. It … | 23 | 3317 | stable |
| WebPageTest The official open-source codebase for WebPageTest, the website performance testing service, which can be self-hosted as a private instance.… | 38 | 3295 | active |
| mljar/mljar-supervised MLJAR AutoML is a Python package for automated machine learning on tabular data, providing feature engineering, hyperparameter tuning, mode… | 85 | 3294 | active |
| oracle/visualvm VisualVM is an all-in-one visual troubleshooting and profiling tool for Java applications, integrating command-line JDK tools with lightwei… | 84 | 3267 | stable |
| pyutils/line_profiler line_profiler is a Python module for line-by-line profiling of functions, with the kernprof script for running profilers. It is the officia… | 74 | 3243 | active |
| DynamoRIO/dynamorio DynamoRIO is a runtime code manipulation system that supports arbitrary code transformations on executing programs, with a powerful instruc… | 95 | 3181 | active |
| alexei-led/pumba Pumba is a chaos testing and network emulation CLI tool for Docker, containerd, and Podman containers. It can kill, stop, pause, or remove … | 95 | 3177 | active |
| Phoronix Test Suite The Phoronix Test Suite is an open-source, cross-platform automated benchmarking and testing platform written in PHP. It provides an extens… | 66 | 3145 | stable |
| thuml/Time-Series-Library TSLib is an open-source Python library providing a unified codebase of advanced deep learning models for general time series analysis. It s… | 63 | 12923 | maintenance |
| felixge/fgprof fgprof is a sampling Go profiler that analyzes both On-CPU and Off-CPU (e.g. I/O) time together, also known as wall-clock profiling. It exp… | 28 | 3112 | stable |
| liboqs liboqs is an open source C library providing implementations of quantum-resistant (post-quantum) key encapsulation mechanisms and digital s… | 85 | 3071 | active |
| zombocom/derailed_benchmarks A Ruby gem providing a suite of benchmarking and profiling tools for Rails and Ruby applications. It measures memory usage of Gemfile depen… | 32 | 3068 | active |
| hegelai/prompttools PromptTools is an open-source Python library for testing, experimenting with, and evaluating prompts, LLMs (OpenAI, Anthropic, LLaMA, etc.)… | 46 | 3056 | active |
| hydradatabase/columnar Hydra Columnar is a Postgres extension providing column-oriented table storage via the table access method API, enabling fast analytical qu… | 23 | 3043 | stable |
| tensorflow/agents TF-Agents is a TensorFlow-based Python library for building, training, and deploying reinforcement learning and contextual bandit algorithm… | 57 | 3028 | active |
| MiniProfiler/dotnet MiniProfiler is a lightweight profiling library and UI for .NET, ASP.NET, and ASP.NET Core applications. It shows where time is spent in re… | 46 | 3004 | stable |
| pmorissette/bt bt is a flexible Python backtesting framework for testing quantitative trading strategies. It uses composable Algos and AlgoStacks with a t… | 96 | 2989 | active |
| sovrasov/flops-counter.pytorch A Python library (ptflops) that computes the theoretical number of multiply-add operations (FLOPs) and parameter counts for neural network … | 36 | 2983 | active |
| bethgelab/foolbox Foolbox is a Python library for generating adversarial examples that fool deep neural networks, with state-of-the-art gradient-based and de… | 44 | 2979 | active |
| ekzhu/datasketch datasketch is a Python library of probabilistic data structures (MinHash, Weighted MinHash, HyperLogLog, HyperLogLog++) and indexes (MinHas… | 90 | 2969 | stable |
| keras-team/keras-tuner KerasTuner is a hyperparameter optimization framework for Keras models with a define-by-run syntax for configuring search spaces. It ships … | 47 | 2923 | active |
| RoboTwin-Platform/RoboTwin RoboTwin 2.0 is a scalable benchmark and data-generation platform for bimanual (dual-arm) robotic manipulation, built on simulated digital … | 66 | 2914 | active |
| RL Baselines3 Zoo RL Baselines3 Zoo is a training framework for reinforcement learning agents built on Stable Baselines3. It provides CLI scripts for trainin… | 88 | 2892 | active |
| gem5 gem5 is the official open-source computer-system architecture simulator for research and teaching, modeling processor microarchitecture and… | 86 | 2871 | active |
| facebook/Ax Ax is a Python platform for adaptive experimentation that uses machine learning (Bayesian and bandit optimization via BoTorch) to efficient… | 93 | 2826 | active |
| symfony/stopwatch A Symfony component that provides a simple stopwatch API for profiling PHP code by timing named events, laps, and sections. It is a lightwe… | 83 | 2812 | stable |
| huawei-noah/HEBO A research monorepo from Huawei Noah's Ark Lab containing official implementations of Bayesian optimization, reinforcement learning, and ge… | 67 | 2812 | active |
| awslabs/amazon-redshift-utils A collection of Python utilities, admin scripts, views, and stored procedures for managing and optimizing Amazon Redshift data warehouse cl… | 47 | 2802 | active |
| ColinIanKing/stress-ng stress-ng is a command-line stress testing tool with 380+ stressors that exercise CPU, memory, filesystem, and OS kernel interfaces. It is … | 94 | 2793 | active |
| vava-nessa/free-coding-models A CLI tool that tracks a large catalog of free coding LLM models across 20+ providers in real time, benchmarking their speed and stability.… | 77 | 2790 | active |
| RazrFalcon/cargo-bloat A cargo subcommand that analyzes Rust executables to show which functions and crates take up the most space in the binary. It supports ELF,… | 32 | 2769 | stable |
| openjdk/jmh JMH is the Java Microbenchmark Harness, a tool for building, running, and analysing nano/micro/milli/macro benchmarks on the JVM. It genera… | 70 | 2684 | stable |
| TheTumultuousUnicornOfDarkness/CPU-X CPU-X is a free and open-source system profiling and monitoring application for GNU/Linux and FreeBSD, similar to CPU-Z on Windows. It repo… | 10 | 2658 | active |
| EMI-Group/evox EvoX is a distributed GPU-accelerated framework for evolutionary computation, compatible with PyTorch and built on JAX. It provides 50+ evo… | 97 | 2640 | active |
| NoiseByNorthwest/php-spx SPX is a PHP profiling extension written in C that collects function-level performance metrics (time, memory, I/O) with full call stack con… | 74 | 2638 | active |
| jmcarpenter2/swifter Swifter is a Python library that speeds up pandas apply operations by automatically choosing the fastest execution strategy, whether vector… | 32 | 2630 | active |
| processone/tsung Tsung is a multi-protocol distributed load testing and benchmarking tool written in Erlang. It simulates thousands of clients to test the s… | 60 | 2628 | active |
| GameTechDev/PresentMon PresentMon is a set of tools for capturing and analyzing the performance characteristics of graphics applications on Windows, tracing frame… | 90 | 2627 | active |
| linux-test-project/ltp The Linux Test Project is a collection of test suites and tools for validating the reliability, robustness, and stability of the Linux kern… | 91 | 2609 | active |
| yandex/yandex-tank Yandex.Tank is an extensible open-source load and performance testing tool for Linux, supporting multiple load generators such as Phantom, … | 67 | 2599 | active |
| rbspy rbspy is a sampling CPU profiler for Ruby that can profile already-running processes by PID or launch new ones, with low overhead safe for … | 97 | 2573 | active |
| jiffyclub/snakeviz SnakeViz is a browser-based graphical viewer for Python cProfile output, serving as a visual alternative to the pstats module. It runs as a… | 59 | 2566 | stable |
| huggingface/lighteval Lighteval is Hugging Face's all-in-one toolkit for evaluating large language models across multiple inference backends (Transformers, vLLM,… | 72 | 2563 | active |
| evanwashere/mitata Mitata is a microbenchmarking library for JavaScript runtimes (Bun, Deno, Node.js) and C++ via a single header. It provides a clean API for… | 34 | 2538 | active |
| google/XNNPACK XNNPACK is a highly optimized C library of floating-point neural network inference operators for ARM, x86, WebAssembly, and RISC-V platform… | 77 | 2481 | active |
| Lightning-AI/torchmetrics TorchMetrics is a PyTorch-native library of machine learning metrics that work in distributed and scalable training pipelines. It provides … | 87 | 2477 | stable |
| stephencookdev/speed-measure-webpack-plugin A webpack plugin that measures and reports the time spent in each loader and plugin during webpack builds. It wraps your webpack config and… | 62 | 2441 | active |
| youngyangyang04/Skiplist-CPP A lightweight key-value storage engine implemented in C++ using a skiplist as the core data structure, supporting insert, delete, search, d… | 32 | 2398 | stable |
| gkamradt/needle-in-a-haystack A benchmarking tool (niah / needlehaystack) that pressure-tests LLM long-context retrieval by sweeping context length x needle depth cells … | 66 | 2397 | active |
| tinylibs/tinybench Tinybench is a tiny (~10KB) JavaScript benchmarking library built on Web APIs with accurate timing via process.hrtime or performance.now. I… | 98 | 2386 | active |
| yse/easy_profiler A lightweight cross-platform profiler library for C++ that instruments arbitrary code blocks with extremely low overhead (~15ns per block).… | 62 | 2375 | stable |
| uptrain-ai/uptrain UpTrain is an open-source Python platform for evaluating and improving LLM applications, offering 20+ preconfigured evaluation checks cover… | 23 | 2366 | active |
| IBM/AssetOpsBench AssetOpsBench is an open-source benchmark and framework from IBM for building, orchestrating, and evaluating domain-specific AI agents for … | 65 | 2362 | active |
| wilicc/gpu-burn GPU Burn is a multi-GPU CUDA stress test tool that pushes NVIDIA GPUs to maximum load for stability and thermal testing. It supports config… | 76 | 2356 | active |
| oneclickvirt/ecs A comprehensive VPS/server benchmarking tool written in Go that tests system information, network performance, and IP quality with zero env… | 83 | 2351 | active |
| mrdoob/stats.js A tiny JavaScript library that displays an on-page info box monitoring rendering performance, showing FPS, frame time in milliseconds, and … | 23 | 9150 | maintenance |
| marmelab/gremlins.js gremlins.js is a JavaScript monkey testing library for Node.js and the browser that unleashes random user-like actions (clicking, typing, s… | 23 | 9096 | maintenance |
| HdrHistogram HdrHistogram is a Java library for recording and analyzing sampled data value counts across a configurable integer value range with configu… | 23 | 2314 | stable |
| fcsonline/drill Drill is a lightweight HTTP load testing tool written in Rust, designed as a JVM-free alternative to tools like JMeter. Benchmark plans are… | 84 | 2311 | active |
| gperftools/gperftools gperftools (Google Performance Tools) is a collection of high-performance multi-threaded malloc implementations (tcmalloc) plus CPU and hea… | 87 | 8982 | maintenance |
| macbre/phantomas A headless Chromium-based web performance metrics collector and monitoring tool built on Node.js and Puppeteer. It loads pages in a headles… | 96 | 2263 | active |
| facebookresearch/fvcore fvcore is a lightweight Python core library providing common functionality shared across FAIR's computer vision frameworks such as Detectro… | 73 | 2254 | stable |
| andikleen/pmu-tools pmu-tools is a collection of Python tools and C libraries for profile collection and performance analysis on Intel CPUs, built on top of Li… | 71 | 2251 | active |
| LloydAsp/NodeQuality A shell-based VPS testing tool that runs benchmark scripts (Yabs, IP quality, network quality) inside a temporary chroot sandbox (BenchOS) … | 47 | 2248 | active |
| cyring/CoreFreq CoreFreq is a CPU monitoring and tuning tool for 64-bit processors that combines a Linux kernel module with a terminal UI to report frequen… | 95 | 2246 | active |
| PragmaticFlow/NBomber NBomber is a distributed load-testing framework for .NET that lets you write load test scenarios in plain C# or F# with no dependency on pr… | 88 | 2241 | active |
| Austin Austin is a frame stack sampler for CPython written in C, used to profile running Python processes without instrumentation. It samples stac… | 68 | 2209 | active |
| BIMK/PlatEMO PlatEMO is a MATLAB-based platform for evolutionary multi-objective optimization developed by BIMK at Anhui University. It provides 300+ op… | 84 | 2207 | active |
| scandum/quadsort Quadsort is a branchless stable adaptive merge sort implemented in C, designed to outperform quicksort on many workloads. It uses a quad sw… | 32 | 2204 | active |
| tmm1/stackprof Stackprof is a sampling call-stack profiler for Ruby 2.2+ inspired by gperftools. It collects CPU, wall, object, and other profile samples … | 62 | 2195 | stable |
| rurban/smhasher SMhasher is a test suite for evaluating the quality and speed of non-cryptographic and cryptographic hash functions. It benchmarks throughp… | 75 | 2177 | active |
| TsingZ0/PFLlib PFLlib is a beginner-friendly Python library and benchmark for traditional and personalized federated learning, offering 39 algorithms, 24 … | 55 | 2166 | active |
| coding-kitties/investing-algorithm-framework A Python framework for developing, backtesting, and deploying quantitative trading algorithms and trading bots. It supports vector and even… | 87 | 2165 | active |
| Open Motion Planning Library (OMPL) OMPL is an open-source C++ library of over 40 sampling-based motion planning algorithms (RRT-Connect, PRM, RRT*, KPIECE, etc.) across more … | 90 | 2156 | stable |
| microsoft/Microsoft.IO.RecyclableMemoryStream A .NET library that pools MemoryStream buffers to reduce garbage collection pressure and Large Object Heap allocations in performance-criti… | 66 | 2152 | stable |
| enarjord/passivbot Passivbot is a cryptocurrency trading bot written in Python and Rust that automatically places and cancels limit orders on perpetual future… | 96 | 2130 | active |
| tairov/llama2.mojo A single-file implementation of Llama 2 transformer inference written in pure Mojo, leveraging SIMD and vectorization for fast CPU inferenc… | 71 | 2129 | active |
| bcoe/c8 c8 is a CLI tool that generates code coverage reports for Node.js applications using Node's built-in V8 coverage output, formatted with Ist… | 83 | 2122 | active |
| automl/auto-sklearn auto-sklearn is an automated machine learning (AutoML) toolkit that acts as a drop-in replacement for scikit-learn estimators, automaticall… | 66 | 8135 | maintenance |
| marlbenchmark/on-policy The official PyTorch implementation of Multi-Agent PPO (MAPPO), an on-policy reinforcement learning algorithm for cooperative multi-agent s… | 32 | 2104 | active |
| Avalanche Avalanche is an end-to-end continual learning library built on PyTorch, developed by ContinualAI. It provides modules for benchmarks, train… | 24 | 2104 | active |