Open-source benchmarking software
817 open-source libraries, tools and applications tagged benchmarking, ranked by how closely benchmarking is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
817 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| cure-lab/LTSF-Linear Official PyTorch implementation of LTSF-Linear (Linear, DLinear, NLinear) from the AAAI-23 paper 'Are Transformers Effective for Time Serie… | 32 | 2520 | maintenance |
| cleanbrowsing/dnsperftest A shell script that benchmarks the latency of popular public DNS resolvers (Cloudflare, Google, Quad9, OpenDNS, etc.) from your machine. It… | 32 | 2517 | maintenance |
| huggingface/evaluate Hugging Face's library for easily evaluating machine learning models and datasets with dozens of standardized metrics, comparisons, and mea… | 74 | 2491 | maintenance |
| zjw-swun/AppMethodOrder An Android library that traces the call order and execution time of all methods in an app without invasive code changes, filtering by packa… | 32 | 2490 | maintenance |
| YellowLabTools/YellowLabTools Yellow Lab Tools is a web performance and front-end code quality auditing tool that loads a webpage, collects metrics via Phantomas/Puppete… | 23 | 2461 | maintenance |
| HobbitLong/RepDistiller RepDistiller is a PyTorch research codebase implementing Contrastive Representation Distillation (CRD) from ICLR 2020. It also benchmarks 1… | 32 | 2444 | maintenance |
| IST-DASLab/gptq Reference implementation of GPTQ, a one-shot post-training weight quantization method for large generative transformer models based on appr… | 32 | 2386 | maintenance |
| alibaba/TProfiler TProfiler is a Java code profiling tool from Alibaba designed to run long-term in production environments. It attaches to a Java 6+ VM to c… | 32 | 2371 | maintenance |
| nitin42/react-perf-devtool A browser developer tool extension for inspecting the rendering performance of React components, based on measures collected via the window… | 23 | 2307 | maintenance |
| Flow Forecast Flow Forecast is a PyTorch-based deep learning framework for time series forecasting, classification, and anomaly detection. It offers stat… | 66 | 2302 | maintenance |
| kconner/KMCGeigerCounter An iOS library that acts as a framerate meter, playing a Geiger-counter-style click whenever the app drops a frame during animation. It als… | 32 | 2245 | maintenance |
| smallnest/go-web-framework-benchmark A benchmark suite comparing the full HTTP request processing performance of dozens of Go web frameworks, including gin, echo, fiber, beego,… | 70 | 2134 | maintenance |
| rlworkgroup/garage garage is a Python toolkit for developing and evaluating reinforcement learning algorithms, with a library of state-of-the-art RL algorithm… | 23 | 2130 | maintenance |
| facebookresearch/SentEval SentEval is a Python library from Facebook Research for evaluating the quality of sentence embeddings via a suite of downstream transfer ta… | 10 | 2111 | maintenance |
| sfujim/TD3 The author's official PyTorch implementation of Twin Delayed Deep Deterministic Policy Gradients (TD3), a reinforcement learning algorithm … | 32 | 2110 | maintenance |
| naver/ngrinder nGrinder is an enterprise-level performance and stress testing platform consisting of a web-based controller and load-generating agents. It… | 10 | 2101 | maintenance |
| miloyip/nativejson-benchmark A benchmark suite evaluating the conformance and performance (speed, memory, code size) of 41 open-source C/C++ JSON parsing and generation… | 23 | 2070 | maintenance |
| google/butteraugli Butteraugli is a C++ library and command-line tool that estimates the psychovisual difference between two images, producing both a scalar s… | 10 | 1999 | maintenance |
| mtrebi/memory-allocators A C++ library implementing several custom memory allocators (linear, stack, pool, and free list) as faster alternatives to malloc/free. It … | 75 | 1994 | maintenance |
| catapult-project/catapult Catapult is a collection of performance tools from the Chromium project for gathering, displaying, and analyzing performance data, includin… | 10 | 1965 | maintenance |
| Tiiiger/bert_score BERTScore is a PyTorch implementation of the BERTScore automatic evaluation metric for text generation, which scores generated text against… | 23 | 1920 | maintenance |
| snap-stanford/GraphGym GraphGym is a platform for designing and evaluating Graph Neural Networks (GNNs), built around a highly modularized pipeline covering data … | 23 | 1908 | maintenance |
| xinshuoweng/AB3DMOT Official Python implementation of AB3DMOT, a simple real-time baseline for 3D multi-object tracking using oriented 3D bounding boxes from L… | 32 | 1846 | maintenance |
| THUDM/CogDL CogDL is a Python/PyTorch toolkit for graph deep learning that lets researchers train and compare GNN and graph embedding models on tasks l… | 23 | 1820 | maintenance |
| google/compare_gan A TensorFlow library from Google Research providing configurable implementations of GAN components: losses, penalties, normalization techni… | 10 | 1815 | maintenance |
| uber-common/jvm-profiler A Java Agent library from Uber that collects CPU, memory, IO metrics and stacktraces from JVM processes in a distributed way, with reporter… | 66 | 1801 | maintenance |
| AlphaPy AlphaPy is a Python machine learning framework built on scikit-learn, pandas, Keras, XGBoost, LightGBM, and CatBoost for building classific… | 36 | 1756 | maintenance |
| GPGPU-Sim GPGPU-Sim is a cycle-level simulator that models contemporary NVIDIA GPUs running CUDA and OpenCL workloads, including support for TensorCo… | 78 | 1750 | maintenance |
| Maratyszcza/NNPACK NNPACK is a C99 acceleration package providing high-performance SIMD and multi-core CPU implementations of neural network layers, especiall… | 32 | 1710 | maintenance |
| jondot/goweight goweight is a command-line tool that analyzes Go binaries and breaks down their size by module or package. It supports Go modules and can o… | 23 | 1707 | maintenance |
| jvm-profiling-tools/perf-map-agent A Java agent (written in C with a Java bootstrap) that generates /tmp/perf-<pid>.map symbol files for JIT-compiled methods so the Linux per… | 32 | 1701 | maintenance |
| open-mmlab/mmrazor MMRazor is OpenMMLab's model compression toolbox and benchmark built on PyTorch. It provides implementations of neural architecture search,… | 23 | 1683 | maintenance |
| topepo/caret caret (Classification And Regression Training) is an R package providing a unified interface for training, tuning, and evaluating classific… | 64 | 1669 | maintenance |
| GoogleChromeLabs/size-plugin A Webpack plugin from GoogleChromeLabs that prints the gzipped sizes of your webpack assets and the changes since the last build. It tracks… | 58 | 1662 | maintenance |
| hfawaz/dl-4-tsc Companion repository for the paper 'Deep learning for time series classification: a review', providing TensorFlow 2 implementations of nine… | 32 | 1660 | maintenance |
| julienschmidt/go-http-routing-benchmark A benchmark suite comparing the performance of Go HTTP request routers and web frameworks by implementing real-world API routing structures… | 32 | 1658 | maintenance |
| benhamner/Metrics A multi-language library providing implementations of supervised machine learning evaluation metrics such as AUC, log loss, MAE, RMSE, and … | 32 | 1651 | maintenance |
| zhengcx/MethodTraceMan MethodTraceMan is an Android performance tool that finds high time-consuming methods causing app jank via bytecode instrumentation. It uses… | 23 | 1650 | maintenance |
| autonomio/talos Talos is a Python library for automating hyperparameter experiments and model evaluation with TensorFlow (tf.keras), Keras, and PyTorch mod… | 98 | 1634 | maintenance |
| google/highwayhash A C++ library providing fast, strong (well-distributed and unpredictable) hash functions: a portable SipHash implementation and HighwayHash… | 10 | 1603 | maintenance |
| maciejczyzewski/libchaos A C++ library implementing randomization, hashing, and statistical analysis algorithms based on the concept of chaos machines. It aims to r… | 23 | 1597 | maintenance |
| andywer/leakage Leakage is a Node.js library for detecting memory leaks by running a function repeatedly and analyzing V8 heap snapshot diffs. It integrate… | 23 | 1588 | maintenance |
| HIPS/Spearmint Spearmint is a Python package for Bayesian optimization that automatically tunes experiment parameters to minimize an objective in as few r… | 32 | 1573 | maintenance |
| frogermcs/AndroidDevMetrics AndroidDevMetrics is a performance metrics library for Android development that measures Activity lifecycle method execution times, frame r… | 32 | 1531 | maintenance |
| Eric-mingjie/rethinking-network-pruning A PyTorch research codebase reproducing the ICLR 2019 paper 'Rethinking the Value of Network Pruning', which shows pruned models trained fr… | 32 | 1513 | maintenance |
| oooldking/script A collection of Linux shell scripts for VPS server benchmarking, including SuperBench (download, I/O, and network speed tests), SuperSpeed … | 33 | 1503 | maintenance |
| Swall0w/torchstat torchstat is a lightweight neural network analyzer for PyTorch that reports parameter counts, FLOPs, multiply-adds, and memory usage per la… | 23 | 1499 | maintenance |
| microsoft/pict PICT is a command-line tool from Microsoft that generates compact pairwise (combinatorial) test cases from a model file of parameters and v… | 65 | 1467 | maintenance |
| kdlucas/byte-unixbench UnixBench is the original BYTE UNIX benchmark suite, a C-based tool that measures overall Unix-like system performance via multiple tests (… | 70 | 1446 | maintenance |
| google-research/disentanglement_lib disentanglement_lib is an open-source Python library for research on learning disentangled representations, supporting models like BetaVAE,… | 10 | 1425 | maintenance |
| pympler/pympler Pympler is a pure-Python development library for measuring, monitoring, and analyzing the memory behavior of Python objects in a running ap… | 40 | 1412 | maintenance |
| safety-research/bloom Bloom is a Python tool that automatically generates behavioral evaluation suites for LLMs, probing target models for behaviors like sycopha… | 52 | 1408 | maintenance |
| JasperSnoek/spearmint Spearmint is a Python package for Bayesian optimization of machine learning algorithms, iteratively tuning parameters to minimize an object… | 32 | 1392 | maintenance |
| Maluuba/nlg-eval A Python library and CLI tool that computes unsupervised automated metrics (BLEU, METEOR, ROUGE, CIDEr, SPICE, SkipThought, and embedding-b… | 23 | 1392 | maintenance |
| facebookresearch/LAMA LAMA is a research probe for analyzing the factual and commonsense knowledge stored in pretrained language models such as BERT, RoBERTa, GP… | 10 | 1392 | maintenance |
| tinkoff-ai/CORL CORL is a research-oriented Python library providing thoroughly benchmarked single-file implementations of state-of-the-art offline and off… | 10 | 1374 | maintenance |
| linkedin/dr-elephant Dr. Elephant is a self-serve performance monitoring and tuning tool for Apache Hadoop and Apache Spark jobs and workflows. It automatically… | 23 | 1370 | maintenance |
| BorealisAI/advertorch AdverTorch is a Python toolbox for adversarial robustness research built on PyTorch. It provides modules for generating adversarial perturb… | 32 | 1365 | maintenance |
| hallatore/Netling Netling is a fast, lightweight load testing client for web testing, available as a WPF GUI, console client, or embeddable .NET library. It … | 23 | 1346 | maintenance |
| desktoppr/wbench WBench is a Ruby command-line tool that benchmarks end-user website load times using the HTML5 performance timing API. It simulates first-t… | 32 | 1330 | maintenance |
| OpenCover/opencover OpenCover is a code coverage tool for .NET 2 and above on Windows, supporting 32 and 64-bit processes with both branch and sequence point c… | 10 | 1323 | maintenance |
| YelpArchive/MOE MOE (Metric Optimization Engine) is a global, black box optimization engine built by Yelp for optimizing expensive, time-consuming objectiv… | 10 | 1320 | maintenance |
| tysam-code/hlb-CIFAR10 A single-file PyTorch implementation that trains a neural network to 94% accuracy on CIFAR-10 in under 6.3 seconds on a single A100 GPU, fo… | 23 | 1310 | maintenance |
| observing/thor Thor is a Node.js CLI tool for benchmarking and load testing WebSocket servers. It generates large numbers of persistent WebSocket connecti… | 32 | 1302 | maintenance |
| jsperf/jsperf.com The source code for jsPerf.com, a web application for creating, sharing, and running JavaScript performance benchmarks in the browser. It i… | 32 | 1296 | maintenance |
| palash1992/GEM GEM is a Python library offering a unified framework for static graph embedding algorithms, including node2vec, HOPE, SDNE, Graph Factoriza… | 23 | 1286 | maintenance |
| joealcorn/laboratory A Python library inspired by GitHub's Scientist for safely refactoring critical code paths by running experiments in production. It execute… | 55 | 1282 | maintenance |
| Nanoscope Nanoscope is an extremely accurate Android method tracing tool from Uber, implemented partly as a fork of AOSP to achieve ~20 nanoseconds o… | 23 | 1275 | maintenance |
| attaswift/Attabench Attabench is a macOS microbenchmarking application for Swift that measures the performance of low-level algorithms across varying input siz… | 23 | 1273 | maintenance |
| jvm-profiling-tools/honest-profiler Honest Profiler is a sampling profiler for the JVM implemented as a native agent, designed to avoid the safepoint sample bias that skews ot… | 32 | 1252 | maintenance |
| patric-r/jvmtop jvmtop is a lightweight top-like console application that monitors all accessible running JVMs on a machine, displaying JVM internal metric… | 23 | 1238 | maintenance |
| VerySleepy/verysleepy Very Sleepy is a free open-source sampling CPU profiler for Windows with a wxWidgets-based GUI. It can attach to any running native Windows… | 23 | 1233 | maintenance |
| tylin/coco-caption The official evaluation code for the Microsoft COCO image captioning benchmark, implementing metrics such as BLEU, METEOR, ROUGE-L, CIDEr, … | 32 | 1225 | maintenance |
| lukejacksonn/perflink Perflink is a frontend-only single page web app for benchmarking JavaScript code snippets, hosted on GitHub Pages at perf.link. It runs tes… | 32 | 1224 | maintenance |
| ucbdrive/few-shot-object-detection FsDet is the official implementation of the ICML 2020 paper 'Frustratingly Simple Few-Shot Object Detection' (TFA), built on detectron2. It… | 23 | 1219 | maintenance |
| uzh-rpg/rpg_trajectory_evaluation A Python toolbox for quantitatively evaluating visual(-inertial) odometry trajectories, supporting multiple alignment methods and standard … | 23 | 1215 | maintenance |
| pschraut/UnityHeapExplorer Heap Explorer is a memory profiler, debugger, and analyzer plugin for Unity 2019.3 through Unity 6, installed via the Unity Package Manager… | 32 | 1210 | maintenance |
| tarekziade/boom Boom! is a simple Python command-line HTTP load testing tool designed as a replacement for Apache Bench, using greenlets to simulate thousa… | 10 | 1209 | maintenance |
| passy/build-time-tracker-plugin A Gradle plugin that continuously tracks and reports build times across builds, with summary bar charts and CSV output. It complements Grad… | 32 | 1204 | maintenance |
| sitespeedio/coach The Coach is a web performance tool that analyzes pages and gives advice on performance problems, acting as a modern successor to YSlow. It… | 10 | 1196 | maintenance |
| haydenjames/bench-scripts A curated collection of Linux server benchmarking scripts and command-line tools for testing CPU, disk I/O, memory, and network performance… | 32 | 1194 | maintenance |
| flow-project/flow Flow is a computational framework for deep reinforcement learning and control experiments in traffic microsimulation, built on top of the S… | 23 | 1192 | maintenance |
| thlorenz/deoptigate Deoptigate is a CLI tool that investigates V8/Node.js function deoptimizations by processing v8 logs and presenting them in a visual viewer… | 32 | 1169 | maintenance |
| svanoort/pyresttest PyRestTest is a Python-based REST API testing and microbenchmarking tool where tests are defined declaratively in YAML or JSON files withou… | 23 | 1164 | maintenance |
| google/active-learning A Python module for running experiments comparing different active learning algorithms on benchmark datasets. It provides a main experiment… | 10 | 1163 | maintenance |
| wyharveychen/CloserLookFewShot Reference PyTorch implementation of the ICLR 2019 paper 'A Closer Look at Few-shot Classification', providing a testbed for empirical evalu… | 32 | 1160 | maintenance |
| DeepGraphLearning/RecommenderSystems A Python research library collecting the authors' recommender systems code, covering sequential, feature-based (CTR), and social recommenda… | 32 | 1156 | maintenance |
| tensorflow/benchmarks A benchmark framework for TensorFlow, containing PerfZero for running and reporting TensorFlow benchmarks and legacy CNN benchmark scripts … | 10 | 1148 | maintenance |
| wasabeef/Takt Takt is an Android library that measures an app's frame rate (FPS) using the Choreographer API and displays it as an on-screen overlay. It … | 32 | 1144 | maintenance |
| LLMPerf LLMPerf is a Python library for benchmarking and validating the performance of LLM APIs. It runs load tests measuring inter-token latency a… | 10 | 1123 | maintenance |
| emmericp/MoonGen MoonGen is a scriptable high-speed packet generator built on DPDK and LuaJIT that can saturate 10 Gbit/s Ethernet links with 64-byte packet… | 53 | 1118 | maintenance |
| CCExtractor/vardbg vardbg is a simple Python debugger and profiler that records variable history and execution times, then generates animated videos visualizi… | 32 | 1113 | maintenance |
| baidu-research/DeepBench DeepBench is a benchmarking suite from Baidu Research that measures the performance of fundamental deep learning operations (matrix multipl… | 32 | 1109 | maintenance |
| torvalds/test-tlb A small C benchmark tool by Linus Torvalds that measures memory access latency and TLB behavior across working-set sizes from a few kilobyt… | 32 | 1065 | maintenance |
| aliutkus/speechmetrics A Python library that wraps several objective speech quality metrics (MOSNet, BSSEval, STOI, PESQ, SRMR, SISDR) behind a unified API. It su… | 32 | 1058 | maintenance |
| graphdeeplearning/graphtransformer Reference implementation of the Graph Transformer architecture from the paper 'A Generalization of Transformer Networks to Graphs' (DLG-AAA… | 32 | 1030 | maintenance |
| pyheat An IPython magic extension that profiles Python code and renders the results as a heat map, wrapping the py-heat library. It exposes a %%he… | 32 | 1029 | maintenance |
| kenjis/php-framework-benchmark A benchmark suite that measures the minimum bootstrap overhead of PHP frameworks using a Hello World application. It compares requests per … | 32 | 1027 | maintenance |
| tmm1/perftools.rb A Ruby binding to Google's gperftools that provides CPU profiling of Ruby code via sampling. It supports profiling blocks, whole applicatio… | 32 | 1025 | maintenance |
| facebookresearch/CompilerGym CompilerGym is a Python library of OpenAI Gym-compatible reinforcement learning environments for compiler and program optimization tasks, c… | 10 | 1021 | maintenance |