Open-source benchmarking software
817 open-source libraries, tools and applications tagged benchmarking, ranked by how closely benchmarking is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
817 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| HongyuanLuke/frequencylaw Official code repository for the paper 'Textual Frequency Law on Large Language Models', implementing TFL, TFD, and CTFT methods for studyi… | 46 | 1428 | active |
| AgibotTech/genie_sim Genie Sim is AgiBot's open-source simulation platform for embodied intelligence, providing environment reconstruction, LLM-driven scene gen… | 70 | 1412 | active |
| ServiceNow/BrowserGym BrowserGym is a Gym-style environment library for web task automation, providing a unified interface to web agent benchmarks like MiniWoB, … | 78 | 1376 | active |
| meridianlabs-ai/inspect_petri Inspect Petri is an alignment auditing agent that automatically probes language models for concerning behaviors like sycophancy, reward hac… | 81 | 1375 | active |
| ZJU-REAL/ClawGUI ClawGUI is a unified Python framework for GUI agents covering the full lifecycle: online reinforcement learning training (ClawGUI-RL with G… | 64 | 1356 | active |
| BytedTsinghua-SIA/CUDA-Agent CUDA-Agent is a large-scale agentic reinforcement learning system from ByteDance Seed and Tsinghua that trains LLMs to generate high-perfor… | 53 | 1338 | active |
| evilsocket/nerve Nerve is a simple Agent Development Kit (ADK) for building, running, evaluating, and orchestrating LLM-based agents using YAML definitions … | 10 | 1318 | active |
| ChenmienTan/RL2 RL2 (Ray Less Reinforcement Learning) is a concise Python library for post-training large language models with reinforcement learning, SFT,… | 54 | 1316 | active |
| PrunaAI/pruna Pruna is an open-source Python model optimization framework that makes AI models faster, smaller, cheaper, and greener via caching, quantiz… | 80 | 1288 | active |
| poetiq-ai/poetiq-arc-agi-solver A Python research codebase that reproduces Poetiq's record-breaking, leaderboard-topping submissions to the ARC-AGI-1 and ARC-AGI-2 abstrac… | 39 | 1283 | active |
| tensorflow/model-analysis TensorFlow Model Analysis (TFMA) is a Python library for evaluating TensorFlow models on large datasets in a distributed manner. It compute… | 77 | 1267 | active |
| plurai-ai/intellagent IntellAgent is a Python framework for diagnosing and optimizing conversational LLM agents through simulated, realistic synthetic user inter… | 55 | 1258 | active |
| meta-pytorch/attention-gym Attention Gym is a collection of tools, examples, and reference implementations for working with PyTorch's FlexAttention API. It provides a… | 91 | 1257 | active |
| Tencent/GT GT (Great Tit) is a portable on-device debugging tool from Tencent that acts as an Integrated Debug Environment running directly on smartph… | 23 | 4395 | maintenance |
| NVlabs/alpasim AlpaSim is an open-source, Python-based autonomous vehicle simulation platform for developing and testing end-to-end AV policies in closed … | 72 | 1245 | active |
| python-adaptive/adaptive Adaptive is an open-source Python library for parallel active learning of mathematical functions. It intelligently selects the most informa… | 87 | 1229 | active |
| sentient-agi/EvoSkill EvoSkill is an open-source Python framework that automatically discovers and synthesizes reusable skills from failed agent trajectories to … | 72 | 1229 | active |
| datawhalechina/torch-rechub Torch-RecHub is a lightweight PyTorch framework for building recommendation system models with 30+ out-of-the-box algorithms covering ranki… | 96 | 1227 | active |
| langchain-ai/openevals OpenEvals is a library of readymade evaluators for testing LLM applications, including LLM-as-judge evaluators and reference-based/referenc… | 86 | 1226 | active |
| yanchunhuo/AutomationTest A Python-based automation testing framework supporting API automation, web UI automation (Selenium), app UI automation (Appium), and perfor… | 59 | 1221 | active |
| aydinnyunus/ai-captcha-bypass A Python command-line tool that uses multimodal LLMs (GPT-4o, Gemini) to automatically solve various CAPTCHA types, including text, reCAPTC… | 58 | 1219 | active |
| Zefan-Cai/R-KV R-KV is a training-free, redundancy-aware KV cache compression method for reasoning LLMs, discarding repetitive tokens on-the-fly during de… | 64 | 1214 | active |
| ubicomplab/rPPG-Toolbox rPPG-Toolbox is an open-source Python toolbox for camera-based physiological sensing (remote photoplethysmography), enabling heart rate and… | 48 | 1214 | active |
| SalesforceAIResearch/enterprise-deep-research Enterprise Deep Research (EDR) is a multi-agent deep research system from Salesforce AI Research that combines a master planning agent, spe… | 52 | 1212 | active |
| SimAI SimAI is a large-scale network simulation toolkit from Alibaba Cloud for modeling AI training and inference workloads on GPU clusters, publ… | 65 | 1201 | active |
| getkeops/keops KeOps (pykeops) is a Python library for computing kernel reductions over large arrays on CPUs and GPUs using efficient C++/CUDA routines wi… | 61 | 1192 | active |
| vladmihalcea/flexy-pool FlexyPool is a Java library that wraps existing JDBC connection pools (HikariCP, DBCP, C3P0, Tomcat CP, Vibur, Druid, Atomikos) to add metr… | 60 | 1181 | active |
| LibCity LibCity is an open-source PyTorch library for urban spatial-temporal data mining, providing a unified pipeline for traffic prediction resea… | 23 | 1172 | active |
| Tencent/hpc-ops HPC-Ops is a production-grade C++/CUDA operator library for high-performance LLM inference, developed by Tencent's Hunyuan AI Infra team. I… | 58 | 1170 | active |
| simpler-env/SimplerEnv SIMPLER (SimplerEnv) is a collection of simulated environments built on SAPIEN/ManiSkill for evaluating real-world robot manipulation polic… | 48 | 1169 | active |
| Object Detection Metrics A Python toolkit implementing the most popular metrics (AP, mAP, precision-recall curves) used to evaluate object detection algorithms, wit… | 46 | 1167 | stable |
| ShiqiYu/OpenGait OpenGait is a flexible and extensible Python framework for gait recognition research, providing implementations of state-of-the-art models … | 65 | 1154 | active |
| cvg/glue-factory Glue Factory is a PyTorch-based library for training and evaluating deep neural networks that detect and match local visual features (point… | 66 | 1153 | active |
| JDAI-CV/fast-reid FastReID is a PyTorch-based research platform implementing state-of-the-art re-identification algorithms for persons, vehicles, and faces. … | 23 | 3995 | maintenance |
| kahlan/kahlan Kahlan is a full-featured Unit and BDD test framework for PHP using a describe-it syntax similar to RSpec/JSpec. It includes built-in stubb… | 95 | 1146 | active |
| InternRobotics/InternNav InternNav is an open-source PyTorch-based toolbox for building embodied navigation foundation models, supporting vision-language navigation… | 52 | 1135 | active |
| XYZ-AI-Lab/AxisAgentic AxisAgentic is an extensible Python runtime for long-horizon AI agents that handles multi-turn execution, tool orchestration, context manag… | 52 | 1120 | active |
| luchris429/purejaxrl PureJaxRL provides end-to-end reinforcement learning training pipelines implemented entirely in JAX, including environments, enabling massi… | 31 | 1118 | active |
| PKU-MARL/DexterousHands Bi-DexHands is a Python library providing bimanual dexterous hand manipulation environments built on NVIDIA Isaac Gym for reinforcement lea… | 32 | 1104 | active |
| agi-brain/xuance XuanCe is an open-source Python library of deep reinforcement learning (DRL) and multi-agent reinforcement learning (MARL) algorithm implem… | 91 | 1087 | active |
| DSE-MSU/DeepRobust DeepRobust is a PyTorch library for adversarial robustness research, providing implementations of attack and defense methods for both image… | 41 | 1086 | active |
| GradientHQ/symphony-coord Symphony-Coord is a decentralized multi-agent framework that treats agent selection as an online multi-armed bandit problem, allowing roles… | 60 | 1081 | active |
| Lazarus-AI/clearwing Clearwing is a dual-mode autonomous offensive-security tool that combines a network-pentest ReAct agent with an LLM-driven source-code vuln… | 62 | 1078 | active |
| NVIDIA-Merlin/HugeCTR HugeCTR is a GPU-accelerated deep learning framework from NVIDIA designed for training and inference of large recommender models, especiall… | 71 | 1073 | active |
| farrokhi/dnsdiag A Python-based toolset for measuring, troubleshooting, and auditing DNS. It includes dnsping for latency measurement, dnstraceroute for tra… | 92 | 1064 | active |
| braintrustdata/autoevals Autoevals is a Python and TypeScript library for automatically evaluating AI model outputs using LLM-as-a-judge, heuristic, and statistical… | 91 | 1053 | active |
| scrya-com/rotorquant RotorQuant is a KV cache compression method for LLM inference that replaces full d×d orthogonal rotations with block-diagonal Clifford roto… | 47 | 1049 | active |
| NVIDIA-NeMo/Skills Nemo-Skills is a collection of Python pipelines for improving the skills of large language models, covering synthetic data generation, mode… | 69 | 1042 | active |
| kuleshov-group/bd3lms BD3-LMs is a research implementation of Block Discrete Denoising Diffusion Language Models that interpolate between autoregressive and diff… | 31 | 1037 | active |
| BRL-CAD/brlcad BRL-CAD is a cross-platform open source combinatorial solid modeling system with an interactive 3D geometry editor, a high-performance netw… | 85 | 1031 | active |
| alex-petrenko/sample-factory Sample Factory is a high-throughput Python reinforcement learning library implementing synchronous and asynchronous policy gradient algorit… | 66 | 1026 | active |
| open-mmlab/mmyolo MMYOLO is the OpenMMLab toolbox and benchmark for the YOLO series of object detection models, implemented on PyTorch. It provides unified i… | 23 | 3477 | maintenance |
| fla-org/native-sparse-attention Efficient Triton kernel implementations of Native Sparse Attention (NSA), a hardware-aligned, natively trainable sparse attention mechanism… | 47 | 1023 | active |
| Overcooked AI Overcooked-AI is a benchmark environment for fully cooperative human-AI task performance, based on the video game Overcooked, where agents … | 25 | 1021 | active |
| tradytics/eiten Eiten is an open-source Python toolkit for building statistical and algorithmic investment portfolios, implementing strategies like Eigen P… | 32 | 3304 | maintenance |
| zaiyunduan123/springboot-seckill A high-concurrency flash-sale (seckill) system built with SpringBoot, MySQL, Redis, RabbitMQ, and Guava. It demonstrates caching, rate limi… | 32 | 3291 | maintenance |
| facebookresearch/vissl VISSL is Facebook AI Research's extensible, modular and scalable PyTorch library for state-of-the-art self-supervised learning with images.… | 10 | 3290 | maintenance |
| weld-project/weld Weld is a language and high-performance runtime that optimizes data-intensive analytics workflows by expressing computations in a common in… | 54 | 3008 | maintenance |
| T5 T5X is a modular, composable framework built on JAX and Flax for high-performance training, evaluation, and inference of sequence models at… | 71 | 3002 | maintenance |
| Cartucho/mAP A Python library and script that computes mean Average Precision (mAP) for object detection models, adapted from the official PASCAL VOC 20… | 23 | 2970 | maintenance |
| 2d-inc/developer_quest Developer Quest is a Flutter game demo built for Google I/O 2019 where players become a tech lead, slay bugs, and manage a team. It also se… | 32 | 2969 | maintenance |
| torch-points3d/torch-points3d A PyTorch-based framework for deep learning on 3D point clouds, supporting models like PointNet, KPConv, and MinkowskiEngine sparse convolu… | 64 | 2717 | maintenance |
| Qihoo360/ArgusAPM ArgusAPM is an open-source Android application performance management (APM) platform from Qihoo 360 that collects UI, network, memory, proc… | 32 | 2685 | maintenance |
| exchange-core/exchange-core An ultra-fast order matching engine core for exchanges written in Java, built on LMAX Disruptor, Agrona, Eclipse Collections, and OpenHFT. … | 23 | 2645 | maintenance |
| jcjohnson/torch-rnn torch-rnn provides high-performance, reusable RNN and LSTM modules for the Torch7 deep learning framework, used for character-level languag… | 32 | 2562 | maintenance |
| dblalock/bolt Bolt is a C++ library (with a Python wrapper) for lossy compression of dense real-valued vectors that supports mathematical operations dire… | 32 | 2520 | maintenance |
| allenai/RL4LMs RL4LMs is a modular Python library from AllenAI for fine-tuning language models with reinforcement learning to align with human preferences… | 32 | 2396 | maintenance |
| salesforce/decaNLP decaNLP is a PyTorch research framework and benchmark for the Natural Language Decathlon, a multitask NLP challenge spanning ten tasks from… | 10 | 2336 | maintenance |
| MineDojo MineDojo is an AI research framework built on Minecraft for training open-ended, generally capable embodied agents. It provides a simulatio… | 23 | 2260 | maintenance |
| facebookresearch/pycls pycls is a PyTorch-based codebase for image classification research developed by Facebook AI Research. It provides implementations of stand… | 10 | 2161 | maintenance |
| Tencent/ObjectDetection-OneStageDet OneStageDet (OSD) is a PyTorch-based single-stage object detection framework from Tencent implementing YOLOv2 and YOLOv3 with plans to unif… | 42 | 1976 | maintenance |
| Yvictor/TradingGym TradingGym is an OpenAI Gym-style Python toolkit providing trading environments for training reinforcement learning agents and backtesting … | 32 | 1920 | maintenance |
| ZhengyaoJiang/PGPortfolio PGPortfolio is a Python library implementing a deep reinforcement learning framework for financial portfolio management, based on the paper… | 23 | 1852 | maintenance |
| OpenPPL/ppq PPQ (PPL Quantization Tool) is an extensible, high-performance offline neural network quantization framework written in Python. It parses a… | 23 | 1822 | maintenance |
| facebookresearch/TensorComprehensions Tensor Comprehensions is a C++ library with a Python API that provides a domain-specific language for expressing machine learning workloads… | 10 | 1769 | maintenance |
| tmm1/rbtrace rbtrace is a Ruby gem and CLI tool that lets you trace method calls inside another running Ruby process in real time, similar to strace but… | 72 | 1726 | maintenance |
| badamczewski/PowerUp PowerUp is a collection of .NET-based productivity utilities including a live IDE watcher that compiles, decompiles, and disassembles code … | 32 | 1659 | maintenance |
| projectNe10/Ne10 Ne10 is an open-source C library of common math, signal processing, image processing, and physics functions heavily optimized for ARM CPUs … | 23 | 1540 | maintenance |
| hysts/pytorch_image_classification A PyTorch library implementing many image classification architectures (ResNet, DenseNet, WRN, PyramidNet, SENet, etc.) and augmentation te… | 10 | 1451 | maintenance |
| BigScience BigScience is a research project training large transformer language models (BERT, GPT-style) at scale, built on a fork of Megatron-LM inte… | 32 | 1448 | maintenance |
| open-mmlab/mmhuman3d MMHuman3D is an open-source PyTorch-based toolbox and benchmark for 3D human parametric models (e.g., SMPL, SMPL-X) in computer vision and … | 23 | 1426 | maintenance |
| advboxes/AdvBox AdvBox is a Baidu open-source toolbox for generating adversarial examples that fool neural networks across frameworks like PaddlePaddle, Py… | 32 | 1405 | maintenance |
| Yifan-Song793/RestGPT RestGPT is an LLM-based autonomous agent that controls real-world applications like TMDB and Spotify by planning and executing RESTful API … | 29 | 1403 | maintenance |
| Cysu/open-reid Open-ReID is a lightweight Python library for person re-identification built on PyTorch, providing a uniform dataset interface, models, and… | 32 | 1371 | maintenance |
| Takin Takin is a Java-based open-source full-link performance testing platform for online and test environments, especially for microservices arc… | 23 | 1321 | maintenance |
| thinkeridea/go-extend go-extend is a Go utility library that collects commonly used helper functions to speed up development and reduce repetitive code. It provi… | 23 | 1308 | maintenance |
| DeepGraphLearning/graphvite GraphVite is a general, high-performance graph embedding engine written in C++ with Python bindings, supporting node embedding, knowledge g… | 32 | 1270 | maintenance |
| cmu-db/ottertune OtterTune is a tool from the CMU Database Group that automatically finds good settings for a DBMS's configuration knobs using machine learn… | 10 | 1233 | maintenance |
| facebookresearch/ToMe ToMe (Token Merging) is a PyTorch library from Meta AI that speeds up existing Vision Transformers by merging similar tokens inside the net… | 10 | 1204 | maintenance |
| KaihuaTang/Scene-Graph-Benchmark.pytorch A PyTorch codebase for Scene Graph Generation (SGG) built on maskrcnn-benchmark, implementing methods from the CVPR 2020 paper 'Unbiased Sc… | 56 | 1199 | maintenance |
| VoltaML/voltaML VoltaML is a lightweight Python library that compiles and optimizes machine learning and deep learning models for high-performance inferenc… | 32 | 1177 | maintenance |
| clovaai/voxceleb_trainer A PyTorch framework for training and evaluating speaker recognition and verification models on the VoxCeleb datasets. It implements multipl… | 64 | 1176 | maintenance |
| JPaulMora/Pyrit Pyrit is a WPA/WPA2-PSK precomputed cracker that builds massive databases of pre-computed Pairwise Master Keys using multi-core CPUs and GP… | 64 | 1140 | maintenance |
| rlancemartin/auto-evaluator A lightweight Streamlit-based evaluation tool for LLM question-answering chains built with LangChain. It auto-generates question-answer pai… | 30 | 1103 | maintenance |
| RUCAIBox/TextBox TextBox 2.0 is a Python/PyTorch library providing a unified pipeline for applying pre-trained language models to text generation tasks. It … | 23 | 1096 | maintenance |
| EleutherAI/math-lm Repository for Llemma, an open language model for mathematics, hosting training code, data preprocessing scripts, and evaluation tooling. I… | 31 | 1095 | maintenance |
| ethanhe42/channel-pruning Reference implementation of the ICCV 2017 channel pruning method for accelerating very deep convolutional neural networks, using LASSO regr… | 23 | 1088 | maintenance |
| SusionSuc/rabbit-client Rabbit is an Android application performance monitoring (APM) framework written in Kotlin. It provides startup timing, FPS and memory analy… | 23 | 1067 | maintenance |
| open-mmlab/mmflow MMFlow is an open-source optical flow toolbox and benchmark built on PyTorch as part of the OpenMMLab project. It provides a unified, modul… | 23 | 1061 | maintenance |
| zdzhaoyong/GSLAM GSLAM is a general Simultaneous Localization and Mapping (SLAM) framework and benchmark written in C++. It provides a unified plugin-based … | 23 | 1059 | maintenance |