IDEA-Research/Rex-Omni

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1561 stars · 111 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Rex-Omni is a 3B-parameter multimodal large language model that unifies object detection, OCR, pointing, keypoint detection, and visual prompting as a next-point-prediction task. It ships with a Python wrapper for inference, fine-tuning (SFT and GRPO), evaluation code, and an AWQ-quantized checkpoint.

Use cases

• detect objects in images by category name
• open-set object detection with text prompts
• locate objects from referring expressions
• point to objects described in natural language
• OCR with bounding boxes and polygons
• keypoint detection for humans and animals
• detect objects matching a visual prompt box
• fine-tune a detection model on custom pointing datasets

When to choose

• you need one unified model for multiple visual grounding tasks
• you want open-vocabulary detection driven by natural language
• you need a lightweight 3B model with quantized weights for GPU inference
• you want to fine-tune detection or pointing behavior with SFT or RL

When to avoid

• you need real-time detection on CPU-only hardware
• you need a mature production detector with a permissive license
• you need video or 3D perception rather than single-image tasks

Facets

library · maturity active

computer-vision ocr machine-learning llm-inference computer-vision image-processing artificial-intelligence large-language-models python cross-platform mllm open-set-detection next-point-prediction visual-grounding keypoint-detection visual-prompting grpo fine-tuning awq-quantization natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="IDEA-Research/Rex-Omni")