marin-community/marin

Health v2 · maintenance only

Flags: prerelease_only

Adoption not part of the score

2428 stars · 212 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Marin is an open-source Python framework and research program for training foundation models, covering the full pipeline from data curation, filtering, and tokenization through pretraining, posttraining, and evaluation. It emphasizes open development, documenting every experiment, dataset, and checkpoint in real time, and includes artifacts like the Delphi scaling suite and frontier mixture-of-experts training runs.

Use cases

• pretrain a large language model from scratch on TPUs
• curate and filter pretraining datasets like Nemotron-CC or StarCoderData
• run scaling law experiments across compute budgets
• run posttraining and SFT experiments on open LLMs
• evaluate and compare LLM checkpoints reproducibly
• train audio-text, DNA, or protein foundation models
• reproduce and share ML experiments with full provenance

When to choose

• you want an open, reproducible pipeline for LLM pretraining and posttraining research
• you need data curation, tokenization, training, and evaluation in one framework
• you want to run scaling suites or mixture-of-experts experiments on TPU research clouds
• you value open science with preregistered experiments and shared checkpoints

When to avoid

• you just need to fine-tune a small model quickly with a high-level API like HF Trainer or LoRA tooling
• you need production inference serving rather than training research
• you lack access to significant TPU/GPU compute for large-scale training

Facets

framework · maturity active

llm-training machine-learning etl data-science benchmarking large-language-models machine-learning deep-learning artificial-intelligence python cloud foundation-models pretraining posttraining data-curation scaling-laws tpu open-science mixture-of-experts tokenization evaluation data-engineering gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="marin-community/marin")