lucidrains/CoCa-pytorch

Health v2 · maintenance only

Adoption not part of the score

1198 stars · 89 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Pytorch implementation of CoCa (Contrastive Captioners), an image-text foundation model that combines contrastive learning with an encoder-decoder transformer for image captioning and CLIP-like embeddings. It adopts the PaLM transformer architecture with parallel SwiGLU feedforwards and requires a pretrained vision transformer as the image encoder.

Use cases

• train a multimodal image-text foundation model in pytorch
• generate captions for images with a transformer model
• get CLIP-style image and text embeddings for contrastive search
• implement contrastive captioning loss for vision-language pretraining
• fine-tune an image-to-text model on my own image-caption dataset
• reproduce CoCa paper results for research

When to choose

• you want a flexible, hackable Pytorch implementation of the CoCa architecture for research
• you need both captioning and contrastive (CLIP-like) objectives in one model
• you already have a pretrained vision transformer encoder to plug in

When to avoid

• you need a pretrained, ready-to-use model with weights out of the box
• you want a production inference service rather than a training library
• you prefer frameworks other than Pytorch

Facets

library · maturity maintenance

machine-learning deep-learning transformers artificial-intelligence deep-learning computer-vision python contrastive-learning multimodal image-to-text image-captioning clip research-implementation vision-transformer natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="lucidrains/CoCa-pytorch")