tjake/Jlama

Health v2 · maintenance only

Adoption not part of the score

1309 stars · 164 forks observed · 2026-10-01

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Jlama is a modern LLM inference engine written in Java, supporting popular transformer models like Llama, Mistral, Gemma, and Qwen with quantization and distributed inference. It ships as an embeddable library plus a CLI that can serve an OpenAI-compatible REST API with a chat UI.

Use cases

• run llm inference in a java application
• serve an openai-compatible api for local models
• chat with huggingface models from the command line
• run quantized llama or mistral models locally
• generate embeddings with a java library
• distributed inference across multiple machines

When to choose

• your application is on the JVM and you want built-in LLM inference without Python
• you want an OpenAI-compatible local REST API with a chat UI
• you need quantized model support (Q8/Q4) in pure Java
• you want distributed inference across a cluster of JVMs

When to avoid

• you need maximum inference throughput available in GPU-native runtimes like vLLM or llama.cpp
• you need broad model architecture support beyond the supported families
• you are not on Java 20+ since the engine requires modern JDK features

Facets

library · maturity active

llm-inference cli http-server machine-learning transformers large-language-models artificial-intelligence machine-learning developer-tools jvm cli cross-platform llm-inference java simd quantization openai-compatible-api huggingface distributed-inference panama-vector safetensors

Member repositories

For agents

markdown · JSON · MCP: product_card(name="tjake/Jlama")