withcatai/node-llama-cpp

Health v2 · maintenance only

Adoption not part of the score

2162 stars · 213 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

node-llama-cpp is a Node.js library providing bindings to llama.cpp for running LLMs locally, with pre-built binaries and automatic GPU support (Metal, CUDA, Vulkan). It includes a CLI for chatting with models, JSON schema-enforced generation, function calling, and embedding/reranking support.

Use cases

• run llama.cpp models from node.js
• chat with local LLMs without writing code
• force model output to follow a JSON schema
• implement function calling with local models
• generate embeddings for semantic search
• run AI models offline on my own machine
• build an Electron app with local LLM inference

When to choose

• you want local, private LLM inference in a Node.js, Bun, or Electron app
• you need structured output enforced via grammar or JSON schema
• you want zero-config GPU acceleration with prebuilt binaries
• you need embeddings or reranking alongside text generation

When to avoid

• you need to serve many concurrent users at scale (use a dedicated inference server)
• your stack is Python and you prefer native llama-cpp-python bindings
• you rely on cloud-hosted LLM APIs rather than local models

Facets

library · maturity active

llm-inference machine-learning rag cli sdk chatbot large-language-models artificial-intelligence machine-learning developer-tools self-hosted windows bun cli llamacpp-bindings gguf -schema-grammar function-calling embeddings reranking local-llm typescript cuda metal vulkan prebuilt-binaries nodejs macos linux electron gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="withcatai/node-llama-cpp")