GoogleCloudPlatform/localllm

Health v2 · maintenance only

Flags: no_releases archived

Adoption not part of the score

1550 stars · 119 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A CLI tool and Docker image for running quantized LLMs locally on Google Cloud Workstations using llama-cpp-python's web server. It packages Hugging Face quantized models behind an OpenAI-compatible local endpoint.

Use cases

• run llms locally on cloud workstations
• serve quantized huggingface models with llama.cpp
• self-host an openai-compatible llm endpoint
• use local llms for code assistance without sending data externally
• spin up a workstation image with an llm tool preinstalled

When to choose

• you use Google Cloud Workstations and want local LLM inference
• you want quantized models served via llama-cpp-python without GPU setup
• you need a private, on-cloud LLM endpoint for developer tooling

When to avoid

• you need high-throughput production inference with GPU acceleration
• you are not on GCP and just want a simple local LLM runner
• you need the latest model architectures beyond llama.cpp support

Facets

cli-tool · maturity maintenance

llm-inference cli http-server large-language-models developer-tools cloud-computing python cloud cli llama-cpp quantized-models huggingface google-cloud cloud-workstations local-inference docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="GoogleCloudPlatform/localllm")