jamesob/local-llm resource

Health v2 · maintenance only

Flags: no_releases young no_license

Adoption not part of the score

1800 stars · 106 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A personal guide and configuration repository for running state-of-the-art LLMs and speech-to-text locally on consumer/prosumer hardware. It documents hardware choices (multi-GPU EPYC rig with PCIe switches), kernel/BIOS tuning, and ready-to-run Docker serving configs.

Use cases

• build a multi-GPU rig for running large LLMs locally
• run speech-to-text models on my own hardware
• configure vLLM to serve a 594B model with tensor parallelism
• benchmark GPU peer-to-peer bandwidth and latency
• learn how to power-limit multiple GPUs on a home circuit
• set up PCIe switch bifurcation and ACS settings for GPU P2P

When to choose

• you want battle-tested, opinionated hardware and configuration notes for a serious local LLM setup
• you need ready-to-run Docker/vLLM serving configs for very large models
• you're troubleshooting NCCL, IOMMU, or PCIe peer-to-peer issues in a multi-GPU box

When to avoid

• you want a turnkey installer or maintained software package - this is a guide plus configs, not a tool
• you're running small models on a single consumer GPU
• you need something with a license or formal support - the repo has no license

Facets

learning-resource · maturity active

llm-inference speech-recognition gpu-computing benchmarking developer-tools large-language-models hardware self-hosted artificial-intelligence speech-processing self-hosted local-llm hardware-guide vllm tensor-parallelism pcie-p2p speech-to-text docker-compose shell-scripts linux docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="jamesob/local-llm")