llama.cpp setup guide: build, run, and troubleshoot GGUF models locally

Updated May 14, 2026. This guide was refreshed against current llama.cpp build/readme docs and Hugging Face GGUF docs. The AMD flag is now GGML_HIP, the current tools are llama-cli and llama-server, and the primary model path uses -hf where possible.

This is the practical path: build llama.cpp, load a GGUF model, run the CLI, start the local server, and verify that it answers through an OpenAI-compatible endpoint. If you are still deciding between Ollama, llama.cpp, LM Studio, and vLLM, start with the local runtime chooser .

what this guide does

By the end, you should have:

• build/bin/llama-cli available;
• build/bin/llama-server available;
• one GGUF model loaded from Hugging Face or a local file;
• a CLI response from llama-cli;
• a JSON response from http://localhost:8080/v1/chat/completions.

This guide does not try to rank every local model or benchmark your machine. Treat the RAM and quant tables as starter heuristics, then test your own model on your own hardware.

prerequisites

You need Git, CMake, a C/C++ build toolchain, and enough RAM for the model you choose. GPU acceleration depends on your backend:

build llama.cpp

Clone the repo and build the default target:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

Check that the tools exist:

./build/bin/llama-cli --help
./build/bin/llama-server --help

If those commands work, the base build is fine. Now pick the backend that matches your machine.

macOS / Metal

On macOS, llama.cpp’s build docs say Metal is enabled by default. Build with the base command above, then use -ngl 99 when running a model if you want to offload layers to the GPU:

./build/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -ngl 99 -p "Say hello in one sentence."

If GPU acceleration is active, startup logs should mention the active Metal path. If the model is too large, reduce model size, quant size, or context length before blaming the runtime.

NVIDIA / CUDA

For NVIDIA GPUs, build with GGML_CUDA=ON:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

Then run the same CLI test:

./build/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -ngl 99 -p "Say hello in one sentence."

If it silently falls back to CPU, check CUDA installation, driver compatibility, and the startup logs.

AMD / HIP

For AMD GPUs on Linux, current llama.cpp build docs use GGML_HIP=ON — not the older HIPBLAS-era flag names.

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release \
  && cmake --build build --config Release -- -j 16

GPU_TARGETS is optional in the upstream docs; set it only when you know your GPU target. If ROCm cannot find device libraries, use the HIP device-library troubleshooting path from the llama.cpp build docs rather than guessing flags from old blog posts.

Windows preset

For Windows builds, use a Visual Studio 2022 Developer Command Prompt or Developer PowerShell. Current llama.cpp build docs show the x64 LLVM preset:

cmake --preset x64-windows-llvm-release
cmake --build build-x64-windows-llvm-release

If you are on Windows and only need a quick local model, a packaged app may be less painful. Use this route when you specifically want llama.cpp binaries and control.

load a GGUF model

The cleanest current path is llama.cpp’s Hugging Face shortcut:

./build/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -p "What is a GGUF file? Answer in one sentence."

You can also start the server directly from Hugging Face:

./build/bin/llama-server -hf ggml-org/gemma-3-1b-it-GGUF --port 8080

For manual downloads, use Hugging Face’s GGUF filter and pick a model card you trust:

# Example only: replace with the GGUF repo and file you selected.
wget https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf -O model.gguf

Manual files are useful when you want reproducible artifacts. The -hf shortcut is better for a first smoke test.

quant and RAM starter table

These are starter ranges for 4-bit-ish GGUF local inference, not permanent laws:

Start with a smaller instruct model and prove the stack works. Then increase model size or quant quality. Debugging a 70B model before proving the server responds is how you turn a setup into a séance.

run the CLI

With a local file:

./build/bin/llama-cli -m model.gguf -p "What is the capital of France?"

With GPU offload:

./build/bin/llama-cli -m model.gguf -ngl 99 -p "Say hello in one sentence."

Interactive mode:

./build/bin/llama-cli -m model.gguf -i

Common flags:

run the server

Start a local OpenAI-compatible HTTP server:

./build/bin/llama-server -m model.gguf --port 8080

Or start directly from Hugging Face:

./build/bin/llama-server -hf ggml-org/gemma-3-1b-it-GGUF --port 8080

The llama.cpp README documents the browser UI at http://localhost:8080 and the chat completion endpoint at /v1/chat/completions.

smoke test

With the server running, send one tiny chat request:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Reply with exactly: llama.cpp works"}],
    "temperature": 0
  }'

A good response is JSON with a choices array and text under choices[0].message.content. The exact wording can vary by model, but the server should not return connection refused, HTML, or a model-loading error.

expected success signals

troubleshooting table

using it from Python

Keep the Python layer boring. Start the server, then point an OpenAI-compatible client at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Say hello."}],
)

print(response.choices[0].message.content)

If the Python client fails but the curl smoke test works, the runtime is fine. Debug the client base URL, request body, or model name.

source-backed notes

• The llama.cpp README documents llama-cli, llama-server, -hf, and the local OpenAI-compatible server.
• The llama.cpp build docs document CPU, Metal, CUDA, HIP, Vulkan, and Windows build paths.
• Hugging Face documents GGUF files and the GGUF model filter .

related

• Local LLM Runtimes — choose between Ollama, llama.cpp, LM Studio, and vLLM
• Model Quantization — understand Q4/Q5/Q8 tradeoffs
• Context Optimization — avoid oversized prompts and context waste
• Multi-Model Routing — route tasks once one local model is not enough
• Sandboxing & Security — isolate local tools and agents

Next: Choose the right local LLM runtime