Ollama

by Ray Svitla

the job

ollama does two things and they should be read together:

it runs an LLM on your machine.
it exposes that model as a local API on http://localhost:11434 that other software can talk to.

the second part is what most write-ups skip. ollama is not just an inference binary. it is a daemon with a stable HTTP surface. once it is running, anything on your machine that speaks HTTP can route through it: a chat UI, a coding agent, a voice pipeline, a script, a cron job. that is what makes it interesting for a personal operator stack rather than for tinkering with models.

if you only want to chat with a local model, that is fine — ollama covers it. if you want a custody layer for your agents, that is the bigger reason to run it.

what it actually is

ollama is a small daemon plus CLI plus model registry. under the hood it leans on llama.cpp for inference on most hardware. quantized weights ship as a single named pull (ollama pull llama3.2). a small modelfile lets you bake a system prompt, parameters, and a base model into a named local model.

three surfaces matter:

CLI: ollama pull, ollama run, ollama list, ollama ps, ollama show, ollama serve. enough to operate a fleet of local models from a terminal.
native HTTP API: documented at the ollama API reference . this is the canonical surface and the most stable.
OpenAI-compatible endpoint: an OpenAI-style HTTP surface so tooling that already speaks an OpenAI base URL can talk to ollama with no rewrite.

that compatibility surface is the part that drags ollama out of “model toy” territory and into “infrastructure”. for an OpenAI-shaped client, swapping a cloud model for a local one becomes a base URL change instead of a code rewrite.

before citing a version, check the ollama releases page . this page deliberately does not pin one — local model runners ship fast and an evergreen page that names a tag rots within weeks.

quickstart

from nothing installed to an OpenAI-shaped client talking to a local model:

# install — see https://ollama.com/download for your OS

# pull a model
ollama pull llama3.2

# one-shot a prompt
ollama run llama3.2 "summarize this paragraph: ..."

# start the daemon (most installs run it as a service already)
ollama serve

# confirm the local API is up
curl http://localhost:11434/api/tags

point an OpenAI-shaped client at the compatibility endpoint:

# base URL for OpenAI-shaped clients: http://localhost:11434/v1
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "hello"}]
  }'

if those commands work, every other tool in your stack that speaks the OpenAI shape can talk to a local model with a single base URL change.

when to use ollama

situation	use ollama
single user on a laptop or workstation	yes
you want a local OpenAI-compatible endpoint without writing one	yes
you want a daemon you can leave running across reboots	yes
you need a chat surface in front of it (open webui , claude desktop, a custom UI)	yes
you are building a personal agent that should mostly stay on the machine	yes
you need to A/B between cloud and local models from the same client	yes

when not to use ollama

situation	use something else
serving 5+ concurrent users from one box	vllm
you need maximum throughput per GPU and are willing to tune	vllm or llama.cpp directly
you want fine control over quantization, sampling, prompt format, batching	llama.cpp directly
you are on apple silicon and want the fastest pure-mac path today	mlx-lm directly
you are deploying to production with SLAs and observability requirements	a real serving stack — not ollama
you want a polished desktop GUI for non-terminal users	lm studio

ollama is the wrong tool for production serving and a slightly heavy tool for “I just want to play with weights for an afternoon”. its sweet spot is the operator middle: a single human running long-lived agents on their own hardware.

how it fits with the rest of the local stack

stop treating “local LLM” as one thing. there are at least four distinct jobs and ollama only does two of them.

layer	job	examples
inference engine	actually run the math	llama.cpp, mlx, vllm
local model runner	manage models, expose an API, hide the engine	ollama, lm studio
chat / cockpit UI	give a human a place to talk and load context	open webui , lm studio
agent harness	use the model to do work in a loop with tools and state	claude code, custom agent

ollama sits at “local model runner”. pair it where it makes sense:

ollama + open webui: ollama runs models and serves the API; open webui is the cockpit — chat, RAG, document extraction, plugin tools. classic local “chatgpt-but-mine” stack.
ollama + an OpenAI-style client: anything that takes an OpenAI base URL can be pointed at ollama. that includes most agent frameworks, IDE plugins, and notebooks built around the OpenAI SDK.
ollama + voice-first agents: pair with a whisper.cpp voice capture pipeline so capture, transcription, and response all stay on the machine.

and a head-to-head when you have to pick one runner:

your situation	pick	why
one user, want a daemon + OpenAI-style API	ollama	batteries included; one stable port
max GPU / sampling / quantization control	llama.cpp directly	georgi gerganov ’s lane; no abstraction tax
many users, throughput per GPU matters	vllm	a real serving stack, not a personal daemon
polished desktop GUI, never opens a terminal	lm studio	app-shaped, not API-shaped
apple silicon, max native speed	mlx-lm	apple’s own kernels

deeper runtime-by-runtime comparison lives in local LLM runtimes .

the shape that emerges: pick one local model runner so your agents have one base URL, then layer the cockpit and the harness on top. running both ollama and lm studio at once on the same machine, with two model directories and two daemons, is a self-inflicted operations problem.

why it matters for custody

custody is the boring word for “what leaves the machine”. ollama’s role here is concrete:

prompts, outputs, and embeddings stay on the box unless you wire ollama to a cloud retriever or browser tool.
the API surface is the boundary. anything talking to localhost:11434 is local. anything pointed at a cloud base URL is not. that boundary is auditable with a packet filter.
model versions are your decision. an ollama tag you pulled is the tag you have until you pull again.
the kill switch is a daemon stop. shutting it down severs the local AI surface entirely.

llama.cpp, mlx, and vllm also keep things local. ollama’s specific contribution is making the custody story easy to operate: a daemon, a model registry, and a stable API that other people’s software already speaks.

operator checklist

before you treat an ollama install as part of your stack, run through this:

version: check ollama --version against the latest release . pin if you care.
bind address: ollama listens on 127.0.0.1:11434 by default. if you change that to a LAN address, you opened a free local-model API to anything on your network. set the bind deliberately.
model dir: know where models live on your OS. disk fills fast — a 70B at Q4 is tens of GB.
memory headroom: ollama ps shows what is loaded. unload with ollama stop <model> when switching.
context length: many models default to a small num_ctx. set it explicitly in the request or modelfile if your agent needs long context.
api surface: pick one — native API or OpenAI-compatible — and standardize. mixing them in one stack causes confused defaults later.
logs: know where ollama writes logs on your OS. that is where errors hide.
firewall: if the daemon is reachable from anywhere outside your machine, that should be a deliberate decision, not an accident.
uninstall path: know how to fully remove it (binary, models dir, launch agent / systemd unit) before you install. local AI installs accumulate.

decision points

three quick decisions that make most ollama-or-not arguments unnecessary:

how many humans will hit this model? one → ollama. several → vllm. this is the cleanest signal.
do you want one base URL for every agent? yes → run ollama (or one alternative, never both). no → you do not need a model runner; talk to llama.cpp / mlx directly.
does the prompt or the file ever need to stay on the machine? yes → local runner of some kind, ollama is the default. no → you do not actually need local; rent the model.

if you cannot answer those three, you are not yet ready to make the ollama call.

what ollama does not solve

worth saying out loud, because the marketing around local AI tends to skip it:

it does not pick the right model for you. quantization, model family, and size are still your problem.
it does not give you cloud-grade reasoning on a laptop. local 7-13B class models are useful, not equivalent to frontier cloud models.
it does not give you observability. there is no built-in trace store, eval harness, or token-spend dashboard. that is your harness.
it does not enforce permissions. ollama will happily serve any prompt any local process sends it. the policy gate is somewhere else.
it does not version your prompts or skills. that lives in your repo and your agent config files .

ollama is the runner. the harness is still your job.

the take

most “local LLM” content treats ollama as the entire story. it is not. it is the unsexy middle layer that makes the rest of the local stack usable: one daemon, one registry, one stable port. the model is the part you rent. ollama is one of the few pieces that makes the part you own actually addressable from your existing tools.

if you are building a personal agent stack — a coding agent on top, open webui for the cockpit, whisper.cpp for voice , llama.cpp underneath — ollama is the simplest answer to “what listens on the local port?”. keep it boring on purpose.