Local LLM inference with Ollama: a practical guide.

Cloud LLM APIs are convenient, but they come with three strings attached: your data leaves your machine, you pay per token, and the service can change or disappear under you. For a lot of work — anything involving sensitive data, high-volume batch processing, or just tinkering — running the model locally is the better trade. Ollama has become the de facto standard for doing that with minimal friction, and local-first LLM tooling is something I care about enough to have built a whole application around. This guide takes you from installation to serving models to your own code.

What Ollama actually is

Ollama is a local model server built on top of llama.cpp, the C/C++ inference engine that made running LLMs on consumer hardware practical. It wraps three things into one tool:

• a model manager — pull, list, and remove models from a curated library, like Docker for LLMs;
• an inference runtime — runs models on your GPU (NVIDIA and AMD on Windows/Linux, Apple Silicon on macOS), spilling over to CPU when the model doesn't fully fit;
• a local API server — an HTTP API on localhost:11434 that any application on your machine can call.

The models it serves are open-weights models — Meta's Llama family, Alibaba's Qwen, Mistral, Google's Gemma, DeepSeek's distills, and many more — in the quantized GGUF format (more on quantization below).

Installation and first run

On Windows and macOS, download the installer from ollama.com/download. On Linux:

curl -fsSL https://ollama.com/install.sh | sh

Then pull and run a model in one command:

ollama run llama3.1

The first run downloads the model (a few GB); after that you're in an interactive chat in your terminal. Exit with /bye. The day-to-day commands:

ollama pull qwen2.5:14b     # download a model without starting a chat
ollama list                 # models on disk
ollama ps                   # models currently loaded in memory
ollama show llama3.1        # details: parameters, context length, quantization
ollama stop llama3.1        # unload from memory
ollama rm qwen2.5:14b       # delete from disk

Model names use a name:tag scheme, where tags select size and variant: llama3.1 defaults to llama3.1:8b, while llama3.1:70b is the big one. Browse what's available at ollama.com/library.

Choosing a model that fits your hardware

The library models are quantized: their weights are compressed from 16-bit floats down to (typically) ~4 bits each, which cuts memory to roughly a quarter with a modest quality loss. That's the trick that makes an 8-billion-parameter model fit comfortably on a laptop. As a rule of thumb, a 4-bit model needs about 0.6–0.7 GB of memory per billion parameters, plus a little headroom for the context window:

If a model doesn't fully fit in GPU memory, Ollama splits it between GPU and CPU — it works, but tokens-per-second drops sharply, so it's usually better to pick a smaller model or a smaller quantization than to spill. On pure CPU, small models (3–8B) are genuinely usable; large ones are not.

A sane starting lineup as of mid-2026: llama3.1:8b or qwen2.5:7b as general-purpose workhorses, qwen2.5-coder for code, gemma3:4b when resources are tight, and a deepseek-r1 distill if you want visible chain-of-thought reasoning locally. Expectations matter: an 8B local model is not a frontier model. It is, however, remarkably capable for summarization, extraction, classification, drafting, and RAG over your own documents — the bread-and-butter tasks that make up most real workloads.

Talking to Ollama from code

Everything Ollama does is exposed through the local HTTP API on port 11434, which means any language can use it:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [{"role": "user", "content": "Why is the sky blue?"}],
  "stream": false
}'

From Python, the official client is nicer. pip install ollama, then:

import ollama

response = ollama.chat(
    model="llama3.1",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "Explain quantization in two sentences."},
    ],
)
print(response["message"]["content"])

Streaming, so output appears token by token:

stream = ollama.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    stream=True,
)
for chunk in stream:
    print(chunk["message"]["content"], end="", flush=True)

Structured output is supported too — pass a JSON schema as the format argument and the model's response is constrained to valid JSON matching it. That turns a local model into a dependable extraction engine:

from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total: float
    currency: str

response = ollama.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": f"Extract the invoice fields:\n\n{invoice_text}"}],
    format=Invoice.model_json_schema(),
)
invoice = Invoice.model_validate_json(response["message"]["content"])

And for RAG systems, Ollama serves embedding models as well — so the entire pipeline, embeddings included, can run offline:

ollama.pull("nomic-embed-text")
result = ollama.embed(model="nomic-embed-text", input=["chunk one", "chunk two"])
vectors = result["embeddings"]   # ready for your vector store

The OpenAI-compatible endpoint

Ollama also exposes an OpenAI-compatible API at http://localhost:11434/v1. Practically, this means most tools and libraries built for cloud APIs can point at your local server by changing two settings — base URL and model name — with any non-empty string as the API key. It's the quickest way to make an existing codebase local-first without rewriting it.

Customizing models with a Modelfile

A Modelfile is to models what a Dockerfile is to images: a small recipe that layers your configuration on top of a base model. Save this as Modelfile:

FROM llama3.1

# Sampling and context settings
PARAMETER temperature 0.3
PARAMETER num_ctx 8192

# Baked-in system prompt
SYSTEM """You are a code review assistant. You answer tersely,
point out concrete defects, and never rewrite code unasked."""

ollama create reviewer -f Modelfile
ollama run reviewer

Two parameters deserve special mention. temperature you know. But num_ctx — the context window — is the one that bites people: Ollama defaults to a modest context length regardless of what the model supports, and input beyond it is silently truncated. If your RAG answers mysteriously ignore the documents you stuffed into the prompt, this is almost always why. Raise num_ctx in a Modelfile (or pass options={"num_ctx": 8192} per request from Python); note that a larger context costs more memory.

Operational notes

• Model loading and keep-alive — the first request after a quiet period pays a load-from-disk cost; by default models unload again a few minutes after last use. For a service, set keep_alive (per request, or OLLAMA_KEEP_ALIVE=-1 to pin the model in memory).
• Serving beyond localhost — by default Ollama listens only on 127.0.0.1. Set OLLAMA_HOST=0.0.0.0 to expose it on your network — and put a reverse proxy with authentication in front if you do, because the API itself has none.
• Where models live — ~/.ollama/models by default; move it with OLLAMA_MODELS if your system drive is small. Models are multi-GB; prune with ollama rm.
• Containers — the official ollama/ollama image plus --gpus all gives you a reproducible, deployable inference server; mount a volume at /root/.ollama so models persist. (Details on GPU containers in the Docker post.)
• When you outgrow it — Ollama is optimized for simplicity and single-machine use. If you need to serve many concurrent users at maximum throughput, that's the point to look at batching-oriented servers like vLLM; the application code barely changes thanks to the OpenAI-compatible interface.

When local inference is the right call

The decision usually comes down to four factors. Privacy: medical, legal, and proprietary data may simply not be allowed to leave your infrastructure — local inference makes the question moot. Cost shape: APIs bill per token, hardware is a fixed cost; at sustained high volume the crossover comes fast. Control: a pinned local model never changes behavior under you, which is worth a lot in regulated or evaluation-heavy settings. Capability: the one factor that favors the cloud — frontier models remain meaningfully stronger, so hard reasoning tasks may justify the API. Many of the best systems I've built are hybrids: local models for the private, high-volume work; a cloud model for the rare hard cases.

All posts