Ollama vs vLLM vs llama.cpp

The one-line verdict

• Trying open models on your own machine? Ollama: one command to download and run a model, with a local API.
• Serving a model to many users on GPUs? vLLM: built for throughput, with PagedAttention and continuous batching.
• Maximum portability and control, from laptops to edge devices? llama.cpp: a dependency-free C/C++ engine with aggressive quantization.

All three run open-weight models on hardware you control, so your data never leaves it, and all three can expose an OpenAI-compatible API.

Why run a model yourself at all?

Hosted APIs are the easiest start, but running a model yourself means:

• Privacy: prompts and documents stay on your machines.
• Predictable cost: you pay for hardware, not per token.
• Offline use: it works on a plane, or inside a locked-down network.
• Control: you pick the exact model version, and it never changes under you.

The price is managing hardware, memory and updates yourself, and open models may trail the best hosted ones on some tasks.

Side by side

	Ollama	vLLM	llama.cpp
Best for	Local development and trying models	High-throughput serving	Portability, CPUs, edge devices
Start a server	ollama run <model>	vllm serve <model>	llama serve -hf <repo>
Default API	REST on port 11434, OpenAI-compatible under /v1	OpenAI-compatible on port 8000	OpenAI-compatible server
Hardware	Laptops and desktops, CPU or GPU	NVIDIA, AMD and Intel GPUs, CPUs, plus plugins (e.g. TPUs)	CPUs, Apple silicon, CUDA, HIP, Vulkan, SYCL
Model format	Its model library; GGUF can be imported	Hugging Face models	GGUF
Main strength	Easiest setup	Throughput under load	Runs almost anywhere, heavy quantization

Ollama: the easy button

ollama run gemma4

That single command downloads the model and starts chatting. In the background, Ollama runs a local server on http://localhost:11434 with its own REST API, plus OpenAI-compatible endpoints such as /v1/chat/completions and /v1/embeddings. A Modelfile lets you set a system prompt and parameters, and you can import your own GGUF models.

It's the fastest way to see what an open model can do, and to develop against a local model before deploying anywhere.

vLLM: the serving engine

vllm serve Qwen/Qwen2.5-1.5B-Instruct

This starts an OpenAI-compatible server on http://localhost:8000, serving one model. vLLM's core idea comes from its 2023 paper: the attention KV cache for each request is large and grows unpredictably, and handling it naively wastes memory, which limits how many requests can run together. PagedAttention manages that memory like an operating system manages pages, with near-zero waste, so more requests fit in each batch. The paper reports 2 to 4 times the throughput of earlier systems at the same latency.

On top of that, vLLM does continuous batching (new requests join a running batch instead of waiting) and prefix caching. Its server also offers an Anthropic Messages API and gRPC, and it supports NVIDIA, AMD and Intel GPUs and CPUs, with plugins for hardware such as Google TPUs.

llama.cpp: the engine that runs anywhere

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

llama.cpp is a plain C/C++ implementation with no dependencies. It treats Apple silicon as a first-class target, and has CUDA (NVIDIA), HIP (AMD), Vulkan and SYCL backends, as well as optimized CPU paths. Older builds name the server binary llama-server.

Its standout feature is quantization: storing weights in 1.5-bit to 8-bit integers instead of 16-bit floats. A 4-bit model needs roughly a quarter of the memory, which is what makes a capable model run on an ordinary laptop, at some cost in quality. Models use the GGUF file format, which much of the local-model ecosystem has adopted.

A worked example: one switch, three stages

Because all three speak the OpenAI API shape, the same client code works against each. Only the base URL and model name change:

from openai import OpenAI

# Local development with Ollama (the key is required by the client but unused)
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

# Production serving with vLLM:
# client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

reply = client.chat.completions.create(
    model="gemma4",
    messages=[{"role": "user", "content": "Explain a cache stampede in one sentence."}],
)
print(reply.choices[0].message.content)

A typical path: prototype on a laptop with Ollama, ship to vLLM on GPUs once real traffic arrives, and use llama.cpp wherever there's no big GPU: an edge box, a CPU-only server, or an app that embeds the model.

Five questions that decide it

1. How many users at once? One developer: Ollama. Many concurrent users: vLLM.
2. What hardware? A MacBook or CPU-only machine favors llama.cpp or Ollama; NVIDIA or AMD GPU servers favor vLLM.
3. How much memory do you have? Too little for full precision means quantized GGUF models via llama.cpp or Ollama.
4. Do you need to embed the model in another program? llama.cpp's library is designed for it.
5. Will your app move between them? Keep to the OpenAI-compatible API, and switching is a configuration change.

Honest limits

• Benchmarks depend heavily on model, quantization, batch size and hardware. Compare on your own workload.
• Quantization trades quality for memory. Test the smaller version on your real tasks before committing to it.
• These tools change fast. Command names, supported models and hardware lists move every few months, so check each project's docs.

Remember this

• Ollama = the easiest local start, with a local API on port 11434.
• vLLM = throughput under load, thanks to PagedAttention and continuous batching.
• llama.cpp = runs almost anywhere, with GGUF and heavy quantization.
• All three can expose an OpenAI-compatible API, so switching is mostly a base URL.