Running an LLM on a CPU-Only VPS in 2026: What Actually Works

I keep seeing people rent GPU VPSs for jobs a CPU box can handle, then pay ten times more than they need to. I run qwen2.5-coder-1.5b and qwen2.5-1.5b-instruct GGUFs on modest hardware, and the lesson is simple: on CPU, size, context, and quantisation matter more than brand names. If you only need extraction, routing, classification, or short drafts, a small quantised model on a cheap VPS is fine. If you want a chatty agent with long context and tool loops, the CPU-only VPS becomes a tax collector.

As of 2026-09-06, the upstream tools are alive and moving: llama.cpp’s latest release is v0.4.0, the repo was pushed again today, and Ollama’s latest release is v0.33.3. That matters because this space changes fast, but the basic physics do not.

A small note before the rest: those token/sec numbers are generation speed, not the whole story. On CPU, the first response is often limited by prompt evaluation and the KV cache, not by how fast the model can stream new tokens.

Why quantisation is the whole game

On a CPU VPS, the model file size is only the starting point. The real question is whether the weights, runtime overhead, and KV cache all fit together without pushing the machine into swap or the OOM killer.

That is why quantisation is the lever that matters. Q4_K_M is the default I reach for because it usually gives the best balance of size, speed, and quality on CPU. Q5_K_M is nice when you have extra headroom and want a little more quality. Q8_0 is usually too expensive for the sort of VPS you would rent for this job.

The other trap is context length. A model that looks fine at 2k context can become painful at 8k or 16k because the KV cache grows with context. That is where people lose RAM, not just quality. If you want a CPU-only VPS to feel usable, keep the context window honest.

My rule is boring but it works:

• use 1.5B to 3B models for the real work
• use Q4_K_M unless you have a clear reason not to
• keep context smaller than your ego wants it to be
• never assume “it fits in RAM” means “it works well”

llama.cpp vs Ollama vs vLLM on CPU

If I only had one sentence here, it would be this: llama.cpp is my default on CPU, Ollama is the convenience layer, and vLLM is the wrong default because it is built around GPUs.

llama.cpp

This is the one I trust when I care about memory, control, and predictable behaviour. It speaks GGUF directly, it has an OpenAI-compatible server, and it does not hide the important knobs from me. On a CPU VPS, that matters more than a pretty wrapper.

I also like that it makes the trade-offs obvious. If generation is slow, I know it is the hardware or the model, not some extra abstraction layer pretending to help.

Ollama

Ollama is the easier path if you want to get something running fast and you do not care about squeezing every last bit of performance. It is good for demos and for people who want a friendly model manager.

I still use llama.cpp more often because I want to choose the quantisation myself and I want to see exactly what the box is doing. Ollama is fine on CPU, just not as transparent.

vLLM

vLLM is excellent when you have a GPU. On a CPU-only VPS, it is usually the wrong answer.

It is not that it is bad software. It is that it is solving a different problem. If the machine has no GPU, I do not start there.

Real install and run commands

I checked the current upstream state before writing this. llama.cpp is active, its latest release is v0.4.0, and Ollama is also current. That does not mean you should chase the newest release every week, but it does mean the tools are alive and maintained.

For a Debian or Ubuntu VPS, this is the clean path I would use for llama.cpp:

sudo apt-get update
sudo apt-get install -y git cmake build-essential pkg-config libopenblas-dev

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

cmake -B build -DGGML_NATIVE=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build -j"$(nproc)"

Then run a small GGUF model with a sensible context window:

./build/bin/llama-server \
  -m /models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -t "$(nproc)" \
  -c 4096 \
  --host 127.0.0.1 \
  --port 8080

If you want the same box to be usable through a browser or another service, I keep the server bound to localhost and put a reverse proxy in front of it. That keeps the model off the public internet unless I explicitly expose it.

Ollama is easier to start with:

curl -fsSL https://ollama.com/install.sh | sh
OLLAMA_NUM_THREADS="$(nproc)" ollama serve

Then pull a small model and keep the context small:

ollama pull qwen2.5:1.5b-instruct

For CPU-only VPS work, I would still start with llama.cpp unless I had a strong reason to prefer the Ollama UX.

Running it as a server with an OpenAI-compatible API

This is the part that makes a local CPU model actually useful. Once the server exposes a /v1/chat/completions style endpoint, you can point scripts, agents, and tools at it without rewriting everything.

A quick test with curl looks like this:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local",
    "messages": [
      {"role": "user", "content": "Extract the invoice total from this text."}
    ],
    "temperature": 0.1
  }'

If you want to point an OpenAI SDK at it, the shape is the same:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")

resp = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Route this ticket to billing or support."}],
)
print(resp.choices[0].message.content)

That is where a CPU VPS starts to earn its keep. It can sit behind your app as a cheap local router, extractor, or summariser without you paying GPU money for every request.

What is actually viable on a small local model

A CPU-only VPS works best when the model does one thing well and does not need to think for long.

Good fits

• classification: spam vs not spam, sales vs support, billing vs technical
• extraction: names, dates, totals, SKUs, email fields, JSON from short text
• routing: pick the next tool, queue, or agent
• cheap drafts: short replies, subject lines, summaries, rewrite passes
• small document triage: the first pass before a bigger model takes over

Bad fits

• long-context reasoning
• coding agents that need many tool calls
• multi-step research with lots of retrieval hops
• huge context windows with dozens of pages of pasted text
• anything where the user expects fast back-and-forth chat

The pattern I use is simple: the CPU model does the cheap front-end work, then a better model handles the hard part if needed. That is much more honest than pretending a 1.5B model can replace a strong hosted model.

What actually goes wrong in practice

The failure modes are not mysterious. They are just annoying.

The OOM killer shows up first

The first time I pushed a small VPS too hard, the box did not politely warn me. It just died. That is what happens when the weights fit but the KV cache and the rest of the process do not.

You will usually see one of three things:

• the process gets killed with no graceful shutdown
• the server stops responding after context grows
• the system starts swapping so hard that everything feels broken

The fix is not clever. Lower the context, reduce the model size, or move up a RAM tier.

Prefill is the real bottleneck, not generation

People love to talk about tokens/sec, but the first response matters more than the stream after it starts. On CPU, prompt evaluation can take far longer than generation, especially when the input prompt is long.

That is why some setups look fine in a short demo and then feel awful in real use. A 300-token prompt can seem okay. A 5,000-token prompt can make the box feel dead.

Context length eats RAM through KV cache

This is the part people forget. Every extra token in context has a memory cost. The bigger the context window, the more the KV cache grows, and the faster the VPS runs out of comfortable headroom.

That is why “I can load the model” is not the same thing as “I can use the model.” The second one is what matters.

The log usually tells on you

When things go wrong, the useful clues are boring:

• memory pressure
• slow prompt eval
• request timeouts
• the server becoming unresponsive under longer prompts

That is also why I do not recommend CPU-only boxes for coding agents. The loop is too chatty, the prompts are too large, and the latency compounds fast.

When you should just pay for an API instead

This is the part people skip because they already rented the VPS and want to justify it. I would rather do the maths.

I verified current pricing for two cheap hosted models before writing this:

• Gemini 2.5 Flash-Lite: $0.10 per million input tokens, $0.40 per million output tokens
• Claude Haiku 4.5: $1.00 per million input tokens, $5.00 per million output tokens

If your workload is 80% input and 20% output, then your blended cost looks like this:

Now compare that with a CPU VPS that costs, say, $15 a month.

• At Gemini Flash-Lite pricing, that VPS only starts to look cheaper once you are using roughly 94 million total tokens a month.
• At Claude Haiku 4.5 pricing, the break-even is around 8.3 million total tokens a month.

If you are nowhere near those numbers, API wins on cost and on time. You are paying for idle CPU 24/7 on the VPS, while the API bills only when you use it.

My rule is blunt:

• use the CPU VPS when privacy, control, or always-on local routing matters
• use the API when you just need cheap intelligence and do not want to babysit a server

Recommendation

If I had to buy one box today, I would not start tiny. A 2 GB VPS is too tight for anything except very small classification jobs. A 4 GB VPS can work if you stay disciplined and keep the model at 1.5B. If I wanted the setup to be pleasant instead of fragile, I would buy the 8 GB CPU VPS tier from Hostinger and run a 1.5B or small 3B GGUF with a short context window.

That is the sweet spot for me: enough RAM to leave breathing room for the OS, the model, and the KV cache, but not so much that I feel tempted to pretend it is a GPU box.

For my own use, I would keep the box boring: llama.cpp server, Q4_K_M quantisation, 4k context or less, localhost-only, and one narrow job. That is the part that actually works.

FAQ

Can I run an LLM on a 2 GB VPS?

Only if you keep the model tiny and the task narrow. A 1.5B model in a tight quantisation can sometimes fit, but there is very little room left for context, logs, and the rest of the OS. For real use, 4 GB is the first tier I would call workable.

What is the best GGUF quantization for CPU-only VPS?

Q4_K_M is my default. It gives the best balance of speed, size, and quality on CPU for most small models. I only move up to Q5_K_M when I have extra RAM and a reason to spend it.

Is llama.cpp better than Ollama on CPU?

For me, yes, when I care about control and predictable performance. Ollama is easier to start, but llama.cpp gives me more direct control over the model, context, and runtime trade-offs. On a CPU VPS, that control matters.

How do I expose llama.cpp as an OpenAI-compatible API?

Use llama-server and point your client at /v1/chat/completions. The OpenAI Python SDK works if you set base_url to your local server and use any placeholder API key. That makes it easy to plug the local model into apps and scripts.

Why is prompt evaluation so slow on CPU?

Because the first pass through the prompt has to process all of the input tokens before the model can start generating. Long prompts, large contexts, and small VPS CPUs make that phase painfully obvious. Generation speed looks better than the first response if you only measure tokens/sec.

Should I self-host or just use an API?

If the job is cheap drafting, extraction, routing, or classification, I would compare the monthly volume against API pricing first. For most small workloads, Gemini Flash-Lite or Claude Haiku is cheaper than running a VPS all month. I only self-host when I want control, locality, or an always-on local service.

Read next

• VPS vs Shared Hosting: Which Should You Choose in 2026?
• Atmospheric Water Generators: The Physics, the Yield, and the Real Cost Per Litre
• What Off-Grid Power Actually Costs Per Month (2026 Breakdown)

Some links on this page are affiliate links. If you buy through them I earn a commission at no extra cost to you. I pay for the services I write about and only recommend what I actually use.