Day 5: Local AI Runtimes — Deploying Ollama and vLLM for Bare-Metal Execution

Loading a model with a native Hugging Face pipeline (AutoModelForCausalLM) inside a web server is simple, and it is inefficient: you carry a heavy Python interpreter, get no real hardware scheduling and tend to fall over under concurrent traffic. Production uses dedicated inference engines instead. This lesson covers the two that matter: Ollama for development and edge compute, and vLLM for datacentre concurrency.

Ollama: development and edge

Ollama wraps llama.cpp and GGUF weights behind a small daemon.

• Memory efficiency. It can split a model's layers between CPU RAM and GPU memory, so models larger than your VRAM still run.
• A Modelfile describes a reproducible model setup.
• OpenAI-compatible endpoints, such as /v1/chat/completions, so existing clients work by changing the base URL.

# Modelfile
FROM gemma2:2b
PARAMETER temperature 0.2
PARAMETER num_ctx 4096
SYSTEM """You are an enterprise infrastructure diagnostic assistant. Output structured JSON only."""

ollama create infra-diag -f Modelfile
ollama run infra-diag "Summarise: link flap on port 12, CRC errors rising"

vLLM: datacentre concurrency

vLLM is built to serve many users at once from a GPU.

PagedAttention. A standard engine allocates the KV-cache in one contiguous block per request, sized for the maximum context, and much of it stays empty. The vLLM authors measured 60 to 80% of KV-cache memory wasted in earlier systems. vLLM instead stores the cache in non-contiguous blocks, like virtual-memory paging in an operating system, and reports under 4% waste.

Continuous batching. Rather than waiting for a whole batch to finish, vLLM interleaves incoming prompts into the running batch at the token level, which keeps the GPU busy and raises throughput.

Deploying vLLM

docker run -d --name enterprise-vllm \
  --gpus all \
  --shm-size="4g" \
  -p 8000:8000 \
  -v /opt/models/gemma-2-9b-awq:/model \
  vllm/vllm-openai:latest \
  --model /model \
  --quantization awq \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90

• --gpu-memory-utilization 0.90 lets vLLM claim 90% of VRAM for the weights and the PagedAttention cache pool. It is a starting point, so leave headroom if other processes share the GPU.
• --max-model-len 4096 caps the context per request, so the cache pool can serve more users at once.
• --quantization awq matches the AWQ weights from Day 4.

vLLM exposes the same OpenAI-compatible API, so your client code does not change.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
reply = client.chat.completions.create(
    model="/model",                       # the name vLLM was started with
    messages=[{"role": "user", "content": "List three causes of CRC errors."}],
)
print(reply.choices[0].message.content)

Pick Ollama while you build and for small or edge machines. Move to vLLM when concurrency, not convenience, is the problem.

Run it yourself

This is the exact example from the video. A NumPy simulation of the idea, not vLLM itself, so use it to understand the effect and measure your own engine for real numbers.

import numpy as np

rng = np.random.default_rng(0)
MAX, BLOCK = 4096, 16
used = rng.integers(100, 1500, 1000)   # tokens really used

contig = np.full(used.size, MAX)  # reserve the max
paged = np.ceil(used / BLOCK) * BLOCK  # whole blocks

def waste(reserved):
    return 100 * (1 - used.sum() / reserved.sum())

print(f"contiguous waste: {waste(contig):.0f}%")
print(f"paged waste: {waste(paged):.1f}%")
slots = 10**6
print("fit in 1M:", slots // MAX, "vs", int(slots // paged.mean()))

Run it:

python paged.py

Output I got while recording the video:

contiguous waste: 80%
paged waste: 0.9%
fit in 1M: 244 vs 1205

Coming Up Next

Day 6: Containerizing AI microservices: building and packaging APIs with FastAPI.

#LocalAI #vLLM #Ollama #ModelServing #DevOps #DataSovereignty #HighPerformanceComputing #OpenSource