Day 4: Optimizing Model Weights — Quantization Basics (GGUF, AWQ) for Efficient Inference

Production AI runs into the memory wall. LLMs are bound by memory bandwidth, because every generated token moves billions of parameters from memory to the compute cores. A standard 8-billion-parameter model in FP16 needs roughly 16 GB of VRAM for the weights alone, before the KV-cache and context. Quantization compresses the weights so these models fit on commodity enterprise hardware without a collapse in quality.

Bytes per parameter

FP16 stores each weight in 16 bits: 1 sign bit, 5 exponent bits and 10 mantissa bits, which is 2 bytes. INT4 uses 4 bits, which is 0.5 bytes, a 75% saving.

The mathematics

Quantization maps a float weight to a small integer with a scale and an offset.

q = round(w / S) + Z
w ≈ S × (q − Z)        (de-quantize)

w is the original weight, S a scale factor, Z a zero-point offset and q the stored integer. At run time the weights are de-quantized on the fly, or used directly by low-bit kernels, which cuts memory by up to 75%.

Here is a tiny runnable demonstration of the rounding error you trade for that saving. It uses simple 4-bit quantization of a random weight vector, so it shows the principle rather than what production formats do (they quantize in small groups and keep extra scales to reduce the error).

import numpy as np

rng = np.random.default_rng(0)
w = rng.normal(0, 0.05, size=4096).astype(np.float32)      # pretend these are weights

bits = 4
levels = 2**bits - 1                                       # 15 steps for 4-bit
S = (w.max() - w.min()) / levels                           # scale factor
Z = round(-w.min() / S)                                    # zero point

q = np.clip(np.round(w / S) + Z, 0, levels).astype(np.uint8)   # quantize: stored as small integers
w_hat = S * (q.astype(np.float32) - Z)                         # de-quantize

print("max level:", q.max(), " memory: 16 bits ->", bits, "bits per weight")
print("mean absolute error:", float(np.abs(w - w_hat).mean()))

Choosing a format

• GGUF: the format used by llama.cpp and Ollama. It targets CPUs and Apple Silicon, supports offloading layers between CPU and GPU, and ships as one portable file.
• AWQ (Activation-aware Weight Quantization): suits high-throughput GPU serving through engines such as vLLM and TensorRT-LLM. It protects the small set of weights that matter most, and holds up well at 4-bit.
• GPTQ: a post-training method for GPUs that calibrates 4-bit weights against a sample dataset.

The format has to match the runtime you plan to use, so decide the engine first.

The quality trade-off

• Q8_0 is very close to FP16 and halves the memory.
• Q4_K_M is a common default for local deployment. An 8B model drops to about 4.8 GB and runs on a 16 GB workstation, usually with a small quality loss.
• Q2_K compresses hardest and can show clear degradation, such as weaker reasoning and malformed JSON output.

How much quality you lose depends on the model and the task, so run your own prompts and a few edge cases before you commit. For example, with Ollama you can pull a specific quantization by tag: ollama pull llama3.1:8b-instruct-q4_K_M.

Run it yourself

This is the exact example from the video. A real script with a real result.

import numpy as np

rng = np.random.default_rng(0)
w = rng.normal(0, 0.05, 4096).astype(np.float32)

bits = 4
levels = 2**bits - 1
S = (w.max() - w.min()) / levels
Z = round(-w.min() / S)

q = np.clip(np.round(w / S) + Z, 0, levels).astype(np.uint8)
w_hat = S * (q.astype(np.float32) - Z)

print("levels used:", int(q.max()) + 1)
print("mean abs error:", round(float(np.abs(w - w_hat).mean()), 5))
print("size in bytes:", w.size * 2, "->", w.size * bits // 8)

Run it:

python quantize.py

Output I got while recording the video:

levels used: 16
mean abs error: 0.00604
size in bytes: 8192 -> 2048

Coming Up Next

Day 5: Local AI runtimes: deploying Ollama and vLLM for bare-metal execution.

#Quantization #MachineLearning #Optimization #LLMs #GGUF #vLLM #DataScience #AIEngineering