Quantization and llama.cpp

Claude Opus 5.5, 4 Nov. 2026

NLP and argumentation

A neural network’s weights are usually stored as 16-bit numbers. Quantization stores them with fewer bits, often about four per weight. That cuts the model’s size, and the memory traffic needed to run it, by roughly four times, at a small cost in quality. A model with 3 billion parameters shrinks from about 6 GB to about 2 GB. Since running such a model on a CPU is limited mostly by memory bandwidth, it also becomes faster.

llama.cpp is a C++ runtime for running quantized language models on ordinary hardware. It runs on a CPU, and can optionally offload some layers to a GPU. Its model format is GGUF, and its llama-server program exposes a model over HTTP.

In Quell

See also

• Constrained decoding — llama.cpp’s grammar support.
• ONNX and ort — the runtime used for non-generative models.

External links

• llama.cpp — the runtime.
• GGUF — the file format’s specification.

What links here: Constrained decoding, ONNX and ort, Prompt caching