qproj wikiConstrained decoding
A language model writes its output one token, a word or word fragment, at a time. At each step it assigns a probability to every possible next token. Constrained decoding masks out every token that would break a given grammar before one is chosen. The output is then guaranteed to have the required shape. A model asked for JSON produces valid JSON, and a field restricted to F, V or P can contain nothing else.
The guarantee covers form, not correctness. The output always parses, but it can still be wrong. Constraints can also make a model’s answers worse if they force it into an awkward format. That is one reason to give the model a compact notation rather than a verbose one.
The llama.cpp runtime accepts constraints as a JSON Schema or as a grammar in its GBNF notation.
In Quell
See also
- Quantization and llama.cpp — the llama.cpp runtime and model format.
- In-context learning — the prompting that accompanies the constraint.
External links
- llama.cpp grammars — GBNF and JSON Schema support.
What links here: Quantization and llama.cpp