Gold maps and measurement

design · draft · NLP and argumentation · edited 2026-11-06

The gold maps are reviewed xAIF files in misc/labs/xAIF/examples. Each one is both game content and a test case for every arm.

What exists

• Reviewed: aquinas-llm.json, engels-alt.json.
• Drafted, awaiting Ada’s critique: aristotle-function.json.
• Every way into an RA: ra-inputs.json, also the conformance suite for any re-implementation of the rules.
• Conventions: judgement-calls.md, the seed of a labelling guide.
• Corpus: 31 excerpts from 29 writers in misc/labs/nlp/data/pta-corpus. Never committed.

How a map gets made

flowchart LR
  P["draft: by hand,<br/>an agent, or an arm"] --> E["review in<br/>the xAIF editor"]
  E -->|accepted / fixed| G[("gold maps")]
  E -->|new kind of decision| J["judgement-calls.md"]
  J --> P
  G -->|"few-shot examples,<br/>thresholds"| P

Cloud models and agents are allowed here, for drafting and labelling. The game never depends on them.

How arms are scored

• Stage-wise. Feed the gold answers to every step before the one under test.
• Split by writer. No writer’s text helps label their own.
• Exemplars first. Look at the worst cases before computing statistics.
• Calibration. When an arm says 80%, is it right 80% of the time? The editor will show confidences.
• A human ceiling. A second annotator on a sample shows how much people disagree.

What links here: Arms and decisions, Open questions, Phases, Quell debate: plan map, Argument maps, Top-k accuracy and calibration, Stage-wise evaluation, Research labs, xAIF editor