qproj wikiStage-wise evaluation
When a multi-step pipeline gets an answer wrong, the score at the end does not show which step failed. An error early on spoils everything after it. Stage-wise evaluation isolates each step by giving it the gold (human-reviewed) answers for every earlier step. Whatever mistakes it then makes are its own.
For argument maps, that means:
- to score segmentation, provide only the text;
- to score reconstruction, provide the correct segments and relations;
- to score lever choice, provide the correct canonical statements.
A model that produces the whole map in one go has no separate steps to feed. Its output is scored instead by part: locutions, relations, statement fields and levers, each compared against the gold map.
In Quell
See also
- Top-k accuracy and calibration — another measure used in the same evaluation.
What links here: Argument mining, oAMF