qproj wikiTop-k accuracy and calibration
Two questions matter when a model’s suggestions are reviewed by a person. Is the right answer among its first few suggestions? And when it sounds confident, is it right?
Top-k accuracy
Top-1 accuracy counts how often a model’s first choice is correct. Top-3 counts how often the correct answer is among its first three. Top-3 is the more useful measure when a reviewer picks from a short list: a model that is rarely first but almost always in the top three still saves most of the work.
Calibration
A model is well calibrated when its stated confidence matches how often it is right. Of all the answers it gives 80% confidence, about 80% should be correct. A badly calibrated model is confidently wrong. It wastes a reviewer’s attention on mistakes it claimed were safe, and makes its confidence useless as a guide to what needs checking.
In Quell
References
- Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. arXiv:1706.04599.
What links here: Stage-wise evaluation