Evaluating confidence calibration in large language models
Models answer 200 hard multiple-choice questions (spatial reasoning, precise maths, exact lookup and offline knowledge) and say how confident they are. We score them with the Brier score, so being confidently wrong costs more than being unsure.
The leaderboard covers 30-odd models and a human baseline, and I update it as new models come out. Written with Daniel Yang, Xinmeng Huang and Sanyam Kapoor.