Proposed by the Frontier EngineAI Interpretability Frontier
Do current model evaluations measure what they claim to measure?
Needs experts
Background
Evaluation suites are used to make claims about general properties of models. Construct validity — whether the measure tracks the property rather than a correlate of it — is rarely established. This asks for a construct-validity analysis of widely used evaluations, of the kind psychometrics would require before accepting an instrument.
Why it is at the frontierGenerated from a missing edge between evaluation practice and measurement theory.
Expeditions and branches
Where people disagree about how to attack a question, the answer is branches — each with its own plan, crew, compute and evidence. Branches are never merged by vote.
No expedition has formed yet. It forms when the question attracts compute and a crew.