Benchmark quality review
We review your tasks, reference answers and scoring code for flaws that penalize correct work or let an incorrect submission pass.
- C1Ground truth or scoring errors
- A reference value is wrong, or a reversed comparison gives a higher score to a less accurate result.
- C2Unstated scoring requirements
- The task never specifies a precision, but the grader accepts only answers rounded to two decimal places.
- C3Missed critical errors
- A test checks that the program runs. A submission can pass even when its main calculation is wrong.
- C4Scoring exploits and answer leaks
- Repeating a keyword earns full credit, or a file visible to the model contains the reference answer.
- C5Other evaluation issues
- The same submission gets different scores under identical conditions, or two instructions in the task contradict each other.