The question
A higher aggregate success rate does not show whether a change helped the same cases, introduced regressions elsewhere, or rests on a small sample. Those distinctions matter when deciding what to trust or investigate next.
The contribution
I added uncertainty and paired-outcome reporting to SkillEvaluator’s Tier 3 results. Readers can inspect uncertainty around each condition’s case-level success rate, and the direction of changes between matched cases.
The change carries those results through collection, machine-readable output and the HTML report, with documentation and regression tests. An exact paired diagnostic is reported only when the cases are completely paired. Existing success values and verdict thresholds remain unchanged.
A score + its uncertainty
Gains + regressions on matched cases
The report provides more evidence for interpreting the configured run.
The accepted result
NVIDIA merged the contribution into SkillEvaluator’s main branch on 25 August 2026. The public review and patch document the implementation, reported validation and statistical limits.
The statistics describe that configured run. They do not establish task independence, remove leakage, or prove performance on new tasks.