Problem
The TOPS-fMRI score combines two subscores with different chance baselines:
- Study4 uses Pearson
r clipped to [0, 1], where an uninformative constant prediction scores about 0;
- Study5 uses raw AUC, where an uninformative constant prediction scores
0.5.
The final score is 0.5 * study4_score + 0.5 * study5_score.
Trigger scenario
A constant, uninformative model receives study4_score = 0 and study5_score = 0.5, producing an overall score of about 0.25.
Impact
- An uninformative model receives a non-zero headline score.
- The effective score range is compressed.
- Study4 and Study5 contribute equally despite using different zero points.
- A TOPS score of
0.5 is difficult to interpret.
Expected fix
In a new TOPS task version, align both subscores to the same chance baseline. For example, transform AUC with max(0, 2 * AUC - 1).
Keep raw Pearson r and AUC values as diagnostic metrics, document the chance baseline and formula, regenerate the baseline for the new task version, and do not retroactively apply the new formula to old results.
Problem
The TOPS-fMRI score combines two subscores with different chance baselines:
rclipped to[0, 1], where an uninformative constant prediction scores about0;0.5.The final score is
0.5 * study4_score + 0.5 * study5_score.Trigger scenario
A constant, uninformative model receives
study4_score = 0andstudy5_score = 0.5, producing an overall score of about0.25.Impact
0.5is difficult to interpret.Expected fix
In a new TOPS task version, align both subscores to the same chance baseline. For example, transform AUC with
max(0, 2 * AUC - 1).Keep raw Pearson
rand AUC values as diagnostic metrics, document the chance baseline and formula, regenerate the baseline for the new task version, and do not retroactively apply the new formula to old results.