Best practices for evaluating ML models on imbalanced datasets?

[yazdanparasthesam]

Hi everyone 👋

I'm interested in how people in the OpenML community approach model evaluation for imbalanced classification problems.

In many real-world ML projects, accuracy can be misleading when one class is significantly underrepresented. I would be interested to hear your opinions on:

• Which metrics do you typically prioritize: F1-score, ROC-AUC, PR-AUC, balanced accuracy, or others?
• When is PR-AUC more informative than ROC-AUC?
• Do you usually use stratified cross-validation?
• How do you handle threshold selection and calibration?
• Do you recommend resampling techniques such as SMOTE, or do you prefer class-weighted models?

I would especially appreciate insights based on practical experience with OpenML datasets and benchmark experiments.

Thanks in advance! 🚀

[JayeshSuryavanshi]

Coming at this from a lot of fraud and rare-event work, so my defaults assume "the positive class is 1% and expensive to miss." Going through your questions:

Metrics. I treat PR-AUC (average precision) as the headline number and basically ignore plain accuracy. The reason is base-rate sensitivity: if 1% of your data is positive, always predicting "negative" gets you 99% accuracy and a respectable ROC-AUC while being useless. F1 and balanced accuracy are fine, but they bake in a threshold, so they describe one operating point rather than the ranking quality of the whole model. What I actually report is PR-AUC for ranking quality, plus precision and recall at the threshold I would deploy at.

PR-AUC vs ROC-AUC. ROC-AUC rewards ranking the true negatives, and when negatives massively outnumber positives that part is easy and inflates the score. PR-AUC only looks at how well you retrieve the positive class, and it moves with the base rate, so it tracks the thing you usually care about. My rule of thumb: the rarer the positive and the more the cost sits on the positive side, the more I lean on PR-AUC. Near balance, ROC-AUC is fine.

Cross-validation. Always stratified so each fold keeps the class ratio, and with very few positives I use repeated stratified CV to cut variance, because one unlucky split can swing the numbers a lot when you only have a few dozen positives. The bigger trap in real data is grouped or leaky records (same user, account, or time window landing in both train and test). If that applies, StratifiedGroupKFold or a time-based split matters far more than the metric choice.

Threshold and calibration. I never evaluate at the default 0.5. I choose the threshold on a validation split to hit a target (a minimum precision, a fixed recall, or minimum expected cost if I have a cost matrix) and report the precision/recall tradeoff around it. Calibration matters when the scores have to mean something, either because you threshold on probabilities that must transfer across time or because the score feeds a downstream cost calculation. Gradient-boosted trees are often miscalibrated out of the box; CalibratedClassifierCV with isotonic or Platt on a held-out split fixes most of it.

SMOTE vs class weights. My default first move is cost-sensitive learning: class_weight="balanced", or scale_pos_weight for XGBoost/LightGBM. It is simpler, adds no synthetic data, and in my experience it matches or beats resampling most of the time. If you do resample, the rule that actually matters is to resample inside the CV fold, never before the split; use an imblearn Pipeline so SMOTE only sees the training portion of each fold, otherwise you leak and the CV score becomes fiction. And be skeptical of SMOTE on high-dimensional, categorical, or fraud-like data, where interpolated points are often unrealistic. Treat it as something you validate beats class weights, not something you assume helps.

OpenML-specific: the predefined stratified splits in tasks are handy exactly because they make runs comparable, and since runs store per-instance predictions you can recompute PR-AUC yourself even when the task default is AUC. That is usually what I do rather than trusting a single default metric.