How to put cutmix/mixup and label smooth together for A1 configuration of RSB?

[0x4f5da2]

Hi @rwightman, nice to see your paper. I'm reproducing the A1 configuration of 'ResNet strikes back'. It seems that BinaryCrossEntropy loss is not compatible with both cutmix/mixup and label smooth enabled. I'm wondering how to put cutmix/mixup and label smooth together.

Thanks a lot.

[rwightman]

@0x4f5da2 there are lots of examples of this if you search a bit

[0x4f5da2]

@0x4f5da2 there are lots of examples of this if you search a bit

You mentioned to add --bce-target-thresh 0.2 in the A2 and A3. Is this the same case for A1? Adding --bce-target-thresh 0.2 seems to be conflict with the label smooth in the mixup.

Based on my understanding, the target is smoothed in Mixup:
https://github.com/rwightman/pytorch-image-models/blob/2c33ca6d8ce5d9257edf8cab5ab7ece81780aaf7/timm/data/mixup.py#L22

However, adding --bce-target-thresh 0.2 will set the target to 1 if above 0.2 and set to 0 if below 0.2:
https://github.com/rwightman/pytorch-image-models/blob/cd638d50a58ac16f7cb312f36e681eae4dcde655/timm/loss/binary_cross_entropy.py#L42

[rwightman]

@0x4f5da2 yes, there is a discrepancy (thresholding disabling the smoothing) that was noticed after the paper was published. A footnote has been added that will be updated in arxiv. To be fair we actually ran more ablations with several types of smoothing (same +ve/-ve and different +ve/-ve eps) and the smoothing actually didn't have significant impact (so original results stand). They were re-run on several seeds and the mean result across them did not appear to be statistically significant between no-smoothing and the best smoothing setup.

That said, all combinations of bce-loss, bce-loss + target thresh, with and w/o smoothing, and also without any bce at all will work with the LAMB optimizer and the rest of the hparams and produce good results. The best overall combo was found for R50 with the mixture in paper, but I have trained some other, larger models since with more augmentation and regularization and found that bce had less impact there... some of my recent self-attn hybrid halo/bottlneck transformer networks used LAMB based hparams with A1 but no BCE and quite a bit more dropout, stronger RA, RamdomErasing, etc.

[Zoe-Wan]

Hi @rwightman , thanks for your nice job!
I am wondering why is BCE loss better than CrossEntropy loss when dealing with this multi-class classification (not multi-label classification) problems? Am I missing any points?