Specific training scripts

[bjfranks]

Hi,

I am looking for specific training scripts for ResNet (right now) and EfficientNet (in the future) to approximately reproduce the accuracies reported by the pretrained variants. For instance do you provide the specific code or command line arguments used for ResNet strikes back? The best I can find are the sample scripts on https://rwightman.github.io/pytorch-image-models/scripts/ or https://rwightman.github.io/pytorch-image-models/training_hparam_examples/, but I suspect these will not reproduce the reported performances. Is the A1 recipes command line provided somewhere for easy reference?

[bjfranks]

Perhaps someone can clarify if the following is a correct A1 recipe
./distributed_train.sh 9 /data/imagenet/ --amp -b=2048 --epochs=600 --drop-path=0.05 --opt=lamb --weight-decay=0.01 --sched='cosine' --lr=5e-3 --warmup-epochs=5 --model=resnet50 --aa=rand-m7-mstd0.5-inc1 -j=16 --reprob=0.0 --remode='pixel' --mixup=0.2 --cutmix=1.0 --bce-loss --bce-target-thresh 0.2 --aug-repeats 3
Its not quite finished training, but I appear to be getting better accuracies than 80.7 already.

[rwightman]

I don't see how those could work as you've got it set for a batch size of 2048 per card (you need to device batch size across devices and adjust LR if the global batch size is != 2048). otherwise looks okay, think weight decay is .02, you can used fusedlamb on GPU if you have apex installed (slightly faster).

[bjfranks]

Ahh, ok thanks, I missunderstood the -b option. In an actual run I reduced the batch-size to 128, but I guess I then modified the learning rate incorrectly. So I suspect a proper setting would perhaps be:
./distributed_train.sh 8 /data/imagenet/ --amp -b=128 --epochs=600 --drop-path=0.05 --opt=lamb --weight-decay=0.02 --sched='cosine' --lr=2.5e-3 --warmup-epochs=5 --model=resnet50 --aa=rand-m7-mstd0.5-inc1 -j=16 --reprob=0.0 --remode='pixel' --mixup=0.2 --cutmix=1.0 --bce-loss --bce-target-thresh 0.2 --aug-repeats 3

[rwightman]

@bjfranks close, but I find it better to scale the lr by sqrt for lamb and adam so I'd do 5e-3 * sqrt(8*128/2048) = .0035 .. not sure what your computer is, but j=16 is a lot of workers when you have 8 dist processes, that'll be 8*16=128 processes, so unless you have 128 CPU cores I'd take it back to a number that doesn't exceed your total CPU core count.

with cosine curve good idea to set your min-lr for the end, and we used a lower starting lr than the default I'd add --warmup-lr 1e-6 --min-lr 1e-6