Swin-T training on ImageNet-1k --> does not converge

[RuslanGreenhead]

Hello everyone!
I try to reproduce the original Swin Transformer paper results on ImageNet-1k classification. I use training configuration as stated in paper:
(batch_size=1024 (in my case --> 2 GPUs * 256 sampes each * 2 accumulation steps), AdamW, initial_lr=1e-3, weight_decay=0.05, grad_clip_norm=1.0, 300 epochs (first 20 - linear warmup, then - cosine decay), drop_path=0.2, other dropouts disabled)
But the model comes out on a plateau of about 35% val top-1 accuracy and does not converge further (train loss doesn't come down either)... Augmentations are same to the authors' ones
What can cause such a problem? And how can I fix it? Would be greatful for any piece of advice!

[rwightman]

@RuslanGreenhead try turning off accumulation, not that well tested and probably not wortwhile here... and lr to 5e-4

[rwightman]

If it works without accum let me know, probably something to look at. Otherwise, possibly something going on with dataset... so detail on which copy of imagenet-1k is being used, how the folders are setup, etc needed. And I'm assuming using the train script here and not elsewhere...