Vit-Small/16 pre-training (IN 1K or 21K) ?

[aelnouby]

Thanks for making the pre-trained weights available. I understand that the converted weights for ViT-B and Vit-L are pretrained using ImageNet-21K. However for the weights provided for Vit-Small, are they pretrained using ImageNet-1K or ImageNet-21K ?

[rwightman]

@aelnouby the converted weights are pretrained on ImageNet-21k and finetuned on ImageNet-1k by the Google group that wrote the paper / released official models.

My 'small' model def was a reduced size model I threw together that was more practical to train on 2xGPU setup than the official 'base' model. It was just trained on ImageNet-1k from scratch, but achieved results comparable to the official base model training results on pure ImageNet-1k mentioned in the paper. I used heavier augmentation.

[mmoayeri]

Hi, thanks very much for your work @rwightman

To clarify, are the models 'vit_tiny_patch16_224', 'vit_small_patch16_224', and 'vit_base_patch16_224' all trained on ImageNet-21k and finetuned on ImageNet-1k?
Also, you mention using heavier augmentation -- is this the same augmentation procedure as in DeiT? If so, how do the non-distilled DeiT models differ from the ViT models? My understanding is that the DeiT models are trained on ImageNet-1k only and use heavy augmentation.

Thank you!

[knelk]

@aelnouby, @mmoayeri Did you happen to find any VIT (preferably VIT-B) with pretrained weights from Imagenet1k-only (not 21k+finetune)?

Thanks!

[rwightman]

@knelk answered here #1400

[knelk]

thanks!