Using pretrained VIT on MAE

[Songloading]

Hi There,
I am currently trying to apply pretrained VIT models on Masked Autoencoders (MAE). I was trying to follow the MAE implementations in https://github.com/lucidrains/vit-pytorch. However, when I do:

v = timm.create_model('vit_base_patch16_224', pretrained=True)
num_ftrs = v.head.in_features
v.head = nn.Linear(num_ftrs, 2) 
model = MAE(
    encoder = v,
    masking_ratio = 0.75,   # the paper recommended 75% masked patches
    decoder_dim = 512,      # paper showed good results with just 512
    decoder_depth = 6       # anywhere from 1 to 8
)

I got

AttributeError: 'VisionTransformer' object has no attribute 'pos_embedding'

which implies the implementations are not compatible. Is there any way I can fix this without changing internal implementations of the MAE or is there any other MAE implementations in pytorch that can be directly used for pretrained VITs. I've asked the same question at their repo, but it seems they had no answer for this. Thanks in advance!

[rwightman]

@Songloading the official MAE impl (https://github.com/facebookresearch/mae) uses timm vision transformer models, however it uses an older ver and needs a small fix to work with latest facebookresearch/mae#17

lucidrains impl is intended to work with his vit implementation