Pre-processing code for video features

[YOURENXIAKE]

First of all, I would like to thank you for open-sourcing your code for TranSTR. It is a valuable resource for the research community.

I am trying to use your code, but I am facing an issue with the pre-processing of the video features.

The code does not provide any pre-processing code, so I am using my own video features, which are 2048 dimensions. When I try to use these features in the multimodal_transformer.py file, I am getting an error in the with_pos_embed function. The error is because the pos variable is 768 dimensions, but my video features are 2048 dimensions.

I would like to request that you provide some pre-processing code for the video features. This would make it easier for users to use your code with their own video features.

I would appreciate any help you can provide with this issue.

[dongfengxijian]

@YOURENXIAKE
I also met this error. If you solve this, please @me. Thank you!

[yl3800]

Hi, thanks for the interest. If you want to use your feature, just change the default feature dim to 2048. Meanwhile, I will try to share the frame feature that I used, but you might have to follow doc-doc/CoVGT#2 to extract the object feature, as these object feature files are too large to share.

[YOURENXIAKE]

Thanks for your reply! We also used CoVGT's features, but the reproduced results were lower after adding a linear layer to map 2048 to 768, so I'd like to ask how CoVGT's features are handled, thanks!