Great work! a few questions for the sake of reproducibility

[ericaweng]

-Thanks for the great work and the very thorough documentation. It has made using your code and building off your code very easy! I appreciate it a lot :)
I had a few questions, just to make sure I'm interpreting and reproducing your results accurately.

• did you try an ablation study on HiVT with no map features (whether bev features or decoded vector map) used at all?
• I notice the numbers you report in Table 1 of your paper are only the results for testing on the ego agent. did you compare numbers for testing all the agents? I tried this, and noticed that the performance improvement of using your method isn't as great when evaluating the score of all agents, probably because most agents are static. in addition, the map seems to not be as useful when evaluating on all agents. when evaluating your pretrained model without bev features (setting the vit_embed in LocalEncoder.forward to 0 before feeding into the next module), i get 0.254 vs. 0.245 for your method (MapTRv2_CL + Bev). on the other hand, the improvement is much more noticeable when evaluating only on ego agents (0.417 vs 0.369 (in your paper you report 0.365; i got 0.369 when i ran your pretrained model and the code)).
• how many seeds did you try for each trajectory experiment (each cell in table 1)? Just 1 each, right?
• are the hyperparameter settings set in the argparse defaults (in hivt.py and train.py) the ones used to produce the reported results? (embed_dim=128 being the only change, as you say in the traj.md doc)

[alfredgu001324]

Thanks for the compliments!

1. Not really, I don't think I have tried this.

2. Yes the evaluation is done on the ego vehicle only, as per the standard of trajectory prediction papers. Also since DenseTNT is pretty old, it can only make ego-agent predictions (if you want to make multi agent prediction, I guess you need to run the model for every single agent, but I am not sure whether that will work as it is also only trained on ego trajectories iirc). In contrast, HiVT can make predictions for all the agents in the scene.

Actually, may I know if the released checkpoints work fine? I tidied up the code a bit which I think might break the checkpoint reading, but I never get a chance to try loading it.

3. Yes that is correct.

4. For the specific hyperparameter settings, you can take a look at the appendix of the paper. I have done some grid search to make the performance optimal for each combination (cannot really find a universal configuration unfortunately. This is actually something what I want to explore a bit more when I finish the paper, to dig into the BEV features a bit more to see why it is helpful, are there any other better ways to encode them etc.)

[alfredgu001324]

If you want to use the BEV features to enhance other vehicles' predictions, you probably need to make some changes to the encoding mechanism. Currently it uses the center (where the ego vehicle is located at) as the query (IIRC, not quite remember anymore....), but if you want to use it for enhancements for other vehicle's predictions, you should use their corresponding BEV patches as the query.