Issues in dataset preprocessing and ADE/FDE results

[InhwanBae]

Hi @CHENGY12

Thank you for your great work. I was able to set up and run code quite quickly!

While reproducing results with your evaluation code using pre-processed dataset pickle file from Trajectron++, I noticed that the results were somewhat different.

I tried to analyze the cause of this, and I found that data_utils.py in your code is based on the eccv2020 branch of trajectron++. In that branch, there was a critical issue in data pre-processing that future two GT coordinates were additionally used while calculating the acceleration of the last observation points. Many issues have been raised in Trajectron++ regarding this problem (issues #26,#40 and #53).

DisDis/trajectron/environment/data_utils.py

Line 27 in 55188a5

dx[~np.isnan(x)] = np.gradient(x[~np.isnan(x)], dt)

Fortunately, this issue was solved in the Trajectron++ code. Below are my PCMD results after fixing it and re-training the DisDis model:

DisDis	ETH	HOTEL	ZARA1	ZARA2	UNIV	AVG
PCMD_ADE	1.02/0.85/0.61	0.42/0.25/0.15	0.43/0.31/0.19	0.34/0.25/0.15	0.54/0.40/0.26	0.55/0.41/0.27
PCMD_FDE	2.13/1.72/1.16	0.80/0.46/0.23	0.96/0.64/0.35	0.77/0.55/0.29	1.20/0.86/0.51	1.17/0.85/0.51

Note that DisDis still performs much better than other comparable models (including reproduced Trajectron++). Also, I did not edit the config file, so there is room for better results when modifying the hyper-parameters.

As your paper compared the DisDis with other SOTA models which use Social-GAN data-loader (Social-GAN, STGAT, Social-STGCNN), I think the authors should fix this issue and update the numbers in the arXiv paper for a fair comparison.

Thank you.

[CHENGY12]

Thanks so much for pointing out this problem, fixing the bugs, and re-running the methods!
We used the eccv2020 branch of trajectron++ as our baseline and applied our method to it. We will update it as the newest version of trajectron++ to solve the problem. Thanks again for re-running the modified code and showing that DisDis still performs better than other comparable models (including reproduced Trajectron++). We will re-run all experiments to double-check the conclusion and will update Table 1, Table 2, and Figure 3 in the Arxiv.

[InhwanBae]

Hi @CHENGY12, thank you for your prompt response. Please let me know once you update your results! :D

[CHENGY12]

Hi @InhwanBae, we have updated the results. Thank you very much!

[InhwanBae]

Hi @CHENGY12, thank you so much for updating the results! I'm closing this issue now. Thanks again!