Results are not reproducible

[Fan-QY]

The results of your work are excellent. However, we reproduced based on your instructions and there is a difference of AP 5.7% (62.5%-56.8%) in scannet val. Could you please give tips and help accordingly?
Thanks a lot!

20250817_165321.log
20250817_221442.log

[AdrianKruse]

Did you load the SSTNet weights? I had a similar issue with MAFT, where the scores are WAY worse training without the pretrained SSTNet weights.

[Fan-QY]

Did you load the SSTNet weights? I had a similar issue with MAFT, where the scores are WAY worse training without the pretrained SSTNet weights.

You can see my log, I added it, but the results are still bad.

[Howard-coder191]

Thank you for your attention. The results are somewhat unexpected, even lower than those of MAFT. To rule out any potential implementation issues, we plan to reproduce the experiments.

[Fan-QY]

Thank you for your attention. The results are somewhat unexpected, even lower than those of MAFT. To rule out any potential implementation issues, we plan to reproduce the experiments.

Thank you very much, I would appreciate it if you would give me a comment after repeating the validation experiment.

[Howard-coder191]

20250919_184321.log Hi, I reproduced the results on another device, and they are consistent with the previous ones, with only about a 1~2% difference, which is acceptable in this task. Could you give me more information?

[Fan-QY]

20250919_184321.log Hi, I reproduced the results on another device, and they are consistent with the previous ones, with only about a 1~2% difference, which is acceptable in this task. Could you give me more information?

Thanks for your reply, I because the code that I reproduced using docker's containers, due to my hardware only supports CUDA 11.8+, cuda version is higher and pytorch 2+. So I upgraded the code of attention_rpe_ops, pointgroup_ops and Relation3D's compiled partial functions. I will downgrade and try again.

[Fan-QY]

I reproduced it from start to finish. pytorch == 1.13, CUDA = 11.7. the code didn't change anything (Just #2), pre-training weights SSTNet. my graphics card is an A6000 Ada should be the same hardware architecture as the 4090 [Ada Lovelace] . But the results are still bad, I haven't reached AP 0.56 after 250 Epoch. But your log has reached AP 0.60.
My final result is still only 56.49. a huge difference from what you have in your paper.

Do you have any tips for training?

20251006_130805.log