Released fine-tuned weights vs. Reproduced SFT results

[seunghan96]

Hello, thank you very much for sharing this excellent research and providing both the code and datasets.
I really appreciate the contribution to the community :)

While experimenting with the fine-tuning scripts, I noticed a significant discrepancy between the performance of the released fine-tuned weights and the weights I obtained by running the provided supervised fine-tuning (SFT) script.

For example, on ETTh1 (4 horizons, average MAE):
• Base (pretrained only): 0.231
• Chat (released fine-tuned weights): 0.144
• Chat-reproduced (pretrained + SFT using provided script & dataset): 0.408

I used the fine-tuning dataset exactly as provided, and I did not make any modifications to the fine-tuning script. Given this, I am wondering:
• Is there any difference between the released fine-tuned weights and the ones obtained by running the current script?
• Could there be additional training steps, hyperparameters, or preprocessing procedures that were applied when producing the released weights but are not reflected in the script?

Any clarification on this would be greatly appreciated. Thank you again for your valuable work!

[seunghan96]

Hello, I would also like to share one potential cause that I noticed.

In the training code in the github (e.g., model.py and the fine-tuning script), the tokenizer is set with:

• tokenizer.padding_side = "right"

However, in the released checkpoint on Hugging Face, the tokenizer_config.json of both Base & Chat specify:

• "padding_side": "left"

This mismatch between "right" in the training code and "left" in the released checkpoint might be related to the performance gap I observed.

Could you please confirm if the fine-tuned weights were originally trained with left padding, and whether this discrepancy could explain the large difference in results?

Thank you again for your time!