25 tarin on three year data

[rogerkuou]

fix #25

did not finish the train-validation-test split in this PR, but made a new issue #28

[rogerkuou]

Hi @SarahAlidoost and @meiertgrootes, I created an exmaple training process on a subset of the two year data., and ran it on Levante. In this PR I included example SLURM training process, with an README on how to config the jobs on Levante.

A copy of the example run can be found on /work/bd0854/b380854/eso4clima. I executed the slurm task on my home dir, and copied the entire experiment here.

[SarahAlidoost]

@rogerkuou thanks for the script. Since the PR #29 fixed a few issues, we need to merge main into this branch. I also left some comments, mainly about the structure of the example.py and the code that should be run with slurm. If something is unclear, please let me know. In meantime, I will work on issue #33.

[rogerkuou]

Hi @SarahAlidoost, thanks for the review! I implemented most of your comments:

1. I separated the example script to two: training and inference
2. Now the training scipt export models with checkpoint. I slightlt modified the class to make it returning the config
3. I used logging to replace the print statements
4. The plotting part has been removed

I did not implemente the training utility function and will leave it to #33 .

Can you give another look?

[SarahAlidoost]

@rogerkuou thanks for addressing the comments 👍 . Here some more suggestions:

• I see that the example notebook has been changed in this PR. I cannot see exactly what is changed, but since this PR is about testing large data on HPC, let's not change the example notebook.
• No need to add inference script in this PR. For now we can skip that one. Let's focus on setup of the training on HPC in this PR. Also, in fixing #32 we can add inefrence script later.
• After implementing these suggestions and re-running the slurm job, can you please add the slurm logfile to the PR as well? Also, can you perhaps give an indication how much resources have been used to complete the job.

If something not clear, please let me know.

[SarahAlidoost]

Hi @SarahAlidoost @meiertgrootes, I updated the example training script with you utility functions, and performed two experiments, both with 128 cores:

• A train on two year dataset and space subset from 30S to 30N and from 30W to 30E. This experiment successfully finished in ~2hrs. Log file eso4clima_24438134_subset.out.
• A train still on two year dataset but with ALMOST global coverage, from 80S to 80N and from 179.99W to 179.99E. Since this experiment is only for evaluate the performance, I only ran a SLURM job of 1hr. The traning finished >20 epochs and killed by SLURM time limit. Log file eso4clima_24449471_full.out.

Based on the two experiments I have several findings, also to address @SarahAlidoost 's comments in pervious review:

1. When including high lattitude data (>80N/S), the training cotinues giving loss=inf even after 40 epochs. I have not investigate the cause, nor the threshold of the lattitude. But I got the impression that the current coverage is something we want to work with. Therefore I left this spatial subsetting in the exmaple script. (maybe we should create an issue?)
2. In the current log file there are many warnings when using open_mfdataset to lazy loading multiple nc files. I did another experiment with zarr and these warnings are gone. I would recommend if possible, convert the files to Zarr. This might be easy for daily datasets but hard for hourly.
3. The STDataset seem to handle the lazy loaded data well without need to interfene.
4. following @SarahAlidoost previous comments, I changed the batch_size = 10 with accumulation_steps = 2. This seems to work well for global datasets. I use a patch_size_training = 120 for patch size.

Please feel free to have another look.

Hi @rogerkuou Thanks! From your comment, I understood that you ran the job for 1 hour on purpose. I don’t think that’s the best way to assess performance, but we can leave it for now. About the open_mfdataset warning and the zarr vs. nc, let’s first make sure we understand the warning; are they actually related to the file format? If so, could you make an issue for it?

I’ve added some comments and made a few issues, no need to run the job at this point. We can improve the slurm job step by step later. For now, let’s keep the PR open.

[rogerkuou]

Hi @rogerkuou Thanks! From your comment, I understood that you ran the job for 1 hour on purpose. I don’t think that’s the best way to assess performance, but we can leave it for now. About the open_mfdataset warning and the zarr vs. nc, let’s first make sure we understand the warning; are they actually related to the file format? If so, could you make an issue for it?

I’ve added some comments and made a few issues, no need to run the job at this point. We can improve the slurm job step by step later. For now, let’s keep the PR open.

Thanks @SarahAlidoost, I replied all the comments in the thread or in the issue. Thanks for making the new issues. Indeed they are good next steps to investigate. For now I will keep this PR open but working on new issues.

[meiertgrootes]

Beyond the issues that were already raised and for which separate issues have been opened this looks fine. See my comment on th eorogin of the NaN values in #41

[CLAassistant]

All committers have signed the CLA.

[SarahAlidoost]

Suggested change

	#SBATCH --nodes=5
	#SBATCH --nodes=1

[SarahAlidoost]

@rogerkuou for training we can only use 1 node as the current training workflow cannot be distributed over several nodes, see this line

[SarahAlidoost]

Instead of hardcoding, can we get these parameters directly from the results of the hyper parameter tuning?