Enhancement of GETKF/LETKF ensemble DA capability.

[mos3r3n]

Description

This PR introduces various new features in LETKF/GETKF:

1. use the linear operator as default for more efficient observer step;
2. Enable inflation within LETKF/GETKF itself, instead of using the RTPP application;
3. Use 'reduce size space' for the satellite data thinning section to reduce memory and IO burden. It also benefits the variational DA.
4. Enable offline bias correction in LETKF/GETKF.
5. Calculate HofX of ensemble analyses (EnKFDiagOMA), separately from the Observer and Solver steps.

Note: The two-step strategy is still used, due to the poor efficiency of using 'Halo' distribution for ensemble HofX calculation; EnKFDiagOMA is now still a part of ensemble DA, making it a dependency for ensemble forecasts (To be improved).

Issue closed

Closes #(if applicable)

Tests completed

Tier 1:

• 3dvar_OIE120km_WarmStart
• 3denvar_OIE120km_IAU_WarmStart
• 3dvar_OIE120km_ColdStart
• 3dvar_O30kmIE60km_ColdStart
• 3denvar_O30kmIE60km_WarmStart
• eda_OIE120km_WarmStart
• getkf_OIE120km_WarmStart
• letkf_OIE120km_WarmStart
• ForecastFromGFSAnalysesMPT

Tier 2 (optional):

• GenerateGFSAnalyses
• GenerateObs

[ibanos90]

Hi @mos3r3n, I thought a bit more about your changes for the static BC for EnKF and I was wondering if in order to make it a somewhat more generic we could add a variable that indicates if bias correction is variational or static instead of making it application dependent. What do you think?

Also, now that we have a folder named biasStatic, would it make sense renaming the current bias => biasVariational?

[mos3r3n]

Hi @mos3r3n, I thought a bit more about your changes for the static BC for EnKF and I was wondering if in order to make it a somewhat more generic we could add a variable that indicates if bias correction is variational or static instead of making it application dependent. What do you think?

Also, now that we have a folder named biasStatic, would it make sense renaming the current bias => biasVariational?

I did the modification to directly set the bias coefficient files in the yaml, not linking them to dbOut or dbIn. That would be much cleaner. For the potential variational DA that doesn't update bias coefficient files, I prefer to leave it to the future. I will not rename 'bias' to 'biasVariational' in this PR unless you all think it is necessary. Thanks!

[ibanos90]

Hi @mos3r3n, I thought a bit more about your changes for the static BC for EnKF and I was wondering if in order to make it a somewhat more generic we could add a variable that indicates if bias correction is variational or static instead of making it application dependent. What do you think?
Also, now that we have a folder named biasStatic, would it make sense renaming the current bias => biasVariational?

I did the modification to directly set the bias coefficient files in the yaml, not linking them to dbOut or dbIn. That would be much cleaner. For the potential variational DA that doesn't update bias coefficient files, I prefer to leave it to the future. I will not rename 'bias' to 'biasVariational' in this PR unless you all think it is necessary. Thanks!

Sounds good, thanks for considering these changes!

[mos3r3n]

Hi @mos3r3n, I thought a bit more about your changes for the static BC for EnKF and I was wondering if in order to make it a somewhat more generic we could add a variable that indicates if bias correction is variational or static instead of making it application dependent. What do you think?
Also, now that we have a folder named biasStatic, would it make sense renaming the current bias => biasVariational?

I did the modification to directly set the bias coefficient files in the yaml, not linking them to dbOut or dbIn. That would be much cleaner. For the potential variational DA that doesn't update bias coefficient files, I prefer to leave it to the future. I will not rename 'bias' to 'biasVariational' in this PR unless you all think it is necessary. Thanks!

Sounds good, thanks for considering these changes!

The calculation of HofXs of ensemble analyses needs to modify dbOut to dbAna. When the bias coefficient files are set in the yaml, their locations will be changed to dbAna, too, and diagOMA will fail. I have modified it to store the HofXs of analyses in dbOut but with different suffixes, so dbAna is no longer needed.

[ibanos90]

Thanks for addressing my comments @mos3r3n!