Skip to content

Commit

Permalink
IM-283 Workaround for chunks with mixed filenames
Browse files Browse the repository at this point in the history
* We have a "cache" (or template) for filenames. That template
  includes the version suffix `-v1.0.nc`.

* If there is a chunk with mixed filenames (`*-v1.0.nc` and
  `*-v1.1.nc`), the aggregation process fails:

2024-02-21T16:40:01.579160980Z Reading /home/klab/.klab/copernicus/sis-agrometeorological-indicators/solar_radiation_178/Solar-Radiation-Flux_C3S-glob-agric_AgERA5_20230916_final-v1.0.nc...
2024-02-21T16:40:01.580784561Z ERROR: /home/klab/.klab/copernicus/sis-agrometeorological-indicators/solar_radiation_178/Solar-Radiation-Flux_C3S-glob-agric_AgERA5_20230916_final-v1.0.nc (No such file or directory)
  • Loading branch information
iperdomo committed Feb 22, 2024
1 parent 834b0e3 commit 1cb1f96
Showing 1 changed file with 19 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -426,6 +426,11 @@ public boolean accept(File pathname) {
+ ".nc");
}

private File adjustDataFilePath(File f) {
String newPath = f.getAbsolutePath().replace("-v1.0.nc", "-v1.1.nc");
return new File(newPath);
}

@Override
protected boolean createAggregatedLayer(String variable, int startTick, int endTick, ITime.Resolution resolution,
File destinationFile) {
Expand All @@ -438,7 +443,20 @@ protected boolean createAggregatedLayer(String variable, int startTick, int endT

List<File> toAggregate = new ArrayList<>();
for (int tick = startTick; tick <= endTick; tick++) {
toAggregate.add(getDataFile(variable, tick));
File dataFile = getDataFile(variable, tick);

if (dataFile.exists()) {
toAggregate.add(dataFile);
continue;
}

// Workaround for chunks with mixed file names
// The file path includes the version of the CDS API that was requested with
// Some chunks contain a mix of *-v1.0.nc and *-v1.1.nc
File newPath = adjustDataFilePath(dataFile);
if (newPath.exists()) {
toAggregate.add(newPath);
}
}

/*
Expand Down

0 comments on commit 1cb1f96

Please sign in to comment.