Repository navigation
ENH: replace the forecast disk cache with reuse in memory - #1219
MateusStano wants to merge 5 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #1219 +/- ##
===========================================
+ Coverage 86.55% 86.61% +0.06%
===========================================
Files 131 130 -1
Lines 18248 17904 -344
===========================================
- Hits 15794 15508 -286
+ Misses 2454 2396 -58 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This reverts 88aed5a ("ENH: cache downloaded atmosphere netCDF datasets (#654) (#1137)"): atmosphere_cache.py, its tests, the cache code in environment.py and the no_cache parameter of set_atmospheric_model. The cache judged a saved forecast by its age alone (6 hours for every model), was on by default, wrote to the home folder and said nothing when it was used. The next commit replaces it. The pylint comment moves that came with #1137 in motor.py, sensor.py, monte_carlo.py and test_plots.py are kept as they are. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Within one Python session, a forecast downloaded with one of the latest forecast shortcuts (GFS, NAM, RAP, HRRR, AIGFS) is kept in memory. When the same model, place and launch time are asked for again, the server is asked which model run supplies the launch time, and the data is downloaded again only if that run changed. - The server is always contacted. If it cannot be reached the request fails as before, so a forecast that cannot be confirmed as current is never used. - Nothing is written to disk and there is no new parameter. - A reused Environment gets its own copies of the results. The gain is small: on GFS a repeated request takes about 2.3 s in place of about 6.5 s, since the check with the server remains. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
174cf23 to
95c36e6
Compare
|
|
||
| .. note:: | ||
|
|
||
| Within one Python session, RocketPy keeps in memory the forecasts it |
There was a problem hiding this comment.
| Within one Python session, RocketPy keeps in memory the forecasts it | |
| Within one Python session, RocketPy keeps in memory (why?) the forecasts it |
will this consume RAM memory? I don't understand why that would be a better option
There was a problem hiding this comment.
Yes, it will use RAM, but its about 50 kB per forecast (measured on GFS). When you do import rocketpy alone is you save 150 MB to RAM, so its a small addition
Why memory and not disk: I don't want RocketPy writing files in people's home folder by default. With #1137 every different request (model, place, launch time) left a file in ~/.rocketpy_cache, and a file was only removed if the same request came back after it expired. A new launch date every day means a new file every day, forever.
Gui-FernandesBR
left a comment
There was a problem hiding this comment.
Eight inline findings for the author to resolve later. The first is a regression (windy without file); the rest are about the reuse path. I checked them by running the PR's real process_forecast_reanalysis on a GFS-shaped netCDF (live GFS for the elevation key). The new tests do not cover them (the reuse tests replace the processing with a fake).
- Windy without a model uses its default again. - A forecast is identified by its model run and by the first and last times of the dataset, and a launch time outside them is never reused. - The elevation given by the user is not part of the request. - The attributes kept for reuse are listed, not found by comparing objects. - A model run that cannot be read means the forecast is processed again. - The wind heading and direction are rebuilt on a reuse, so they read the Environment's own data. - The 100 most recent forecasts are kept. - The reuse tests read small GFS-shaped netCDF files with the real processing. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Addressed all your comments in the last 2 commits @Gui-FernandesBR |
Pylint counted too many statements in it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Pull request type
Checklist
ruff,pylint) has passed locallyCHANGELOG.md: no action needed here, see the note at the endWhat this is
This covers the forecast cache item of the v1.14.0 release checklist (#1209, "Environment: #1137 cache fixes"), following the discussion in #1194. It replaces the disk cache of #1137 with a much smaller reuse in memory.
Current behavior
#1137 saves the profiles of every forecast download to
~/.rocketpy_cache/atmosphereand loads them back for the next identical request:no_cacheparameter and two environment variables.New behavior
Two commits:
atmosphere_cache.py, its tests, the cache code inenvironment.py,no_cache). The pylint comment moves that came with that PR in other files are kept.file="GFS", NAM, RAP, HRRR, AIGFS) is kept in memory. When the same model, place and launch time are asked for again:reftimeof the dataset at that forecast time);What this guarantees:
The gain is marginal
Measured on GFS: a first request takes about 6.5 s and a repeated one about 2.3 s, because the check with the server remains. So this saves roughly 4 s per repeated forecast, within one session, for five models. It is kept because it is small and safe, and it answers #654 without the risks of a disk cache. To keep a forecast between sessions, save the
Environment.Tests
tests/unit/environment/test_forecast_reuse.py, 17 tests with fake datasets and no network: same run reused, new run downloaded, a run that is only partly published, a different place, date or elevation, an unreachable server, a run that cannot be identified, a dataset given by the user, and that reused results are copies. An autouse fixture clears the memory between tests.Breaking change
no_cache,ROCKETPY_CACHEandROCKETPY_CACHE_TTLexist only ondevelop.Additional information
Closes #654in Updates master with develop before releasing v1.14.0 #1194 can stay.Refs #1209, #1194, #1137, #654.
🤖 Generated with Claude Code