Skip to content

ENH: replace the forecast disk cache with reuse in memory - #1219

Open
MateusStano wants to merge 5 commits into
developfrom
enh/forecast-reuse
Open

MateusStano wants to merge 5 commits into
developfrom
enh/forecast-reuse

Conversation

@MateusStano

Copy link
Copy Markdown
Member

Pull request type

  • Code changes (bugfix, features)
  • Code maintenance (refactoring, formatting, tests)

Checklist

  • Tests for the changes have been added
  • Docs have been reviewed and updated
  • Lint (ruff, pylint) has passed locally
  • Environment unit tests have passed locally
  • CHANGELOG.md: no action needed here, see the note at the end

What this is

This covers the forecast cache item of the v1.14.0 release checklist (#1209, "Environment: #1137 cache fixes"), following the discussion in #1194. It replaces the disk cache of #1137 with a much smaller reuse in memory.

Current behavior

#1137 saves the profiles of every forecast download to ~/.rocketpy_cache/atmosphere and loads them back for the next identical request:

  • A saved forecast is judged by its age alone, 6 hours for every model, so a newer model run is ignored until the file expires.
  • It is on by default, writes to the home folder, and says nothing when a saved forecast is used.
  • It adds about 950 lines, a no_cache parameter and two environment variables.

New behavior

Two commits:

  1. Revert of ENH: cache downloaded atmosphere netCDF datasets (#654) #1137 (atmosphere_cache.py, its tests, the cache code in environment.py, no_cache). The pylint comment moves that came with that PR in other files are kept.
  2. Reuse in memory, about 110 lines. Within one Python session, a forecast downloaded with one of the latest forecast shortcuts (file="GFS", NAM, RAP, HRRR, AIGFS) is kept in memory. When the same model, place and launch time are asked for again:
    • the server is asked which model run supplies the launch time (the reftime of the dataset at that forecast time);
    • same run: the kept results are reused, as copies;
    • different run, or a run that cannot be identified: the data is downloaded again.

What this guarantees:

  • The server is always contacted. If it cannot be reached, the request fails as it does today. A forecast that cannot be confirmed as current is never used.
  • Nothing is written to disk, and there is no new parameter or environment variable.
  • A file or address given by the user is never kept. Ensembles, reanalysis, Windy, Open-Meteo and Meteomatics are untouched.
  • While a new run is being published, near forecast times switch to it before far ones. The run is read at the launch time, so the download happens as soon as the new run reaches it.

The gain is marginal

Measured on GFS: a first request takes about 6.5 s and a repeated one about 2.3 s, because the check with the server remains. So this saves roughly 4 s per repeated forecast, within one session, for five models. It is kept because it is small and safe, and it answers #654 without the risks of a disk cache. To keep a forecast between sessions, save the Environment.

Tests

tests/unit/environment/test_forecast_reuse.py, 17 tests with fake datasets and no network: same run reused, new run downloaded, a run that is only partly published, a different place, date or elevation, an unreachable server, a run that cannot be identified, a dataset given by the user, and that reused results are copies. An autouse fixture clears the memory between tests.

Breaking change

  • No

no_cache, ROCKETPY_CACHE and ROCKETPY_CACHE_TTL exist only on develop.

Additional information

Refs #1209, #1194, #1137, #654.

🤖 Generated with Claude Code

@MateusStano
MateusStano requested a review from a team as a code owner November 17, 2026 01:15
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.82540% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.61%. Comparing base (9bf63a0) to head (3e18f79).

Files with missing lines Patch % Lines
rocketpy/environment/tools.py 90.47% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1219      +/-   ##
===========================================
+ Coverage    86.55%   86.61%   +0.06%     
===========================================
  Files          131      130       -1     
  Lines        18248    17904     -344     
===========================================
- Hits         15794    15508     -286     
+ Misses        2454     2396      -58     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

MateusStano and others added 2 commits November 19, 2026 21:14
This reverts 88aed5a ("ENH: cache downloaded atmosphere netCDF datasets
(#654) (#1137)"): atmosphere_cache.py, its tests, the cache code in
environment.py and the no_cache parameter of set_atmospheric_model.

The cache judged a saved forecast by its age alone (6 hours for every
model), was on by default, wrote to the home folder and said nothing when
it was used. The next commit replaces it.

The pylint comment moves that came with #1137 in motor.py, sensor.py,
monte_carlo.py and test_plots.py are kept as they are.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Within one Python session, a forecast downloaded with one of the latest
forecast shortcuts (GFS, NAM, RAP, HRRR, AIGFS) is kept in memory. When
the same model, place and launch time are asked for again, the server is
asked which model run supplies the launch time, and the data is downloaded
again only if that run changed.

- The server is always contacted. If it cannot be reached the request
  fails as before, so a forecast that cannot be confirmed as current is
  never used.
- Nothing is written to disk and there is no new parameter.
- A reused Environment gets its own copies of the results.

The gain is small: on GFS a repeated request takes about 2.3 s in place
of about 6.5 s, since the check with the server remains.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

.. note::

Within one Python session, RocketPy keeps in memory the forecasts it

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Within one Python session, RocketPy keeps in memory the forecasts it
Within one Python session, RocketPy keeps in memory (why?) the forecasts it

will this consume RAM memory? I don't understand why that would be a better option

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it will use RAM, but its about 50 kB per forecast (measured on GFS). When you do import rocketpy alone is you save 150 MB to RAM, so its a small addition

Why memory and not disk: I don't want RocketPy writing files in people's home folder by default. With #1137 every different request (model, place, launch time) left a file in ~/.rocketpy_cache, and a file was only removed if the same request came back after it expired. A new launch date every day means a new file every day, forever.

@Gui-FernandesBR Gui-FernandesBR left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eight inline findings for the author to resolve later. The first is a regression (windy without file); the rest are about the reuse path. I checked them by running the PR's real process_forecast_reanalysis on a GFS-shaped netCDF (live GFS for the elevation key). The new tests do not cover them (the reuse tests replace the processing with a fake).

Comment thread rocketpy/environment/environment.py Resolved
Comment thread rocketpy/environment/environment.py Resolved
Comment thread rocketpy/environment/environment.py Resolved
Comment thread rocketpy/environment/environment.py Resolved
Comment thread rocketpy/environment/tools.py Resolved
Comment thread rocketpy/environment/environment.py Resolved
Comment thread rocketpy/environment/environment.py Resolved
@Gui-FernandesBR Gui-FernandesBR mentioned this pull request Nov 20, 2026
4 of 11 tasks
MateusStano and others added 2 commits November 20, 2026 11:12
- Windy without a model uses its default again.
- A forecast is identified by its model run and by the first and last
  times of the dataset, and a launch time outside them is never reused.
- The elevation given by the user is not part of the request.
- The attributes kept for reuse are listed, not found by comparing objects.
- A model run that cannot be read means the forecast is processed again.
- The wind heading and direction are rebuilt on a reuse, so they read the
  Environment's own data.
- The 100 most recent forecasts are kept.
- The reuse tests read small GFS-shaped netCDF files with the real
  processing.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MateusStano

Copy link
Copy Markdown
Member Author

Addressed all your comments in the last 2 commits @Gui-FernandesBR

Pylint counted too many statements in it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants