Skip to content

ENH: let a convergence study be reproduced from one seed - #1198

Open
thc1006 wants to merge 5 commits into
RocketPy-Team:developfrom
thc1006:enh/a-convergence-study-from-one-seed-v2
Open

thc1006 wants to merge 5 commits into
RocketPy-Team:developfrom
thc1006:enh/a-convergence-study-from-one-seed-v2

Conversation

@thc1006

@thc1006 thc1006 commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Addresses #1077.

While this was open, #1209 put the seed pass-through and its changelog entry on develop (9e3cc63). I merged develop and took its side of both conflicts, so what this PR adds now is the follow-up from review.

Pull request type

  • Code changes (bugfix, features)

Checklist

  • Tests for the changes have been added (if needed)
  • Docs have been reviewed and added / updated
  • Lint (ruff check / ruff format --check, pylint) has passed locally
  • All tests have passed locally
  • CHANGELOG.md: develop already has the entry for this PR.

Current behavior

simulate_convergence passes random_seed to every batch but bootstraps each confidence interval from SciPy's global generator. Two runs with the same seed can return different widths and, near the tolerance, stop after a different number of batches. A bad random_seed fails only inside the first batch, and not at all if the logs already hold max_simulations.

New behavior

The bootstrap is seeded from the study's seed, so a study run again with the same seed stops after the same batch. That includes a study that drew its own seed, run again with the one it printed. random_seed is checked with the other inputs, before any log is read, and the docstring says what the seed does not fix: where the study stops can depend on batch_size, and a finished study runs nothing.

Breaking change

  • No

Additional information

The tests sit in test_monte_carlo_release.py, beside the convergence tests develop added, and the cross-batch one there now compares whole input rows. Taking out the bootstrap seed, seeding it from the argument instead of the study, or dropping the up-front check each makes one of them fail.

Unit and integration tests pass on Python 3.10 and 3.14 with .[all], alone and merged with #1197. ruff 0.17.6 and pylint 4.1.4 are clean.

Two things I found along the way are filed separately as #1229 and #1230.

Each batch derived a root of its own, so a study could not be repeated and
the same max_simulations reached different samples per batch size.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006
thc1006 requested a review from a team as a code owner September 16, 2026 20:54
@codecov

codecov Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.55%. Comparing base (9bf63a0) to head (1176b3e).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1198   +/-   ##
========================================
  Coverage    86.55%   86.55%           
========================================
  Files          131      131           
  Lines        18248    18252    +4     
========================================
+ Hits         15794    15798    +4     
  Misses        2454     2454           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Gui-FernandesBR Gui-FernandesBR left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nine findings inline. The one on random_seed=random_seed affects what the feature promises; the rest are smaller.

include_function_data=False,
parallel=parallel,
n_workers=n_workers,
random_seed=random_seed,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The seed fixes the samples but not the convergence result. estimate_confidence_interval(...) a few lines below is called without random_state, so the bootstrap draws from scipy's global RNG: two runs with the same random_seed return different confidence_interval_history, and when a width lands near tolerance they can stop after a different number of batches.

Deriving one int from the study seed, on a stream the per-simulation children don't use, and passing it as random_state= would close it. The new tests use tolerance=1e-9, which never stops early, and don't compare the returned history; a test comparing the history at a tolerance the run actually crosses would catch this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The bootstrap now gets random_state=_bootstrap_seed(self.__study_seed), a 32-bit int from SeedSequence(study_seed, spawn_key=(0,)), which is not one of the [seed, index] sequences the simulations draw from. It has to be 32 bits: SciPy builds a RandomState from an int, and that stops at 2**32 - 1.

It comes from the seed the study ran with, not from the argument, so a study that drew its own seed can be repeated with the one it printed. Two new tests cover this, one at a tolerance the run crosses after its second batch. Without random_state, both fail.

Comment thread rocketpy/simulation/monte_carlo.py Outdated
Fixes the whole study rather than one batch. Every batch continues
the same root, so a given ``max_simulations`` reaches the same set
of samples whatever ``batch_size`` divided it up. Keyword-only.
Default is None, which draws fresh entropy and reproduces nothing.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rerunning a seeded study on the same filename returns [] without using random_seed. This predates the PR: simulate_convergence loads the existing outputs first, and if they already hold max_simulations rows the loop never runs, so even a different seed passes silently. Worth saying in this docstring that reproducing a study needs a fresh filename.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to Returns, with the note about a fresh filename.

On develop this is down to a finished study. One short of max_simulations that gets a different seed raises "Cannot append with a different root seed". If you'd rather the finished case raise too, I can add that here.

parallel=False,
n_workers=None,
*,
random_seed=None,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

random_seed isn't checked in the "Validate inputs up-front" block. A Generator, a negative int or a float only fails inside the first simulate() call, after import_outputs, and not at all when the loop is skipped. Calling _root_seed_sequence(random_seed) next to the other checks, plus a case in test_simulate_convergence_validates_inputs, would cover it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. On develop the check is _study_seed, now called with the other input checks, before import_outputs. A negative seed is a case in test_simulate_convergence_validates_inputs, and a separate test passes bad seeds to a study whose loop never runs.

Comment thread rocketpy/simulation/monte_carlo.py Outdated
Comment on lines +1090 to +1092
Fixes the whole study rather than one batch. Every batch continues
the same root, so a given ``max_simulations`` reaches the same set
of samples whatever ``batch_size`` divided it up. Keyword-only.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Same set of samples whatever batch_size" only holds if the study runs to max_simulations. The tolerance check runs at batch boundaries, so a study that converges early stops at a different count per batch_size: with a tolerance met after the first batch, batch_size=2 stops at 2 and batch_size=4 at 4.

Suggested change
Fixes the whole study rather than one batch. Every batch continues
the same root, so a given ``max_simulations`` reaches the same set
of samples whatever ``batch_size`` divided it up. Keyword-only.
Fixes the whole study rather than one batch. Every batch continues
the same root, so runs with the same seed draw a common prefix of
one stream of samples; where the study stops can still depend on
``batch_size``. Keyword-only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I took develop's docstring in the merge, which doesn't say this, and added your sentence to it.

def _drawn(analysis):
with open(analysis.input_file, "r", encoding="utf-8") as written:
rows = [json.loads(line) for line in written if line.strip()]
return {row["index"]: row.get("mass") for row in rows}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests only compare mass, which comes from the rocket's child stream, so a regression in the environment or flight draws still passes. Keying the dict by index also collapses duplicated rows, including in the sorted(...) == [0, 1, 2, 3] check on line 70. Comparing every input field, row by row as a list, would cover both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests moved into test_monte_carlo_release.py and compare whole input rows, in order. develop's cross-batch test there compared mass alone too, so it does the same now.

return stochastic_environment, stochastic_calisto, stochastic_flight


def _converge(analysis, batch_size, seed):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These three tests run 20 real Calisto flights with no @pytest.mark.slow, while the convergence tests in tests/integration/simulation/test_monte_carlo.py are marked slow. Marking them, or trimming the runs, keeps the default suite fast.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They run on release_study now, whose flights stop at one second, so each takes under half a second. I left them unmarked like the rest of that file.

from rocketpy.simulation.monte_carlo import MonteCarlo


def _study(tmp_path, stem, models):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_study, _drawn and the models fixture are copied verbatim from tests/unit/simulation/test_append_lineage.py, and models is also in test_monte_carlo_seeding.py. Moving them to a shared conftest or helper keeps the three copies from drifting when the row format changes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file is gone. The new tests use release_study and _rows from the file they live in now. develop's lineage tests no longer carry _study or _drawn, so the one copy left is models in test_monte_carlo_seeding.py.



def test_a_convergence_study_repeats_from_one_seed(models, tmp_path):
"""The same seed gives the study back, which it could not before."""

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This docstring and the comment on lines 52-53 describe the code before this PR, which won't exist after merge. The history fits the commit message; the test can just state the property.

Suggested change
"""The same seed gives the study back, which it could not before."""
"""The same seed gives back the same study."""

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed with it. The new docstrings state the property.

Comment thread CHANGELOG.md Outdated

### Added

- ENH: Reproduce a convergence study from one seed [#1077](https://github.com/RocketPy-Team/RocketPy/issues/1077)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The entry links only the issue. The header asks for the PR title and number, like the neighbouring entries.

Suggested change
- ENH: Reproduce a convergence study from one seed [#1077](https://github.com/RocketPy-Team/RocketPy/issues/1077)
- ENH: let a convergence study be reproduced from one seed [#1198](https://github.com/RocketPy-Team/RocketPy/pull/1198) [#1077](https://github.com/RocketPy-Team/RocketPy/issues/1077)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

develop already has an entry for this PR (9e3cc63, linking #1198 and #1209), so I kept it in the merge and dropped mine. I can add the seeded bootstrap to it if you want.

develop now carries the seed pass-through and its changelog entry
(9e3cc63, RocketPy-Team#1209), so both conflicts take develop's side.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
simulate_convergence handed its seed to every batch but bootstrapped each
confidence interval from SciPy's global generator, so the same seed gave a
different confidence_interval_history and, near the tolerance, a different
stopping batch. The bootstrap now takes a 32-bit seed spawned off the study
seed. That also covers a study that drew its own seed and is run again with
the one it recorded.

A random_seed of the wrong type or sign is now refused with the other inputs,
before the logs are read, including when they already hold every simulation
and nothing would run. The docstring says what the seed does not fix: where
the study stops can depend on batch_size, and a finished study runs nothing
whatever seed it is given.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The cross-batch test compared mass alone, which comes from the rocket's
stream, so a change in the environment or flight draws would still pass. It
now compares each input row in full and in order.

The three tests this branch added ran twenty Calisto flights, keyed the rows
they compared by index, and copied their helpers from other files. What they
checked is now covered next to the convergence tests develop already has, on
its light release_study fixture, so the file is gone.

Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
@thc1006
thc1006 force-pushed the enh/a-convergence-study-from-one-seed-v2 branch from 3f7b767 to 1176b3e Compare November 20, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants