Repository navigation
Conversation
Each batch derived a root of its own, so a study could not be repeated and the same max_simulations reached different samples per batch size. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1198 +/- ##
========================================
Coverage 86.55% 86.55%
========================================
Files 131 131
Lines 18248 18252 +4
========================================
+ Hits 15794 15798 +4
Misses 2454 2454 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Gui-FernandesBR
left a comment
There was a problem hiding this comment.
Nine findings inline. The one on random_seed=random_seed affects what the feature promises; the rest are smaller.
| include_function_data=False, | ||
| parallel=parallel, | ||
| n_workers=n_workers, | ||
| random_seed=random_seed, |
There was a problem hiding this comment.
The seed fixes the samples but not the convergence result. estimate_confidence_interval(...) a few lines below is called without random_state, so the bootstrap draws from scipy's global RNG: two runs with the same random_seed return different confidence_interval_history, and when a width lands near tolerance they can stop after a different number of batches.
Deriving one int from the study seed, on a stream the per-simulation children don't use, and passing it as random_state= would close it. The new tests use tolerance=1e-9, which never stops early, and don't compare the returned history; a test comparing the history at a tolerance the run actually crosses would catch this.
There was a problem hiding this comment.
Fixed. The bootstrap now gets random_state=_bootstrap_seed(self.__study_seed), a 32-bit int from SeedSequence(study_seed, spawn_key=(0,)), which is not one of the [seed, index] sequences the simulations draw from. It has to be 32 bits: SciPy builds a RandomState from an int, and that stops at 2**32 - 1.
It comes from the seed the study ran with, not from the argument, so a study that drew its own seed can be repeated with the one it printed. Two new tests cover this, one at a tolerance the run crosses after its second batch. Without random_state, both fail.
| Fixes the whole study rather than one batch. Every batch continues | ||
| the same root, so a given ``max_simulations`` reaches the same set | ||
| of samples whatever ``batch_size`` divided it up. Keyword-only. | ||
| Default is None, which draws fresh entropy and reproduces nothing. |
There was a problem hiding this comment.
Rerunning a seeded study on the same filename returns [] without using random_seed. This predates the PR: simulate_convergence loads the existing outputs first, and if they already hold max_simulations rows the loop never runs, so even a different seed passes silently. Worth saying in this docstring that reproducing a study needs a fresh filename.
There was a problem hiding this comment.
Added to Returns, with the note about a fresh filename.
On develop this is down to a finished study. One short of max_simulations that gets a different seed raises "Cannot append with a different root seed". If you'd rather the finished case raise too, I can add that here.
| parallel=False, | ||
| n_workers=None, | ||
| *, | ||
| random_seed=None, |
There was a problem hiding this comment.
random_seed isn't checked in the "Validate inputs up-front" block. A Generator, a negative int or a float only fails inside the first simulate() call, after import_outputs, and not at all when the loop is skipped. Calling _root_seed_sequence(random_seed) next to the other checks, plus a case in test_simulate_convergence_validates_inputs, would cover it.
There was a problem hiding this comment.
Done. On develop the check is _study_seed, now called with the other input checks, before import_outputs. A negative seed is a case in test_simulate_convergence_validates_inputs, and a separate test passes bad seeds to a study whose loop never runs.
| Fixes the whole study rather than one batch. Every batch continues | ||
| the same root, so a given ``max_simulations`` reaches the same set | ||
| of samples whatever ``batch_size`` divided it up. Keyword-only. |
There was a problem hiding this comment.
"Same set of samples whatever batch_size" only holds if the study runs to max_simulations. The tolerance check runs at batch boundaries, so a study that converges early stops at a different count per batch_size: with a tolerance met after the first batch, batch_size=2 stops at 2 and batch_size=4 at 4.
| Fixes the whole study rather than one batch. Every batch continues | |
| the same root, so a given ``max_simulations`` reaches the same set | |
| of samples whatever ``batch_size`` divided it up. Keyword-only. | |
| Fixes the whole study rather than one batch. Every batch continues | |
| the same root, so runs with the same seed draw a common prefix of | |
| one stream of samples; where the study stops can still depend on | |
| ``batch_size``. Keyword-only. |
There was a problem hiding this comment.
Agreed. I took develop's docstring in the merge, which doesn't say this, and added your sentence to it.
| def _drawn(analysis): | ||
| with open(analysis.input_file, "r", encoding="utf-8") as written: | ||
| rows = [json.loads(line) for line in written if line.strip()] | ||
| return {row["index"]: row.get("mass") for row in rows} |
There was a problem hiding this comment.
These tests only compare mass, which comes from the rocket's child stream, so a regression in the environment or flight draws still passes. Keying the dict by index also collapses duplicated rows, including in the sorted(...) == [0, 1, 2, 3] check on line 70. Comparing every input field, row by row as a list, would cover both.
There was a problem hiding this comment.
The tests moved into test_monte_carlo_release.py and compare whole input rows, in order. develop's cross-batch test there compared mass alone too, so it does the same now.
| return stochastic_environment, stochastic_calisto, stochastic_flight | ||
|
|
||
|
|
||
| def _converge(analysis, batch_size, seed): |
There was a problem hiding this comment.
These three tests run 20 real Calisto flights with no @pytest.mark.slow, while the convergence tests in tests/integration/simulation/test_monte_carlo.py are marked slow. Marking them, or trimming the runs, keeps the default suite fast.
There was a problem hiding this comment.
They run on release_study now, whose flights stop at one second, so each takes under half a second. I left them unmarked like the rest of that file.
| from rocketpy.simulation.monte_carlo import MonteCarlo | ||
|
|
||
|
|
||
| def _study(tmp_path, stem, models): |
There was a problem hiding this comment.
_study, _drawn and the models fixture are copied verbatim from tests/unit/simulation/test_append_lineage.py, and models is also in test_monte_carlo_seeding.py. Moving them to a shared conftest or helper keeps the three copies from drifting when the row format changes.
There was a problem hiding this comment.
The file is gone. The new tests use release_study and _rows from the file they live in now. develop's lineage tests no longer carry _study or _drawn, so the one copy left is models in test_monte_carlo_seeding.py.
|
|
||
|
|
||
| def test_a_convergence_study_repeats_from_one_seed(models, tmp_path): | ||
| """The same seed gives the study back, which it could not before.""" |
There was a problem hiding this comment.
This docstring and the comment on lines 52-53 describe the code before this PR, which won't exist after merge. The history fits the commit message; the test can just state the property.
| """The same seed gives the study back, which it could not before.""" | |
| """The same seed gives back the same study.""" |
There was a problem hiding this comment.
Removed with it. The new docstrings state the property.
|
|
||
| ### Added | ||
|
|
||
| - ENH: Reproduce a convergence study from one seed [#1077](https://github.com/RocketPy-Team/RocketPy/issues/1077) |
There was a problem hiding this comment.
The entry links only the issue. The header asks for the PR title and number, like the neighbouring entries.
| - ENH: Reproduce a convergence study from one seed [#1077](https://github.com/RocketPy-Team/RocketPy/issues/1077) | |
| - ENH: let a convergence study be reproduced from one seed [#1198](https://github.com/RocketPy-Team/RocketPy/pull/1198) [#1077](https://github.com/RocketPy-Team/RocketPy/issues/1077) |
develop now carries the seed pass-through and its changelog entry (9e3cc63, RocketPy-Team#1209), so both conflicts take develop's side. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
simulate_convergence handed its seed to every batch but bootstrapped each confidence interval from SciPy's global generator, so the same seed gave a different confidence_interval_history and, near the tolerance, a different stopping batch. The bootstrap now takes a 32-bit seed spawned off the study seed. That also covers a study that drew its own seed and is run again with the one it recorded. A random_seed of the wrong type or sign is now refused with the other inputs, before the logs are read, including when they already hold every simulation and nothing would run. The docstring says what the seed does not fix: where the study stops can depend on batch_size, and a finished study runs nothing whatever seed it is given. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The cross-batch test compared mass alone, which comes from the rocket's stream, so a change in the environment or flight draws would still pass. It now compares each input row in full and in order. The three tests this branch added ran twenty Calisto flights, keyed the rows they compared by index, and copied their helpers from other files. What they checked is now covered next to the convergence tests develop already has, on its light release_study fixture, so the file is gone. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
3f7b767 to
1176b3e
Compare
Addresses #1077.
While this was open, #1209 put the seed pass-through and its changelog entry on develop (9e3cc63). I merged develop and took its side of both conflicts, so what this PR adds now is the follow-up from review.
Pull request type
Checklist
ruff check/ruff format --check,pylint) has passed locallyCHANGELOG.md: develop already has the entry for this PR.Current behavior
simulate_convergencepassesrandom_seedto every batch but bootstraps each confidence interval from SciPy's global generator. Two runs with the same seed can return different widths and, near the tolerance, stop after a different number of batches. A badrandom_seedfails only inside the first batch, and not at all if the logs already holdmax_simulations.New behavior
The bootstrap is seeded from the study's seed, so a study run again with the same seed stops after the same batch. That includes a study that drew its own seed, run again with the one it printed.
random_seedis checked with the other inputs, before any log is read, and the docstring says what the seed does not fix: where the study stops can depend onbatch_size, and a finished study runs nothing.Breaking change
Additional information
The tests sit in
test_monte_carlo_release.py, beside the convergence tests develop added, and the cross-batch one there now compares whole input rows. Taking out the bootstrap seed, seeding it from the argument instead of the study, or dropping the up-front check each makes one of them fail.Unit and integration tests pass on Python 3.10 and 3.14 with
.[all], alone and merged with #1197. ruff 0.17.6 and pylint 4.1.4 are clean.Two things I found along the way are filed separately as #1229 and #1230.