Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some love to model comparison #315

Open
wants to merge 6 commits into
base: dev
Choose a base branch
from

Conversation

Kucharssim
Copy link
Collaborator

Model comparison still had some rough edges, so I am taking the liberty to make it smoother.

  • Implement a .predict method for the model comparison approximator
  • Fixes to the mc_confusion_matrix_plot
  • Changed some behavior of the model comparison simulator, which makes it much more useable especially in the case when we want to use mixed batches
  • Added a simple example notebook

@codecov-commenter
Copy link

codecov-commenter commented Feb 13, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 12.50000% with 42 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...low/approximators/model_comparison_approximator.py 13.63% 19 Missing ⚠️
bayesflow/simulators/model_comparison_simulator.py 13.63% 19 Missing ⚠️
bayesflow/diagnostics/plots/mc_confusion_matrix.py 0.00% 4 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Files with missing lines Coverage Δ
bayesflow/diagnostics/plots/mc_calibration.py 22.72% <ø> (ø)
bayesflow/diagnostics/plots/mc_confusion_matrix.py 22.50% <0.00%> (-1.19%) ⬇️
...low/approximators/model_comparison_approximator.py 23.89% <13.63%> (-2.48%) ⬇️
bayesflow/simulators/model_comparison_simulator.py 24.07% <13.63%> (-1.51%) ⬇️

@paul-buerkner
Copy link
Contributor

Thank you! Looks good to me. Can you sync this branch with #310 where I made some general interface improvements to the diagnostic functions? Once #310 is merged, I will merge your PR too.

@paul-buerkner paul-buerkner added this to the BayesFlow 2.0 milestone Feb 14, 2025
@Kucharssim
Copy link
Collaborator Author

Once #310 is merged, I will merge your PR too.

Sure, I can wait until #310 is merged, it'd easier for me to then get up to date with the dev branch.

- individual models can rely on a shared simulations (makes mixed batches much more useable)
- mixed batches are sampled more efficiently (batched samples per model, rather than batching individual simulations)
@Kucharssim Kucharssim force-pushed the model-comparison-improvements branch from fb49715 to 3a4c422 Compare February 14, 2025 14:46
@Kucharssim
Copy link
Collaborator Author

it seems the linter again failed for something not in this PR...

@stefanradev93
Copy link
Contributor

That's on me. It's fixed on dev now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants