Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model Comparison #184

Merged
merged 75 commits into from
Aug 14, 2024
Merged

Model Comparison #184

merged 75 commits into from
Aug 14, 2024

Conversation

LarsKue
Copy link
Contributor

@LarsKue LarsKue commented Aug 9, 2024

This branch implements:

  1. A better way to implement new architectures as compared to streamlined-backend
  2. An architecture for Model Comparison
  3. Modularized Data Adapters (previously Configurators)
    3.1. These are now moved into the datasets, allowing this code to be run in a worker process
  4. Fully CPU simulation with numpy, improving compatibility of GPU setups with multiprocessing (to be fixed upstream by keras)
  5. Easier creation of online datasets and some smart defaults for training parameters (e.g. batch size)
  6. Improved logging utilities and some general clean-ups

There are some drawbacks:

  1. Since data adapters are now generalized instead of specific to each architecture, there is no longer a way to override individual variable configuration
  2. Data adapters must be explicitly created by the user (but this is a simple one-liner)
  3. The sample and log_prob methods are not yet finished (this is WIP)

This got much bigger than I had anticipated originally, so please just take a general look in your reviews and we can set a meeting to discuss in more detail 😄

LarsKue added 30 commits July 17, 2024 10:55
adds workflows, restructures workflow inheritance, adds smart defaults for training parameters
also add preliminary batch_size auto search
- no longer accidentally usable as a class or constructor
- now correctly recognizes subclasses of backend-specific Tensors as valid types (e.g. torch.IntTensor)
- cleaned up namespace in bayesflow.types
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@codecov-commenter
Copy link

codecov-commenter commented Aug 9, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 57.69231% with 385 lines in your changes missing coverage. Please review.

Please upload report for BASE (streamlined-backend@b4afea0). Learn more about missing BASE report.

Files Patch % Lines
bayesflow/utils/optimal_transport/sinkhorn.py 16.90% 59 Missing ⚠️
...low/approximators/model_comparison_approximator.py 30.37% 55 Missing ⚠️
bayesflow/approximators/continuous_approximator.py 60.97% 32 Missing ⚠️
bayesflow/simulators/model_comparison_simulator.py 21.95% 32 Missing ⚠️
bayesflow/utils/dict_utils.py 22.85% 27 Missing ⚠️
bayesflow/utils/io.py 6.89% 27 Missing ⚠️
bayesflow/approximators/approximator.py 59.52% 17 Missing ⚠️
bayesflow/simulators/simulator.py 25.00% 15 Missing ⚠️
...esflow/data_adapters/flow_matching_data_adapter.py 46.15% 14 Missing ⚠️
bayesflow/datasets/rounds_dataset.py 23.52% 13 Missing ⚠️
... and 22 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@                  Coverage Diff                   @@
##             streamlined-backend     #184   +/-   ##
======================================================
  Coverage                       ?   59.32%           
======================================================
  Files                          ?      117           
  Lines                          ?     2532           
  Branches                       ?        0           
======================================================
  Hits                           ?     1502           
  Misses                         ?     1030           
  Partials                       ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@marvinschmitt marvinschmitt removed their request for review August 9, 2024 15:59
@LarsKue LarsKue marked this pull request as ready for review August 14, 2024 16:12
@LarsKue LarsKue merged commit 7c66f97 into streamlined-backend Aug 14, 2024
1 of 13 checks passed
@LarsKue LarsKue deleted the model-comparison branch August 14, 2024 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants