42 fit gp model on existing population from optimization problem

[flo-schu]

Implements a Surrogate class for fitting a GP on existing data from optimizations

implements

• fitting the GP on an existing population of samples
• estimating the objectives functions
• estimating the feasible objectives space
• conditioning the optimization_problem on fixed variables
• find x0 at the edges of the problem

plots

• conditional
• feasible space
• minimum boundary
• partial dependence
• overlay contour plot with true population

tests

• test if surrogate model breaks or not
• test convergence of surrogate model to known solutions

implementation of surrogates for different optimiztation tasks

⚡ priority

• linear inequality constraints
• linear equality constraints
• (pseudo) multi-objective,
• non-linear constraints

performance

• implement algorithm that finds a very good starting point for the task
• find a faster method to test if the conditioned problem contains feasible space (creating a hopsy problem needs a lot of time 1/3 of total runtime) This depends currently on hopsy

code quality

• Docstrings (GPT add docstrings, include warnings and errors, use numpy style, imperative style)
• refactor condition_optimization_problem method to a separate class that is an OptimizationProblem

open questions

• actually sampled points overlaid over mean + uncertainty of GP
  GP.predict(std=True) -> mu, sigma
• contours? Lösen über fill between mit verschiedenen quantilen

partial dependence

the function partial_dependence of sklearn returns a value that is not the actual response of the objective function but another value. This behavior is not ideal if it should be overlaid over the true objective space and the true evaluated points.

minimum boundary plot

This plot draws the minimum boundary of the objective function over a given parameter.
There are two main approaches to achieve this

1. drawn a large number of samples and iterate over a moving window that always returns the minimum point. This is a pragmatic approach but it will be inexact
2. create a linear space over the target parameter x that should be drawn. Then optimize wrt all other parameters and find minimum. Ideally a good starting point would be that point which is close to the ones that are already close to the minimum by weighting these can be points from the surrogate or the true model evaluation (but surrogate is probably smarter, because there will be more points available)

this problem is solved by conditioning the optimization problem on one variable. This is done by fixing the conditioning variable at a specific value and substituting that value in the matrix and moving it to the RHS (b).

closes additional issues

• closes Implement partial dependence plots #33

[flo-schu]

trying out new features

currently the fixtures for optimization problems are not on the dev branch. @schmoelder, maybe you can change this?

Until then, there is a "fix". I am playing around with the surrogate model in the paper repository: https://github.com/flo-schu/bayesian-optimization
There I have just copy pasted the fixtures and work on the plots. If you clone this repo, you can try out the surrogate features

[schmoelder]

currently the fixtures for optimization problems are not on the dev branch. @schmoelder, maybe you can change this?

Yes, I will try to move it to dev.
In the meantime, you could also try to base the branch off of the ax branch.

Edit: Moved fixtures into dev.

[flo-schu]

I've made a very first approach for integration of Surrogate in post_processing and OptimizationResults it works for U_NSGA but it is still very WIPy.

Let's talk it through tomorrow and see which additional features we want and what can be improved, refactored, etc. :)

[flo-schu]

not sure if the merge of dev into 42... was the right thing to do. Probably rebase would have been your preferred choice? If possible you can walk me through correcting this and doing it differently.

[flo-schu]

okay, I have cleaned up surrogate module and removed it from OptimizationResults and Optimizer. Next week I will continue developing it for further optimization tasks and write some tests