Updating Benchmark Sets

[AdityaPandeyCN]

Checklist

• Appropriate tests were added
• Any code changes were done in a way that does not break public API
• All documentation related to code changes were updated
• The new code follows the
  contributor guidelines, in particular the SciML Style Guide and
  COLPRAC.
• Any new documentation only uses public API

Additional context

This PR adds the recently implemented SciPy global optimization algorithms from OptimizationSciPy.jl (#927) to the black-box global optimizer benchmarks.

[ChrisRackauckas]

It's going to need to wait for the registration to finish JuliaRegistries/General#133534 JuliaRegistries/General#133535

Also could you add the new PyCMA one which just merged around the same time? SciML/Optimization.jl#933

[AdityaPandeyCN]

Sure

[ChrisRackauckas]

https://buildkite.com/julialang/scimlbenchmarks-dot-jl/builds/3473#0197b061-b0dd-4ffe-8fc5-2e2d200a57c8

@mxpoch it looks like PyCMA is giving a segfault and having some IO issues as part of the benchmark. Does PyCMA have to do IO in order to give the results? Seems like it could just all be memory?

[AdityaPandeyCN]

Hi @ChrisRackauckas Do you think we should run the Python-based optimizers sequentially to avoid the errors, this maybe happening because of multithreaded Python calls in Julia?

[mxpoch]

@ChrisRackauckas Yep the default behaviour of PyCMA is to store logs in an 'outcmaes' folder. If you pass verb_log=0 as an argument it shouldn't do any writes.

[AdityaPandeyCN]

I have made changes to run the SciPy optimizers sequentially, I have tried to run this on my local machine but this became overwhelming for my machine:-

1. In one of the iteration the sphere function test ran successfully and moved to the ellipsoidal but I interrupted as it ran for hours, the sphere test function was the one which corrupted so maybe running sequentially can fix this.
2. I scaled down the iteration and they ran correctly but I don't think this verifies anything? as the error happened at approx 60 billion iterations(https://buildkite.com/julialang/scimlbenchmarks-dot-jl/builds/3474/steps/canvas?sid=0197b0e1-97d8-43ef-866b-8264b670d97c)
   Any suggestion @ChrisRackauckas

[ChrisRackauckas]

as the error happened at approx 60 billion iterations

That seems like it would be over the limit? Is the maxiters not being passed on?

[AdityaPandeyCN]

Are we referring to the run_length here?, if so than yes seems like the SciPy optimizers are ignoring it and we have to set an explicit maxiters here.

[ChrisRackauckas]

Yes run_length is maxiters https://github.com/jonathanBieler/BlackBoxOptimizationBenchmarking.jl/blob/3026c725f4303f05a0a558386c485e689428bff2/src/benchmark.jl#L100

[ChrisRackauckas]

It looks like it is handled though: https://github.com/SciML/Optimization.jl/blob/master/lib/OptimizationSciPy/src/OptimizationSciPy.jl#L331.

It might be best to try and just isolate one problem

[AdityaPandeyCN]

I think I have found the root cause in my ScipyBasinhopping wrapper (inner optimiser runs un-bounded). I’ve dropped Basinhopping from the current benchmark and running this on my machine if this works well I will push it and fix the wrapper code.

[ChrisRackauckas]

oh nice! Does the nested optimizer need something like inner vs outer iterations?

[AdityaPandeyCN]

Basinhopping counts hops with niter (outer loop), then calls a local minimizer by default L-BFGS-B that has its own gradient iterations. We already set niter, but we also need to pass something like minimizer_kwargs[:options]["maxiter"] = … so each inner L-BFGS-B run is limited, otherwise a single hop can spin for billions.
This is what I am understanding as of now for this

[AdityaPandeyCN]

Hello @ChrisRackauckas I commented out two of the global optimizers the ScipyBasinhopping( reason we are discussing here) and ScipyDualAnnealing(this was really slow on my computer) I ran the benchmark and this is the result https://github.com/AdityaPandeyCN/juliabenchmark/blob/main/GlobalOptimization/blackbox_global_optimizers.md

Please have a look, also note that the DualAnnealling one was really slow it didn't crash so commented it out... it ran till here
┌ Info: Chain(ScipyDualAnnealing → NelderMead) └ Sphere
┌ Info: Chain(ScipyDualAnnealing → NelderMead) └ Ellipsoidal
I pressed ctrl c after this because this was taking really long time.