Include source code for CBC libraries

[pmla]

The CBC libraries in the git repository are pre-built. This makes it hard to debug any issues occuring within the library since we don't have the source.

It also makes the git repository very large (it is currently over 200mb). Git is designed with text in mind. It will store a diff of text files, but for binary files it stores a copy of every version. Here is some more info.

Fortunately it is usually not too difficult to build a shared library from source. It can be specified in the setup.py file. If you put the CBC source files somewhere I can try to do it.

[jGaboardi]

@pmla, would this also make #56 more manageable?

[h-g-s]

Hi Peter, This is an interesting discussion.  Right now we provide binaries so that there is no need of compiling anything when installing Python-MIP. I had some bad experiences where numpy would not compile on my machine depending on the versions of other packages that I have installed (such as blas and lapack if I remember correctly). Building is easy, but building it statically, so that no additional dependency is need is complicated. The build script now is in scripts/buildCBCLinux.sh I'm open to discussions/proposals. Cheers Em 3/13/20 12:08 PM, Peter Larsen escreveu:

The CBC libraries in the git repository are pre-built. This makes it hard to debug any issues occuring within the library since we don't have the source. It also makes the git repository very large (it is currently over 200mb). Git is designed with text in mind. It will store a diff of text files, but for binary files it stores a copy of every version. Here is some more info. ***@***.***/why-you-should-not-push-binary-files-to-git-and-what-to-do-if-you-do-29f34692cef0> Fortunately it is usually not too difficult to build a shared library from source. It can be specified in the |setup.py| file. If you put the CBC source files somewhere I can try to do it. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#70>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4VZOVLPBJGTFJLGSK43BLRHJEAHANCNFSM4LHEPHVA>.

-- ============================================================= Haroldo Gambini Santos Computing Department Universidade Federal de Ouro Preto - UFOP email: haroldo@ufop.edu.br home/research page: www.decom.ufop.br/haroldo It has long been an axiom of mine that the little things are infinitely the most important. -- Sir Arthur Conan Doyle, "A Case of Identity"

[pmla]

would this also make #56 more manageable?

@jGaboardi, perhaps! I don't know much about conda though.

[pmla]

@h-g-s OK, thanks. It looks quite complex :) I will have a poke around.

[tkralphs]

I just want to mention for completeness that Cbc is open source and hosted here in the COIN-OR Github organization. The topic of building Cbc from source, especially for the purpose of using it with Python modules, is well-explored territory. A lot of effort has been put into making Cbc as easy to build on the major platforms as possible and there are multiple Python packages that wrap it. Building it from source is a bit complex, since it its comprised of multiple independent packages, each of which has to be built separately. However, there is a build script coinbrew which automates the process and makes building easy.

There are also already multiple conda recipes (see here and particularly here), as well as Linux packages and homebrew recipes for OS X. If you want to improve the situation on your favorite platform, the best idea would be to contribute to what already exists rather than starting something new.

As for binaries versus building from source, it seems like a good idea to make it easy for most users to get Cbc in binary form, as most people will not end up wanting to debug it and anyone who wants to build from source and debug can easily do so. But it is indeed not a good idea to host binaries in git. It seems better for the setup script to just download the binaries from bintray or for the user to be asked to install Cbc themselves.

[pgoelz]

To add to the conversation, we have been chasing a bug where CBC silently crashes and takes the entire program with it. The bug only appears on Windows and is not triggered by any specific ILP. Our usage case optimizes a fixed sequence of ILPs and will crash at a different ILP each time (I have verified that the ILPs are indeed the same). The cause seems to be an access violation within the call to cbclib.Cbc_solve() in cbc-c-windows-x86-64.dll, but I don’t know how to debug further.
I was hoping to recompile CBC with debug flags, but coinbrew fails with checking how to get verbose linking output from ... configure: WARNING: compilation failed. I also tried using your compilation scripts (./downloadCBC.sh, then ./buildCBCWin.sh, on a linux machine), but got the following error output:

...
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/ec2-user/python-mip/scripts/Cbc':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check 
See `config.log' for more details
make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target `install'.  Stop.

I’m attaching config.log. Am I using the scripts wrong? So far, I have not modified any flags.

[esieke]

Hello Everyone,
in a project i try to use MIP on an aarch64 platform. That means that I have to compile the CBC package each time. The biggest disadvantage for me is that MIP isn't related to an official CBC release. If I use downloadCBC.sh, the code base can change from day to day. Or did I miss something?

Suggestion:
Use cowbrew in downloadCBC.sh to fetch the source code related to a stable CBC release.
e.g. coinbrew fetch Cbc:stable/2.10

UPDATE
Sorry I didn't see issue #35! The questions are more or less the same!

[pmla]

Thanks for all the feedback. A quick summary:

1. We are duplicating the efforts of the Cbc project by building our own binaries.
2. We do not support all platforms.
3. The binaries are not tied to an official release. If a user does some debugging and finds a problem, it is quite likely that we all have different Cbc versions. See issue Coin versions #35.
4. The git repository is very large because git does not handle binary files well, and the binaries are replaced frequently.
5. The binaries are difficult to debug.

One solution would be to fetch the binaries from bintray as @tkralphs suggests. Unfortunately PyPI / pip no longer allows external files to be downloaded during installation, as this feature was often misused by malicious packages.

Alternatively we could replace the binaries in python-mip/mip/libraries/ with the binaries built by the Cbc project (for all platforms). This would solve issues 1-3. Since the official releases do not come too frequently this would also help with issue 4. Issue #35 is the obstacle to this.

We could also consider building everything on the user's machine, by switching from setuptools to scikit-build. This summary of distribution tools is helpful. This proposal is probably unworkable, as building on Windows is very difficult.

I think solving #35 is the way to go. Opinions?

[tkralphs]

The biggest problem with tying binaries to a release is just that it is only recently, with the new version of the BuildTools, that we can easily build DLLs with the Visual Studio compiler on Windows with coinbrew in an automated fashion. This can't really be done easily with any existing releases. We're working on getting a new stable version and new releases out from the current master, but there is still some work to be done there. In the meantime, we can at least tie releases to a particular SHA. Once we are rolling with releases from a new stable version based on the current master version and things stabilize, then I don't think there will be much question but that just installing the latest release will be the right thing to do. Users shouldn't be doing anything with master in normal times. So we need a bit of patience to pull this all together.

There is a solution for one part that is mentioned in #35---building a single DLL using coinbrew---but I'm not sure we quite got all the pieces of the puzzle in place. @h-g-s did you ever get all the changes committed for this? It seemed like all that was needed was to remove some code from header files.

The best solution overall is probably going to be different on different platforms.

• On OS X, homebrew is widely used, easy, and builds on the user's machine.
• On Linux, building is easy and most users on Linux are comfortable with building from source. On the other hand, there are also packages for Cbc on major Linux platforms and if we keep those up-to-date, Linux won't be an issue. Currently, however, the Cbc package on Debian is orphaned. I don't know the status of the Fedora package. I have some volunteers to help with that, but this needs to be organized, and more help is always welcome.
• For Windows, the best solutions seem to be conda or a GUI installer. As I pointed out, there are many conda recipes for Cbc, but I don't know how many work on Windows. Python users on Windows seem to be comfortable with conda. On the other hand, a nice GUI installer would also solve the problem on Windows. I had a nice cross-platform installer for Windows, but the open source project I was using to build the installer was abandoned and I got distracted from the effort by other things (like getting the building of DLLs working). I don't think it would be difficult to make a decent installer. If nothing else, a Powershell script that downloads from bintray and sets up some environment variables would suffice.

I think there is a path to make all of this work smoothly on all platforms and we've already been slowly working towards that in support of the many projects that use Cbc. If anyone wants to contribute to filling in any of the missing pieces, that would be great. Here are things that could help.

• Help with making the process of building a single DLL using coinbrew as smooth as possible. I think this could even result in an easy way of building on Windows from a single batch script eventually.
• Help testing various conda recipes to see if there are any that work well on Windows and could make a good solution.
• Help developing a GUI installer for Windows.
• Help with maintaining Homebrew recipes on OS X.
• Help with maintaining Linux packages.

Let me know if you want to contribute! I guess I should advertise some of this to the Cbc community at large as well.

[tkralphs]

@pgoelz Please open a separate issue and provide more details. What compiler are you trying to build with?

[tuliotoffolo]

Hi @pmla, just wanted to mention that we want and eventually will remove the binaries from the repo. However, we must first find the best way of adding them to the pip package, since we must keep Python-MIP straightforward to install in most windows, linux and mac machines. We are currently looking into that.