Replace sage.libs.giac with new optional package sagemath-giac

[orlitzky]

As the last step in #38668, we move sage/libs/giac into a separate package,

• https://github.com/sagemath/sagemath-giac

and make it optional. This module is already mostly independent of sage -- there is nothing in sagelib that depends on it unconditionally. Its main use is as an integration backend, where it is technically optional because we try whatever backends are available in succession and expect some to fail.

In fact, the modules themselves are already optional if you are using meson. The sage/libs/giac directory is simply skipped when meson cannot find giac. It is not possible to do the same thing using the build system for sage-the-distro, however, which motivates the separate package. Many reasons are given in #38668 for why you might want to avoid giac, but the main reason for me right now is because it's not portable: it doesn't build and/or run on systems where the rest of sage works fine, including the one I am using most of the time.

The new package uses the namespace sagemath_giac to work around missing namespace support in cython, but sage.libs.giac re-exports everything under the old name for backwards compatibility.

[github-actions]

Documentation preview for this PR (built with commit d7a5f54; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[antonio-rojas]

In fact, the modules themselves are already optional if you are using meson. The sage/libs/giac directory is simply skipped when meson cannot find giac. It is not possible to do the same thing using the build system for sage-the-distro, however, which motivates the separate package.

How is giac integration different from bliss/meataxe/sirocco that makes it impossible for it to be optional while still living in the main sage source tree?

[orlitzky]

How is giac integration different from bliss/meataxe/sirocco that makes it impossible for it to be optional while still living in the main sage source tree?

It's not. Matthias has a sagemath_giac in his fork of Sage along those same lines, but however you go about it, it's a good amount of work. (You can't just add a --disable-giac flag that skips sage/libs/giac.)

What you'll find at https://github.com/orlitzky/sagemath-giac is a normal cython package. You can clone it, build it, test it, package it, etc. independent of sage-the-distro, using what you already know as a python developer. There's no magic, and even the doctests work with a simple python -m doctest or pytest. Considering that both approaches involve a significant amount of work, I find this to be a much friendlier design.

[antonio-rojas]

What you'll find at https://github.com/orlitzky/sagemath-giac is a normal cython package. You can clone it, build it, test it, package it, etc. independent of sage-the-distro, using what you already know as a python developer. There's no magic, and even the doctests work with a simple python -m doctest or pytest. Considering that both approaches involve a significant amount of work, I find this to be a much friendlier design.

I disagree - thinking outside the sage-the-distro framework I see a few disadvantages of this approach:

• This makes sagemath-giac a de-facto third-party package. When there are breaking changes in sagelib, you need to port stuff in two different places instead of having all code centralized in a single repo (adds burden to developers), and make sure everything is released in sync (adds burden to upstream maintainers) - we're seeing this now with the pari 2.17 upgrade and SnapPy.
• You can no longer build and package all components (standard and optional) from a single repo - you have to either combine several sources at build time or create separate packages (adds burden to downstream packagers).

Since, as you said, this works properly in meson and we're moving towards making meson the default buildsystem, won't this eventually fix itself?

[orlitzky]

• This makes sagemath-giac a de-facto third-party package. When there are breaking changes in sagelib, you need to port stuff in two different places instead of having all code centralized in a single repo (adds burden to developers), and make sure everything is released in sync (adds burden to upstream maintainers) - we're seeing this now with the pari 2.17 upgrade and SnapPy.
• You can no longer build and package all components (standard and optional) from a single repo - you have to either combine several sources at build time or create separate packages (adds burden to downstream packagers).

Fair concerns, but in this case I would not be too worried:

• These modules are very lightly integrated with sage and should not break when sage upgrades. If you look through the git log, you'll see a few module names changing in import statements, but nothing else. (And we should have a year to sync those while the old names are deprecated.)
• On the contrary, because it is largely independent of sage, it will be less work to package it once than it would be to package (the same exact code) multiple times with each release of sage, like we do with sagemath_bliss and friends. The most common changes are updates to the doctest output due to changes in giac, which become much easier to fix and release in a minor bump to sagemath-giac without having to wait 6 months for the next version of sage while your tests fail.
• It's becoming an optional package, so it requires no extra work. You can do nothing and upgrade never :)

Since, as you said, this works properly in meson and we're moving towards making meson the default buildsystem, won't this eventually fix itself?

Absolutely, this was a tempting approach. But giac is completely broken on both types of systems I have. It doesn't build on anything other than x86_64/arm64, and even on x86_64, it crashes under a hardened glibcxx. How long is eventually?

[orlitzky]

It is maybe worth mentioning that, once upon a time, this was already a separate package. It was bundled with sagelib when it was made a standard package to avoid a circular dependency sagelib -> sagemath_giac -> sagelib. But today, there is no sagelib -> sagemath_giac dependency. All of the integration backends are drop-ins and can fail without issue.

[tobiasdiez]

Can you explain the problem with the cython compilation in more detail please. You mentioned in the other PR:

Basically, you can't compile external cython packages against sage-the-distro because cython will use sys.path from the system python, and thus all of sage is hidden from it.

So, you have sage installed in venv A, but cython in venv B; and try to build sagemath-giac in venv A?

I think the reason why this is not working is because you don't declare sage as a dependency here:
https://github.com/orlitzky/sagemath-giac/blob/23d0d7d6073d9034d7fd87bd5aca0657c98cdd89/src/sagemath_giac/meson.build#L52
You probably need to add a similar construction for sage as you do for gmpy2.

To make this easier we may add a sage-config tool similar to what numpy did: numpy/numpy#25730

[orlitzky]

So, you have sage installed in venv A, but cython in venv B; and try to build sagemath-giac in venv A?

Almost. The system cython is being used to build sagemath-giac against sagelib installed in a venv (I am building sage-the-distro "normally" to make sure it works). Without the sage-cython wrapper, this is what happens:

[sagemath_giac-0.1.0] [spkg-install] [1/3] Compiling Cython source /home/mjo/src/sage.git/local/var/tmp/sage/build/sagemath_giac-0.1.0/src/src/sagemath_giac/giac.pyx
[sagemath_giac-0.1.0] [spkg-install] FAILED: src/sagemath_giac/giac.cpython-312-x86_64-linux-gnu.so.p/src/sagemath_giac/giac.pyx.cpp 
[sagemath_giac-0.1.0] [spkg-install] cython -M --fast-fail -3 --cplus /home/mjo/src/sage.git/local/var/tmp/sage/build/sagemath_giac-0.1.0/src/src/sagemath_giac/giac.pyx -o src/sagemath_giac/giac.cpython-312-x86_64-linux-gnu.so.p/src/sagemath_giac/giac.pyx.cpp
[sagemath_giac-0.1.0] [spkg-install] 
[sagemath_giac-0.1.0] [spkg-install] Error compiling Cython file:
[sagemath_giac-0.1.0] [spkg-install] ------------------------------------------------------------
[sagemath_giac-0.1.0] [spkg-install] ...
[sagemath_giac-0.1.0] [spkg-install] from sys import maxsize as Pymaxint, version_info as Pyversioninfo
[sagemath_giac-0.1.0] [spkg-install] import os
[sagemath_giac-0.1.0] [spkg-install] import math
[sagemath_giac-0.1.0] [spkg-install] 
[sagemath_giac-0.1.0] [spkg-install] # sage includes
[sagemath_giac-0.1.0] [spkg-install] from sage.ext.stdsage cimport PY_NEW
[sagemath_giac-0.1.0] [spkg-install] ^
[sagemath_giac-0.1.0] [spkg-install] ------------------------------------------------------------
[sagemath_giac-0.1.0] [spkg-install] 
[sagemath_giac-0.1.0] [spkg-install] /home/mjo/src/sage.git/local/var/tmp/sage/build/sagemath_giac-0.1.0/src/src/sagemath_giac/giac.pyx:144:0: 'sage/ext/stdsage.pxd' not found

That's a "cythonize" failure that occurs because the sage.* namespace is not visible to cython, which gets its search path from python's sys.path. Meson is run using the venv python where everything looks OK because the venv is magically added to its sys.path. But then the system cython is ultimately run (from its shebang) using /usr/bin/python, which does not include the venv in its sys.path. As a result, all cimport statements for modules in the venv fail.

I think the reason why this is not working is because you don't declare sage as a dependency here: https://github.com/orlitzky/sagemath-giac/blob/23d0d7d6073d9034d7fd87bd5aca0657c98cdd89/src/sagemath_giac/meson.build#L52 You probably need to add a similar construction for sage as you do for gmpy2.

If you look further down, I am including a similar thing for sage.cpython. This addresses the compilation stage (where some headers are required), but I don't think it affects the cythonize phase?

[tobiasdiez]

If the issue is at the cythonize stage, then adding something like this should work (with the path to the root of sage):

sage/src/meson.build

Lines 128 to 131 in dc99dc8

	# Meson currently ignores include_directories for Cython modules, so we
	# have to add them manually.
	# https://github.com/mesonbuild/meson/issues/9562
	add_project_arguments('-I', meson.current_source_dir(), language: 'cython')

[orlitzky]

If the issue is at the cythonize stage, then adding something like this should work (with the path to the root of sage):

Sure, but this is just one of potentially many external cython packages, and the issue only arises because sage-the-distro hides sagelib in a venv. Wouldn't it be better to address the problem once, in sage-the-distro?

sagemath-standard is declared as a dependency of sagemath-giac (in pyproject.toml), so it is expected to be available to python. As with any build system, you can put it in an unusual place, but then it is up to you to inform the build system where to find it (at build time) and eventually ensure that it is available at run time. I am thinking along the lines of -L/usr/local/lib (build time) and LD_LIBRARY_PATH=/usr/local/lib (run time).

At build time, sage-the-distro is using the venv python, which has all of the dependencies in its path. At run time, sage-the-distro uses the venv python for the same reason. It's only the call to the system cython with an un-fudged path (at build time) that is a problem. And IMO sage-the-distro should be responsible for ensuring that the cython it's using can find the dependencies of the packages it is installing. We could provide include directories by hand in sage-the-distro, but the simplest way I could think of was just to launch cython using the venv python which makes everything consistent.

[orlitzky]

Reviewers should take a look at https://github.com/sagemath/sagemath-giac too since obviously a big part of this was separating the package.

[orlitzky]

@dimpase I don't know why but you aren't in the reviewer dropdown

[dimpase]

I screamed too loudly on #39467.
Please get @tobiasdiez to review.

[orlitzky]

Since, as you said, this works properly in meson and we're moving towards making meson the default buildsystem, won't this eventually fix itself?

Thinking more about this recently, there is another reason why "wait for meson" isn't as nice of a solution. On Gentoo (and on sage-the-distro), sagelib usually "breaks" whenever giac is updated. And, to fix it, you have to rebuild all of sagelib. With sagemath-giac separate, you can instead do a 30s rebuild of the affected modules only.

[dimpase]

"wait for meson" is worse than doing a proper Pythonic modularisation, IMHO.

[tobiasdiez]

If the issue is at the cythonize stage, then adding something like this should work (with the path to the root of sage):

Sure, but this is just one of potentially many external cython packages, and the issue only arises because sage-the-distro hides sagelib in a venv. Wouldn't it be better to address the problem once, in sage-the-distro?

At build time, sage-the-distro is using the venv python, which has all of the dependencies in its path. At run time, sage-the-distro uses the venv python for the same reason. It's only the call to the system cython with an un-fudged path (at build time) that is a problem. And IMO sage-the-distro should be responsible for ensuring that the cython it's using can find the dependencies of the packages it is installing. We could provide include directories by hand in sage-the-distro, but the simplest way I could think of was just to launch cython using the venv python which makes everything consistent.

Sorry for the late reply. This makes a lot more sense now. So, I agree, having a "sage cython" in sage-the-distro as a wrapper around cython with the correct path/deps handling looks like a good solution. But this should then be a script specific to sage-the-distro, and not something that is installed as part of sagelib, right?

[orlitzky]

But this should then be a script specific to sage-the-distro, and not something that is installed as part of sagelib, right?

Right.

[orlitzky]

(The Gentoo CI failures related to giac are a pre-existing condition)

[kiwifb]

Regardless of modularisation, having to recompile sage to enable an option (in the sage packaging context - I am aware that if you are pottering in your local tree install, it is not the same) has always been a bit of a downer. Having it being a plug and play thing is much more effective and easy to turn on and off.
I do not care so much about the rest of modularisation, but I have been packaging sage for 15+ years now and separating optional stuff makes my life easier and align better with best practice in the python ecosystem where functionality is available at runtime if the right package is installed.

[orlitzky]

Gone again, no rebuild. And sagelib doesn't need an := dependency on giac any more.

for := to be useful, we would need giac to start using soname that are not 0.0.0.

We can set SLOT=0/${PV} on giac now to indicate that all upgrades potentially break ABI. In the past this would have been too painful, but now you can put the := dep in sagemath-giac (which rebuilds quickly) rather than in sagemath-standard.

[orlitzky]

Could you then please create this sage-cython wrapper in build and not reuse the one in src that is still shipped to all users? Thanks!

This one is not so easy because sage-cython needs to be inside the venv for it to magically pick up the venv python/sitedir when it is launched.

src/setup.cfg already contains one line,

# Only makes sense in sage-the-distribution. TODO: Move to another installation script.
bin/sage-list-packages

but I've no idea what such an installation script might look like.

[orlitzky]

Force push:

• Rebase onto develop
• Update sagemath-giac to v0.1.1
• Eliminate the sage-cython hack (we still need the hack, but it happens in meson now)

[tobiasdiez]

Thanks for all the work you put into this!

[orlitzky]

Thanks! I've updated the release tour, and with this, sage now runs on risc-v hardware out-of-the-box:

• https://github.com/sagemath/sage/wiki/Sage-10.6-Release-Tour

[user202729]

Looks like there's a little problem in terms of documentation:

• when the meson/conda environment is created, giac is still installed by default. (if sagemath_giac is not installed by default anyway, what is this for?)
• yet trying to do something like solve(…, algorithm="giac") will raise an unhelpful error message ModuleNotFoundError: No module named 'sagemath_giac'
• normally at least it should say Run "sage -i sagemath_giac" (if user uses the sage-distro)?
• there's no information whatsoever in https://doc.sagemath.org/html/en/reference//spkg/sagemath_giac.html that explains how to install the package for those using meson on conda/mamba.
• it's not that recommended to mix packages installed with conda and packages installed with pip as well.

So far the build/install instruction doesn't work out of the box even in an environment that I already install sage (with --editable)…

Error compiling Cython file:
------------------------------------------------------------
...
from sys import maxsize as Pymaxint, version_info as Pyversioninfo
import os
import math

# sage includes
from sage.ext.stdsage cimport PY_NEW
^
------------------------------------------------------------

...src/sagemath_giac/giac.pyx:144:0: 'sage/ext/stdsage.pxd' not found

In fact in editable mode some hacks are needed to inject the paths to sage headers in #39275

(is there no equivalent of maximal test on meson that this bug goes uncaught?)

I needed to apply the following patch

diff --git a/src/sagemath_giac/meson.build b/src/sagemath_giac/meson.build
index eb15018..f7ecf9b 100644
--- a/src/sagemath_giac/meson.build
+++ b/src/sagemath_giac/meson.build
@@ -57,7 +57,8 @@ cython_include_args = run_command(
     '-c',
     '''
 import sys
-for p in sys.path:
+import sage.env
+for p in sys.path + [sage.env.SAGE_SRC]:
     if p:
         print("--include-dir")
         print(p)

[orlitzky]

• yet trying to do something like solve(…, algorithm="giac") will raise an unhelpful error message ModuleNotFoundError: No module named 'sagemath_giac'
• normally at least it should say Run "sage -i sagemath_giac" (if user uses the sage-distro)?

These two at least are relatively easy to fix. The docs for solve() should mention that sagemath-giac (and not just giac) must be installed. A feature test before the attempted import would also provide a better error message.

[tobiasdiez]

Looks like there's a little problem in terms of documentation:

* when the meson/conda environment is created, `giac` is still installed by default. (if `sagemath_giac` is not installed by default anyway, what is this for?)

Should be fixed by #40001 (once the conda env lock files are updated). Please have a look.