Revive doc-release

[kwankyu]

On every release, "doc-release" (full sage documentation with live examples and pdf docs) had been published. That was broken by #40379.

The up-to-date doc-release was available through the Documentation section of the Wiki. Currently available doc-release was manually uploaded by me.

By the way, "doc-release" is also essential for "other version" selector (see below the Sage logo on the left panel) to work. It is currently broken. Note that "doc-release" is the source of "other versions" info. See https://doc-release--sagemath.netlify.app/html/en/versions.txt.

Now doc-release is built in the doc-build-pdf workflow, and published in the new doc-publish-pdf workflow, separated from the doc-publish workflow.

Test run:

doc-build-pdf: https://github.com/kwankyu/sage/actions/runs/17028284341

doc-publish-pdf: https://github.com/kwankyu/sage/actions/runs/16993049457

Deployed doc-release:

https://doc-release--sagemath-test.netlify.app/html/en/reference/

Try "Sage Live" tabs at https://doc-release--sagemath-test.netlify.app/html/en/reference/plot3d/sage/plot/plot3d/plot3d

Fixes #40463.

📝 Checklist

• The title is concise and informative.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation and checked the documentation preview.

⌛ Dependencies

[github-actions]

Documentation preview for this PR (built with commit bdf6897; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[user202729]

actually what is livedoc release? I mostly just use the Documentation preview link for development which still works… I think.

and what rule determines which user can deploy to which netlify URL?

[kwankyu]

actually what is livedoc release? I mostly just use the Documentation preview link for development which still works… I think.

See the updated PR description. The doc-release is published on every release (so about weekly).

and what rule determines which user can deploy to which netlify URL?

No user deploys. The netlify URL is determined by the sage repo secrets NETLIFY_SITE_ID and NETLIFY_AUTH_TOKEN which currently points to my netlify site.

[user202729]

You can add "Fixes #40463" to the PR description.

[kwankyu]

Thanks. A test running at my repo is not yet finished; hence "Draft" state yet.

[user202729]

Is there a published version of the built PDF documentation accessible somewhere? I can't find it.

[kwankyu]

The test is building pdfs. I will show you when it finished.

[user202729]

I assume meson builds things faster than make (but doc-build is not 100% meson either, it still uses bootstrap at some parts), it's unfortunate that meson live doc doesn't work. If I understood correctly the logic is moved to doc-build-pdf just because doc-build-pdf still uses make (while doc-build uses meson).

Still, working slow code is better than broken code.

(also, any idea which part of the code is responsible for deploying the documentation preview on each pull request, and why does that one actually work?)

[kwankyu]

I assume meson builds things faster than make (but doc-build is not 100% meson either, it still uses bootstrap at some parts), it's unfortunate that meson live doc doesn't work.

It does work. But doc-release (=live doc + pdf docs) needs pdf docs, for which meson does not work yet (right?).

If I understood correctly the logic is moved to doc-build-pdf just because doc-build-pdf still uses make (while doc-build uses meson).

Yes.

Still, working slow code is better than broken code.

For once a weak release, "slow" does not matter much.

(also, any idea which part of the code is responsible for deploying the documentation preview on each pull request, and why does that one actually work?)

See doc-publish.yml

[user202729]

But doc-release (=live doc + pdf docs) needs pdf docs, for which meson does not work yet (right?).

not really. sage --docbuild en/tutorial pdf works just fine in meson, provided you install the sage_docbuild package into the Python environment first. So is sage --docbuild reference/stats pdf.

sage --docbuild en/reference pdf, however, is currently broken… (with or without the en)

It does work.

I assume the original # The following fails randomly comments (that you deleted) means that it does not work. I haven't checked what's the error.

[tobiasdiez]

Thanks for fixing this!

A few initial questions/comments:

• What's the reason for moving the livedocs to the pdf run? They are just a different 'flavor' of the html build, right? Perhaps it's easier to just move the pdf build to the html (as a separate job) and then rename the workflow simply to "doc build".
• I would prefer if it uses meson directly, if that's now working.
• The released html seems to have broken styles, eg https://doc-release--sagemath.netlify.app/html/en/reference/rings_standard/ doesn't render nicely for me.
• Could you please also add this "doc-release" url somewhere in the documentation and readme, so that it's easier to find.

[user202729]

• What's the reason for moving the livedocs to the pdf run? They are just a different 'flavor' of the html build, right? Perhaps it's easier to just move the pdf build to the html (as a separate job) and then rename the workflow simply to "doc build".
• I would prefer if it uses meson directly, if that's now working.

I think this is already discussed above: none of us knows why livedoc on meson fails (yet), so a temporary workaround is to use it in the pdf, which is still based on make.

• The released html seems to have broken styles, eg https://doc-release--sagemath.netlify.app/html/en/reference/rings_standard/ doesn't render nicely for me.

probably just a mistake caused while doing the manual upload, if you change the sagemath to sagemath-test (automated upload) then it works well. In any case I don't see it as an evidence of the code being incorrect.

• Could you please also add this "doc-release" url somewhere in the documentation and readme, so that it's easier to find.

As mentioned in the first message, it's linked from the wiki. I agree it's hard to find though, having in the README would be good.

The README, however, links to https://doc.sagemath.org/html/en/index.html—which makes sense, since this is user-facing, and unreleased versions are not visible to most users. Maybe https://doc.sagemath.org/html/en/developer/index.html#section-updating-documentation would be suitable.

[kwankyu]

It does work.

I assume the original # The following fails randomly comments (that you deleted) means that it does not work. I haven't checked what's the error.

It works with "-j1", as you suggested as a fix.

[user202729]

Okay, so currently we have meson builds HTML doc, and make builds PDF doc.

Why do we need to build the HTML again in the "Build live doc" step, if meson can already build HTML with -j1 workaround?

[kwankyu]

• I would prefer if it uses meson directly, if that's now working.

I think this is already discussed above: none of us knows why livedoc on meson fails (yet), so a temporary workaround is to use it in the pdf, which is still based on make.

Right. Later if everything works well with meson, all could be combined into single "doc-build.yml" and "doc-publish.yml". However, since building pdf docs takes more time, having separate workflows might be still preferred.

• The released html seems to have broken styles, eg https://doc-release--sagemath.netlify.app/html/en/reference/rings_standard/ doesn't render nicely for me.

probably just a mistake caused while doing the manual upload, if you change the sagemath to sagemath-test (automated upload) then it works well. In any case I don't see it as an evidence of the code being incorrect.

Correct.

• Could you please also add this "doc-release" url somewhere in the documentation and readme, so that it's easier to find.

As mentioned in the first message, it's linked from the wiki. I agree it's hard to find though, having in the README would be good.

The README, however, links to https://doc.sagemath.org/html/en/index.html—which makes sense, since this is user-facing, and unreleased versions are not visible to most users.

Right. "doc-release" is for developers. That provides a sample to be eventually released for users.

On the other hand, https://doc.sagemath.org provides the official doc for stable releases for ordinary users. I hope that that also provides live doc...

Maybe https://doc.sagemath.org/html/en/developer/index.html#section-updating-documentation would be suitable.

I looked around. But I don't see a suitable place; the section is focused on local development.

Perhaps Wiki itself should be advertised, but isn't it wildly visible?

[kwankyu]

For your information, "doc-release" is also essential for "other version" selector (see below the Sage logo on the left panel) to work. It is currently broken.

"doc-release" is the source of "other versions" info. See https://doc-release--sagemath.netlify.app/html/en/versions.txt

[kwankyu]

I updated the Wiki to emphasize the doc-release.

[kwankyu]

Why do we need to build the HTML again in the "Build live doc" step, if meson can already build HTML with -j1 workaround?

"again" because it is "live doc". And in doc-build-pdf because doc-release should contain pdf docs.

An alternative approach is to build live doc with '-j1' in "doc-build" and publish doc-release with pdf docs built in "doc-build-pdf" (after wasting time waiting for pdf docs) But this approach would slow down doc previews for PRs.

[user202729]

"again" because it is "live doc"

Just to clarify, the point is the Sage Live tab on the documentation page right?

I've actually never heard of the feature before. It needs more advertisement, I suppose.

Perhaps Wiki itself should be advertised, but isn't it wildly visible?

I've never heard of the wiki before either.

I think one useful place to introduce it is in the message by github-actions itself. Currently it says

            [Documentation preview for this PR](${{ steps.deploy-netlify.outputs.NETLIFY_URL }}/html/en) (built with commit ${{ steps.source-run-info.outputs.sourceHeadSha }}; [changes](${{ steps.deploy-netlify.outputs.NETLIFY_URL }}/CHANGES.html)) is ready! :tada:
            This preview will update shortly after each push to this PR.

we could add a

This message is automatically generated by <something>.
Refer to https://doc.sagemath.org/html/en/developer/[...] for an explanation how it works.

[tobiasdiez]

An alternative approach is to build live doc with '-j1' in "doc-build" and publish doc-release with pdf docs built in "doc-build-pdf" (after wasting time waiting for pdf docs) But this approach would slow down doc previews for PRs.

I like this approach better. The live docs are only build on releases and take quite some time, right? So in this case, it would be okay to wait for the live docs + pdf build before publishing them together.

Btw, what is the difference between the doc-release and doc-develop netlify sites?

I would be fine changing the first sentence on https://doc-develop--sagemath.netlify.app/html/en/index.html to "Documentations in other languages and the most recent beta release are available too." and then link the doc-release. And a similar sentence can be added in the readme.

For your info, in #40597, I've now fixed the pdf-meson build and migrated the CI to it.

[kwankyu]

I've actually never heard of the feature before. It needs more advertisement, I suppose.

It was first advertised here: https://github.com/sagemath/sage/wiki/Sage-10.2-Release-Tour#documentation

... we could add a

This message is automatically generated by <something>.
Refer to https://doc.sagemath.org/html/en/developer/[...] for an explanation how it works.

I guess we are talking about doc-release and live doc; this thing is different from doc previews for PRs.

I think we may add a subsection about the automatic generation of various docs for developers available in github interface (perhaps also about the Wiki :-) in the section about the github repo: https://doc-release--sagemath.netlify.app/html/en/developer/github# However, then this issue is somewhat beyond the present PR. I may do this work in a new PR.

[kwankyu]

An alternative approach is to build live doc with '-j1' in "doc-build" and publish doc-release with pdf docs built in "doc-build-pdf" (after wasting time waiting for pdf docs) But this approach would slow down doc previews for PRs.

I like this approach better. The live docs are only build on releases and take quite some time, right? So in this case, it would be okay to wait for the live docs + pdf build before publishing them together.

I gave up that approach and took the present approach, because that approach would slow down generation of doc previews for PRs since the doc-publish action needs to wait for the completion of building pdf docs, which takes long.

Btw, what is the difference between the doc-release and doc-develop netlify sites?

doc-develop is used when doc preview diffs (changes) for a PR is generated, by comparing it with doc-pr-xxxxx.

I would be fine changing the first sentence on https://doc-develop--sagemath.netlify.app/html/en/index.html to "Documentations in other languages and the most recent beta release are available too." and then link the doc-release. And a similar sentence can be added in the readme.

I don't agree. That sentence is about the specific doc version. It is inappropriate to talk about other versions in the same sentence.

Anyway, let's deal with the advertisement issue in a new PR mentioned in the previous comment.

For your info, in #40597, I've now fixed the pdf-meson build and migrated the CI to it.

I don't know meson and how things are going about meson transition. Hopefully in a sequel to this PR, you may convert pdf doc build stuffs to meson and perhaps then merge doc-build.yml and doc-build-pdf.yml (or not because of the delayed doc previews issue). Let's get this in for now.

[tobiasdiez]

An alternative approach is to build live doc with '-j1' in "doc-build" and publish doc-release with pdf docs built in "doc-build-pdf" (after wasting time waiting for pdf docs) But this approach would slow down doc previews for PRs.

I like this approach better. The live docs are only build on releases and take quite some time, right? So in this case, it would be okay to wait for the live docs + pdf build before publishing them together.

I gave up that approach and took the present approach, because that approach would slow down generation of doc previews for PRs since the doc-publish action needs to wait for the completion of building pdf docs, which takes long.

I'm not sure I quite follow. The "-j1" approach would consist in just uncommenting the currently commented code and adding that not-parallel flag, right? I don't see how this is related to the pdf-doc building.

Let's figure out the best way to get the live docs functional using meson. In the best case, even with parallel jobs. If not, then I'm fine with another workaround. But using make in place of meson is just a step backwards and will brake sooner than later again.

[user202729]

Let's figure out the best way to get the live docs functional using meson. In the best case, even with parallel jobs. If not, then I'm fine with another workaround. But using make in place of meson is just a step backwards and will brake sooner than later again.

We have [no live doc < make live doc < meson live doc].

The current situation is no live doc, so make live doc is better than that.
If you want to improve from make live doc to meson live doc, you can take the step later.
This PR is opened before yours that migrates doc-build-pdf to meson, so I think it is reasonable to get it in before.

That said…

---

in my understanding, the ideal dependency is like

doc-html (currently on meson) ---------> deploy doc-html preview
                                 \
                                  \
doc-pdf (currently on make) -----------> build live doc -> deploy live doc

but the current one, somehow, have deploy preview depend on doc-pdf.

In order to separate this, one need to spin up a separate job.
While this doesn't necessarily slow down deploy doc-html preview, it runs more jobs, and need some intermediate steps e.g. zip up the content and send it to the other job.

At least that's my current understanding. Without actually trying to implement it, it's probably difficult to tell what's the bottleneck.

[kwankyu]

I'm not sure I quite follow. The "-j1" approach would consist in just uncommenting the currently commented code and adding that not-parallel flag, right? I don't see how this is related to the pdf-doc building.

Right.

But note that In the situation where doc-publish deals with both publishing doc previews for PRs and doc-release, waiting for doc-build-pdf to finish would slow down generation of doc previews for PRs, which do not need pdf docs.

Hence I separated out doc-publish-pdf, which only deals with doc-release. Then it is only natural that livedoc (which contains pdf docs) is generated in doc-build-pdf.

[kwankyu]

in my understanding, the ideal dependency is like

....

but the current one, somehow, have deploy preview depend on doc-pdf.

No. "deploy preview" (= doc preview for PRs in my terminology) does not depend on doc-build-pdf.

... By the way, If you somehow are confused by the pdf icons shown in doc previews (for example, https://doc-pr-40610--sagemath.netlify.app/html/en/), that is another thing broken by the recent upheaval in the documentation building process. (There are no pdf docs linked; The icons should not appear.)

The current approach is

doc-html (currently on meson) ---------> deploy doc-html preview                                  

doc-pdf (currently on make) -----------> build live doc -> deploy live doc

(no dependency)

... and the ideal approach (for meson enthusiasts :-) seems to be

doc-html (currently on meson) ---------> deploy doc-html preview                                  

doc-pdf (currently on meson) -----------> build live doc -> deploy live doc

(no dependency)

[kwankyu]

doc-build-pdf workflow is failing since 10.8.beta0:

https://github.com/sagemath/sage/actions/workflows/doc-build-pdf.yml

[user202729]

Wait, I don't understand. Why does build live doc require doc-pdf? Surely it should require doc-html instead (since it appears on html page)?

[kwankyu]

"building live doc" does not require pdfs. But, doc-release is published from live-doc artifact, which contains both htmls (live doc) and pdfs.

By the way, building live doc is a complete rebuilding and does not use the previously built htmls.

[kwankyu]

It seems that something happened with jupyter kernel "sagemath" in 10.8.beta0:

...
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] Extension error!
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] Versions
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] ========
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * Platform:         linux; (Linux-6.11.0-1018-azure-x86_64-with-glibc2.39)
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * Python version:   3.12.3 (CPython)
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * Sphinx version:   8.2.3
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * Docutils version: 0.21.2
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * Jinja2 version:   3.1.4
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * Pygments version: 2.18.0
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] Last Messages
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] =============
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ]     interfaces
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ]     introduction
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ]     latex
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] Loaded Extensions
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] =================
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * sphinx.ext.mathjax (8.2.3)
...
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * matplotlib.sphinxext.plot_directive (3.10.1)
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] * jupyter_sphinx (0.5.3)
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] Traceback
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ] =========
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ]       File "/sage/local/var/lib/sage/venv-python3.12/lib/python3.12/site-packages/jupyter_sphinx/utils.py", line 15, in blank_nb
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ]         raise ExtensionError("Unable to find kernel", orig_exc=e)
  [sagemath_doc_pdf-none] [spkg-install] [tutorial ]     sphinx.errors.ExtensionError: Unable to find kernel (exception: No such kernel named sagemath)