[Experimental] Build the lectures with QuantEcon/mystmd instead of jupyter-book #1538
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Build Project [using QuantEcon/mystmd] | |
| on: [pull_request] | |
| jobs: | |
| preview: | |
| runs-on: "runs-on=${{ github.run_id }}/family=g4dn.2xlarge/image=quantecon_ubuntu2404/volume=80gb/spot=false" | |
| steps: | |
| - uses: actions/checkout@v7 | |
| # The key hashes everything cached execution outputs depend on: the | |
| # lectures themselves, the Python requirements, and this workflow file | |
| # (which pins the mystmd engine SHA, the JAX version and the XLA env). | |
| # Hashing only the lectures is how the b9cc649 engine swap shipped | |
| # green without ever executing a notebook: the cache replayed outputs | |
| # from the old engine. Any engine/env/requirements change now forces a | |
| # cold build automatically. | |
| - name: Cache Notebook Execution | |
| uses: actions/cache@v6 | |
| id: cache-execution | |
| with: | |
| path: ./lectures/_build/execute | |
| key: ${{ runner.os }}-execute-cache-${{ hashFiles('lectures/**/*.md', 'myst_requirements.txt', '.github/workflows/ci.yml') }} | |
| # Node 24 matches the theme's .nvmrc (quantecon-theme.mystmd), which runs | |
| # as a Remix server during `myst build --html`. | |
| - uses: actions/setup-node@v7 | |
| with: | |
| node-version: '24' | |
| - uses: oven-sh/setup-bun@v2 | |
| - name: Install Python | |
| if: steps.cache-execution.outputs.cache-hit != 'true' | |
| uses: actions/setup-python@v7 | |
| with: | |
| python-version: '3.12' | |
| cache: 'pip' | |
| cache-dependency-path: 'myst_requirements.txt' | |
| - name: Install execution requirements | |
| if: steps.cache-execution.outputs.cache-hit != 'true' | |
| run: python -m pip install -r myst_requirements.txt | |
| # The GPU lectures (jax_intro, autodiff, numpy_vs_numba_vs_jax) need JAX | |
| # on the g4dn runner. Without it those notebooks execute to | |
| # ModuleNotFoundError cells. Pinned to the version validated on | |
| # 2026-08-03; unpinned, the executed environment drifts between runs of | |
| # the same commit. Bumping it here busts the execution cache (the key | |
| # hashes this file), so the bump is validated by a cold build. | |
| - name: Install JAX | |
| if: steps.cache-execution.outputs.cache-hit != 'true' | |
| run: | | |
| pip install "jax[cuda13]==0.11.0" | |
| python scripts/test-jax-install.py | |
| # Pinned to the exact SHA of the fork's main (unmoved since 2026-06-12, | |
| # so this is the same engine the 2026-08-03 cold-run failures used). | |
| # A branch name here makes the engine a moving target between runs of | |
| # the same lecture commit, which makes failures impossible to bisect. | |
| - name: Install mystmd (QuantEcon fork) | |
| run: | | |
| mkdir -p /tmp/qe-mystmd | |
| cd /tmp/qe-mystmd | |
| git init -q | |
| git remote add origin https://github.com/QuantEcon/mystmd.git | |
| git fetch -q --depth 1 origin 96ee78f550ff2e0410730f328352e98d3e29d338 | |
| git checkout -q FETCH_HEAD | |
| git rev-parse HEAD | |
| bun install | |
| bun run build | |
| npm install -g /tmp/qe-mystmd/packages/mystmd | |
| - name: Log QuantEcon/mystmd build identifier | |
| run: cat /tmp/qe-mystmd/quantecon/VERSION.yml | |
| - name: Verify mystmd version | |
| run: myst --version | |
| # Standing telemetry: sample host RAM, top-RSS processes and GPU memory | |
| # every 5s for the rest of the job. The nohup'd loop survives step | |
| # boundaries; the runner reaps it at job end. Read back in the always() | |
| # step after the build. This is what diagnosed the 2026-08-03 jax_intro | |
| # kernel death (GPU preallocation race — see the Build HTML env below); | |
| # it costs nothing, so it stays for the next one. | |
| - name: Start memory monitor | |
| run: | | |
| nohup bash -c 'while true; do | |
| echo "=== $(date -u +%H:%M:%S)" | |
| free -m | sed -n "2,3p" | |
| ps -eo pid,rss,etime,comm --sort=-rss | head -8 | |
| nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader 2>/dev/null | |
| sleep 5 | |
| done' >/tmp/memmon.log 2>&1 & | |
| echo "monitor started" | |
| # Built with the fork's `myst` CLI rather than `jupyter book build`. | |
| # jupyter-book 2.x is a Python shim that execs a mystmd bundle vendored | |
| # inside its own wheel (jupyter_book/dist/jupyter-book.cjs) and offers | |
| # no supported way to point it at another CLI — so `jupyter book build` | |
| # silently ignored the fork we install above and built with upstream | |
| # mystmd (visible as 23 `unknown export output extension: *.ipynb` | |
| # errors, since ipynb export is a fork-only capability). | |
| - name: Build HTML | |
| working-directory: ./lectures | |
| env: | |
| # Three JAX lectures (jax_intro, autodiff, numpy_vs_numba_vs_jax) | |
| # execute concurrently on the g4dn's single T4. By default the first | |
| # JAX process to touch the GPU preallocates 75% of it (11.3GiB of | |
| # 15.3GiB); the 2026-08-03 diagnostics showed jax_intro's kernel then | |
| # clamped to the ~3.4GiB remainder and aborted with no Python | |
| # traceback once a later cell outgrew that pool (dmesg had no | |
| # OOM-killer or segfault records — an XLA fatal, not a host kill). | |
| # On-demand allocation lets the concurrent kernels share the GPU. | |
| XLA_PYTHON_CLIENT_PREALLOCATE: 'false' | |
| run: myst build --html --execute | |
| # Reads the telemetry back. dmesg covers the whole job because runs-on | |
| # boots a fresh instance per run — an OOM-killed process appears there | |
| # with its RSS, so this cleanly separates host OOM kills from in-process | |
| # aborts (the 2026-08-03 death was the latter: dmesg was clean while the | |
| # GPU sampler showed the preallocation collision). Start from this | |
| # output for any future kernel death. | |
| - name: Execution diagnostics | |
| if: always() | |
| run: | | |
| echo "::group::memory timeline (5s samples)" | |
| cat /tmp/memmon.log || true | |
| echo "::endgroup::" | |
| echo "::group::kernel log (OOM killer / segfaults)" | |
| (sudo dmesg -T || dmesg -T) 2>/dev/null | grep -iE "out of memory|oom|killed process|segfault" || echo "no OOM/segfault records" | |
| echo "::endgroup::" | |
| echo "::group::final state" | |
| free -m | |
| nvidia-smi 2>/dev/null || true | |
| echo "::endgroup::" | |
| - name: Upload build output | |
| uses: actions/upload-artifact@v7 | |
| with: | |
| path: './lectures/_build/html' | |
| - name: Preview Deploy to Netlify | |
| uses: nwtgck/actions-netlify@v4 | |
| with: | |
| publish-dir: './lectures/_build/html' | |
| production-branch: main | |
| github-token: ${{ secrets.GITHUB_TOKEN }} | |
| deploy-message: "Preview Deploy from GitHub Actions" | |
| env: | |
| NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }} | |
| NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }} |