[Experimental] Build the lectures with QuantEcon/mystmd instead of jupyter-book #1537
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Build Project [using QuantEcon/mystmd] | |
| on: [pull_request] | |
| jobs: | |
| preview: | |
| runs-on: "runs-on=${{ github.run_id }}/family=g4dn.2xlarge/image=quantecon_ubuntu2404/volume=80gb/spot=false" | |
| steps: | |
| - uses: actions/checkout@v7 | |
| - name: Cache Notebook Execution | |
| uses: actions/cache@v6 | |
| id: cache-execution | |
| with: | |
| path: ./lectures/_build/execute | |
| key: ${{ runner.os }}-execute-cache-${{ hashFiles('lectures/**/*.md') }} | |
| # Node 24 matches the theme's .nvmrc (quantecon-theme.mystmd), which runs | |
| # as a Remix server during `myst build --html`. | |
| - uses: actions/setup-node@v7 | |
| with: | |
| node-version: '24' | |
| - uses: oven-sh/setup-bun@v2 | |
| - name: Install Python | |
| if: steps.cache-execution.outputs.cache-hit != 'true' | |
| uses: actions/setup-python@v7 | |
| with: | |
| python-version: '3.12' | |
| cache: 'pip' | |
| cache-dependency-path: 'myst_requirements.txt' | |
| - name: Install execution requirements | |
| if: steps.cache-execution.outputs.cache-hit != 'true' | |
| run: python -m pip install -r myst_requirements.txt | |
| # The GPU lectures (jax_intro, autodiff, numpy_vs_numba_vs_jax) need JAX | |
| # on the g4dn runner — same install as the jupyter-book 1.x workflow on | |
| # main. Without it those notebooks execute to ModuleNotFoundError cells. | |
| - name: Install JAX | |
| if: steps.cache-execution.outputs.cache-hit != 'true' | |
| run: | | |
| pip install -U "jax[cuda13]" | |
| python scripts/test-jax-install.py | |
| # Pinned to the exact SHA of the fork's main (unmoved since 2026-06-12, | |
| # so this is the same engine the 2026-08-03 cold-run failures used). | |
| # A branch name here makes the engine a moving target between runs of | |
| # the same lecture commit, which makes failures impossible to bisect. | |
| - name: Install mystmd (QuantEcon fork) | |
| run: | | |
| mkdir -p /tmp/qe-mystmd | |
| cd /tmp/qe-mystmd | |
| git init -q | |
| git remote add origin https://github.com/QuantEcon/mystmd.git | |
| git fetch -q --depth 1 origin 96ee78f550ff2e0410730f328352e98d3e29d338 | |
| git checkout -q FETCH_HEAD | |
| git rev-parse HEAD | |
| bun install | |
| bun run build | |
| npm install -g /tmp/qe-mystmd/packages/mystmd | |
| - name: Log QuantEcon/mystmd build identifier | |
| run: cat /tmp/qe-mystmd/quantecon/VERSION.yml | |
| - name: Verify mystmd version | |
| run: myst --version | |
| # DIAGNOSIS (PR #363 cold-cache jax_intro kernel death): sample host | |
| # RAM, top-RSS processes and GPU memory every 5s for the rest of the | |
| # job. The nohup'd loop survives step boundaries; the runner reaps it | |
| # at job end. Read back in the always() step after the build. | |
| - name: Start memory monitor | |
| run: | | |
| nohup bash -c 'while true; do | |
| echo "=== $(date -u +%H:%M:%S)" | |
| free -m | sed -n "2,3p" | |
| ps -eo pid,rss,etime,comm --sort=-rss | head -8 | |
| nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader 2>/dev/null | |
| sleep 5 | |
| done' >/tmp/memmon.log 2>&1 & | |
| echo "monitor started" | |
| # Built with the fork's `myst` CLI rather than `jupyter book build`. | |
| # jupyter-book 2.x is a Python shim that execs a mystmd bundle vendored | |
| # inside its own wheel (jupyter_book/dist/jupyter-book.cjs) and offers | |
| # no supported way to point it at another CLI — so `jupyter book build` | |
| # silently ignored the fork we install above and built with upstream | |
| # mystmd (visible as 23 `unknown export output extension: *.ipynb` | |
| # errors, since ipynb export is a fork-only capability). | |
| - name: Build HTML | |
| working-directory: ./lectures | |
| env: | |
| # Three JAX lectures (jax_intro, autodiff, numpy_vs_numba_vs_jax) | |
| # execute concurrently on the g4dn's single T4. By default the first | |
| # JAX process to touch the GPU preallocates 75% of it (11.3GiB of | |
| # 15.3GiB); the 2026-08-03 diagnostics showed jax_intro's kernel then | |
| # clamped to the ~3.4GiB remainder and aborted with no Python | |
| # traceback once a later cell outgrew that pool (dmesg had no | |
| # OOM-killer or segfault records — an XLA fatal, not a host kill). | |
| # On-demand allocation lets the concurrent kernels share the GPU. | |
| XLA_PYTHON_CLIENT_PREALLOCATE: 'false' | |
| run: myst build --html --execute | |
| # A kernel death with no Python traceback is consistent with a SIGKILL | |
| # from the host OOM killer, but that has never been confirmed. dmesg | |
| # covers the whole job because runs-on boots a fresh instance per run. | |
| - name: Execution diagnostics | |
| if: always() | |
| run: | | |
| echo "::group::memory timeline (5s samples)" | |
| cat /tmp/memmon.log || true | |
| echo "::endgroup::" | |
| echo "::group::kernel log (OOM killer / segfaults)" | |
| (sudo dmesg -T || dmesg -T) 2>/dev/null | grep -iE "out of memory|oom|killed process|segfault" || echo "no OOM/segfault records" | |
| echo "::endgroup::" | |
| echo "::group::final state" | |
| free -m | |
| nvidia-smi 2>/dev/null || true | |
| echo "::endgroup::" | |
| - name: Upload build output | |
| uses: actions/upload-artifact@v7 | |
| with: | |
| path: './lectures/_build/html' | |
| - name: Preview Deploy to Netlify | |
| uses: nwtgck/actions-netlify@v4 | |
| with: | |
| publish-dir: './lectures/_build/html' | |
| production-branch: main | |
| github-token: ${{ secrets.GITHUB_TOKEN }} | |
| deploy-message: "Preview Deploy from GitHub Actions" | |
| env: | |
| NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }} | |
| NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }} |