Summary
Add caching support for container mode in setup-environment to speed up lecture-specific package installation.
Current State
Container mode currently installs packages fresh each run via conda env update. This takes ~30-60 seconds per build.
Problem with actions/cache
The naive approach of using actions/cache to cache /opt/conda/envs/... doesn't work because:
actions/cache is a JavaScript action - It runs on the host runner, not inside the container
- Container filesystem is isolated - Paths like
/opt/conda/... exist inside the container but are not visible to the host
- Cache restore/save fails silently - The action may appear to succeed but won't actually cache anything
Potential Solutions
Option 1: Workspace-relative cache
Copy installed packages to a workspace directory (e.g., .cache/conda-pkgs/), then restore them on subsequent runs:
- name: Cache packages in workspace
uses: actions/cache@v4
with:
path: .cache/conda-pkgs
key: lecture-pkgs-${{ hashFiles('environment.yml') }}
- name: Restore packages from workspace cache
if: steps.cache.outputs.cache-hit == 'true'
run: cp -r .cache/conda-pkgs/* /opt/conda/envs/quantecon/lib/python*/site-packages/
Pros: Works with existing actions/cache
Cons: Complex, fragile, may cause inconsistent conda state
Option 2: Use pip cache only
Cache pip's download cache instead of installed packages:
- name: Cache pip downloads
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ hashFiles('environment.yml') }}
Pros: Simple, pip cache is often workspace-accessible
Cons: Only speeds up pip packages, not conda packages
Option 3: GitHub Actions cache mounted volumes
Use Docker volume mounts to expose cache directories to the container:
container:
image: ghcr.io/quantecon/quantecon-build:latest
volumes:
- /home/runner/.cache:/root/.cache
Pros: Clean solution
Cons: Requires workflow changes, not transparent
Option 4: Pre-bake more packages into container
Include more lecture-specific packages directly in the container image, reducing what needs to be installed at runtime.
Pros: Fastest builds, no caching complexity
Cons: Larger container, harder to customize per-lecture
Recommended Approach
Start with Option 4 - analyze which packages are commonly installed across lectures and add them to quantecon-build container. This provides the best build time improvement with minimal complexity.
If further optimization is needed, investigate Option 3 for pip caching.
Acceptance Criteria
Related
Summary
Add caching support for container mode in
setup-environmentto speed up lecture-specific package installation.Current State
Container mode currently installs packages fresh each run via
conda env update. This takes ~30-60 seconds per build.Problem with
actions/cacheThe naive approach of using
actions/cacheto cache/opt/conda/envs/...doesn't work because:actions/cacheis a JavaScript action - It runs on the host runner, not inside the container/opt/conda/...exist inside the container but are not visible to the hostPotential Solutions
Option 1: Workspace-relative cache
Copy installed packages to a workspace directory (e.g.,
.cache/conda-pkgs/), then restore them on subsequent runs:Pros: Works with existing
actions/cacheCons: Complex, fragile, may cause inconsistent conda state
Option 2: Use pip cache only
Cache pip's download cache instead of installed packages:
Pros: Simple, pip cache is often workspace-accessible
Cons: Only speeds up pip packages, not conda packages
Option 3: GitHub Actions cache mounted volumes
Use Docker volume mounts to expose cache directories to the container:
Pros: Clean solution
Cons: Requires workflow changes, not transparent
Option 4: Pre-bake more packages into container
Include more lecture-specific packages directly in the container image, reducing what needs to be installed at runtime.
Pros: Fastest builds, no caching complexity
Cons: Larger container, harder to customize per-lecture
Recommended Approach
Start with Option 4 - analyze which packages are commonly installed across lectures and add them to
quantecon-buildcontainer. This provides the best build time improvement with minimal complexity.If further optimization is needed, investigate Option 3 for pip caching.
Acceptance Criteria
Related