Skip to content

feat: Add caching support for container mode builds #18

Description

@mmcky

Summary

Add caching support for container mode in setup-environment to speed up lecture-specific package installation.

Current State

Container mode currently installs packages fresh each run via conda env update. This takes ~30-60 seconds per build.

Problem with actions/cache

The naive approach of using actions/cache to cache /opt/conda/envs/... doesn't work because:

  1. actions/cache is a JavaScript action - It runs on the host runner, not inside the container
  2. Container filesystem is isolated - Paths like /opt/conda/... exist inside the container but are not visible to the host
  3. Cache restore/save fails silently - The action may appear to succeed but won't actually cache anything

Potential Solutions

Option 1: Workspace-relative cache

Copy installed packages to a workspace directory (e.g., .cache/conda-pkgs/), then restore them on subsequent runs:

- name: Cache packages in workspace
  uses: actions/cache@v4
  with:
    path: .cache/conda-pkgs
    key: lecture-pkgs-${{ hashFiles('environment.yml') }}

- name: Restore packages from workspace cache
  if: steps.cache.outputs.cache-hit == 'true'
  run: cp -r .cache/conda-pkgs/* /opt/conda/envs/quantecon/lib/python*/site-packages/

Pros: Works with existing actions/cache
Cons: Complex, fragile, may cause inconsistent conda state

Option 2: Use pip cache only

Cache pip's download cache instead of installed packages:

- name: Cache pip downloads
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ hashFiles('environment.yml') }}

Pros: Simple, pip cache is often workspace-accessible
Cons: Only speeds up pip packages, not conda packages

Option 3: GitHub Actions cache mounted volumes

Use Docker volume mounts to expose cache directories to the container:

container:
  image: ghcr.io/quantecon/quantecon-build:latest
  volumes:
    - /home/runner/.cache:/root/.cache

Pros: Clean solution
Cons: Requires workflow changes, not transparent

Option 4: Pre-bake more packages into container

Include more lecture-specific packages directly in the container image, reducing what needs to be installed at runtime.

Pros: Fastest builds, no caching complexity
Cons: Larger container, harder to customize per-lecture

Recommended Approach

Start with Option 4 - analyze which packages are commonly installed across lectures and add them to quantecon-build container. This provides the best build time improvement with minimal complexity.

If further optimization is needed, investigate Option 3 for pip caching.

Acceptance Criteria

  • Profile current build times to establish baseline
  • Analyze common packages across lecture repos
  • Implement chosen caching strategy
  • Document any workflow changes required
  • Measure improvement in build times

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions