Skip to content

Adds caching functionality #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 37 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
2bf4e14
try caching in CircleCI
tsharmaMW Jun 19, 2025
a239850
fix linting issues
tsharmaMW Jun 19, 2025
b123923
add debugging
tsharmaMW Jun 19, 2025
71ac1ca
add new line
tsharmaMW Jun 19, 2025
21c19d6
remove cache steps
tsharmaMW Jun 19, 2025
a5f7492
fix
tsharmaMW Jun 19, 2025
230cc29
update tmpDIR
tsharmaMW Jun 19, 2025
9f5877e
remove relative temp dir
tsharmaMW Jun 19, 2025
4c817fb
make caching conditional
tsharmaMW Jun 19, 2025
515841b
change ownership of the install dir to current user
tsharmaMW Jun 20, 2025
e4c472e
move command to end of script
tsharmaMW Jun 20, 2025
b9e4f21
update
tsharmaMW Jun 20, 2025
83df8a4
update temp directory for run-tests
tsharmaMW Jun 20, 2025
bfea32c
debug
tsharmaMW Jun 20, 2025
a5499da
seperate cache and tmp directories
tsharmaMW Jun 23, 2025
adfc254
sort the products to generate the key
tsharmaMW Jun 23, 2025
09c6139
indent
tsharmaMW Jun 23, 2025
21c6bf7
update key and cache param
tsharmaMW Jun 23, 2025
873198d
update key and cache validation
tsharmaMW Jun 23, 2025
8c8029e
update cache dir and add test
tsharmaMW Jun 24, 2025
1b839b6
move parsing of release to another script
tsharmaMW Jun 24, 2025
d64b7ea
update
tsharmaMW Jun 24, 2025
e50bf36
disable shellcheck warnings
tsharmaMW Jun 24, 2025
dc37346
update paths
tsharmaMW Jun 24, 2025
795298f
make sourced script path relative
tsharmaMW Jun 25, 2025
5794cb0
remove sourcing the common script
tsharmaMW Jun 25, 2025
5dd3f31
eval the script instead of including
tsharmaMW Jun 25, 2025
4db2af5
update checksum
tsharmaMW Jun 26, 2025
148ab74
add shellcheck warning
tsharmaMW Jun 26, 2025
e2c94d8
refactor code
tsharmaMW Jun 26, 2025
5940d2e
add cached MATLAB to path instead of copying
Jul 1, 2025
34f8eb4
remove copy step from install script
tsharmaMW Jul 1, 2025
1c36b07
fix
tsharmaMW Jul 1, 2025
8b2aa55
Update command
tsharmaMW Jul 1, 2025
7f9885a
update dir
tsharmaMW Jul 11, 2025
6b31371
add product in install-with-cache test
tsharmaMW Jul 11, 2025
885ba49
update test
tsharmaMW Jul 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .circleci/test-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,42 @@ jobs:
- matlab/run-command:
command: "exp='<<parameters.release>>'; assert(strcmp(version('-release'),exp(2:6)))"

integration-test-install-with-cache:
parameters:
executor:
type: executor
executor: <<parameters.executor>>
steps:
- checkout
- run:
name: Clean cache directory before first install
command: rm -rf $HOME/matlab_ci
shell: bash
- matlab/install:
no-output-timeout: 30m
products: Simulink
cache: true
- run:
name: Verify files exist in cache
command: |
set -e
matlab_binary="$HOME/matlab_ci/matlab_root"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like the matlab root not the binary?

if [[ "$(uname)" == "Darwin" ]]; then
matlab_binary="$matlab_binary/MATLAB.app"
fi
test -x "$matlab_binary/bin/matlab" || {
echo "Missing MATLAB binary at expected cache path!"
exit 1
}
test -x "$HOME/matlab_ci/matlab-batch/matlab-batch"* || {
echo "MATLAB Batch not found in cache!"
exit 1
}
echo "All expected files found in cache."
shell: bash
- matlab/run-command:
command: version

integration-test-run-command:
parameters:
executor:
Expand Down Expand Up @@ -500,6 +536,11 @@ workflows:
executor: [linux, windows, macos]
release: [R2023bU1]

- integration-test-install-with-cache:
matrix:
parameters:
executor: [linux, windows, macos]

- integration-test-run-command:
matrix:
parameters:
Expand Down
35 changes: 35 additions & 0 deletions src/commands/install.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,48 @@ parameters:
maximum time a job is allowed to run.
type: string
default: 10m
cache:
description: >
Option to enable caching, specified as false or true. By default, the value is false and the command does not store MATLAB and the specified products in a cache for future use.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks fine, Tushar. Are there any other user-facing messages in this submission that I should look at?

type: boolean
default: false

steps:
- when:
condition:
equal: [ << parameters.cache >>, true ]
steps:
- run:
name: Prepare cache metadata
environment:
PARAM_RELEASE: <<parameters.release>>
RESOLVE_RELEASE_SH: <<include(scripts/resolve-release.sh)>>
command: |
eval "$RESOLVE_RELEASE_SH"
echo "$mpmrelease" > install-metadata.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this writing install-metadata.txt into the user's working directory (where their source code is)? If so, we probably want to put it somewhere else. I think it would be pretty unexpected for the orb to drop files in your source root and more likely for the user to delete or mess with them.

echo "<< parameters.products >>" | tr ' ' '\n' | sort | xargs >> install-metadata.txt
shell: bash
- restore_cache:
name: Restore Cache
keys:
- matlab-cache-{{ arch }}-{{ checksum "install-metadata.txt" }}
- run:
name: Install MATLAB
environment:
PARAM_RELEASE: <<parameters.release>>
PARAM_PRODUCTS: <<parameters.products>>
PARAM_CACHE: <<parameters.cache>>
RESOLVE_RELEASE_SH: <<include(scripts/resolve-release.sh)>>
command: <<include(scripts/install.sh)>>
shell: bash
no_output_timeout: <<parameters.no-output-timeout>>
- when:
condition:
equal: [ << parameters.cache >>, true ]
steps:
- save_cache:
name: Save Cache
key: matlab-cache-{{ arch }}-{{ checksum "install-metadata.txt" }}
paths:
- ~/matlab_ci/matlab_root
- ~/matlab_ci/matlab-batch
82 changes: 27 additions & 55 deletions src/scripts/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,26 +17,6 @@ sudoIfAvailable() {
fi
}

stream() {
local url="$1"
local status=0

if command -v wget >/dev/null 2>&1; then
wget --retry-connrefused --waitretry=5 -qO- "$url" || status=$?
elif command -v curl >/dev/null 2>&1; then
curl --retry 5 --retry-connrefused --retry-delay 5 -sSL "$url" || status=$?
else
echo "Could not find wget or curl command" >&2
return 1
fi

if [ $status -ne 0 ]; then
echo "Error streaming file from $url" >&2
fi

return $status
}

download() {
local url="$1"
local filename="$2"
Expand All @@ -61,34 +41,21 @@ download() {
os=$(uname)
arch=$(uname -m)
binext=""
tmpdir=$(mktemp -d 2>/dev/null || mktemp -d -t 'install')
rootdir="$tmpdir/matlab_root"
batchdir="$tmpdir/matlab-batch"
mpmdir="$tmpdir/mpm"
matlabcidir="$HOME/matlab_ci"
mkdir -p "$matlabcidir"
rootdir="$matlabcidir/matlab_root"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's technically possible for a user to use matlab/install twice in their pipeline to install two different releases of MATLAB, so we probably need to consider getting the release name in this path. Any reason not to install to the standard /usr/local/MATLAB/<release> location used by mpm?

batchdir="$matlabcidir/matlab-batch"
mpmdir="$matlabcidir/mpm"
batchbaseurl="https://ssd.mathworks.com/supportfiles/ci/matlab-batch/v1"
mpmbaseurl="https://www.mathworks.com/mpm"
releasestatus=""

# resolve release
parsedrelease=$(echo "$PARAM_RELEASE" | tr '[:upper:]' '[:lower:]')
if [[ "$parsedrelease" = "latest" ]]; then
mpmrelease=$(stream https://ssd.mathworks.com/supportfiles/ci/matlab-release/v0/latest)
elif [[ "$parsedrelease" = "latest-including-prerelease" ]]; then
mpmrelease=$(stream https://ssd.mathworks.com/supportfiles/ci/matlab-release/v0/latest-including-prerelease)
releasestatus="--release-status=Prerelease"
else
mpmrelease="$parsedrelease"
fi

# validate release is supported
if [[ "$mpmrelease" < "r2020b" ]]; then
echo "Release '${mpmrelease}' is not supported. Use 'R2020b' or a later release.">&2
exit 1
fi
eval "$RESOLVE_RELEASE_SH"

# install system dependencies
if [[ "$os" = "Linux" ]]; then
# install MATLAB dependencies
# shellcheck disable=SC2154
release=$(echo "${mpmrelease}" | grep -ioE "(r[0-9]{4}[a-b])")
stream https://ssd.mathworks.com/supportfiles/ci/matlab-deps/v0/install.sh | sudoIfAvailable -s -- "$release"
# install mpm depencencies
Expand All @@ -102,7 +69,7 @@ elif [[ "$os" = "Darwin" && "$arch" = "arm64" ]]; then
sudoIfAvailable -c "softwareupdate --install-rosetta --agree-to-license"
else
# install Java runtime
jdkpkg="$tmpdir/jdk.pkg"
jdkpkg="$matlabcidir/jdk.pkg"
download https://corretto.aws/downloads/latest/amazon-corretto-8-aarch64-macos-jdk.pkg "$jdkpkg"
sudoIfAvailable -c "installer -pkg '$jdkpkg' -target /"
fi
Expand Down Expand Up @@ -131,20 +98,25 @@ mkdir -p "$rootdir"
mkdir -p "$batchdir"
mkdir -p "$mpmdir"

# install mpm
download "$mpmbaseurl/$mwarch/mpm" "$mpmdir/mpm$binext"
chmod +x "$mpmdir/mpm$binext"

# install matlab-batch
download "$batchbaseurl/$mwarch/matlab-batch$binext" "$batchdir/matlab-batch$binext"
chmod +x "$batchdir/matlab-batch$binext"

# install matlab
"$mpmdir/mpm$binext" install \
--release="$mpmrelease" \
--destination="$rootdir" \
${releasestatus} \
--products ${PARAM_PRODUCTS} MATLAB
# Short-circuit if MATLAB & matlab-batch already exist and PARAM_CACHE is true
if [[ "$PARAM_CACHE" == "1" && -x "$batchdir/matlab-batch$binext" && -x "$rootdir/bin/matlab" ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't cache matlab-batch because it is important that we always make sure we're using the latest version of it. It's a fairly small binary, so just downloading it every time isn't bad.

echo "Skipping fresh installation and restoring from Cache."
else
# install mpm
download "$mpmbaseurl/$mwarch/mpm" "$mpmdir/mpm$binext"
chmod +x "$mpmdir/mpm$binext"

# install matlab-batch
download "$batchbaseurl/$mwarch/matlab-batch$binext" "$batchdir/matlab-batch$binext"
chmod +x "$batchdir/matlab-batch$binext"

# install matlab
"$mpmdir/mpm$binext" install \
--release="$mpmrelease" \
--destination="$rootdir" \
${releasestatus} \
--products ${PARAM_PRODUCTS} MATLAB
fi

# add MATLAB and matlab-batch to path
echo 'export PATH="'$rootdir'/bin:'$batchdir':$PATH"' >> $BASH_ENV
Expand Down
37 changes: 37 additions & 0 deletions src/scripts/resolve-release.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/bin/bash

stream() {
local url="$1"
local status=0

if command -v wget >/dev/null 2>&1; then
wget --retry-connrefused --waitretry=5 -qO- "$url" || status=$?
elif command -v curl >/dev/null 2>&1; then
curl --retry 5 --retry-connrefused --retry-delay 5 -sSL "$url" || status=$?
else
echo "Could not find wget or curl command" >&2
return 1
fi

if [ $status -ne 0 ]; then
echo "Error streaming file from $url" >&2
fi

return $status
}

parsedrelease=$(echo "$PARAM_RELEASE" | tr '[:upper:]' '[:lower:]')
if [[ "$parsedrelease" = "latest" ]]; then
mpmrelease=$(stream https://ssd.mathworks.com/supportfiles/ci/matlab-release/v0/latest)
elif [[ "$parsedrelease" = "latest-including-prerelease" ]]; then
mpmrelease=$(stream https://ssd.mathworks.com/supportfiles/ci/matlab-release/v0/latest-including-prerelease)
# shellcheck disable=SC2034
releasestatus="--release-status=Prerelease"
else
mpmrelease="$parsedrelease"
fi

if [[ "$mpmrelease" < "r2020b" ]]; then
echo "Release '${mpmrelease}' is not supported. Use 'R2020b' or a later release." >&2
exit 1
fi