Skip to content

Commit 7ff9f6b

Browse files
committed
update-template*.sh: refactor update-template-ubuntu.sh add update-template{,-debian}.sh
cache-common-inc.sh: Prevent `download_to_cache` from rechecking URLs within 10 minutes Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: add `ubuntu_` prefix to some symbols Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: change some parameters order in some function Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: avoid reusing parameters for flags Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: add `ubuntu_{flavor,version}_from_location` Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: add `ubuntu_location_url_spec` Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: temporarily remove URL caching for refactoring Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: add `ubuntu_{arch,path_suffix}_from_location` Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: change some functions to return json Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: refactor checking `url_spec` condition Signed-off-by: Norio Nomura <[email protected]> hack/cache-common-inc.sh: change `download_to_cache` to use JSON instead of tsv Since `unpacked/SHA256SUMS` has no `Content-Type`, the output of `curl -w` formatted as TSV causes `IFS=$'\t' read -r` to fail during parsing. Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: change downloading `unpacked/SHA256SUMS` to use cache Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: add `ubuntu_image_entry_for_image_kernel_flavor_version` Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: change to `ubuntu_image_entry_with_kernel_info` that returns the image entry with the kernel info instead of returning the kernel info. Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: change to use `json_vars` for creating JSON Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: support version aliases: latest, lts Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: change to fully utilize `set -e` Functions in this script assume error handling with 'set -e'. To ensure 'set -e' works correctly: - Use 'set +e' before assignments and '$(set -e; <function>)' to capture output without exiting on errors. - Avoid calling functions directly in conditions to prevent disabling 'set -e'. - Use 'shopt -s inherit_errexit' (Bash 4.4+) to avoid repeated 'set -e' in all '$(...)'. Changes: - Add `error_exit` and more error messages - Remove `|| return` after `$(...)` - Use `jq -e` Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: simplify script for `limactl edit --set` Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: add `image_entry` caching Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: check for `.kernel` in `image_entry` before adding `.kernel.cmdline` Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: add error message for when `com.ubuntu.cloud:released:download.json` does not contain a matching image Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: include `_with_kernel` to cache key if kernel location is not null Signed-off-by: Norio Nomura <[email protected]> cache-common-inc.sh: move `error_exit` from `update-template-ubuntu.sh` Signed-off-by: Norio Nomura <[email protected]> update-template.sh: add `update-template.sh` script to update all distributions - Move functions common to all distributions here. - Scripts for specific distributions accept distribution-specific options and perform additional actions. Signed-off-by: Norio Nomura <[email protected]> hack/(cache-common-inc|update-template*).sh: fix some comments - change `return` to `print` in comments - remove stale comment Signed-off-by: Norio Nomura <[email protected]> update-template-ubuntu.sh: rename the functions to indicate they are using location basename Signed-off-by: Norio Nomura <[email protected]> update-template-debian.sh: support updating Debian image information in template files ```console $ hack/update-template-debian.sh update-template-debian.sh: Update the Debian image location in the specified templates Usage: update-template-debian.sh [--backports[=<bool>]] [--daily[=<bool>]] [--timestamped[=<bool>]] [--version <version>] <template.yaml>... Description: This script updates the Debian image location in the specified templates. If the image location in the template contains a release date in the URL, the script replaces it with the latest available date. If no flags are specified, the script uses the version from the image location basename in the template. Image location basename format: debian-<version>[-backports]-genericcloud-<arch>[-daily][-<timestamp>].qcow2 Published Debian image information is fetched from the following URLs: https://cloud.debian.org/images/cloud/<codename>[-backports]/[daily/](latest|<timestamp>)/debian-<version>[-backports]-genericcloud-<arch>[-daily][-<timestamp>].json The downloaded JSON file will be cached in the Lima cache directory. Examples: Update the Debian image location in templates/**.yaml: $ update-template-debian.sh templates/**.yaml Update the Debian image location in ~/.lima/debian/lima.yaml: $ update-template-debian.sh ~/.lima/debian/lima.yaml Update the Debian image location to debian-13-genericcloud-<arch>.qcow2 in ~/.lima/debian/lima.yaml: $ update-template-debian.sh --version trixie ~/.lima/debian/lima.yaml Flags: --backports[=<bool>] Use the backports image The boolean value can be true, false, 1, or 0 --daily[=<bool>] Use the daily image --timestamped[=<bool>] Use the timestamped image --version <version> Use the specified version The version can be a codename, version number, or alias (testing, stable, oldstable) -h, --help Print this help message ``` Signed-off-by: Norio Nomura <[email protected]> update-template{,-debian}.sh: fix update-template.sh did not sourced `update-template-debian.sh` Signed-off-by: Norio Nomura <[email protected]> update-template{,-debian,-ubuntu}.sh: add `limactl factory-reset` to examples in help Signed-off-by: Norio Nomura <[email protected]>
1 parent 8eb855a commit 7ff9f6b

File tree

4 files changed

+1018
-198
lines changed

4 files changed

+1018
-198
lines changed

hack/cache-common-inc.sh

Lines changed: 42 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
#!/usr/bin/env bash
22

3+
# print the error message and exit with status 1
4+
function error_exit() {
5+
echo "Error: $*" >&2
6+
exit 1
7+
}
8+
39
# e.g.
410
# ```console
511
# $ download_template_if_needed templates/default.yaml
@@ -97,7 +103,7 @@ function size_from_location() {
97103
)
98104
}
99105

100-
# Check the remote location and return the http code and size.
106+
# Check the remote location and print the http code and size.
101107
# If GITHUB_ACTIONS is true, the result is not cached.
102108
# e.g.
103109
# ```console
@@ -113,7 +119,7 @@ function check_location() {
113119
fi
114120
}
115121

116-
# Check the remote location and return the http code and size.
122+
# Check the remote location and print the http code and size.
117123
# The result is cached in .check_location-response-cache.yaml
118124
# e.g.
119125
# ```console
@@ -209,8 +215,7 @@ function location_to_sha256() {
209215
elif command -v shasum >/dev/null; then
210216
sha256="$(echo -n "${location}" | shasum -a 256 | cut -d' ' -f1)"
211217
else
212-
echo "sha256sum or shasum not found" >&2
213-
exit 1
218+
error_exit "sha256sum or shasum not found"
214219
fi
215220
echo "${sha256}"
216221
)
@@ -351,16 +356,32 @@ function hash_file() {
351356
# /Users/user/Library/Caches/lima/download/by-url-sha256/346ee1ff9e381b78ba08e2a29445960b5cd31c51f896fc346b82e26e345a5b9a/data # on macOS
352357
# /home/user/.cache/lima/download/by-url-sha256/346ee1ff9e381b78ba08e2a29445960b5cd31c51f896fc346b82e26e345a5b9a/data # on others
353358
function download_to_cache() {
354-
local code_time_type_url
355-
code_time_type_url=$(
356-
curl -sSLI -w "%{http_code}\t%header{Last-Modified}\t%header{Content-Type}\t%{url_effective}" "$1" -o /dev/null
357-
)
359+
local cache_path
360+
cache_path=$(location_to_cache_path "$1")
361+
# before checking remote location, check if the data file is already downloaded and the time file is updated within 10 minutes
362+
if [[ -f ${cache_path}/data && -n "$(find "${cache_path}/time" -mmin -10 || true)" ]]; then
363+
echo "${cache_path}/data"
364+
return
365+
fi
366+
367+
# check the remote location
368+
local curl_info_json write_out
369+
write_out='{
370+
"http_code":%{http_code},
371+
"last_modified":"%header{Last-Modified}",
372+
"content_type":"%{content_type}",
373+
"url":"%{url_effective}",
374+
"filename":"%{filename_effective}"
375+
}'
376+
curl_info_json=$(curl -sSLI -w "${write_out}" "$1" -o /dev/null)
358377

359378
local code time type url
360-
IFS=$'\t' read -r code time type url filename <<<"${code_time_type_url}"
361-
[[ ${code} == 200 ]] || exit 1
379+
code=$(jq -r '.http_code' <<<"${curl_info_json}")
380+
time=$(jq -r '.last_modified' <<<"${curl_info_json}")
381+
type=$(jq -r '.content_type' <<<"${curl_info_json}")
382+
url=$(jq -r '.url' <<<"${curl_info_json}")
383+
[[ ${code} == 200 ]] || error_exit "Failed to download $1"
362384

363-
local cache_path
364385
cache_path=$(location_to_cache_path "${url}")
365386
[[ -d ${cache_path} ]] || mkdir -p "${cache_path}"
366387

@@ -369,18 +390,23 @@ function download_to_cache() {
369390
[[ -f ${cache_path}/time && "$(<"${cache_path}/time")" == "${time}" ]] || needs_download=1
370391
[[ -f ${cache_path}/type && "$(<"${cache_path}/type")" == "${type}" ]] || needs_download=1
371392
if [[ ${needs_download} -eq 1 ]]; then
372-
local code_time_type_url_filename
373-
code_time_type_url_filename=$(
393+
curl_info_json=$(
374394
echo "downloading ${url}" >&2
375-
curl -SL -w "%{http_code}\t%header{Last-Modified}\t%header{Content-Type}\t%{url_effective}\t%{filename_effective}" --no-clobber -o "${cache_path}/data" "${url}"
395+
curl -SL -w "${write_out}" --no-clobber -o "${cache_path}/data" "${url}"
376396
)
377397
local filename
378-
IFS=$'\t' read -r code time type url filename <<<"${code_time_type_url_filename}"
379-
[[ ${code} == 200 ]] || exit 1
398+
code=$(jq -r '.http_code' <<<"${curl_info_json}")
399+
time=$(jq -r '.last_modified' <<<"${curl_info_json}")
400+
type=$(jq -r '.content_type' <<<"${curl_info_json}")
401+
url=$(jq -r '.url' <<<"${curl_info_json}")
402+
filename=$(jq -r '.filename' <<<"${curl_info_json}")
403+
[[ ${code} == 200 ]] || error_exit "Failed to download ${url}"
380404
[[ "${cache_path}/data" == "${filename}" ]] || mv "${filename}" "${cache_path}/data"
381405
# sha256.digest seems existing if expected digest is available. so, not creating it here.
382406
# sha256sum "${cache_path}/data" | awk '{print "sha256:"$1}' >"${cache_path}/sha256.digest"
383407
echo -n "${time}" >"${cache_path}/time"
408+
else
409+
touch "${cache_path}/time"
384410
fi
385411
[[ -f ${cache_path}/type ]] || echo -n "${type}" >"${cache_path}/type"
386412
[[ -f ${cache_path}/url ]] || echo -n "${url}" >"${cache_path}/url"

0 commit comments

Comments
 (0)