Skip to content

Commit dbadf8c

Browse files
domenicdomenic
authored
Add a --remote option to run the full build remotely
1 parent 24db54a commit dbadf8c

File tree

2 files changed

+165
-19
lines changed

2 files changed

+165
-19
lines changed

‎README.md

+28-6
Original file line numberDiff line numberDiff line change
@@ -12,19 +12,29 @@ Then, clone this ([html-build](https://github.com/whatwg/html-build)) repo:
1212
git clone https://github.com/whatwg/html-build.git && cd html-build
1313
```
1414

15-
You then have a decision to make as to how you want to do your builds: locally, on your computer, or using a [Docker](https://www.docker.com/) container. We suggest going the Docker route if and only if you are already comfortable with Docker.
15+
## Performing a build
1616

17-
## Building locally
17+
You have a decision to make as to how you want to do your builds:
1818

19-
### Prerequisites
19+
- Locally on your computer
20+
- Remotely, using the [build server](https://github.com/whatwg/build.whatwg.org)
21+
- Using a [Docker](https://www.docker.com/) container
22+
23+
Local builds will be fastest, but require installing a lot of prerequisites. Using the build server is easiest, but slowest. Docker has speed close to a local build, and only requires Docker as a prerequisite.
24+
25+
### Building locally
26+
27+
#### Prerequisites
2028

2129
To build locally, you'll need the following commands installed on your system:
2230

2331
- `curl`, `grep`, `perl`, `unzip`
2432

25-
Optionally, for faster builds, you can install [Wattsi](https://github.com/whatwg/wattsi). If you don't bother with that, the build will use [Wattsi Server](https://github.com/whatwg/build.whatwg.org), which requires an internet connection. If you do use a local build of Wattsi, you'll likely also want Python 3.7+ with [pipx](https://pypa.github.io/pipx/), to enable syntax highlighting of `pre` contents.
33+
Optionally, for faster builds, you can install [Wattsi](https://github.com/whatwg/wattsi). If you don't bother with that, we will use the [build server](https://github.com/whatwg/build.whatwg.org), which requires an internet connection.
2634

27-
### Running the build
35+
If you're using a local install of Wattsi, then optionally, you can install Python 3.7+ with [pipx](https://pypa.github.io/pipx/), to enable syntax highlighting of `pre` contents.
36+
37+
#### Running the build
2838

2939
Run the `build.sh` script from inside your `html-build` working directory, like this:
3040

@@ -40,7 +50,19 @@ You may also set the environment variable `$HTML_SOURCE` to use a custom locatio
4050
HTML_SOURCE=~/hacks/dhtml ./build.sh
4151
```
4252

43-
## Building using a Docker container
53+
### Building using the build server
54+
55+
To use the build server, use the `--remote` flag:
56+
57+
```bash
58+
./build.sh --remote
59+
```
60+
61+
This will ZIP up most of the files in the `html/` directory, send them to the build server, get back another ZIP file with the output, and unzip those into the output folder.
62+
63+
You will need `zip` and `unzip` commands available in your `$PATH`.
64+
65+
### Building using a Docker container
4466

4567
The Dockerized version of the build allows you to run the build entirely inside a "container" (lightweight virtual machine). This includes tricky dependencies like a local copy of Wattsi and Python.
4668

‎build.sh

+137-13
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ DO_LINT=true
1919
DO_HIGHLIGHT=true
2020
SINGLE_PAGE_ONLY=false
2121
USE_DOCKER=false
22+
USE_SERVER=false
2223
VERBOSE=false
2324
QUIET=false
2425
SERVE=false
@@ -66,17 +67,28 @@ function main {
6667

6768
clearCacheIfNecessary
6869

70+
HTML_GIT_DIR="$HTML_SOURCE/.git/"
71+
HTML_SHA=${SHA_OVERRIDE:-$(git --git-dir="$HTML_GIT_DIR" rev-parse HEAD)}
72+
6973
if [[ $USE_DOCKER == "true" ]]; then
7074
doDockerBuild
7175
exit 0
7276
fi
7377

78+
if [[ $USE_SERVER == "true" ]]; then
79+
doServerBuild
80+
81+
if [[ $SERVE == "true" ]]; then
82+
cd "$HTML_OUTPUT"
83+
python3 -m http.server "$SERVE_PORT"
84+
fi
85+
86+
exit 0
87+
fi
88+
7489
checkWattsi
7590
ensureHighlighterInstalled
7691

77-
HTML_GIT_DIR="$HTML_SOURCE/.git/"
78-
HTML_SHA=${SHA_OVERRIDE:-$(git --git-dir="$HTML_GIT_DIR" rev-parse HEAD)}
79-
8092
doLint
8193

8294
updateRemoteDataFiles
@@ -98,7 +110,7 @@ function main {
98110
else
99111
echo ""
100112
echo "Skipping review draft production as the .git directory is not present"
101-
echo "(This always happens if you use the --docker option.)"
113+
echo "(This always happens if you use the --docker or --remote options.)"
102114
fi
103115

104116
$QUIET || echo
@@ -132,6 +144,7 @@ function processCommandLineArgs {
132144
echo
133145
echo "Build options:"
134146
echo " -d|--docker Use Docker to build in a container."
147+
echo " -r|--remote Use the build server."
135148
echo " -s|--serve After building, serve the results on http://localhost:$SERVE_PORT."
136149
echo " -n|--no-update Don't update before building; just build."
137150
echo " -l|--no-lint Don't lint before building; just build."
@@ -163,6 +176,9 @@ function processCommandLineArgs {
163176
-d|--docker)
164177
USE_DOCKER=true
165178
;;
179+
-r|--remote)
180+
USE_SERVER=true
181+
;;
166182
-q|--quiet)
167183
QUIET=true
168184
VERBOSE=false
@@ -179,6 +195,11 @@ function processCommandLineArgs {
179195
;;
180196
esac
181197
done
198+
199+
if [[ $USE_DOCKER == "true" && $USE_SERVER == "true" ]]; then
200+
echo "Error: --docker and --remote are mutually exclusive."
201+
exit 1
202+
fi
182203
}
183204

184205
# Checks if the html-build repository is up to date
@@ -472,6 +493,95 @@ function doDockerBuild {
472493
"${DOCKER_RUN_ARGS[@]}"
473494
}
474495

496+
# Performs the build using the build server, zipping up the input, sending it to the server, and
497+
# unzipping the output.
498+
# Output: the $HTML_OUTPUT directory will contain the built files
499+
function doServerBuild {
500+
clearDir "$HTML_TEMP"
501+
502+
local input_zip="build-server-input.zip"
503+
local build_server_output="build-server-output"
504+
local build_server_headers="build-server-headers.txt"
505+
506+
# Keep include list in sync with `processSource`
507+
#
508+
# We use an allowlist (--include) instead of a blocklist (--exclude) to avoid accidentally
509+
# sending files that the user might not anticipate sending to a remote server, e.g. their
510+
# private-notes-on-current-pull-request.txt.
511+
#
512+
# The contents of fonts/, images/, and dev/ are not round-tripped to the server, but instead
513+
# copied below in this function. (We still send the directories to avoid the build script on the
514+
# server getting confused about their absence.) demos/ needs to be sent in full for inlining.
515+
local zip_args=(
516+
--recurse-paths "$HTML_TEMP/$input_zip" . \
517+
--include ./source ./404.html ./link-fixup.js ./html-dfn.js ./styles.css \
518+
./fonts/ ./images/ ./dev/ ./demos/\*
519+
)
520+
$QUIET && zip_args+=( --quiet )
521+
(cd "$HTML_SOURCE" && zip "${zip_args[@]}")
522+
523+
local query_params=()
524+
$QUIET && query_params+=( quiet )
525+
$VERBOSE && query_params+=( verbose )
526+
$DO_UPDATE || query_params+=( no-update )
527+
$DO_LINT || query_params+=( no-lint )
528+
$DO_HIGHLIGHT || query_params+=( no-highlight )
529+
$SINGLE_PAGE_ONLY && query_params+=( single-page )
530+
531+
$QUIET || echo
532+
$QUIET || echo "Sending files to the build server..."
533+
534+
local query_string
535+
query_string="$(joinBy "\&" "${query_params[@]-''}")"
536+
local curl_url="https://build.whatwg.org/html-build?${query_string}"
537+
local curl_args=( "$curl_url" \
538+
--form "html=@$HTML_TEMP/$input_zip" \
539+
--form "sha=$HTML_SHA" \
540+
--dump-header "$HTML_TEMP/$build_server_headers" \
541+
--output "$HTML_TEMP/$build_server_output" )
542+
if [[ "$VERBOSE" == "true" ]]; then
543+
curl_args+=( --verbose )
544+
elif [[ "$QUIET" == "true" ]]; then
545+
curl_args+=( --silent )
546+
fi
547+
curl "${curl_args[@]}"
548+
549+
# Read exit code from the Exit-Code header and assume failure if not found
550+
local build_server_result=1
551+
while IFS=":" read -r NAME VALUE; do
552+
shopt -s nocasematch
553+
if [[ $NAME == "Exit-Code" ]]; then
554+
build_server_result=$(echo "$VALUE" | tr -d ' \r\n')
555+
break
556+
fi
557+
shopt -u nocasematch
558+
done < "$HTML_TEMP/$build_server_headers"
559+
560+
if [[ $build_server_result != "0" ]]; then
561+
cat "$HTML_TEMP/$build_server_output"
562+
exit "$build_server_result"
563+
else
564+
local unzip_args=()
565+
# Note: Don't use the -v flag; it doesn't work in combination with -d
566+
if [[ "$VERBOSE" == "false" ]]; then
567+
unzip_args+=( -qq )
568+
fi
569+
unzip_args+=( "$HTML_TEMP/$build_server_output" -d "$HTML_OUTPUT" )
570+
unzip "${unzip_args[@]}"
571+
cp -pR "$HTML_SOURCE/fonts" "$HTML_OUTPUT"
572+
cp -pR "$HTML_SOURCE/images" "$HTML_OUTPUT"
573+
574+
if [[ "$SINGLE_PAGE_ONLY" == "false" ]]; then
575+
cp -pR "$HTML_SOURCE/dev" "$HTML_OUTPUT"
576+
fi
577+
578+
$QUIET || echo
579+
$QUIET || echo "Build server output:"
580+
cat "$HTML_OUTPUT/output.txt"
581+
rm "$HTML_OUTPUT/output.txt"
582+
fi
583+
}
584+
475585
# Clears the $HTML_CACHE directory if the build tools have been updated since last run.
476586
# Arguments: none
477587
# Output:
@@ -572,6 +682,8 @@ function processSource {
572682
exit "$WATTSI_RESULT"
573683
fi
574684

685+
# Keep the list of files copied from $HTML_SOURCE in sync with `doServerBuild`
686+
575687
if [[ $BUILD_TYPE == "default" ]]; then
576688
# Singlepage HTML
577689
mv "$HTML_TEMP/wattsi-output/index-html" "$HTML_OUTPUT/index.html"
@@ -664,16 +776,16 @@ function runWattsi {
664776
$QUIET || echo
665777
$QUIET || echo "Local wattsi not present; trying the build server..."
666778

667-
CURL_URL="https://build.whatwg.org/wattsi"
668-
if [[ "$QUIET" == "true" && "$SINGLE_PAGE_ONLY" == "true" ]]; then
669-
CURL_URL="$CURL_URL?quiet&single-page-only"
670-
elif [[ "$QUIET" == "true" ]]; then
671-
CURL_URL="$CURL_URL?quiet"
672-
elif [[ "$SINGLE_PAGE_ONLY" == "true" ]]; then
673-
CURL_URL="$CURL_URL?single-page-only"
674-
fi
675779

676-
CURL_ARGS=( "$CURL_URL" \
780+
local query_params=()
781+
$QUIET && query_params+=( quiet )
782+
$SINGLE_PAGE_ONLY && query_params+=( single-page-only )
783+
784+
local query_string
785+
query_string="$(joinBy "\&" "${query_params[@]}")"
786+
local curl_url="https://build.whatwg.org/wattsi?${query_string}"
787+
788+
CURL_ARGS=( "$curl_url" \
677789
--form "source=@$1" \
678790
--form "sha=$HTML_SHA" \
679791
--form "build=$BUILD_TYPE" \
@@ -754,4 +866,16 @@ function clearDir {
754866
find "$1" -mindepth 1 -delete
755867
}
756868

869+
# Joins parameters $2 onward with the separator given in $1
870+
# Arguments:
871+
# - $1: the separator string
872+
# - $2...: the strings to join
873+
# Output: echoes the joined string
874+
function joinBy {
875+
local d=${1-} f=${2-}
876+
if shift 2; then
877+
printf %s "$f" "${@/#/$d}"
878+
fi
879+
}
880+
757881
main "$@"

0 commit comments

Comments
 (0)