Skip to content

Commit 7dcefb5

Browse files
h4lh4l
authored
Arm64 Docker support (ConsenSysDiligence#1770)
* Rework docker image to add arm64 support The mythril/myth image now builds for linux/arm64 as well as linux/amd64. To achieve this, we now use docker buildx to build the images and handle create a multi-platform image manifest. The build config is defined in a buildx bake file. By default it'll build both platforms at once, but you can build just one by overriding the platform on the command line: $ docker buildx bake --set='*.platform=linux/arm64' The solcx Python package doesn't support downloading solc for arm64, so the image now includes the svm command-line tool, which does. (svm is used by foundry to provide solc versions.) Integration with solcx is not automatic, so currently the image's docker-entrypoint.sh handles symlinking solc versions from svm into solcx's directory. In addition to supporting arm64, the image is now quite a bit smaller. ~400M vs 1.3G before. * Update docker image build script for new image * Remove the z3-solver pip platform hack When installing wheels in the Docker image, we previously used an ugly hack to force pip to install the z3-solver wheel, despite it having invalid platform metadata. Instead of bodging pip install in this way, we now fix the z3-solver wheel's metadata after building it, using `auditwheel addtag` to infer and apply compatible platform metadata, which allows pip to install the wheel normally.
1 parent d531d8b commit 7dcefb5

File tree

6 files changed

+246
-56
lines changed

6 files changed

+246
-56
lines changed

‎.dockerignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
/.*
2+
/build
3+
/docker-bake.hcl
4+
/Dockerfile

‎Dockerfile

Lines changed: 138 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,51 +1,143 @@
1-
FROM ubuntu:focal
2-
3-
ARG DEBIAN_FRONTEND=noninteractive
4-
5-
# Space-separated version string without leading 'v' (e.g. "0.4.21 0.4.22")
6-
ARG SOLC
7-
8-
RUN apt-get update \
9-
&& apt-get install -y \
10-
libsqlite3-0 \
11-
libsqlite3-dev \
12-
&& apt-get install -y \
13-
apt-utils \
14-
build-essential \
15-
locales \
16-
python-pip-whl \
17-
python3-pip \
18-
python3-setuptools \
19-
software-properties-common \
20-
&& add-apt-repository -y ppa:ethereum/ethereum \
21-
&& apt-get update \
22-
&& apt-get install -y \
23-
solc \
24-
libssl-dev \
25-
python3-dev \
26-
pandoc \
27-
git \
28-
wget \
29-
&& ln -s /usr/bin/python3 /usr/local/bin/python
30-
31-
COPY ./requirements.txt /opt/mythril/requirements.txt
32-
33-
RUN cd /opt/mythril \
34-
&& pip3 install -r requirements.txt
35-
36-
RUN locale-gen en_US.UTF-8
37-
ENV LANG en_US.UTF-8
38-
ENV LANGUAGE en_US.en
39-
ENV LC_ALL en_US.UTF-8
40-
41-
COPY . /opt/mythril
42-
RUN cd /opt/mythril \
43-
&& python setup.py install
1+
# syntax=docker/dockerfile:1
2+
ARG PYTHON_VERSION=3.10
3+
ARG INSTALLED_SOLC_VERSIONS
444

5+
6+
FROM python:${PYTHON_VERSION:?} AS python-wheel
7+
WORKDIR /wheels
8+
9+
10+
FROM python-wheel AS python-wheel-with-cargo
11+
# Enable cargo sparse-registry to prevent it using large amounts of memory in
12+
# docker builds, and speed up builds by downloading less.
13+
# https://github.com/rust-lang/cargo/issues/10781#issuecomment-1163819998
14+
ENV CARGO_UNSTABLE_SPARSE_REGISTRY=true
15+
16+
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
17+
ENV PATH=/root/.cargo/bin:$PATH
18+
19+
20+
# z3-solver needs to build from src on arm, and it takes a long time, so
21+
# building it in a separate stage helps parallelise the build and helps it stay
22+
# in the build cache.
23+
FROM python-wheel AS python-wheel-z3-solver
24+
RUN pip install auditwheel
25+
RUN --mount=source=requirements.txt,target=/run/requirements.txt \
26+
pip wheel "$(grep z3-solver /run/requirements.txt)"
27+
# The wheel z3-solver builds does not install in arm64 because it generates
28+
# incorrect platform compatibility metadata for arm64 builds. (It uses the
29+
# platform manylinux1_aarch64 but manylinux1 is only defined for x86 systems,
30+
# not arm: https://peps.python.org/pep-0600/#legacy-manylinux-tags). To work
31+
# around this, we use pypa's auditwheel tool to infer and apply a compatible
32+
# platform tag.
33+
RUN ( auditwheel addtag ./z3_solver-* \
34+
# replace incorrect wheel with the re-tagged one
35+
&& rm ./z3_solver-* && mv wheelhouse/z3_solver-* . ) \
36+
# addtag exits with status 1 if no tags need adding, which is fine
37+
|| true
38+
39+
40+
FROM python-wheel-with-cargo AS python-wheel-blake2b
41+
# blake2b-py doesn't publish ARM builds, and also don't publish source packages
42+
# on PyPI (other than the old 0.1.3 version) so we need to build from from a git
43+
# tag. They do publish binaries for linux amd64, but their binaries only support
44+
# certain platform versions and the amd64 python image isn't supported, so we
45+
# have to build from src for that as well.
46+
47+
# Try to get a binary build or a source release on PyPI first, then fall back
48+
# to building from the git repo.
49+
RUN pip wheel 'blake2b-py>=0.2.0,<1' \
50+
|| pip wheel git+https://github.com/ethereum/[email protected]
51+
52+
53+
FROM python-wheel AS mythril-wheels
54+
# cython is needed to build some wheels, such as cytoolz
55+
RUN pip install cython
56+
RUN --mount=source=requirements.txt,target=/run/requirements.txt \
57+
# ignore blake2b and z3-solver as we've already built them
58+
grep -v -e blake2b -e z3-solver /run/requirements.txt > /tmp/requirements-remaining.txt
59+
RUN pip wheel -r /tmp/requirements-remaining.txt
60+
61+
COPY . /mythril
62+
RUN pip wheel --no-deps /mythril
63+
64+
COPY --from=python-wheel-blake2b /wheels/blake2b* /wheels
65+
COPY --from=python-wheel-z3-solver /wheels/z3_solver* /wheels
66+
67+
68+
# Solidity Compiler Version Manager. This provides cross-platform solc builds.
69+
# It's used by foundry to provide solc. https://github.com/roynalnaruto/svm-rs
70+
FROM python-wheel-with-cargo AS solidity-compiler-version-manager
71+
RUN cargo install svm-rs
72+
# put the binaries somewhere obvious for later stages to use
73+
RUN mkdir -p /svm-rs/bin && cd ~/.cargo/bin/ && cp svm solc /svm-rs/bin/
74+
75+
76+
FROM python:${PYTHON_VERSION:?}-slim AS myth
77+
ARG PYTHON_VERSION
78+
# Space-separated version string without leading 'v' (e.g. "0.4.21 0.4.22")
79+
ARG INSTALLED_SOLC_VERSIONS
80+
81+
COPY --from=solidity-compiler-version-manager /svm-rs/bin/* /usr/local/bin/
82+
83+
RUN --mount=from=mythril-wheels,source=/wheels,target=/wheels \
84+
export PYTHONDONTWRITEBYTECODE=1 && pip install /wheels/*.whl
85+
86+
RUN adduser --disabled-password mythril
87+
USER mythril
4588
WORKDIR /home/mythril
4689

47-
RUN ( [ ! -z "${SOLC}" ] && set -e && for ver in $SOLC; do python -m solc.install v${ver}; done ) || true
90+
# pre-install solc versions
91+
RUN set -x; [ -z "${INSTALLED_SOLC_VERSIONS}" ] || svm install ${INSTALLED_SOLC_VERSIONS}
92+
93+
COPY --chown=mythril:mythril \
94+
./mythril/support/assets/signatures.db \
95+
/home/mythril/.mythril/signatures.db
96+
97+
COPY --chown=root:root --chmod=755 ./docker/docker-entrypoint.sh /
98+
COPY --chown=root:root --chmod=755 \
99+
./docker/sync-svm-solc-versions-with-solcx.sh \
100+
/usr/local/bin/sync-svm-solc-versions-with-solcx
101+
ENTRYPOINT ["/docker-entrypoint.sh"]
102+
103+
104+
# Basic sanity checks to make sure the build is functional
105+
FROM myth AS myth-smoke-test-execution
106+
SHELL ["/bin/bash", "-euo", "pipefail", "-c"]
107+
WORKDIR /smoke-test
108+
COPY --chmod=755 <<"EOT" /smoke-test.sh
109+
#!/usr/bin/env bash
110+
set -x -euo pipefail
111+
112+
# Check solcx knows about svm solc versions
113+
svm install 0.5.0
114+
sync-svm-solc-versions-with-solcx
115+
python -c '
116+
import solcx
117+
print("\n".join(str(v) for v in solcx.get_installed_solc_versions()))
118+
' | grep -P '^0\.5\.0$' || {
119+
echo "solcx did not report svm-installed solc version";
120+
exit 1
121+
}
122+
123+
# Check myth can run
124+
myth version
125+
myth function-to-hash 'function transfer(address _to, uint256 _value) public returns (bool success)'
126+
myth analyze /solidity_examples/timelock.sol > timelock.log || true
127+
grep 'SWC ID: 116' timelock.log || {
128+
error "Failed to detect SWC ID: 116 in timelock.sol";
129+
exit 1
130+
}
131+
132+
# Check that the entrypoint works
133+
[[ $(/docker-entrypoint.sh version) == $(myth version) ]]
134+
[[ $(/docker-entrypoint.sh echo hi) == hi ]]
135+
[[ $(/docker-entrypoint.sh bash -c "printf '>%s<' 'foo bar'") == ">foo bar<" ]]
136+
EOT
137+
138+
RUN --mount=source=./solidity_examples,target=/solidity_examples \
139+
/smoke-test.sh 2>&1 | tee smoke-test.log
48140

49-
COPY ./mythril/support/assets/signatures.db /home/mythril/.mythril/signatures.db
50141

51-
ENTRYPOINT ["/usr/local/bin/myth"]
142+
FROM scratch as myth-smoke-test
143+
COPY --from=myth-smoke-test-execution /smoke-test/* /

‎docker-bake.hcl

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
variable "REGISTRY" {
2+
default = "docker.io"
3+
}
4+
5+
variable "VERSION" {
6+
default = "dev"
7+
}
8+
9+
variable "PYTHON_VERSION" {
10+
default = "3.10"
11+
}
12+
13+
variable "INSTALLED_SOLC_VERSIONS" {
14+
default = "0.8.19"
15+
}
16+
17+
function "myth-tags" {
18+
params = [NAME]
19+
result = formatlist("${REGISTRY}/${NAME}:%s", split(",", VERSION))
20+
}
21+
22+
group "default" {
23+
targets = ["myth", "myth-smoke-test"]
24+
}
25+
26+
target "_myth-base" {
27+
target = "myth"
28+
args = {
29+
PYTHON_VERSION = PYTHON_VERSION
30+
INSTALLED_SOLC_VERSIONS = INSTALLED_SOLC_VERSIONS
31+
}
32+
platforms = [
33+
"linux/amd64",
34+
"linux/arm64"
35+
]
36+
}
37+
38+
target "myth" {
39+
inherits = ["_myth-base"]
40+
tags = myth-tags("mythril/myth")
41+
}
42+
43+
target "myth-dev" {
44+
inherits = ["_myth-base"]
45+
tags = myth-tags("mythril/myth-dev")
46+
}
47+
48+
target "myth-smoke-test" {
49+
inherits = ["_myth-base"]
50+
target = "myth-smoke-test"
51+
output = ["build/docker/smoke-test"]
52+
}

‎docker/docker-entrypoint.sh

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
# Install extra solc versions if SOLC is set
5+
if [[ ${SOLC:-} != "" ]]; then
6+
read -ra solc_versions <<<"${SOLC:?}"
7+
svm install "${solc_versions[@]}"
8+
fi
9+
# Always sync versions, as the should be at least one solc version installed
10+
# in the base image, and we may be running as root rather than the mythril user.
11+
sync-svm-solc-versions-with-solcx
12+
13+
# By default we run myth with options from arguments we received. But if the
14+
# first argument is a valid program, we execute that instead so that people can
15+
# run other commands without overriding the entrypoint (e.g. bash).
16+
if command -v "${1:-}" > /dev/null; then
17+
exec -- "$@"
18+
fi
19+
exec -- myth "$@"

‎docker/sync-svm-solc-versions-with-solcx.sh

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
# Let solcx know about the solc versions installed by svm.
5+
# We do this by symlinking svm's solc binaries into solcx's solc dir.
6+
[[ -e ~/.svm ]] || exit 0
7+
mkdir -p ~/.solcx
8+
readarray -t svm_solc_bins <<<"$(find ~/.svm -type f -name 'solc-*')"
9+
[[ ${svm_solc_bins[0]} != "" ]] || exit 0
10+
for svm_solc in "${svm_solc_bins[@]}"; do
11+
name=$(basename "${svm_solc:?}")
12+
version="${name#"solc-"}" # strip solc- prefix
13+
solcx_solc=~/.solcx/"solc-v${version:?}"
14+
if [[ ! -e $solcx_solc ]]; then
15+
ln -s "${svm_solc:?}" "${solcx_solc:?}"
16+
fi
17+
done

‎docker_build_and_deploy.sh

Lines changed: 16 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,29 @@
1-
#!/bin/sh
1+
#!/bin/bash
22

33
set -eo pipefail
44

55
NAME=$1
66

7+
if [[ ! $NAME =~ ^mythril/myth(-dev)?$ ]];
8+
then
9+
echo "Error: unknown image name: $NAME" >&2
10+
exit 1
11+
fi
12+
713
if [ ! -z $CIRCLE_TAG ];
814
then
9-
VERSION=${CIRCLE_TAG#?}
15+
GIT_VERSION=${CIRCLE_TAG#?}
1016
else
11-
VERSION=${CIRCLE_SHA1}
17+
GIT_VERSION=${CIRCLE_SHA1}
1218
fi
1319

14-
VERSION_TAG=${NAME}:${VERSION}
15-
LATEST_TAG=${NAME}:latest
16-
17-
docker build -t ${VERSION_TAG} .
18-
docker tag ${VERSION_TAG} ${LATEST_TAG}
20+
# Build and test all versions of the image. (The result will stay in the cache,
21+
# so the next build should be almost instant.)
22+
docker buildx bake myth-smoke-test
1923

2024
echo "$DOCKERHUB_PASSWORD" | docker login -u $DOCKERHUB_USERNAME --password-stdin
2125

22-
docker push ${VERSION_TAG}
23-
docker push ${LATEST_TAG}
26+
# strip mythril/ from NAME, e.g. myth or myth-dev
27+
BAKE_TARGET="${NAME#mythril/}"
28+
29+
VERSION="${GIT_VERSION:?},latest" docker buildx bake --push "${BAKE_TARGET:?}"

0 commit comments

Comments
 (0)