Fix SDK Docker build scripts, update Emscripten (#172)

* Refresh and fix SDK Docker image build scripts

The way the script is currently it does not work (anymore). I'm aware
of two issues with the current setup:

1. emsdk underneath installs and uses Node JS and the version of Node JS
   is not pinned by the script and isn't tied to the version of the SDK
   used.

   As a result, emsdk moved to a newer version of Node JS than the one
   that was probably used when the scripts were created. This new
   version (it seems, the version currently being used is 18.20.3)
   requires a relatively fresh version of glibc which just isn't
   available in Ubuntu Bionic used as a base image. That results in
   obscure errors from CMake (see
   #170)
2. for whatever reason, the version of the protobuf static libraries
   currently in the repo, don't seem to work. I don't really know
   how the issue happened, but there were at least 2 users that faced
   the problem (see
   #161).

To resolve the first issue, we have a bunch of options:

1. Fix the version of emsdk
2. Fix the version of Node JS
3. Update the version of glibc

I figured I can combine options 1 and 3 together for the following
reasons:

1. Fixing the version of emsdk fixes the problem
2. The problem arised from the fact that the versions of software
   used in the build script are a bit old, so an update might be
   in order, even though we have other solutions to the problem.

> NOTE: It's my understanding that fixing emsdk version should also
> pin Node JS version, so if we deploy option 1, option 2 seem
> redundant.

For the second problem, I think there is only one ultimate solution
- not store binary artifacts in the repository and instead build
them from the sources in the repo.

It seems that in the past there was a concern that building protobuf
libraries takes a long time - it's still certainly the case. However,
I think we can compensate for that in two ways:

1. Drop WAVM - it does not seem like WAVM is still needed (the project
   itself appears to be dead and hasn't had any updates for at least 2
   years), but also one of the comments in
   #158, that
   removed the WAVM from the docs, also suggests that three doesn't
   seem to be a good reason to keep WAVM in the SDK build script.
2. Take advantage of potential hardware threads in the system when
   calling make - most laptops or servers these days have multiple
   cores, so if we have to build protobuf libraries twice, we can
   speed it up by using more cores.

With all those changes made, the build time adds up to something like
32 minutes on my laptop, compared to the 48m without building
protobuf libraries from sources (and without adding -j option to make).
So I think the additional time spent on building protobuf library is
compensated by other changes.

Signed-off-by: Mikhail Krinkin <[email protected]>

* Update emsdk version to 3.1.67 in Bazel configuration

This CL mostly follows the instructions in
https://github.com/emscripten-core/emsdk/blob/main/bazel/README.md.
Even so, there are a few things that are worth mentioning:

1. I tested the changes locally and in my tests
   incompatible_enable_cc_toolchain_resolution bazel flag made no
   difference (e.g. everything builds with or without the flag);
   I still keep it though because instructions say that it should
   be there and it does not cause any harm.
2. I don't think that we still need to patch emsdk to exclude npm
   modules from the toolchain, because it appears that upstream
   already removed those as a toolchain dependency in
   emscripten-core/emsdk#1045.

It's worth noting, that even though I don't think that emsdk patch
is still needed, I actually wasn't able to reproduce the problem
reported in #149
locally without the emsdk.patch even with the current version of
emsdk used by C++ SDK.

Signed-off-by: Mikhail Krinkin <[email protected]>

* Update building.md to match updated sdk_container.sh

This is a followup to the previous commits that refreshed the build
scripts and emsdk. This change reconciles the docs with the current
state of the build scripts.

Signed-off-by: Mikhail Krinkin <[email protected]>

---------

Signed-off-by: Mikhail Krinkin <[email protected]>

Author: krinkinmu

.bazelrc

@@ -0,0 +1,9 @@
+# NOTE: incompatible_enable_cc_toolchain_resolution is set by default on Bazel
+# versions 7.0+, see https://github.com/bazelbuild/bazel/issues/7260.
+#
+# emsdk documentation asks to set this flag (see
+# https://github.com/emscripten-core/emsdk/blob/main/bazel/README.md). And even
+# though things seem to work even without this flag, given that this flag
+# enables improved Bazel C++ toolchain resolution method that became the
+# default in newer Bazel version and does not cause problems, we keep it.
+build --incompatible_enable_cc_toolchain_resolution

Dockerfile-sdk

@@ -1,7 +1,7 @@
-FROM ubuntu:bionic
+FROM ubuntu:noble
 
 COPY ./sdk_container.sh /
 COPY ./build_wasm.sh /
-COPY *.cc *.h *.js *.proto Makefile* *.a /sdk/
+COPY *.cc *.h *.js *.proto Makefile* /sdk/
 
 RUN ./sdk_container.sh

Makefile

@@ -12,11 +12,10 @@ proxy_wasm_intrinsics_lite.pb.h struct_lite.pb.h: proxy_wasm_intrinsics_lite.pro
 	protoc --cpp_out=. struct_lite.proto
 
 ${CPP_API}/libprotobuf.a ${CPP_API}/libprotobuf-lite.a:
-	rm -rf protobuf-wasm && git clone https://github.com/protocolbuffers/protobuf protobuf-wasm \
-	&& cd protobuf-wasm && git checkout v3.9.1 \
-	&& rm -rf wasm-patches && git clone https://github.com/kwonoj/protobuf-wasm wasm-patches \
-	&& cd wasm-patches && git checkout 4bba8b2f38b5004f87489642b6ca4525ae72fe7f \
-	&& cd .. && git apply wasm-patches/*.patch \
+	rm -rf protobuf-wasm \
+	&& git clone https://github.com/protocolbuffers/protobuf protobuf-wasm \
+	&& cd protobuf-wasm \
+	&& git checkout v3.9.1 \
 	&& ./autogen.sh \
 	&& emconfigure ./configure --disable-shared CXXFLAGS="-O3 -flto" \
 	&& emmake make \

bazel/dependencies_extra.bzl

@@ -13,7 +13,9 @@
 #  limitations under the License.
 
 load("@emsdk//:emscripten_deps.bzl", "emscripten_deps")
+load("@emsdk//:toolchains.bzl", "register_emscripten_toolchains")
 
 # Requires proxy_wasm_cpp_sdk_dependencies() to be loaded first.
 def proxy_wasm_cpp_sdk_dependencies_extra():
     emscripten_deps()
+    register_emscripten_toolchains()

bazel/emsdk.patch

@@ -19,12 +19,9 @@ def proxy_wasm_cpp_sdk_repositories():
     maybe(
         http_archive,
         name = "emsdk",
-        sha256 = "1ca0ff918d476c55707bb99bc0452be28ac5fb8f22a9260a8aae8a38d1bc0e27",
-        # v3.1.7 with Bazel fixes
-        strip_prefix = "emsdk-0ea8f8a8707070e9a7c83fbb4a3065683bcf1799/bazel",
-        url = "https://github.com/emscripten-core/emsdk/archive/0ea8f8a8707070e9a7c83fbb4a3065683bcf1799.tar.gz",
-        patches = ["@proxy_wasm_cpp_sdk//bazel:emsdk.patch"],
-        patch_args = ["-p2"],
+        sha256 = "0cb0eabd6e3ceb1a970a2363e67f2b1689c2d83fbeae1e75901213c1f84de2e2",
+        strip_prefix = "emsdk-3.1.67/bazel",
+        url = "https://github.com/emscripten-core/emsdk/archive/refs/tags/3.1.67.tar.gz",
     )
 
     maybe(

bazel/repositories.bzl

@@ -163,9 +163,10 @@ sudo make install
 ```bash
 git clone https://github.com/emscripten-core/emsdk.git
 cd emsdk
-./emsdk update-tags
-./emsdk install 3.1.7
-./emsdk activate 3.1.7
+git checkout 3.1.67
+
+./emsdk install 3.1.67
+./emsdk activate 3.1.67
 
 source ./emsdk_env.sh
 ```
@@ -178,21 +179,19 @@ It is possible later versions will work, e.g.
 ./emsdk activate latest
 ```
 
-However 3.1.7 is known to work.
+However 3.1.67 is known to work.
 
 ### Rebuilding the libprotobuf.a files
 
-If want to rebuild the libprotobuf.a files using a version of protobuf prior to
-3.15, see the instructions at https://github.com/kwonoj/protobuf-wasm. Commit
-4bba8b2f38b5004f87489642b6ca4525ae72fe7f works for protobuf v3.9.x.
+To build the protobuf static libraries for use in your proxy-wasm wasm module,
+if you need them, you may use Emscripten in the same way sdk\_container.sh does
+it:
 
 ```bash
 git clone https://github.com/protocolbuffers/protobuf protobuf-wasm
 cd protobuf-wasm
 git checkout v3.9.1
-git clone https://github.com/kwonoj/protobuf-wasm wasm-patches
-cd wasm-patches && git checkout 4bba8b2f38b5004f87489642b6ca4525ae72fe7f && cd ..
-git apply wasm-patches/*.patch
+git submodule update --init --recursive
 ./autogen.sh
 emconfigure ./configure --disable-shared CXXFLAGS="-O3 -flto"
 emmake make

docs/building.md

@@ -20,16 +20,20 @@ set -e
 export DEBIAN_FRONTEND=noninteractive
 apt-get update
 apt-get upgrade -y
-apt-get install -y --no-install-recommends apt-utils ca-certificates
 apt-get autoremove -y
 apt-get clean
-apt-get install -y --no-install-recommends software-properties-common apt-transport-https git wget curl pkg-config autoconf autotools-dev automake libtool cmake python zlib1g-dev
+apt-get install -y --no-install-recommends ca-certificates git autoconf autotools-dev automake libtool cmake python-is-python3 zlib1g-dev make xz-utils libzstd-dev
 
-# gcc-7
-apt-get install -y --no-install-recommends gcc-7 g++-7 cpp-7
-export CC=gcc-7
-export CXX=g++-7
-export CPP=cpp-7
+# The specific version of GCC does not actually matter as long as it's confirmed to work.
+# That's why we explicitly pin gcc version (in this case to gcc 13) - it's the version
+# which was tested to work.
+apt-get install -y --no-install-recommends gcc-13 g++-13 cpp-13
+export CC=gcc-13
+export CXX=g++-13
+export CPP=cpp-13
+
+NUM_CPUS=$(nproc)
+JOBS=$((NUM_CPUS>1 ? NUM_CPUS-1 : NUM_CPUS))
 
 # get $HOME
 cd
@@ -41,36 +45,43 @@ git checkout v3.9.1
 git submodule update --init --recursive
 ./autogen.sh
 ./configure
-make
+make -j $JOBS
 make check
 make install
 cd
 rm -rf protobuf
 
+# This makes sure that installed dynamic libraries are visible to the dynamic
+# linker, because it seems like make install does not take care of that
+ldconfig
+
 # emscripten
 git clone https://github.com/emscripten-core/emsdk.git
 cd emsdk
-./emsdk update-tags
-./emsdk install 3.1.7
-./emsdk activate 3.1.7
+git checkout 3.1.67
+./emsdk install --shallow 3.1.67
+./emsdk activate 3.1.67
 source ./emsdk_env.sh
 cd
 
+git clone https://github.com/protocolbuffers/protobuf protobuf-wasm
+cd protobuf-wasm
+git checkout v3.9.1
+git submodule update --init --recursive
+./autogen.sh
+emconfigure ./configure --disable-shared CXXFLAGS="-O3 -flto"
+emmake make -j $JOBS
+cd
+
+cp protobuf-wasm/src/.libs/libprotobuf-lite.a /sdk/libprotobuf-lite.a
+cp protobuf-wasm/src/.libs/libprotobuf.a /sdk/libprotobuf.a
+rm -rf protobuf-wasm
+
 # abseil (optional)
 git clone https://github.com/abseil/abseil-cpp
 cd abseil-cpp
-git checkout 14550beb3b7b97195e483fb74b5efb906395c31e -b Jul302019 # Jul 30 2019
+git checkout 4447c7562e3bc702ade25105912dce503f0c4010 -b lts20240722 # Abseil LTS release 20240722.0
 emcmake cmake -DCMAKE_CXX_STANDARD=17 "."
-emmake make
+emmake make -j $JOBS
 cd
 
-# WAVM (optional)
-apt-get install -y --no-install-recommends llvm-6.0-dev
-git clone https://github.com/WAVM/WAVM
-cd WAVM
-git checkout 1ec06cd202a922015c9041c5ed84f875453c4dc7 -b Oct152019 # Oct 15 2019
-cmake "."
-make
-make install
-cd
-rm -rf WAVM