Skip to content

Commit e7e8082

Browse files
SteNicholasclaude
andcommitted
feat(build): support Linux aarch64 builds
Resolve the target architecture once and key the architecture specific build logic on it, replacing PAIMON_CPU_FLAG and PAIMON_ARMV8_ARCH, neither of which was ever defined. Arm64 therefore gets a real -march= flag where it previously got none. - TargetArchitecture.cmake derives PAIMON_TARGET_PROCESSOR and PAIMON_TARGET_CPU_FAMILY. It runs no compiler probe and reads nothing beyond its inputs, so cmake -P tests it without a toolchain. An unrecognized processor gets no tuning flags rather than an error. - PAIMON_AARCH64_MARCH selects the Arm64 -march= value, default armv8-a. DefineOptions.cmake repeats that default as a fallback, since the option exists only for a top-level build and an add_subdirectory() consumer would otherwise configure with a bare "-march=". - The SSE4.2 CRC32C kernel is removed: it computed Castagnoli, while the checksum persisted in the SST block trailer and the B-tree global index is zlib CRC-32. No on-disk value changes, since the kernel hung off the never-defined PAIMON_SIMD_LEVEL and no build ever selected it. The tests now pin the class to values taken from zlib. - Lumina is prebuilt for linux-x86_64 only, so it now fails to configure elsewhere with an explicit error, and CI disables it off x86_64. - build_and_package.sh labels artifacts with the platform, so output/paimon-cpp.tar.gz becomes paimon-cpp-linux-x86_64.tar.gz. --platform overrides the label alone: cross compilation is not wired up end to end, so build natively on each target. - CI gains gcc-debug-arm64, an Arm64 release verification entry, and a script-tests job for the new CMake and packaging tests; all three are required checks. - docs: a supported platform matrix, and a rule against relying on the signedness of plain char. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 92e7e55 commit e7e8082

19 files changed

Lines changed: 796 additions & 102 deletions

.asf.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,8 @@ github:
5454
app_slug: -1
5555
- name: "rat-license-check"
5656
app_slug: -1
57+
- name: "script-tests"
58+
app_slug: -1
5759
- name: "asan-ubsan"
5860
app_slug: -1
5961
- name: "tsan"
@@ -66,6 +68,8 @@ github:
6668
app_slug: -1
6769
- name: "gcc-release"
6870
app_slug: -1
71+
- name: "gcc-debug-arm64"
72+
app_slug: -1
6973
- name: "gcc8-test"
7074
app_slug: -1
7175
pull_requests:

.github/workflows/build_and_test.yaml

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,23 @@ permissions:
3333
contents: read
3434

3535
jobs:
36+
script-tests:
37+
name: script-tests
38+
runs-on: ubuntu-24.04
39+
timeout-minutes: 10
40+
steps:
41+
- name: Checkout paimon-cpp
42+
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
43+
- name: Run CMake module tests
44+
shell: bash
45+
run: ci/scripts/test_cmake_modules.sh
46+
- name: Run packaging argument tests
47+
shell: bash
48+
run: ci/scripts/test_packaging_args.sh
49+
3650
build-and-test:
3751
name: ${{ matrix.name }}
38-
runs-on: ubuntu-24.04
52+
runs-on: ${{ matrix.runner || 'ubuntu-24.04' }}
3953
timeout-minutes: 120
4054
strategy:
4155
fail-fast: false
@@ -60,6 +74,10 @@ jobs:
6074
- name: tsan
6175
skip_rust: true
6276
build_args: --enable_tsan
77+
- name: gcc-debug-arm64
78+
runner: ubuntu-24.04-arm
79+
cc: gcc-14
80+
cxx: g++-14
6381
steps:
6482
- name: Checkout paimon-cpp
6583
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1

.github/workflows/release_candidate.yaml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,21 +99,26 @@ jobs:
9999
name: Verify source archive (${{ matrix.compiler }})
100100
if: github.ref_type == 'tag'
101101
needs: archive
102-
runs-on: ubuntu-24.04
102+
runs-on: ${{ matrix.runner || 'ubuntu-24.04' }}
103103
timeout-minutes: 180
104104
strategy:
105105
fail-fast: false
106106
matrix:
107107
compiler:
108108
- gcc-14
109109
- clang
110+
- gcc-14-arm64
110111
include:
111112
- compiler: gcc-14
112113
cc: gcc-14
113114
cxx: g++-14
114115
- compiler: clang
115116
cc: clang
116117
cxx: clang++
118+
- compiler: gcc-14-arm64
119+
cc: gcc-14
120+
cxx: g++-14
121+
runner: ubuntu-24.04-arm
117122
steps:
118123
- name: Checkout source
119124
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,8 @@ Paimon C++ currently provides:
4949
> Paimon C++ therefore currently treats the `bitmap` global index type as unsupported. The legacy
5050
> implementation remains in the codebase pending migration to the Java-compatible format.
5151
52-
Note: Only Linux x86_64 builds are currently supported and verified.
52+
Note: Linux `x86_64` and `aarch64` builds are supported and verified in CI. See the supported
53+
platform matrix in [docs/source/building.rst](docs/source/building.rst) for other platforms.
5354

5455
## Building
5556

build_and_package.sh

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ MAKE_CLEAN=false
2626
PACKAGE=false
2727
CMAKE_OPTIONS=()
2828
JOBS=""
29+
PACKAGE_PLATFORM=""
30+
PRINT_NAME=false
2931

3032
show_help() {
3133
cat << EOF
@@ -37,6 +39,12 @@ Options:
3739
-c, --clean Clean build directory before building
3840
-p, --package Package creation
3941
-j, --jobs <num> Number of parallel jobs for building (default: auto-detect)
42+
--platform <label>
43+
Platform label used in the package name (default: the host,
44+
e.g. linux-aarch64). This only labels the artifact; it does
45+
not configure a cross build. Must match
46+
[A-Za-z0-9][A-Za-z0-9._-]*, i.e. a single path component.
47+
--print-name Print the resolved package name and exit
4048
-h, --help Show this help message
4149
4250
CMake Options:
@@ -46,6 +54,7 @@ CMake Options:
4654
Examples:
4755
$0 -r -p -j 8 -DPAIMON_BUILD_SHARED=ON -DPAIMON_BUILD_STATIC=OFF
4856
$0 --debug --clean --package --jobs 4
57+
$0 -r -p --platform linux-aarch64
4958
5059
EOF
5160
}
@@ -82,6 +91,24 @@ while [[ $# -gt 0 ]]; do
8291
exit 1
8392
fi
8493
;;
94+
--platform)
95+
shift
96+
# The label becomes a path component of both the install prefix and
97+
# the tarball name, so it must stay a single safe component: the
98+
# pattern admits no separator and no shell metacharacter. Requiring a
99+
# leading letter or digit additionally stops a following option from
100+
# being swallowed as the value.
101+
if [[ $# -eq 0 || ! $1 =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]]; then
102+
echo "Error: --platform requires a label matching [A-Za-z0-9][A-Za-z0-9._-]*" >&2
103+
exit 1
104+
fi
105+
PACKAGE_PLATFORM="$1"
106+
shift
107+
;;
108+
--print-name)
109+
PRINT_NAME=true
110+
shift
111+
;;
85112
-h|--help)
86113
show_help
87114
exit 0
@@ -94,8 +121,27 @@ while [[ $# -gt 0 ]]; do
94121
esac
95122
done
96123

124+
# Defaults to the host. Note this only names the artifact: the bundled third-party
125+
# builds receive just CMAKE_C_COMPILER / CMAKE_CXX_COMPILER (see
126+
# EP_COMMON_TOOLCHAIN in cmake_modules/ThirdpartyToolchain.cmake) and no toolchain
127+
# file, sysroot or find-root, so a cross build is not wired up end to end.
128+
if [ -z "$PACKAGE_PLATFORM" ]; then
129+
HOST_OS=$(uname -s | tr '[:upper:]' '[:lower:]')
130+
if [ "$HOST_OS" = "darwin" ]; then
131+
HOST_OS="macos"
132+
fi
133+
PACKAGE_PLATFORM="$HOST_OS-$(uname -m)"
134+
fi
135+
BUILD_NAME="$BUILD_NAME-$PACKAGE_PLATFORM"
136+
137+
if [ "$PRINT_NAME" = true ]; then
138+
echo "$BUILD_NAME"
139+
exit 0
140+
fi
141+
97142
echo "========== Build Configuration =========="
98143
echo "Build Type: $BUILD_TYPE"
144+
echo "Package Platform: $PACKAGE_PLATFORM"
99145
echo "Package Name: $BUILD_NAME"
100146
echo "Clean Build: $MAKE_CLEAN"
101147
echo "Package: $PACKAGE"

ci/scripts/build_paimon.sh

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,12 @@ fi
136136
if [[ "${enable_tsan}" == "true" ]]; then
137137
ENABLE_TANTIVY="OFF" # Tantivy's Rust library is not TSAN-instrumented.
138138
fi
139+
# CI always builds natively, so the host architecture is the target architecture.
140+
host_arch=$(uname -m)
141+
if [[ "${host_arch}" != "x86_64" ]]; then
142+
ENABLE_LUMINA="OFF"
143+
echo "=== Lumina disabled: no prebuilt artifacts for ${host_arch} ==="
144+
fi
139145

140146
CMAKE_ARGS=(
141147
"-G Ninja"

ci/scripts/test_cmake_modules.sh

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/usr/bin/env bash
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
#
18+
# Runs the CMake module tests under cmake_modules/tests. These are script-mode
19+
# tests: they need neither a toolchain nor a configured build tree, so they stay
20+
# separate from ctest and run in seconds.
21+
22+
set -euo pipefail
23+
24+
source_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
25+
26+
status=0
27+
for test_script in "${source_dir}"/cmake_modules/tests/*_test.cmake; do
28+
echo "=== ${test_script##*/} ==="
29+
if ! cmake -P "${test_script}"; then
30+
status=1
31+
fi
32+
done
33+
34+
exit "${status}"

ci/scripts/test_packaging_args.sh

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
#!/usr/bin/env bash
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
#
18+
# Argument-handling tests for build_and_package.sh. They only exercise
19+
# --print-name, which resolves the package name and exits before any build step,
20+
# so no toolchain is needed.
21+
22+
# No `set -e`: every check runs the script under test and inspects its exit
23+
# status, so a non-zero status is data here, not a reason to abort.
24+
set -uo pipefail
25+
26+
source_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
27+
script="${source_dir}/build_and_package.sh"
28+
29+
checks=0
30+
failures=0
31+
32+
# A private directory: a fixed /tmp path would collide with a concurrent run and
33+
# could make the script delete a file it did not create.
34+
work_dir=$(mktemp -d)
35+
trap 'rm -rf "${work_dir}"' EXIT
36+
sentinel="${work_dir}/must-not-exist"
37+
38+
# --print-name is passed FIRST, never last: the arguments under test have to keep
39+
# their own position, otherwise a trailing --print-name would become the value of
40+
# a preceding --platform and the argument-exhaustion branch could never be
41+
# reached. Parsing is order independent, so leading it changes nothing else.
42+
expect_name() {
43+
local description=$1 expected=$2
44+
shift 2
45+
local actual
46+
checks=$((checks + 1))
47+
if ! actual=$("${script}" --print-name "$@" 2>&1); then
48+
failures=$((failures + 1))
49+
echo "FAIL ${description}: exited non-zero: ${actual}"
50+
elif [[ "${actual}" != "${expected}" ]]; then
51+
failures=$((failures + 1))
52+
echo "FAIL ${description}: expected '${expected}', got '${actual}'"
53+
fi
54+
}
55+
56+
# Asserting the diagnostic, not just a non-zero exit: the script runs under
57+
# `set -u`, so a missing validation branch would still abort -- with an "unbound
58+
# variable" crash instead of a usable message.
59+
expect_rejected() {
60+
local description=$1
61+
shift
62+
local output status
63+
checks=$((checks + 1))
64+
output=$("${script}" --print-name "$@" 2>&1)
65+
status=$?
66+
if [[ "${status}" -eq 0 ]]; then
67+
failures=$((failures + 1))
68+
echo "FAIL ${description}: expected a non-zero exit, but the value was accepted"
69+
elif [[ "${output}" != *"--platform requires a label"* ]]; then
70+
failures=$((failures + 1))
71+
echo "FAIL ${description}: expected the --platform diagnostic, got: ${output}"
72+
fi
73+
}
74+
75+
# The default label is derived from `uname`, so stub it on PATH to check the
76+
# derivation for hosts this machine is not, including the Darwin -> macos mapping.
77+
expect_host_default() {
78+
local description=$1 uname_s=$2 uname_m=$3 expected=$4
79+
shift 4
80+
local stub_dir actual status
81+
checks=$((checks + 1))
82+
stub_dir=$(mktemp -d "${work_dir}/stub.XXXXXX")
83+
cat > "${stub_dir}/uname" <<EOF
84+
#!/bin/sh
85+
case "\$1" in
86+
-s) echo "${uname_s}" ;;
87+
-m) echo "${uname_m}" ;;
88+
*) echo "unexpected uname argument: \$1" >&2; exit 1 ;;
89+
esac
90+
EOF
91+
chmod +x "${stub_dir}/uname"
92+
actual=$(PATH="${stub_dir}:${PATH}" "${script}" --print-name "$@" 2>&1)
93+
status=$?
94+
if [[ "${status}" -ne 0 ]]; then
95+
failures=$((failures + 1))
96+
echo "FAIL ${description}: exited ${status}: ${actual}"
97+
elif [[ "${actual}" != "${expected}" ]]; then
98+
failures=$((failures + 1))
99+
echo "FAIL ${description}: expected '${expected}', got '${actual}'"
100+
fi
101+
}
102+
103+
host_platform="$(uname -s | tr '[:upper:]' '[:lower:]')"
104+
if [[ "${host_platform}" == "darwin" ]]; then
105+
host_platform="macos"
106+
fi
107+
host_platform="${host_platform}-$(uname -m)"
108+
109+
expect_name "release defaults to the host platform" "paimon-cpp-${host_platform}"
110+
expect_name "debug keeps its own prefix" "paimon-cpp-debug-${host_platform}" --debug
111+
expect_name "release is the default build type" "paimon-cpp-${host_platform}" --release
112+
expect_name "--platform overrides the host" "paimon-cpp-linux-aarch64" --platform linux-aarch64
113+
expect_name "--platform applies to debug too" "paimon-cpp-debug-macos-arm64" -d --platform \
114+
macos-arm64
115+
expect_name "the last --platform wins" "paimon-cpp-linux-aarch64" --platform linux-x86_64 \
116+
--platform linux-aarch64
117+
118+
# The documented pattern admits dots, underscores and hyphens; keep that a contract.
119+
expect_name "a dotted label is accepted" "paimon-cpp-linux.arm64" --platform linux.arm64
120+
expect_name "an underscored label is accepted" "paimon-cpp-linux_musl-aarch64" --platform \
121+
linux_musl-aarch64
122+
123+
# Default labels for hosts other than this one.
124+
expect_host_default "linux aarch64 host" Linux aarch64 "paimon-cpp-linux-aarch64"
125+
expect_host_default "darwin arm64 host maps to macos" Darwin arm64 "paimon-cpp-macos-arm64"
126+
expect_host_default "darwin x86_64 host maps to macos" Darwin x86_64 "paimon-cpp-macos-x86_64"
127+
expect_host_default "debug on a linux aarch64 host" Linux aarch64 \
128+
"paimon-cpp-debug-linux-aarch64" --debug
129+
expect_host_default "--platform still wins over the host" Linux aarch64 \
130+
"paimon-cpp-linux-x86_64" --platform linux-x86_64
131+
132+
# --platform as the final argument must be reported as a missing value rather than
133+
# consuming whatever follows.
134+
expect_rejected "--platform without a value" --platform
135+
136+
# Nothing that turns the package name into a path, or that a shell would treat as
137+
# anything but a literal, may be accepted.
138+
expect_rejected "--platform followed by an option" --platform --debug
139+
expect_rejected "bare parent directory" --platform ".."
140+
expect_rejected "parent directory traversal" --platform "../evil"
141+
expect_rejected "absolute path" --platform "/etc/passwd"
142+
expect_rejected "nested path" --platform "linux/x86_64"
143+
expect_rejected "leading dash" --platform "-linux"
144+
expect_rejected "leading dot" --platform ".linux"
145+
expect_rejected "empty label" --platform ""
146+
expect_rejected "command separator" --platform "linux;touch ${sentinel}"
147+
expect_rejected "command substitution" --platform 'linux$(touch '"${sentinel}"')'
148+
expect_rejected "whitespace" --platform "linux x86_64"
149+
150+
if [[ -e "${sentinel}" ]]; then
151+
failures=$((failures + 1))
152+
echo "FAIL a rejected label was still evaluated by a shell"
153+
fi
154+
155+
if [[ "${failures}" -gt 0 ]]; then
156+
echo "${failures} of ${checks} packaging argument checks failed"
157+
exit 1
158+
fi
159+
echo "All ${checks} packaging argument checks passed"

0 commit comments

Comments
 (0)