Skip to content

Commit 7f08dbd

Browse files
czentgrrobandpdx
czentgr
and
robandpdx
committed
[native] Convert CircleCI jobs for PrestoC++ to Github actions
This pull request converts the CircleCI workflows to GitHub actions workflows. The workflows use the standard runners with Ubuntu 22.04. Some modifications were made to allow a build with the limited resources available: |name|build type|comments|expected runtime (no cache)|unit tests|E2E tests| |----|----------|--------|----------------|----------|---------| | prestocpp-linux-build | debug | All adapters enabled | 45m | N | N | | prestocpp-linux-build-and-unit-test | release | JWT, remote function, Parquet enabled | 90m | Y | Y | | prestocpp-macos-build | debug | Setup script plus build | 120m | N | N | | prestocpp-linux-adapters-build | release | Adapters only build | 20m | N | N | Caching is currently enabled for the prestocpp-linux-build-and-unit-test. If the cache can be used, the time to run tests drops to 50m. Additional workflows are prestocpp-format-check and prestocpp-header-check That run in a few seconds. For the original conversion see: #21545 Co-authored-by: Rob Anderson <[email protected]>
1 parent ed176b9 commit 7f08dbd

5 files changed

+484
-0
lines changed

Diff for: ‎.github/workflows/prestocpp-format-and-header-check.yml

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
name: prestocpp-format-and-header-check
2+
3+
on:
4+
workflow_dispatch:
5+
pull_request:
6+
paths:
7+
- 'presto-native-execution/**'
8+
9+
jobs:
10+
prestocpp-format-check:
11+
runs-on: ubuntu-latest
12+
container:
13+
image: public.ecr.aws/oss-presto/velox-dev:check
14+
steps:
15+
- uses: actions/checkout@v4
16+
- name: Fix git permissions
17+
# Usually actions/checkout does this but as we run in a container
18+
# it doesn't work
19+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
20+
- name: Check formatting
21+
run: |
22+
git fetch origin master
23+
cd presto-native-execution
24+
make format-check
25+
- name: Check license headers
26+
run: |
27+
git fetch origin master
28+
cd presto-native-execution
29+
make header-check

Diff for: ‎.github/workflows/prestocpp-linux-adapters-build.yml

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: prestocpp-linux-adapters-build
2+
3+
on:
4+
workflow_dispatch:
5+
pull_request:
6+
paths:
7+
- 'presto-native-execution/scripts/setup-adapters.sh'
8+
9+
jobs:
10+
prestocpp-linux-build-adapters:
11+
runs-on: ubuntu-22.04
12+
container:
13+
image: prestodb/presto-native-dependency:0.290-20241014120930-e1fc090
14+
steps:
15+
- uses: actions/checkout@v4
16+
- name: Fix git permissions
17+
# Usually actions/checkout does this but as we run in a container
18+
# it doesn't work
19+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
20+
- name: Update submodules
21+
run: |
22+
cd presto-native-execution
23+
make submodules
24+
- name: Build all adapter dependencies
25+
run: |
26+
mkdir -p ${GITHUB_WORKSPACE}/adapter-deps/install
27+
source /opt/rh/gcc-toolset-12/enable
28+
set -xu
29+
cd presto-native-execution
30+
DEPENDENCY_DIR=${GITHUB_WORKSPACE}/adapter-deps PROMPT_ALWAYS_RESPOND=n ./scripts/setup-adapters.sh

Diff for: ‎.github/workflows/prestocpp-linux-build-and-unit-test.yml

+333
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,333 @@
1+
name: prestocpp-linux-build-and-unit-test
2+
3+
on:
4+
workflow_dispatch:
5+
pull_request:
6+
7+
jobs:
8+
prestocpp-linux-build-for-test:
9+
runs-on: ubuntu-22.04
10+
container:
11+
image: prestodb/presto-native-dependency:0.290-20241014120930-e1fc090
12+
env:
13+
CCACHE_DIR: "${{ github.workspace }}/ccache"
14+
steps:
15+
- uses: actions/checkout@v4
16+
- name: Fix git permissions
17+
# Usually actions/checkout does this but as we run in a container
18+
# it doesn't work
19+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
20+
21+
- name: Update velox
22+
run: |
23+
cd presto-native-execution
24+
make velox-submodule
25+
26+
- name: Install prestocpp adapter dependencies
27+
run: |
28+
mkdir -p ${HOME}/adapter-deps/install
29+
source /opt/rh/gcc-toolset-12/enable
30+
set -xu
31+
cd presto-native-execution
32+
DEPENDENCY_DIR=${HOME}/adapter-deps PROMPT_ALWAYS_RESPOND=n ./scripts/setup-adapters.sh
33+
34+
- name: Calculate merge-base date for ccache
35+
run: |
36+
git fetch origin master
37+
git show -s --format=%cd --date="format:%Y%m%d" $(git merge-base origin/master HEAD) | tee merge-base-date
38+
39+
- name: Install Github CLI for using apache/infrastructure-actions/stash
40+
run: |
41+
curl -L https://github.com/cli/cli/releases/download/v2.63.2/gh_2.63.2_linux_amd64.rpm > gh_2.63.2_linux_amd64.rpm
42+
rpm -iv gh_2.63.2_linux_amd64.rpm
43+
44+
- uses: apache/infrastructure-actions/stash/restore@4ab8682fbd4623d2b4fc1c98db38aba5091924c3
45+
with:
46+
path: '${{ env.CCACHE_DIR }}'
47+
key: ccache-linux-prestocpp-${{ runner.os }}-${{ hashFiles('merge-base-date') }}
48+
49+
- name: Zero ccache statistics
50+
run: ccache -sz
51+
52+
- name: Build engine
53+
run: |
54+
source /opt/rh/gcc-toolset-12/enable
55+
cd presto-native-execution
56+
cmake \
57+
-B _build/release \
58+
-GNinja \
59+
-DTREAT_WARNINGS_AS_ERRORS=1 \
60+
-DENABLE_ALL_WARNINGS=1 \
61+
-DCMAKE_BUILD_TYPE=Release \
62+
-DPRESTO_ENABLE_PARQUET=ON \
63+
-DPRESTO_ENABLE_REMOTE_FUNCTIONS=ON \
64+
-DPRESTO_ENABLE_JWT=ON \
65+
-DPRESTO_STATS_REPORTER_TYPE=PROMETHEUS \
66+
-DPRESTO_MEMORY_CHECKER_TYPE=LINUX_MEMORY_CHECKER \
67+
-DCMAKE_PREFIX_PATH=/usr/local \
68+
-DThrift_ROOT=/usr/local \
69+
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
70+
-DMAX_LINK_JOBS=4
71+
ninja -C _build/release -j 4
72+
73+
- name: Ccache after
74+
run: ccache -s
75+
76+
- uses: apache/infrastructure-actions/stash/save@4ab8682fbd4623d2b4fc1c98db38aba5091924c3
77+
with:
78+
path: '${{ env.CCACHE_DIR }}'
79+
key: ccache-linux-prestocpp-${{ runner.os }}-${{ hashFiles('merge-base-date') }}
80+
81+
- name: Run Unit Tests
82+
run: |
83+
# Ensure transitive dependency libboost-iostreams is found.
84+
ldconfig /usr/local/lib
85+
cd presto-native-execution/_build/release
86+
ctest -j 4 -VV --output-on-failure --exclude-regex velox.*
87+
88+
- name: Upload artifacts
89+
uses: actions/upload-artifact@v4
90+
with:
91+
name: presto-native-build
92+
path: |
93+
presto-native-execution/_build/release/presto_cpp/main/presto_server
94+
presto-native-execution/_build/release/velox/velox/functions/remote/server/velox_functions_remote_server_main
95+
96+
prestocpp-linux-presto-e2e-tests:
97+
needs: prestocpp-linux-build-for-test
98+
runs-on: ubuntu-22.04
99+
container:
100+
image: prestodb/presto-native-dependency:0.290-20241014120930-e1fc090
101+
env:
102+
MAVEN_OPTS: "-Xmx4G -XX:+ExitOnOutOfMemoryError"
103+
MAVEN_FAST_INSTALL: "-B -V --quiet -T 1C -DskipTests -Dair.check.skip-all -Dmaven.javadoc.skip=true"
104+
MAVEN_TEST: "-B -Dair.check.skip-all -Dmaven.javadoc.skip=true -DLogTestDurationListener.enabled=true --fail-at-end"
105+
steps:
106+
- uses: actions/checkout@v4
107+
- name: Fix git permissions
108+
# Usually actions/checkout does this but as we run in a container
109+
# it doesn't work
110+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
111+
112+
- name: Download artifacts
113+
uses: actions/download-artifact@v4
114+
with:
115+
name: presto-native-build
116+
path: presto-native-execution/_build/release
117+
118+
# Permissions are lost when uploading. Details here: https://github.com/actions/upload-artifact/issues/38
119+
- name: Restore execute permissions and library path
120+
run: |
121+
chmod +x ${GITHUB_WORKSPACE}/presto-native-execution/_build/release/presto_cpp/main/presto_server
122+
chmod +x ${GITHUB_WORKSPACE}/presto-native-execution/_build/release/velox/velox/functions/remote/server/velox_functions_remote_server_main
123+
# Ensure transitive dependency libboost-iostreams is found.
124+
ldconfig /usr/local/lib
125+
126+
- name: Install OpenJDK8
127+
uses: actions/setup-java@v4
128+
with:
129+
distribution: 'temurin'
130+
java-version: '8'
131+
132+
- name: Cache local Maven repository
133+
id: cache-maven
134+
uses: actions/cache@v4
135+
with:
136+
path: ~/.m2/repository
137+
key: ${{ runner.os }}-maven-2-${{ hashFiles('**/pom.xml') }}
138+
restore-keys: |
139+
${{ runner.os }}-maven-2-
140+
141+
- name: Populate maven cache
142+
if: steps.cache-maven.outputs.cache-hit != 'true'
143+
run: ./mvnw de.qaware.maven:go-offline-maven-plugin:resolve-dependencies
144+
145+
- name: Maven install
146+
env:
147+
# Use different Maven options to install.
148+
MAVEN_OPTS: "-Xmx2G -XX:+ExitOnOutOfMemoryError"
149+
run: |
150+
for i in $(seq 1 3); do ./mvnw clean install $MAVEN_FAST_INSTALL -pl 'presto-native-execution' -am && s=0 && break || s=$? && sleep 10; done; (exit $s)
151+
152+
- name: Run presto-native e2e tests
153+
run: |
154+
export PRESTO_SERVER_PATH="${GITHUB_WORKSPACE}/presto-native-execution/_build/release/presto_cpp/main/presto_server"
155+
export TESTFILES=`find ./presto-native-execution/src/test -type f -name 'TestPrestoNative*.java'`
156+
# Convert file paths to comma separated class names
157+
export TESTCLASSES=
158+
for test_file in $TESTFILES
159+
do
160+
tmp=${test_file##*/}
161+
test_class=${tmp%%\.*}
162+
export TESTCLASSES="${TESTCLASSES},$test_class"
163+
done
164+
export TESTCLASSES=${TESTCLASSES#,}
165+
echo "TESTCLASSES = $TESTCLASSES"
166+
# TODO: neeed to enable remote function tests with
167+
# "-Ppresto-native-execution-remote-functions" once
168+
# > https://github.com/facebookincubator/velox/discussions/6163
169+
# is fixed.
170+
171+
mvn test \
172+
${MAVEN_TEST} \
173+
-pl 'presto-native-execution' \
174+
-Dtest="${TESTCLASSES}" \
175+
-DPRESTO_SERVER=${PRESTO_SERVER_PATH} \
176+
-DDATA_DIR=${RUNNER_TEMP} \
177+
-Duser.timezone=America/Bahia_Banderas \
178+
-T1C
179+
180+
prestocpp-linux-spark-e2e-tests:
181+
needs: prestocpp-linux-build-for-test
182+
runs-on: ubuntu-22.04
183+
container:
184+
image: prestodb/presto-native-dependency:0.290-20241014120930-e1fc090
185+
env:
186+
MAVEN_OPTS: "-Xmx4G -XX:+ExitOnOutOfMemoryError"
187+
MAVEN_FAST_INSTALL: "-B -V --quiet -T 1C -DskipTests -Dair.check.skip-all -Dmaven.javadoc.skip=true"
188+
MAVEN_TEST: "-B -Dair.check.skip-all -Dmaven.javadoc.skip=true -DLogTestDurationListener.enabled=true --fail-at-end"
189+
steps:
190+
- uses: actions/checkout@v4
191+
- name: Fix git permissions
192+
# Usually actions/checkout does this but as we run in a container
193+
# it doesn't work
194+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
195+
196+
- name: Download artifacts
197+
uses: actions/download-artifact@v4
198+
with:
199+
name: presto-native-build
200+
path: presto-native-execution/_build/release
201+
202+
# Permissions are lost when uploading. Details here: https://github.com/actions/upload-artifact/issues/38
203+
- name: Restore execute permissions and library path
204+
run: |
205+
chmod +x ${GITHUB_WORKSPACE}/presto-native-execution/_build/release/presto_cpp/main/presto_server
206+
chmod +x ${GITHUB_WORKSPACE}/presto-native-execution/_build/release/velox/velox/functions/remote/server/velox_functions_remote_server_main
207+
# Ensure transitive dependency libboost-iostreams is found.
208+
ldconfig /usr/local/lib
209+
210+
- name: Check Java
211+
run: java --version
212+
213+
- name: Cache local Maven repository
214+
id: cache-maven
215+
uses: actions/cache@v4
216+
with:
217+
path: ~/.m2/repository
218+
key: ${{ runner.os }}-maven-2-${{ hashFiles('**/pom.xml') }}
219+
restore-keys: |
220+
${{ runner.os }}-maven-2-
221+
222+
- name: Populate maven cache
223+
if: steps.cache-maven.outputs.cache-hit != 'true'
224+
run: ./mvnw de.qaware.maven:go-offline-maven-plugin:resolve-dependencies
225+
226+
- name: Maven install
227+
env:
228+
# Use different Maven options to install.
229+
MAVEN_OPTS: "-Xmx2G -XX:+ExitOnOutOfMemoryError"
230+
run: |
231+
for i in $(seq 1 3); do ./mvnw clean install $MAVEN_FAST_INSTALL -pl 'presto-native-execution' -am && s=0 && break || s=$? && sleep 10; done; (exit $s)
232+
233+
- name: Run spark e2e tests
234+
run: |
235+
export PRESTO_SERVER_PATH="${GITHUB_WORKSPACE}/presto-native-execution/_build/release/presto_cpp/main/presto_server"
236+
export TESTFILES=`find ./presto-native-execution/src/test -type f -name 'TestPrestoSpark*.java'`
237+
# Convert file paths to comma separated class names
238+
export TESTCLASSES=
239+
for test_file in $TESTFILES
240+
do
241+
tmp=${test_file##*/}
242+
test_class=${tmp%%\.*}
243+
export TESTCLASSES="${TESTCLASSES},$test_class"
244+
done
245+
export TESTCLASSES=${TESTCLASSES#,}
246+
echo "TESTCLASSES = $TESTCLASSES"
247+
mvn test \
248+
${MAVEN_TEST} \
249+
-pl 'presto-native-execution' \
250+
-Dtest="${TESTCLASSES}" \
251+
-DPRESTO_SERVER=${PRESTO_SERVER_PATH} \
252+
-DDATA_DIR=${RUNNER_TEMP} \
253+
-Duser.timezone=America/Bahia_Banderas \
254+
-T1C
255+
256+
prestocpp-linux-presto-sidecar-tests:
257+
needs: prestocpp-linux-build-for-test
258+
runs-on: ubuntu-22.04
259+
container:
260+
image: prestodb/presto-native-dependency:0.290-20241014120930-e1fc090
261+
env:
262+
MAVEN_OPTS: "-Xmx4G -XX:+ExitOnOutOfMemoryError"
263+
MAVEN_FAST_INSTALL: "-B -V --quiet -T 1C -DskipTests -Dair.check.skip-all -Dmaven.javadoc.skip=true"
264+
MAVEN_TEST: "-B -Dair.check.skip-all -Dmaven.javadoc.skip=true -DLogTestDurationListener.enabled=true --fail-at-end"
265+
steps:
266+
- uses: actions/checkout@v4
267+
- name: Fix git permissions
268+
# Usually actions/checkout does this but as we run in a container
269+
# it doesn't work
270+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
271+
272+
- name: Download artifacts
273+
uses: actions/download-artifact@v4
274+
with:
275+
name: presto-native-build
276+
path: presto-native-execution/_build/release
277+
278+
# Permissions are lost when uploading. Details here: https://github.com/actions/upload-artifact/issues/38
279+
- name: Restore execute permissions and library path
280+
run: |
281+
chmod +x ${GITHUB_WORKSPACE}/presto-native-execution/_build/release/presto_cpp/main/presto_server
282+
chmod +x ${GITHUB_WORKSPACE}/presto-native-execution/_build/release/velox/velox/functions/remote/server/velox_functions_remote_server_main
283+
# Ensure transitive dependency libboost-iostreams is found.
284+
ldconfig /usr/local/lib
285+
286+
- name: Install OpenJDK8
287+
uses: actions/setup-java@v4
288+
with:
289+
distribution: 'temurin'
290+
java-version: '8'
291+
292+
- name: Cache local Maven repository
293+
id: cache-maven
294+
uses: actions/cache@v4
295+
with:
296+
path: ~/.m2/repository
297+
key: ${{ runner.os }}-maven-2-${{ hashFiles('**/pom.xml') }}
298+
restore-keys: |
299+
${{ runner.os }}-maven-2-
300+
301+
- name: Populate maven cache
302+
if: steps.cache-maven.outputs.cache-hit != 'true'
303+
run: ./mvnw de.qaware.maven:go-offline-maven-plugin:resolve-dependencies
304+
305+
- name: Maven install
306+
env:
307+
# Use different Maven options to install.
308+
MAVEN_OPTS: "-Xmx2G -XX:+ExitOnOutOfMemoryError"
309+
run: |
310+
for i in $(seq 1 3); do ./mvnw clean install $MAVEN_FAST_INSTALL -pl 'presto-native-execution' -am && s=0 && break || s=$? && sleep 10; done; (exit $s)
311+
312+
- name: Run presto-native sidecar tests
313+
run: |
314+
export PRESTO_SERVER_PATH="${GITHUB_WORKSPACE}/presto-native-execution/_build/release/presto_cpp/main/presto_server"
315+
export TESTFILES=`find ./presto-native-sidecar-plugin/src/test -type f -name 'Test*.java'`
316+
# Convert file paths to comma separated class names
317+
export TESTCLASSES=
318+
for test_file in $TESTFILES
319+
do
320+
tmp=${test_file##*/}
321+
test_class=${tmp%%\.*}
322+
export TESTCLASSES="${TESTCLASSES},$test_class"
323+
done
324+
export TESTCLASSES=${TESTCLASSES#,}
325+
echo "TESTCLASSES = $TESTCLASSES"
326+
mvn test \
327+
${MAVEN_TEST} \
328+
-pl 'presto-native-sidecar-plugin' \
329+
-Dtest="${TESTCLASSES}" \
330+
-DPRESTO_SERVER=${PRESTO_SERVER_PATH} \
331+
-DDATA_DIR=${RUNNER_TEMP} \
332+
-Duser.timezone=America/Bahia_Banderas \
333+
-T1C

0 commit comments

Comments
 (0)