Skip to content

Python: Update docker file #7164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 22, 2023
Merged

Python: Update docker file #7164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0002050
Update docker file image
Mar 21, 2023
d1fe2b9
fix failure
Mar 22, 2023
aaeb1ad
address review feedback
Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/ISSUE_TEMPLATE/iceberg_bug_report.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ body:
description: What Apache Iceberg version are you using?
multiple: false
options:
- "1.1.0 (latest release)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should make this part of the How to release process? https://iceberg.apache.org/how-to-release/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, I have a PR open for updating release process, looks like this and also REV API update needs to be added apache/iceberg-docs#210

- "1.2.0 (latest release)"
- "1.1.0"
- "1.0.0"
- "0.14.1"
- "0.14.0"
Expand Down
14 changes: 8 additions & 6 deletions python/dev/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,22 +37,24 @@ RUN mkdir -p ${HADOOP_HOME} && mkdir -p ${SPARK_HOME} && mkdir -p /home/iceberg/
WORKDIR ${SPARK_HOME}

ENV SPARK_VERSION=3.3.2
ENV ICEBERG_VERSION=1.2.0
ENV AWS_SDK_VERSION=2.20.18
Comment on lines +40 to +41
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for cleaning this up!


RUN curl -s https://dlcdn.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop3.tgz -o spark-${SPARK_VERSION}-bin-hadoop3.tgz \
&& tar xzf spark-${SPARK_VERSION}-bin-hadoop3.tgz --directory /opt/spark --strip-components 1 \
&& rm -rf spark-${SPARK_VERSION}-bin-hadoop3.tgz

# Download iceberg spark runtime
RUN curl -s https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/1.1.0/iceberg-spark-runtime-3.3_2.12-1.1.0.jar -Lo iceberg-spark-runtime-3.3_2.12-1.1.0.jar \
&& mv iceberg-spark-runtime-3.3_2.12-1.1.0.jar /opt/spark/jars
RUN curl -s https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.3_2.12/${ICEBERG_VERSION}/iceberg-spark-runtime-3.3_2.12-${ICEBERG_VERSION}.jar -Lo iceberg-spark-runtime-3.3_2.12-${ICEBERG_VERSION}.jar \
&& mv iceberg-spark-runtime-3.3_2.12-${ICEBERG_VERSION}.jar /opt/spark/jars

# Download Java AWS SDK
RUN curl -s https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.17.165/bundle-2.17.165.jar -Lo bundle-2.17.165.jar \
&& mv bundle-2.17.165.jar /opt/spark/jars
RUN curl -s https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/${AWS_SDK_VERSION}/bundle-${AWS_SDK_VERSION}.jar -Lo bundle-${AWS_SDK_VERSION}.jar \
&& mv bundle-${AWS_SDK_VERSION}.jar /opt/spark/jars

# Download URL connection client required for S3FileIO
RUN curl -s https://repo1.maven.org/maven2/software/amazon/awssdk/url-connection-client/2.17.165/url-connection-client-2.17.165.jar -Lo url-connection-client-2.17.165.jar \
&& mv url-connection-client-2.17.165.jar /opt/spark/jars
RUN curl -s https://repo1.maven.org/maven2/software/amazon/awssdk/url-connection-client/${AWS_SDK_VERSION}/url-connection-client-${AWS_SDK_VERSION}.jar -Lo url-connection-client-${AWS_SDK_VERSION}.jar \
&& mv url-connection-client-${AWS_SDK_VERSION}.jar /opt/spark/jars

COPY spark-defaults.conf /opt/spark/conf
ENV PATH="/opt/spark/sbin:/opt/spark/bin:${PATH}"
Expand Down
17 changes: 15 additions & 2 deletions python/dev/docker-compose-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ services:
image: python-integration
container_name: pyiceberg-spark
build: .
networks:
iceberg_net:
depends_on:
- rest
- minio
Expand All @@ -37,15 +39,17 @@ services:
- rest:rest
- minio:minio
rest:
image: tabulario/iceberg-rest:0.2.0
image: tabulario/iceberg-rest
container_name: pyiceberg-rest
networks:
iceberg_net:
ports:
- 8181:8181
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3a://warehouse/wh/
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- CATALOG_S3_ENDPOINT=http://minio:9000
minio:
Expand All @@ -54,6 +58,11 @@ services:
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
- MINIO_DOMAIN=minio
networks:
iceberg_net:
aliases:
- warehouse.minio
ports:
- 9001:9001
- 9000:9000
Expand All @@ -63,6 +72,8 @@ services:
- minio
image: minio/mc
container_name: pyiceberg-mc
networks:
iceberg_net:
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
Expand All @@ -74,3 +85,5 @@ services:
/usr/bin/mc policy set public minio/warehouse;
tail -f /dev/null
"
networks:
iceberg_net:
1 change: 1 addition & 0 deletions python/dev/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ services:
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
- MINIO_DOMAIN=minio
ports:
- 9001:9001
- 9000:9000
Expand Down
4 changes: 2 additions & 2 deletions python/mkdocs/docs/verify-release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,8 @@ This will include a Minio S3 container being spun up.

# Cast the vote

Votes are cast by replying to the release candidate announcement email on the dev mailing list with either `+1`, `0`, or `-1`.
Votes are cast by replying to the release candidate announcement email on the dev mailing list with either `+1`, `0`, or `-1`. For example :

> \[ \] +1 Release this as Apache Iceberg 1.1.0 \[ \] +0 \[ \] -1 Do not release this because…
> \[ \] +1 Release this as PyIceberg 0.3.0 \[ \] +0 \[ \] -1 Do not release this because…

In addition to your vote, it’s customary to specify if your vote is binding or non-binding. Only members of the Project Management Committee have formally binding votes. If you’re unsure, you can specify that your vote is non-binding. To read more about voting in the Apache framework, checkout the [Voting](https://www.apache.org/foundation/voting.html) information page on the Apache foundation’s website.