Skip to content

Commit

Permalink
KFP 1.7.0 Rebase (#713)
Browse files Browse the repository at this point in the history
* KFP 1.7.0-rc.4 Rebase

* Resolve API and Backend conflicts

* Apply 1.7.0 changes since rc.4

* Revert unwanted SDK changes

* Revert unwanted samples changes

* Revert unwanted components changes

* Fix 1.7.0 backend conflicts

* Revert unwanted SDK changes (2)

* Fix SDK bugs in backend/Dockerfile

* Fix frontend conflicts
  • Loading branch information
kfp-tekton-bot authored Aug 30, 2021
1 parent 17ddeca commit 0708a56
Show file tree
Hide file tree
Showing 1,024 changed files with 228,467 additions and 71,070 deletions.
11 changes: 4 additions & 7 deletions .cloudbuild.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2018 Google LLC
# Copyright 2018 The Kubeflow Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -180,7 +180,7 @@ steps:
# * manifests/kustomize/base/metadata/base/metadata-grpc-deployment.yaml
# * test/tag_for_hosted.sh
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/tfx-oss-public/ml_metadata_store_server:0.25.1']
args: ['pull', 'gcr.io/tfx-oss-public/ml_metadata_store_server:1.0.0']
id: 'pullMetadataServer'
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/minio:RELEASE.2019-08-14T20-37-41Z-license-compliance']
Expand All @@ -192,10 +192,10 @@ steps:
args: ['pull', 'gcr.io/cloudsql-docker/gce-proxy:1.14']
id: 'pullCloudsqlProxy'
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/argoexec:v2.12.9-license-compliance']
args: ['pull', 'gcr.io/ml-pipeline/argoexec:v3.1.6-patch-license-compliance']
id: 'pullArgoExecutor'
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/workflow-controller:v2.12.9-license-compliance']
args: ['pull', 'gcr.io/ml-pipeline/workflow-controller:v3.1.6-patch-license-compliance']
id: 'pullArgoWorkflowController'

# Tag for Hosted - SemVersion to Major.Minor parsing
Expand Down Expand Up @@ -246,9 +246,6 @@ images:
- 'gcr.io/$PROJECT_ID/ml-pipeline-kubeflow-tf-trainer-gpu:$COMMIT_SHA'
- 'gcr.io/$PROJECT_ID/ml-pipeline-kubeflow-tfjob:$COMMIT_SHA'

# Images for the GCP generic pipeline components
- 'gcr.io/$PROJECT_ID/ml-pipeline-gcp:$COMMIT_SHA'

# Images for the local components
- 'gcr.io/$PROJECT_ID/ml-pipeline-local-confusion-matrix:$COMMIT_SHA'
- 'gcr.io/$PROJECT_ID/ml-pipeline-local-roc:$COMMIT_SHA'
Expand Down
4 changes: 3 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
.git
bower_components
dist
node_modules
**/node_modules
backend/build
v2/build
6 changes: 0 additions & 6 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,3 @@ Resolves #
* `test: fix CI failure. Part of #1234`
Use `part of` to indicate that a PR is working on an issue, but shouldn't close the issue when merged.
-->
- [ ] Do you want this pull request (PR) cherry-picked into the current release branch?

[Learn more about cherry-picking updates into the release branch](https://github.com/kubeflow/pipelines/blob/master/RELEASE.md#cherry-picking-pull-requests-to-release-branch).
<!--
**(Recommended.)** Ask the PR approver to add the `cherrypick-approved` label to this PR. The release manager adds this PR to the release branch in a batch update before release.
-->
27 changes: 27 additions & 0 deletions .github/semantic.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Configuration for https://github.com/zeke/semantic-pull-requests
#
# Kubeflow Pipelines PR title convention is documented in
# https://github.com/kubeflow/pipelines/blob/master/CONTRIBUTING.md#pull-request-title-convention

# Always validate the PR title, and ignore the commits.
titleOnly: true

# TODO: define a list of valid scopes.
# scopes:
# - scope1
# - scope2
# ...

# By default types specified in commitizen/conventional-commit-types is used.
# See: https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json
types:
- feat
- fix
- docs
- style
- refactor
- perf
- test
- build
- chore
- revert
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -104,3 +104,6 @@ docs/_build

# python sdk package
*.tar.gz

# Copy from kubeflow/frontend
coverage/
2 changes: 1 addition & 1 deletion .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ sphinx:
python:
version: 3.7
install:
- requirements: sdk/python/requirements.txt
- requirements: sdk/python/requirements.txt
35 changes: 17 additions & 18 deletions .release.cloudbuild.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2018 Google LLC
# Copyright 2018 The Kubeflow Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -40,7 +40,6 @@ steps:
"ml-pipeline-kubeflow-tfjob"
"ml-pipeline-local-confusion-matrix"
"ml-pipeline-local-roc"
"ml-pipeline-gcp"
)
for image in "${images[@]}"
do
Expand Down Expand Up @@ -260,14 +259,14 @@ steps:
# * manifests/kustomize/base/metadata/base/metadata-grpc-deployment.yaml
# * test/tag_for_hosted.sh
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/tfx-oss-public/ml_metadata_store_server:0.25.1']
args: ['pull', 'gcr.io/tfx-oss-public/ml_metadata_store_server:1.0.0']
id: 'pullMetadataServer'
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/tfx-oss-public/ml_metadata_store_server:0.25.1', 'gcr.io/ml-pipeline/google/pipelines/metadataserver:$TAG_NAME']
args: ['tag', 'gcr.io/tfx-oss-public/ml_metadata_store_server:1.0.0', 'gcr.io/ml-pipeline/google/pipelines/metadataserver:$TAG_NAME']
id: 'tagMetadataServerForMarketplace'
waitFor: ['pullMetadataServer']
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/tfx-oss-public/ml_metadata_store_server:0.25.1', 'gcr.io/ml-pipeline/google/pipelines-test/metadataserver:$TAG_NAME']
args: ['tag', 'gcr.io/tfx-oss-public/ml_metadata_store_server:1.0.0', 'gcr.io/ml-pipeline/google/pipelines-test/metadataserver:$TAG_NAME']
id: 'tagMetadataServerForMarketplaceTest'
waitFor: ['pullMetadataServer']
- id: 'tagMetadataServerForMarketplaceMajorMinor'
Expand All @@ -277,8 +276,8 @@ steps:
args:
- -ceux
- |
docker tag gcr.io/tfx-oss-public/ml_metadata_store_server:0.25.1 gcr.io/ml-pipeline/google/pipelines/metadataserver:$(cat /workspace/mm.ver)
docker tag gcr.io/tfx-oss-public/ml_metadata_store_server:0.25.1 gcr.io/ml-pipeline/google/pipelines-test/metadataserver:$(cat /workspace/mm.ver)
docker tag gcr.io/tfx-oss-public/ml_metadata_store_server:1.0.0 gcr.io/ml-pipeline/google/pipelines/metadataserver:$(cat /workspace/mm.ver)
docker tag gcr.io/tfx-oss-public/ml_metadata_store_server:1.0.0 gcr.io/ml-pipeline/google/pipelines-test/metadataserver:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines/metadataserver:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines-test/metadataserver:$(cat /workspace/mm.ver)
Expand Down Expand Up @@ -479,14 +478,14 @@ steps:
docker push gcr.io/ml-pipeline/google/pipelines-test/cloudsqlproxy:$(cat /workspace/mm.ver)
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/argoexec:v2.12.9-license-compliance']
args: ['pull', 'gcr.io/ml-pipeline/argoexec:v3.1.6-patch-license-compliance']
id: 'pullArgoExecutor'
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/ml-pipeline/argoexec:v2.12.9-license-compliance', 'gcr.io/ml-pipeline/google/pipelines/argoexecutor:$TAG_NAME']
args: ['tag', 'gcr.io/ml-pipeline/argoexec:v3.1.6-patch-license-compliance', 'gcr.io/ml-pipeline/google/pipelines/argoexecutor:$TAG_NAME']
id: 'tagArgoExecutorForMarketplace'
waitFor: ['pullArgoExecutor']
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/ml-pipeline/argoexec:v2.12.9-license-compliance', 'gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$TAG_NAME']
args: ['tag', 'gcr.io/ml-pipeline/argoexec:v3.1.6-patch-license-compliance', 'gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$TAG_NAME']
id: 'tagArgoExecutorForMarketplaceTest'
waitFor: ['pullArgoExecutor']
- id: 'tagArgoExecutorForMarketplaceMajorMinor'
Expand All @@ -496,20 +495,20 @@ steps:
args:
- -ceux
- |
docker tag gcr.io/ml-pipeline/argoexec:v2.12.9-license-compliance gcr.io/ml-pipeline/google/pipelines/argoexecutor:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/argoexec:v2.12.9-license-compliance gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/argoexec:v3.1.6-patch-license-compliance gcr.io/ml-pipeline/google/pipelines/argoexecutor:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/argoexec:v3.1.6-patch-license-compliance gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines/argoexecutor:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines-test/argoexecutor:$(cat /workspace/mm.ver)
- name: 'gcr.io/cloud-builders/docker'
args: ['pull', 'gcr.io/ml-pipeline/workflow-controller:v2.12.9-license-compliance']
args: ['pull', 'gcr.io/ml-pipeline/workflow-controller:v3.1.6-patch-license-compliance']
id: 'pullArgoWorkflowController'
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/ml-pipeline/workflow-controller:v2.12.9-license-compliance', 'gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$TAG_NAME']
args: ['tag', 'gcr.io/ml-pipeline/workflow-controller:v3.1.6-patch-license-compliance', 'gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$TAG_NAME']
id: 'tagArgoWorkflowControllerForMarketplace'
waitFor: ['pullArgoWorkflowController']
- name: 'gcr.io/cloud-builders/docker'
args: ['tag', 'gcr.io/ml-pipeline/workflow-controller:v2.12.9-license-compliance', 'gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$TAG_NAME']
args: ['tag', 'gcr.io/ml-pipeline/workflow-controller:v3.1.6-patch-license-compliance', 'gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$TAG_NAME']
id: 'tagArgoWorkflowControllerForMarketplaceTest'
waitFor: ['pullArgoWorkflowController']
- id: 'tagArgoWorkflowControllerForMarketplaceMajorMinor'
Expand All @@ -519,8 +518,8 @@ steps:
args:
- -ceux
- |
docker tag gcr.io/ml-pipeline/workflow-controller:v2.12.9-license-compliance gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/workflow-controller:v2.12.9-license-compliance gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/workflow-controller:v3.1.6-patch-license-compliance gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker tag gcr.io/ml-pipeline/workflow-controller:v3.1.6-patch-license-compliance gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines/argoworkflowcontroller:$(cat /workspace/mm.ver)
docker push gcr.io/ml-pipeline/google/pipelines-test/argoworkflowcontroller:$(cat /workspace/mm.ver)
Expand Down Expand Up @@ -648,6 +647,6 @@ images:
- 'gcr.io/ml-pipeline/google/pipelines-test/cachedeployer:$TAG_NAME'
- 'gcr.io/ml-pipeline/google/pipelines-test/deployer:$TAG_NAME'
- 'gcr.io/ml-pipeline/google/pipelines-test:$TAG_NAME'
timeout: '1200s'
timeout: '2000s'
tags:
- release-on-tag
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ for demos.

## Architecture

We are currently using [Kubeflow Pipelines 1.5.0](https://github.com/kubeflow/pipelines/releases/tag/1.5.0) and
We are currently using [Kubeflow Pipelines 1.7.0](https://github.com/kubeflow/pipelines/releases/tag/1.7.0) and
[Tekton >= 0.27.0](https://github.com/tektoncd/pipeline/releases/tag/v0.27.0)
for this project.

Expand Down
2 changes: 2 additions & 0 deletions api/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Dependencies need to download via Makefile.
v2alpha1/google
85 changes: 81 additions & 4 deletions api/Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,84 @@
PROTO_PATH=/usr/local/include/
# Copyright 2021 The Kubeflow Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

gen_proto: v2alpha1/pipeline_spec.proto
cd v2alpha1 && protoc -I=$(PROTO_PATH) -I=. \
--go_out=go \
# Contact one of Bobgy, capri-xiyue or zijianjoy if this remote image needs an update.
PREBUILT_REMOTE_IMAGE=gcr.io/ml-pipeline-test/api-generator:latest

.PHONY: all
all: golang python

.PHONY: clean
clean: clean-go clean-python

# Generate proto packages using a pre-built api-generator image.
.PHONY: golang
golang: v2alpha1/*.proto
docker run --interactive --rm \
--user $$(id -u):$$(id -g) \
--mount type=bind,source="$$(pwd)/..",target=/go/src/github.com/kubeflow/pipelines \
$(PREBUILT_REMOTE_IMAGE) \
sh -c 'cd /go/src/github.com/kubeflow/pipelines/api && make generate'

# Delete all generated proto go packages.
.PHONY: clean-go
clean-go:
rm -rf v2alpha1/go
rm -f v2alpha1/google/rpc/status.proto

# Generate Python package.
.PHONY: python
python: v2alpha1/pipeline_spec.proto v2alpha1/google/rpc/status.proto
cd v2alpha1/python && python3 setup.py bdist_wheel

# Delete all generated Python packages
.PHONY: clean-python
clean-python:
rm -rf v2alpha1/python/build
rm -rf v2alpha1/python/dist
rm -f v2alpha1/python/kfp/pipeline_spec/pipeline_spec_pb2.py
rm -f v2alpha1/google/rpc/status.proto

##########################
# The following are IMPLEMENTATION DETAILS.
##########################

# Generates proto packages locally, this should only be called:
# * during development
# * inside the prebuilt docker container
.PHONY: generate
generate: go_pipelinespec go_cachekey

go_pipelinespec: v2alpha1/pipeline_spec.proto v2alpha1/google/rpc/status.proto
mkdir -p v2alpha1/go/pipelinespec
cd v2alpha1 && protoc -I=. \
--go_out=go/pipelinespec \
--go_opt=paths=source_relative \
pipeline_spec.proto

go_cachekey: v2alpha1/pipeline_spec.proto v2alpha1/cache_key.proto
mkdir -p v2alpha1/go/cachekey
cd v2alpha1 && protoc -I=. \
--go_out=go/cachekey \
--go_opt=paths=source_relative \
cache_key.proto

# Fetch dependency proto
v2alpha1/google/rpc/status.proto:
mkdir -p v2alpha1/google/rpc
wget -O v2alpha1/google/rpc/status.proto https://raw.githubusercontent.com/googleapis/googleapis/047d3a8ac7f75383855df0166144f891d7af08d9/google/rpc/status.proto

# protoc-gen-go is already installed in api-generator image
.PHONY: protoc-gen-go
protoc-gen-go:
go install google.golang.org/protobuf/cmd/protoc-gen-go
24 changes: 18 additions & 6 deletions api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,30 @@

## Generate golang proto code

Documentation: <https://developers.google.com/protocol-buffers/docs/reference/go-generated>
Generate golang proto code:

```bash
make clean-go golang
```

## Generate Python proto package

Download `protoc` compiler binary from: <https://github.com/protocolbuffers/protobuf/releases/tag/v3.14.0>.
Generate kfp-pipeline-spec:

Install proto code generator:
Update `VERSION` in [v2alpha1/python/setup.py](https://github.com/kubeflow/pipelines/blob/master/api/v2alpha1/python/setup.py) if applicable.

```bash
go install google.golang.org/protobuf/cmd/protoc-gen-go
make clean-python python
```

Generate golang proto code:
## Generate both Python and golang proto code

Generate both Python and golang proto:

```bash
make gen_proto
make clean all
```

Note, there are no prerequisites, because the generation uses a prebuilt docker image with all the tools necessary.

Documentation: <https://developers.google.com/protocol-buffers/docs/reference/go-generated>
5 changes: 5 additions & 0 deletions api/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
module github.com/kubeflow/pipelines/api

go 1.16

require google.golang.org/protobuf v1.27.1
8 changes: 8 additions & 0 deletions api/go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 38 additions & 0 deletions api/v2alpha1/cache_key.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
// Copyright 2021 The Kubeflow Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

syntax = "proto3";

option go_package = "github.com/kubeflow/pipelines/api/v2alpha1/go/cachekey";
package ml_pipelines;

import "google/protobuf/any.proto";
import "pipeline_spec.proto";

message CacheKey {
map<string, ArtifactNameList> inputArtifactNames = 1;
map<string, Value> inputParameters = 2;
map<string, RuntimeArtifact> outputArtifactsSpec = 3;
map<string, string> outputParametersSpec=4;
ContainerSpec containerSpec=5;
}

message ContainerSpec {
string image = 1;
repeated string cmdArgs = 2;
}

message ArtifactNameList {
repeated string artifactNames = 1;
}
Loading

0 comments on commit 0708a56

Please sign in to comment.