Add the cos2cos sample for CodeEngine #196

subhasree91 · 2025-06-11T19:29:19Z

This application processes objects (files) in IBM Cloud Object Storage (COS) between a primary and secondary bucket. The sample can be tried out using CodeEngine.

Copied from https://github.ibm.com/ibmcloud-codeengine-internship/code-engine-cos2cos.

Signed-off-by: Shailza Thakur <[email protected]> Signed-off-by: Hamza Bharmal <[email protected]> Signed-off-by: Subhasree Das <[email protected]> Signed-off-by: Subhasree-Das <[email protected]>

reggeenr

Hi, thanks for putting so much work into this great sample. All in all it looks very very good. Apart from the comments that I already published, I would like to ask you whether you could some focus on making it really easy for users to get started with this example. Think about which property values can be inferred (if not set). Make assumptions. Be opinionated. Keep it easy for users to get it going... While more complex configurations should be applicable, we shouldn't confront all users with it right-away.

Keep up the great work and thanks for this contribution!!!

reggeenr · 2025-06-30T07:36:34Z

code-engine-cos2cos/Dockerfile

@@ -0,0 +1,25 @@
+FROM golang:1.23-alpine as builder


Hi, can you please adjust the Dockerfile and make sure that a) it does not pull any artifact from dockerhub (if no hostname is specified docker.io is used) and by the multi-staged build uses a distroless image in the end?

You can take the following one as an example
https://github.com/IBM/CodeEngine/blob/main/trusted-profiles/go/Dockerfile

reggeenr · 2025-06-30T07:37:00Z

code-engine-cos2cos/README.md

+### Setup & Configuration
+1. **Clone the Repository**:
+   ```bash
+   git clone https://github.ibm.com/ibmcloud-codeengine-internship/code-engine-cos2cos


This URL must be adjusted

Changed as follows

bash git clone https://github.com/IBM/CodeEngine.git cd code-engine-cos2cos

Now, that I read this, I noticed that we should call the folder cos2cos. Please wait with that renaming until we processed the entire review, otherwise this will be a mess.

However, in this PR the readme should already be adjusted

git clone https://github.com/IBM/CodeEngine.git cd cos2cos

We agree with your point. For now we have updated the readme with above changes.

code-engine-cos2cos/README.md

reggeenr · 2025-06-30T07:40:20Z

code-engine-cos2cos/README.md

+#### 2. Build Using Source Code (Local Source)
+
+To build the image from local source code using IBM Cloud Code Engine:
+
+```bash
+ibmcloud ce build create --name ${BUILD_NAME} --build-type local --image ${REGISTRY}/${NAMESPACE}/${IMAGE_NAME}
+ibmcloud ce buildrun submit --build ${BUILD_NAME} --name ${BUILD_NAME}-build-run
+```
+
+#### 3. Build Using Git-based Source
+
+To build the image using a Git repository:
+
+1. Create a deploy key or user-access key in your GitHub repository.
+2. Add the private key by creating an SSH secret in IBM Cloud Code Engine.
+3. Create a build using the Git repository:
+
+```bash
+ibmcloud ce build create \
+    --name ${BUILD_NAME} \
+    --image ${REGISTRY}/${NAMESPACE}/${IMAGE_NAME} \
+    --source ${GIT_SSH_URL} \
+    --context-dir / \
+    --strategy dockerfile \
+    --git-repo-secret ${GIT_SSH_SECRET}
+```
+
+4. Submit the build:
+
+```bash
+ibmcloud ce buildrun submit --build ${BUILD_NAME}
+```


I suggest to make use of our integrated build and building the container image along with creating the job template:

ibmcloud ce job create --name ... \ --src "." \ --wait

Suggested change implemented in config.sh file from line number 185.

code-engine-cos2cos/bucketOperations/bucketOps.go

code-engine-cos2cos/metadata/local-timestamp.go

code-engine-cos2cos/process/process.go

reggeenr · 2025-06-30T07:54:46Z

code-engine-cos2cos/process/process_test.go

+	// ginkgo.Context("should handle errors", func() {
+
+	// })


If the code isn't needed, please remove it

code-engine-cos2cos/run.sh

…perties

reggeenr

Thanks for the fast turnarounds. Really appreciate the effort.

I will take a look at mechanics of this asset more closely while trying it out.

reggeenr · 2025-07-01T13:54:27Z

code-engine-cos2cos/Dockerfile

+# Stage 2: Final stage (minimal image)
+FROM gcr.io/distroless/static-debian12
+
+# RUN apk --no-cache add ca-certificates


If not needed, would you mind to remove it?

reggeenr · 2025-07-01T13:59:49Z

code-engine-cos2cos/main.go

+	os.Setenv("IBM_COS_CRTokenFilePath_PRIMARY", "/var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token")
+	os.Setenv("IBM_COS_CRTokenFilePath_SECONDARY", "/var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token")


The Trusted Profile token path will always be the same, I don't see a reason to specify it twice and make it configurable.

Now we are using a single trusted profile and we have also changed the CRTokenFilePath to a static variable within the code and accordingly made the changes in main.go.

reggeenr · 2025-07-01T14:01:51Z

code-engine-cos2cos/config.sh

+  --from-literal IBM_COS_CRTokenFilePath_PRIMARY=${IBM_COS_CRTokenFilePath_PRIMARY} \
+  --from-literal IBM_COS_CRTokenFilePath_SECONDARY=${IBM_COS_CRTokenFilePath_SECONDARY} \


I don't see the need to have the token path configurable, please just use a static variable

Changed as requested to a single variable.

reggeenr · 2025-07-01T14:03:31Z

code-engine-cos2cos/config.sh

+  --from-literal IBM_COS_TRUSTED_PROFILE_ID_PRIMARY=${COS_TRUSTED_PROFILE_ID_PRIMARY} \
+  --from-literal IBM_COS_TRUSTED_PROFILE_ID_SECONDARY=${COS_TRUSTED_PROFILE_ID_SECONDARY}


For complexity reasons, please use a single trusted profile that grants permissions to the primary and secondary COS bucket. In the blog post that I wrote about trusted profiles, I just split it up into two trusted profiles to make it clear that is possible in general

Changed as requested to a single variable.
Note:- There has been a slight modification in main.go specifically while creating the COS client. Instead of fetching CRTokenFIlePath for primary and secondary from env file, a static value is passed which is mentioned in main.go

reggeenr · 2025-07-01T14:14:57Z

code-engine-cos2cos/config.sh

+  --env-from-secret ${AUTH_SECRET} \
+  --argument true 2>/dev/null \
+  --wait
+  # --registry-secret ${CONTAINER_REGISTRY_SECRET} \


shall be removed as not longer needed

reggeenr · 2025-07-01T14:15:02Z

code-engine-cos2cos/config.sh

+  # echo "Job '${JOB_NAME}' already exists. Exiting"
+  # exit 1


shall be removed as not longer needed

Instead of this line, we have added an if statement asking the user if they want to override the existing job or create a new one.

reggeenr · 2025-07-01T14:15:06Z

code-engine-cos2cos/config.sh

+  --env-from-secret ${BASE_SECRET} \
+  --env-from-secret ${AUTH_SECRET} \
+  --argument true 2>/dev/null
+  # --registry-secret ${CONTAINER_REGISTRY_SECRET} \


shall be removed as not longer needed

reggeenr · 2025-07-01T14:16:13Z

code-engine-cos2cos/env_sample

+
+# Primary Bucket Trusted Profile Credentials
+IBM_COS_TRUSTED_PROFILE_ID_PRIMARY= <>
+IBM_COS_CRTokenFilePath_PRIMARY=/var/run/secrets/tokens/service-account-token


I suggest to remove that one

We have removed the env_sample file since it is no longer in use.

reggeenr · 2025-07-01T14:18:18Z

code-engine-cos2cos/config.sh

+echo "Step 8.4: Compute Resource Token"
+curl \
+  --request PATCH "https://api.${PROJECT_REGION}.codeengine.cloud.ibm.com/v2/projects/$(ibmcloud ce project current --output json | jq -r .guid)/jobs/${JOB_NAME}" \
+  --header 'Accept: application/json' \
+  --header "Authorization: $(ibmcloud iam oauth-tokens --output json | jq -r '.iam_token')" \
+  --header 'Content-Type: application/merge-patch+json' \
+  --header 'If-Match: *' \
+  --data-raw "{
+    \"run_compute_resource_token_enabled\": true
+  }" 2>/dev/null


If I am not mistaken, this code can be replaced with the corresponding option in the job create/update operation, as we added support for trusted profiles for jobs

Thank you for pointing this out. We replaced it with option in job create/update --trusted-profile-enabled.

…uth-secret

subhasree91 force-pushed the cos2cos-internship branch 2 times, most recently from f7c5043 to 35704c2 Compare June 11, 2025 19:48

Add the cos2cos sample for CodeEngine

1ec8829

Signed-off-by: Shailza Thakur <[email protected]> Signed-off-by: Hamza Bharmal <[email protected]> Signed-off-by: Subhasree Das <[email protected]> Signed-off-by: Subhasree-Das <[email protected]>

subhasree91 force-pushed the cos2cos-internship branch from 35704c2 to 1ec8829 Compare June 11, 2025 19:51

subhasree91 requested a review from reggeenr June 11, 2025 19:57

subhasree91 assigned subhasree91, H-Bharmal and ShailzaThakur7 Jun 11, 2025

reggeenr requested changes Jun 30, 2025

View reviewed changes

Shailza Thakur added 10 commits July 1, 2025 18:10

Update readme

122e6b7

Update data.sh with fewer variables

f504dcf

Remove container registry secret creation

2bcc988

Delete build.sh

eaf359c

Remove dead-code

6aaf438

Update readme

b36dc7b

Remove option for service-credentials. Set 3 variables to default pro…

bc4cd41

…perties

Update Dockerfile to use distroless image

012d944

Remove comments

7bd01f2

Remove comments

f9b4398

reggeenr requested changes Jul 1, 2025

View reviewed changes

Hamza added 7 commits July 2, 2025 18:04

update script to use single trusted profile, Update base-secret and a…

66eaf33

…uth-secret

Remove extra variables to default values

4b80155

Remove dead-code

55713e0

Delete sample env

dde1e3f

Update timestamp.go to use default filename

7f2462d

Update to use default values, Support single-trusted-profile

6c828db

Update readme

30b9e00

		os.Setenv("IBM_COS_CRTokenFilePath_PRIMARY", "/var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token")
		os.Setenv("IBM_COS_CRTokenFilePath_SECONDARY", "/var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token")

		--from-literal IBM_COS_CRTokenFilePath_PRIMARY=${IBM_COS_CRTokenFilePath_PRIMARY} \
		--from-literal IBM_COS_CRTokenFilePath_SECONDARY=${IBM_COS_CRTokenFilePath_SECONDARY} \

		--from-literal IBM_COS_TRUSTED_PROFILE_ID_PRIMARY=${COS_TRUSTED_PROFILE_ID_PRIMARY} \
		--from-literal IBM_COS_TRUSTED_PROFILE_ID_SECONDARY=${COS_TRUSTED_PROFILE_ID_SECONDARY}

Add the cos2cos sample for CodeEngine #196

Are you sure you want to change the base?

Add the cos2cos sample for CodeEngine #196

Conversation

subhasree91 commented Jun 11, 2025

Uh oh!

reggeenr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reggeenr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reggeenr Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reggeenr Jul 1, 2025 •

edited

Loading