Skip to content

added pipelines for prtncon source type #682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
covesturtevant merged 4 commits into from
Jun 4, 2025
Merged

added pipelines for prtncon source type #682

covesturtevant merged 4 commits into from
Jun 4, 2025

Conversation

@visalachundru
Copy link
Contributor

@visalachundru visalachundru commented Jun 2, 2025

No description provided.

@visalachundru visalachundru self-assigned this Jun 2, 2025
covesturtevant
covesturtevant requested changes Jun 3, 2025
View reviewed changes
Copy link
Contributor

@covesturtevant covesturtevant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Please also create DEV github actions for prtncon. Apologies, I forgot to include that in the story.

@visalachundru
Copy link
Contributor Author

visalachundru commented Jun 3, 2025

Cove, I missed adding the git hub actions, but now I added them..

github-advanced-security[bot]
github-advanced-security bot found potential problems Jun 3, 2025
View reviewed changes
.github/workflows/DEV_prtncon_site_list.yml
Comment on lines +11 to +34
runs-on: arc-neon-gke
#runs-on: ubuntu-latest
env:
PACHD_ADDRESS: grpcs://pachyderm-dev.transitions-nonprod.gcp.neoninternal.org:443
PACH_TOKEN: ${{ secrets.RepoOwnerPachydermDev }}
REPO: prtncon_site_list # Pachyderm repo
BRANCH: master
IN_PATHS: 'pipe/prtncon/site-list.json' # Comma-separated list (no spaces) to one or more paths or directories. Length must match OUT_PATHS. If directory, all files in directory will be placed in pachyderm at corresponding entry of OUT_PATHS.
OUT_PATHS: 'site-list.json' # Comma-separated list (no spaces) of corresponding path(s) to place the files(s) in Pachyderm. Must be same length as IN_PATHS. If corresponding entry in IN_PATHS is a file, specify to the file. If corresponding entry in IN_PATHS is a directory, specify to the directory.
steps:
- uses: actions/checkout@v4
- run: ls -la

- name: Put file
uses: ./.github/actions/put-files
with:
pachd_address: ${{ env.PACHD_ADDRESS }}
pach_token: ${{ env.PACH_TOKEN }}
repo_name: ${{ env.REPO }}
branch_name: ${{ env.BRANCH }}
in_paths: ${{ env.IN_PATHS }}
out_paths: ${{ env.OUT_PATHS }}


No newline at end of file

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 7 months ago

To fix the issue, we will add a permissions block at the root level of the workflow file. This block will specify the least privileges required for the workflow to function correctly. Based on the provided workflow, the most likely required permission is contents: read, as the workflow primarily interacts with repository files. If the custom action (put-files) requires additional permissions, they can be added later after further analysis.

Suggested changeset 1
.github/workflows/DEV_prtncon_site_list.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/DEV_prtncon_site_list.yml b/.github/workflows/DEV_prtncon_site_list.yml
--- a/.github/workflows/DEV_prtncon_site_list.yml
+++ b/.github/workflows/DEV_prtncon_site_list.yml
@@ -1,2 +1,4 @@
 name: DEV-prtncon-site-list
+permissions:
+  contents: read
 on:
EOF
@@ -1,2 +1,4 @@
name: DEV-prtncon-site-list
permissions:
contents: read
on:
Copilot is powered by AI and may make mistakes. Always verify output.
.github/workflows/DEV_prtncon_update_dag.yml
Comment on lines +23 to +37
runs-on: ubuntu-latest # windows-latest || macos-latest
outputs:
# Use this changed_file_list if you plan to use get-changed-files-action
changed_file_list: ${{ steps.changed-files-action.outputs.changed_file_list }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # OR "2" -> To retrieve the preceding commit.

# Using get-changed-files-action
- name: Get changed files action
id: changed-files-action
uses: ./.github/actions/get-changed-files

update_pipelines:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 7 months ago

To fix the issue, add a permissions block to the workflow to explicitly limit the GITHUB_TOKEN permissions. Based on the workflow's functionality, it primarily reads repository contents and processes changes, so contents: read is sufficient. This block should be added at the root level of the workflow to apply to all jobs unless overridden.

Suggested changeset 1
.github/workflows/DEV_prtncon_update_dag.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/DEV_prtncon_update_dag.yml b/.github/workflows/DEV_prtncon_update_dag.yml
--- a/.github/workflows/DEV_prtncon_update_dag.yml
+++ b/.github/workflows/DEV_prtncon_update_dag.yml
@@ -1,2 +1,4 @@
 name: DEV-prtncon-update-dag
+permissions:
+  contents: read
 on:
EOF
@@ -1,2 +1,4 @@
name: DEV-prtncon-update-dag
permissions:
contents: read
on:
Copilot is powered by AI and may make mistakes. Always verify output.
.github/workflows/DEV_prtncon_update_dag.yml
Comment on lines +38 to +61
needs: changed_files
runs-on: arc-neon-gke
#runs-on: ubuntu-latest
env:
PACHD_ADDRESS: grpcs://pachyderm-dev.transitions-nonprod.gcp.neoninternal.org:443
PACH_TOKEN: ${{ secrets.RepoOwnerPachydermDev }}
PATHS: 'pipe/prtncon=pipe_list_prtncon.txt' # Format: '<directory>=<pipe_list_file>'. Separate multiple with comma (e.g. 'pipe/pqs1=pipe_list_pqs1.txt,pipe/parWaterSurface=pipe_list_parWaterSurface.txt'). Order matters.
TRANSACTION: True
UPDATE_SCOPE: changed # 'all' or 'changed'. If not specified, all will be updated. 'changed' will update/create any changed/non-existent pipelines.
CHANGED_FILES: ${{needs.changed_files.outputs.changed_file_list}}
steps:
- uses: actions/checkout@v4
- run: ls -la

- name: Update pipelines
uses: ./.github/actions/update-pipelines
with:
pachd_address: ${{ env.PACHD_ADDRESS }}
pach_token: ${{ env.PACH_TOKEN }}
paths: ${{ env.PATHS }}
transaction: ${{ env.TRANSACTION }}
update_scope: ${{ env.UPDATE_SCOPE }}
changed_files: ${{ env.CHANGED_FILES }}

No newline at end of file

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 7 months ago

To fix the issue, we will add a permissions block to the workflow. This block will specify the minimal permissions required for the workflow to function correctly. Based on the workflow's operations, the contents: read permission is sufficient for the changed_files job, as it only checks for changed files. For the update_pipelines job, additional permissions might be required depending on its interaction with the repository or external systems. However, since the provided code does not indicate any repository write operations, we will start with contents: read for both jobs.

Suggested changeset 1
.github/workflows/DEV_prtncon_update_dag.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/DEV_prtncon_update_dag.yml b/.github/workflows/DEV_prtncon_update_dag.yml
--- a/.github/workflows/DEV_prtncon_update_dag.yml
+++ b/.github/workflows/DEV_prtncon_update_dag.yml
@@ -1,2 +1,4 @@
 name: DEV-prtncon-update-dag
+permissions:
+  contents: read
 on:
EOF
@@ -1,2 +1,4 @@
name: DEV-prtncon-update-dag
permissions:
contents: read
on:
Copilot is powered by AI and may make mistakes. Always verify output.
covesturtevant
covesturtevant requested changes Jun 3, 2025
View reviewed changes
Copy link
Contributor

@covesturtevant covesturtevant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Thanks for adding the github actions. One final change needed - the site list is incomplete. You can use the "Site-product Matrix" tab in Blizzard and search for DP1.00013.001 to get the sites applicable for this source type. Or use a different PDR query of your choice.

@visalachundru
Copy link
Contributor Author

visalachundru commented Jun 4, 2025

Cove, made changes to the site list according to the "Site-product Matrix" tab in Blizzard and reran both trino and kafka pipelines and they ran successfully.

covesturtevant
covesturtevant approved these changes Jun 4, 2025
View reviewed changes
Copy link
Contributor

@covesturtevant covesturtevant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks.

@covesturtevant covesturtevant merged commit ef6e2f7 into master Jun 4, 2025
3 checks passed
@covesturtevant covesturtevant deleted the NSE-10840-prtncon branch June 4, 2025 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@covesturtevant covesturtevant covesturtevant approved these changes

Assignees

@visalachundru visalachundru

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@visalachundru @covesturtevant