Skip to content

Distributed Execution #1260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Distributed Execution #1260

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ad70084
Distributed Execution
thomhurst Sep 30, 2025
739a34d
feat(distributed): implement OS and tool requirements for module exec…
thomhurst Sep 30, 2025
b44a055
feat(distributed): implement distributed execution for unit tests acr…
thomhurst Sep 30, 2025
d510d00
feat(tunnel): improve tunnel establishment logic and error handling
thomhurst Sep 30, 2025
22b1153
feat(build): update target frameworks to net9.0
thomhurst Sep 30, 2025
e69e076
feat(distributed): specify .NET 9.0 framework for orchestrator and wo…
thomhurst Sep 30, 2025
d2b7a1d
Formatting Markdown
thomhurst Sep 30, 2025
0bf1acc
feat(distributed): enhance orchestrator and worker setup with tunnel …
thomhurst Sep 30, 2025
f3d7b20
feat(distributed): improve console output for orchestrator and worker…
thomhurst Sep 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .claude/settings.local.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
"allow": [
"WebFetch(domain:github.com)",
"Bash(dotnet test:*)",
"Bash(dotnet build:*)"
"Bash(dotnet build:*)",
"WebFetch(domain:stackoverflow.com)",
"WebSearch"
],
"deny": []
}
Expand Down
267 changes: 241 additions & 26 deletions .github/workflows/dotnet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@
workflow_dispatch:
inputs:
publish-packages:
description: Publish packages?
description: Publish packages?
type: boolean
required: true
default: false
is-alpha:
description: Alpha version?
description: Alpha version?
type: boolean
required: true
default: true
Expand All @@ -22,66 +22,281 @@
SOLUTIONS: ModularPipelines.sln ModularPipelines.Examples.sln src/ModularPipelines.Azure/ModularPipelines.Azure.sln src/ModularPipelines.AmazonWebServices/ModularPipelines.AmazonWebServices.sln src/ModularPipelines.Google/ModularPipelines.Google.sln

jobs:
pipeline:
environment: ${{ github.ref == 'refs/heads/main' && 'Production' || 'Pull Requests' }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
fail-fast: false
runs-on: ${{ matrix.os }}
env:
NUGET_PACKAGES: ${{ matrix.os == 'windows-latest' && 'E:\nuget' || null }}

build:
runs-on: ubuntu-latest
steps:
- name: Add mask
run: |
echo "::add-mask::${{ secrets.DOTNET_FORMAT_PUSH_TOKEN }}"
echo "::add-mask::${{ secrets.DOTNET_FORMAT_PUSH_TOKEN }}"
echo "::add-mask::${{ secrets.NuGet__ApiKey }}"
echo "::add-mask::${{ secrets.ADMIN_TOKEN }}"
echo "::add-mask::${{ secrets.CODACY_APIKEY }}"
- uses: actions/checkout@v5
with:
fetch-depth: 0
persist-credentials: false
- name: Set Up Performant Windows Drive
if: matrix.os == 'windows-latest'
uses: samypr100/setup-dev-drive@v3
with:
drive-size: 5GB

- name: Setup .NET
uses: actions/setup-dotnet@v5
with:
dotnet-version: 9.0.x

- name: Cache NuGet
uses: actions/cache@v4
with:
path: ${{ matrix.os == 'windows-latest' && format('{0}\{1}', env.DEV_DRIVE, 'nuget') || '~/.nuget/packages' }}
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}
restore-keys: |
${{ runner.os }}-nuget- }}
${{ runner.os }}-nuget-
- name: Build ModularPipelines.Analyzers.sln
run: dotnet build ModularPipelines.Analyzers.sln -c Release
- name: Build

- name: Build All Solutions
shell: bash
run: |
for SOLUTION in ${{ env.SOLUTIONS }}
do
dotnet build $SOLUTION -c Release
done
- name: Run Pipeline
run: dotnet run -c Release --framework net8.0
working-directory: "src/ModularPipelines.Build"
- name: Upload Build Artifacts
uses: actions/upload-artifact@v4
with:
name: build-output
path: |
**/bin/Release/**
src/ModularPipelines.Build/appsettings.json
retention-days: 1

orchestrator:
Comment on lines +26 to +73

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 7 days ago

To fix the problem, add an explicit permissions key at the top level of the workflow file (.github/workflows/dotnet.yml) so that all jobs get minimal privileges unless overridden. The vast majority of .NET build/test/package steps do not require write permissions to the repository; contents: read is usually sufficient. If any job specifically requires additional permissions (e.g., to make PR comments or create releases), that job should get a specific permissions override, but that is not evident in the provided snippet.

Steps:

  • Insert a top-level permissions: block right after the workflow name: and before on: (best practice and visually separates configuration sections).
  • Set contents: read as the explicit minimal starting permission, following CodeQL’s recommendation.
  • No code logic, steps, secrets, or jobs need to change—just workflow YAML configuration.
Suggested changeset 1
.github/workflows/dotnet.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml
--- a/.github/workflows/dotnet.yml
+++ b/.github/workflows/dotnet.yml
@@ -1,5 +1,8 @@
 name: .NET
 
+permissions:
+  contents: read
+
 on:
   push:
     branches: ["main"]
EOF
@@ -1,5 +1,8 @@
name: .NET

permissions:
contents: read

on:
push:
branches: ["main"]
Copilot is powered by AI and may make mistakes. Always verify output.
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0

- uses: actions/download-artifact@v4
with:
name: build-output

- name: Setup .NET
uses: actions/setup-dotnet@v5
with:
dotnet-version: 9.0.x

- name: Setup Cloudflared Tunnel
id: tunnel
run: |
# Download and setup cloudflared
wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
chmod +x cloudflared-linux-amd64
# Start tunnel in background
./cloudflared-linux-amd64 tunnel --url http://localhost:8080 > tunnel.log 2>&1 &
TUNNEL_PID=$!
echo "TUNNEL_PID=$TUNNEL_PID" >> $GITHUB_ENV
# Wait for tunnel to be ready and extract URL
echo "Waiting for cloudflared tunnel to establish..."
TUNNEL_URL=""
for i in {1..60}; do
sleep 3
if [ -f tunnel.log ]; then
# Look for the tunnel URL in the log
TUNNEL_URL=$(grep -oE 'https://[a-zA-Z0-9-]+\.trycloudflare\.com' tunnel.log | head -1 || true)
if [ -n "$TUNNEL_URL" ]; then
echo "✓ Tunnel established: $TUNNEL_URL"
echo "url=$TUNNEL_URL" >> $GITHUB_OUTPUT
echo "$TUNNEL_URL" > tunnel-url.txt
break
fi
fi
echo " Waiting for tunnel... ($i/60)"
done
if [ -z "$TUNNEL_URL" ]; then
echo "✗ Failed to get tunnel URL after 3 minutes"
echo "Tunnel log contents:"
cat tunnel.log || echo "No log file found"
exit 1
fi
# Give the tunnel a moment to stabilize
sleep 5
- name: Upload Tunnel URL Artifact
uses: actions/upload-artifact@v4
with:
name: tunnel-url
path: tunnel-url.txt
retention-days: 1

- name: Run Orchestrator
run: dotnet run --framework net9.0 orchestrator 8080
working-directory: src/ModularPipelines.Build
timeout-minutes: 30
env:
GITHUB_TOKEN: ${{ github.token }}
DOTNET_ENVIRONMENT: ${{ github.ref == 'refs/heads/main' && 'Production' || 'Development' }}
NuGet__ApiKey: ${{ github.ref == 'refs/heads/main' && secrets.NuGet__ApiKey || null }}
GitHub__Actor: ${{ github.actor }}
GitHub__Repository__Id: ${{ github.repository_id }}
GitHub__StandardToken: ${{ secrets.DOTNET_FORMAT_PUSH_TOKEN }}
GitHub__AdminToken: ${{ secrets.ADMIN_TOKEN }}
Publish__ShouldPublish: ${{ (github.event.inputs.publish-packages || false) && matrix.os == 'ubuntu-latest' }}
Publish__ShouldPublish: ${{ (github.event.inputs.publish-packages || false) }}
Publish__IsAlpha: ${{ github.event.inputs.is-alpha || true }}
Codacy__ApiKey: ${{ secrets.CODACY_APIKEY }}
CodeCov__Token: ${{ secrets.CODECOV_TOKEN }}
EMAIL_PASSWORD: ${{ secrets.EMAIL_PASSWORD }}

worker-windows:
Comment on lines 74 to 156

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 7 days ago

The best fix is to add a permissions block at the root (top) level of the workflow file .github/workflows/dotnet.yml, so that all jobs in the workflow inherit the minimal permissions of contents: read. This will ensure that GITHUB_TOKEN in all jobs only has read access to repository contents, preventing unintended write operations. Place the block above the jobs: key for clarity, typically right under the workflow name: and on: sections.

Add the following block:

permissions:
  contents: read

No imports or additional definitions are required for this workflow-level configuration.

Suggested changeset 1
.github/workflows/dotnet.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml
--- a/.github/workflows/dotnet.yml
+++ b/.github/workflows/dotnet.yml
@@ -18,6 +18,8 @@
         required: true
         default: true
 
+permissions:
+  contents: read
 env:
   SOLUTIONS: ModularPipelines.sln ModularPipelines.Examples.sln src/ModularPipelines.Azure/ModularPipelines.Azure.sln src/ModularPipelines.AmazonWebServices/ModularPipelines.AmazonWebServices.sln src/ModularPipelines.Google/ModularPipelines.Google.sln
 
EOF
@@ -18,6 +18,8 @@
required: true
default: true

permissions:
contents: read
env:
SOLUTIONS: ModularPipelines.sln ModularPipelines.Examples.sln src/ModularPipelines.Azure/ModularPipelines.Azure.sln src/ModularPipelines.AmazonWebServices/ModularPipelines.AmazonWebServices.sln src/ModularPipelines.Google/ModularPipelines.Google.sln

Copilot is powered by AI and may make mistakes. Always verify output.
runs-on: windows-latest
needs: build
steps:
- uses: actions/checkout@v5

- uses: actions/download-artifact@v4
with:
name: build-output

- name: Setup .NET
uses: actions/setup-dotnet@v5
with:
dotnet-version: 9.0.x

- name: Cache NuGet
uses: actions/cache@v4
with:
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}

- name: Wait for Orchestrator and Get Tunnel URL
shell: bash
run: |
echo "Waiting for orchestrator to publish tunnel URL..."
TUNNEL_URL=""
for i in {1..120}; do
# Try to download the tunnel URL artifact
if gh run download ${{ github.run_id }} -n tunnel-url 2>/dev/null; then
TUNNEL_URL=$(cat tunnel-url.txt)
echo "✓ Got tunnel URL: $TUNNEL_URL"
echo "TUNNEL_URL=$TUNNEL_URL" >> $GITHUB_ENV
break
fi
echo " Waiting for tunnel URL artifact... ($i/120)"
sleep 5
done
if [ -z "$TUNNEL_URL" ]; then
echo "✗ Failed to get tunnel URL after 10 minutes"
exit 1
fi
env:
GH_TOKEN: ${{ github.token }}

- name: Run Worker
run: dotnet run --framework net9.0 worker "${{ env.TUNNEL_URL }}" "worker-windows"
working-directory: src/ModularPipelines.Build
timeout-minutes: 25

worker-macos:
Comment on lines +157 to +206

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 7 days ago

To fix the detected issue, we need to explicitly declare the minimal required permissions for the workflow.

  • The recommended baseline is contents: read, which only allows the workflow to read repository contents.
  • You should add a permissions block at the top level of the workflow (right below name: and before on:) if the same permissions suffice for all jobs, or per job if jobs differ in their needs.
  • For this workflow, unless you know a job genuinely needs write permissions for a resource (e.g. pull-requests: write), start with contents: read.
  • Edit .github/workflows/dotnet.yml to insert the following after the name: field: 
    permissions:
  contents: read
  • No changes to external methods, imports, or definitions are required.
Suggested changeset 1
.github/workflows/dotnet.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml
--- a/.github/workflows/dotnet.yml
+++ b/.github/workflows/dotnet.yml
@@ -1,4 +1,6 @@
 name: .NET
+permissions:
+  contents: read
 
 on:
   push:
EOF
@@ -1,4 +1,6 @@
name: .NET
permissions:
contents: read

on:
push:
Copilot is powered by AI and may make mistakes. Always verify output.
runs-on: macos-latest
needs: build
steps:
- uses: actions/checkout@v5

- uses: actions/download-artifact@v4
with:
name: build-output

- name: Setup .NET
uses: actions/setup-dotnet@v5
with:
dotnet-version: 9.0.x

- name: Cache NuGet
uses: actions/cache@v4
with:
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}

- name: Wait for Orchestrator and Get Tunnel URL
shell: bash
run: |
echo "Waiting for orchestrator to publish tunnel URL..."
TUNNEL_URL=""
for i in {1..120}; do
# Try to download the tunnel URL artifact
if gh run download ${{ github.run_id }} -n tunnel-url 2>/dev/null; then
TUNNEL_URL=$(cat tunnel-url.txt)
echo "✓ Got tunnel URL: $TUNNEL_URL"
echo "TUNNEL_URL=$TUNNEL_URL" >> $GITHUB_ENV
break
fi
echo " Waiting for tunnel URL artifact... ($i/120)"
sleep 5
done
if [ -z "$TUNNEL_URL" ]; then
echo "✗ Failed to get tunnel URL after 10 minutes"
exit 1
fi
env:
GH_TOKEN: ${{ github.token }}

- name: Run Worker
run: dotnet run --framework net9.0 worker "${{ env.TUNNEL_URL }}" "worker-macos"
working-directory: src/ModularPipelines.Build
timeout-minutes: 25

worker-linux-2:
Comment on lines +207 to +256

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 7 days ago

To fix the problem, set a top-level permissions block in your workflow YAML file to restrict the GITHUB_TOKEN's scope. The most minimal, generally safe starting point for typical .NET build/test workflows is contents: read, which permits read-only access to repository contents. You should add this block immediately after the name key and before on: at the root of your workflow. If any jobs or steps in the workflow require escalated permissions, you can override or expand the block later, either per-job or per-step. For now, adding permissions: contents: read at the top ensures that token permissions are not wider than necessary by default.

Suggested changeset 1
.github/workflows/dotnet.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml
--- a/.github/workflows/dotnet.yml
+++ b/.github/workflows/dotnet.yml
@@ -1,4 +1,6 @@
 name: .NET
+permissions:
+  contents: read
 
 on:
   push:
EOF
@@ -1,4 +1,6 @@
name: .NET
permissions:
contents: read

on:
push:
Copilot is powered by AI and may make mistakes. Always verify output.
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/checkout@v5

- uses: actions/download-artifact@v4
with:
name: build-output

- name: Setup .NET
uses: actions/setup-dotnet@v5
with:
dotnet-version: 9.0.x

- name: Cache NuGet
uses: actions/cache@v4
with:
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}

- name: Wait for Orchestrator and Get Tunnel URL
run: |
echo "Waiting for orchestrator to publish tunnel URL..."
TUNNEL_URL=""
for i in {1..120}; do
# Try to download the tunnel URL artifact
if gh run download ${{ github.run_id }} -n tunnel-url 2>/dev/null; then
TUNNEL_URL=$(cat tunnel-url.txt)
echo "✓ Got tunnel URL: $TUNNEL_URL"
echo "TUNNEL_URL=$TUNNEL_URL" >> $GITHUB_ENV
break
fi
echo " Waiting for tunnel URL artifact... ($i/120)"
sleep 5
done
if [ -z "$TUNNEL_URL" ]; then
echo "✗ Failed to get tunnel URL after 10 minutes"
exit 1
fi
env:
GH_TOKEN: ${{ github.token }}

- name: Run Worker
run: dotnet run --framework net9.0 worker "$TUNNEL_URL" "worker-linux-2"
working-directory: src/ModularPipelines.Build
timeout-minutes: 25
6 changes: 6 additions & 0 deletions .idea/.idea.ModularPipelines/.idea/copilot.data.migration.agent.xml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/.idea.ModularPipelines/.idea/copilot.data.migration.ask.xml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/.idea.ModularPipelines/.idea/copilot.data.migration.ask2agent.xml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/.idea.ModularPipelines/.idea/copilot.data.migration.edit.xml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading