Distributed Execution #1260

thomhurst · 2025-09-30T19:12:05Z

No description provided.

github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

…ution

…oss multiple OS

.github/workflows/dotnet.yml

+    runs-on: ubuntu-latest
    steps:
      - name: Add mask
        run: |
-          echo "::add-mask::${{ secrets.DOTNET_FORMAT_PUSH_TOKEN }}" 
+          echo "::add-mask::${{ secrets.DOTNET_FORMAT_PUSH_TOKEN }}"
          echo "::add-mask::${{ secrets.NuGet__ApiKey }}"
          echo "::add-mask::${{ secrets.ADMIN_TOKEN }}"
          echo "::add-mask::${{ secrets.CODACY_APIKEY }}"
+
      - uses: actions/checkout@v5
        with:
          fetch-depth: 0
          persist-credentials: false
-      - name: Set Up Performant Windows Drive
-        if: matrix.os == 'windows-latest'
-        uses: samypr100/setup-dev-drive@v3
-        with:
-          drive-size: 5GB
+
      - name: Setup .NET
        uses: actions/setup-dotnet@v5
        with:
          dotnet-version: 9.0.x
+
      - name: Cache NuGet
        uses: actions/cache@v4
        with:
-          path: ${{ matrix.os == 'windows-latest' && format('{0}\{1}', env.DEV_DRIVE, 'nuget') || '~/.nuget/packages' }}
+          path: ~/.nuget/packages
          key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}
          restore-keys: |
-            ${{ runner.os }}-nuget- }}
+            ${{ runner.os }}-nuget-
+
      - name: Build ModularPipelines.Analyzers.sln
        run: dotnet build ModularPipelines.Analyzers.sln -c Release
-      - name: Build
+
+      - name: Build All Solutions
        shell: bash
        run: |
          for SOLUTION in ${{ env.SOLUTIONS }}
          do
            dotnet build $SOLUTION -c Release
          done
-      
-      - name: Run Pipeline
-        run: dotnet run -c Release --framework net8.0
-        working-directory: "src/ModularPipelines.Build"
+
+      - name: Upload Build Artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: build-output
+          path: |
+            **/bin/Release/**
+            src/ModularPipelines.Build/appsettings.json
+          retention-days: 1
+
+  orchestrator:


To fix the problem, add an explicit permissions key at the top level of the workflow file (.github/workflows/dotnet.yml) so that all jobs get minimal privileges unless overridden. The vast majority of .NET build/test/package steps do not require write permissions to the repository; contents: read is usually sufficient. If any job specifically requires additional permissions (e.g., to make PR comments or create releases), that job should get a specific permissions override, but that is not evident in the provided snippet.

Steps:

Insert a top-level permissions: block right after the workflow name: and before on: (best practice and visually separates configuration sections).

Set contents: read as the explicit minimal starting permission, following CodeQL’s recommendation.

No code logic, steps, secrets, or jobs need to change—just workflow YAML configuration.

.github/workflows/dotnet.yml

…rker execution

…URL handling

.github/workflows/dotnet.yml

+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - uses: actions/checkout@v5
+        with:
+          fetch-depth: 0
+
+      - uses: actions/download-artifact@v4
+        with:
+          name: build-output
+
+      - name: Setup .NET
+        uses: actions/setup-dotnet@v5
+        with:
+          dotnet-version: 9.0.x
+
+      - name: Setup Cloudflared Tunnel
+        id: tunnel
+        run: |
+          # Download and setup cloudflared
+          wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
+          chmod +x cloudflared-linux-amd64
+
+          # Start tunnel in background
+          ./cloudflared-linux-amd64 tunnel --url http://localhost:8080 > tunnel.log 2>&1 &
+          TUNNEL_PID=$!
+          echo "TUNNEL_PID=$TUNNEL_PID" >> $GITHUB_ENV
+
+          # Wait for tunnel to be ready and extract URL
+          echo "Waiting for cloudflared tunnel to establish..."
+          TUNNEL_URL=""
+          for i in {1..60}; do
+            sleep 3
+            if [ -f tunnel.log ]; then
+              # Look for the tunnel URL in the log
+              TUNNEL_URL=$(grep -oE 'https://[a-zA-Z0-9-]+\.trycloudflare\.com' tunnel.log | head -1 || true)
+              if [ -n "$TUNNEL_URL" ]; then
+                echo "✓ Tunnel established: $TUNNEL_URL"
+                echo "url=$TUNNEL_URL" >> $GITHUB_OUTPUT
+                echo "$TUNNEL_URL" > tunnel-url.txt
+                break
+              fi
+            fi
+            echo "  Waiting for tunnel... ($i/60)"
+          done
+
+          if [ -z "$TUNNEL_URL" ]; then
+            echo "✗ Failed to get tunnel URL after 3 minutes"
+            echo "Tunnel log contents:"
+            cat tunnel.log || echo "No log file found"
+            exit 1
+          fi
+
+          # Give the tunnel a moment to stabilize
+          sleep 5
+
+      - name: Upload Tunnel URL Artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: tunnel-url
+          path: tunnel-url.txt
+          retention-days: 1
+
+      - name: Run Orchestrator
+        run: dotnet run --framework net9.0 orchestrator 8080
+        working-directory: src/ModularPipelines.Build
+        timeout-minutes: 30
        env:
          GITHUB_TOKEN: ${{ github.token }}
          DOTNET_ENVIRONMENT: ${{ github.ref == 'refs/heads/main' && 'Production' || 'Development' }}
          NuGet__ApiKey: ${{ github.ref == 'refs/heads/main' && secrets.NuGet__ApiKey || null }}
          GitHub__Actor: ${{ github.actor }}
          GitHub__Repository__Id: ${{ github.repository_id }}
          GitHub__StandardToken: ${{ secrets.DOTNET_FORMAT_PUSH_TOKEN }}
          GitHub__AdminToken: ${{ secrets.ADMIN_TOKEN }}
-          Publish__ShouldPublish: ${{ (github.event.inputs.publish-packages || false) && matrix.os == 'ubuntu-latest' }}
+          Publish__ShouldPublish: ${{ (github.event.inputs.publish-packages || false) }}
          Publish__IsAlpha: ${{ github.event.inputs.is-alpha || true }}
          Codacy__ApiKey: ${{ secrets.CODACY_APIKEY }}
          CodeCov__Token: ${{ secrets.CODECOV_TOKEN }}
          EMAIL_PASSWORD: ${{ secrets.EMAIL_PASSWORD }}
+
+  worker-windows:
+    runs-on: windows-latest


The best fix is to add a permissions block at the root (top) level of the workflow file .github/workflows/dotnet.yml, so that all jobs in the workflow inherit the minimal permissions of contents: read. This will ensure that GITHUB_TOKEN in all jobs only has read access to repository contents, preventing unintended write operations. Place the block above the jobs: key for clarity, typically right under the workflow name: and on: sections.

Add the following block:

permissions: contents: read

No imports or additional definitions are required for this workflow-level configuration.

.github/workflows/dotnet.yml

+    needs: build
+    steps:
+      - uses: actions/checkout@v5
+
+      - uses: actions/download-artifact@v4
+        with:
+          name: build-output
+
+      - name: Setup .NET
+        uses: actions/setup-dotnet@v5
+        with:
+          dotnet-version: 9.0.x
+
+      - name: Cache NuGet
+        uses: actions/cache@v4
+        with:
+          path: ~/.nuget/packages
+          key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}
+
+      - name: Wait for Orchestrator and Get Tunnel URL
+        shell: bash
+        run: |
+          echo "Waiting for orchestrator to publish tunnel URL..."
+          TUNNEL_URL=""
+          for i in {1..120}; do
+            # Try to download the tunnel URL artifact
+            if gh run download ${{ github.run_id }} -n tunnel-url 2>/dev/null; then
+              TUNNEL_URL=$(cat tunnel-url.txt)
+              echo "✓ Got tunnel URL: $TUNNEL_URL"
+              echo "TUNNEL_URL=$TUNNEL_URL" >> $GITHUB_ENV
+              break
+            fi
+            echo "  Waiting for tunnel URL artifact... ($i/120)"
+            sleep 5
+          done
+
+          if [ -z "$TUNNEL_URL" ]; then
+            echo "✗ Failed to get tunnel URL after 10 minutes"
+            exit 1
+          fi
+        env:
+          GH_TOKEN: ${{ github.token }}
+
+      - name: Run Worker
+        run: dotnet run --framework net9.0 worker "${{ env.TUNNEL_URL }}" "worker-windows"
+        working-directory: src/ModularPipelines.Build
+        timeout-minutes: 25
+
+  worker-macos:
+    runs-on: macos-latest


To fix the detected issue, we need to explicitly declare the minimal required permissions for the workflow.

The recommended baseline is contents: read, which only allows the workflow to read repository contents.

You should add a permissions block at the top level of the workflow (right below name: and before on:) if the same permissions suffice for all jobs, or per job if jobs differ in their needs.

For this workflow, unless you know a job genuinely needs write permissions for a resource (e.g. pull-requests: write), start with contents: read.

Edit .github/workflows/dotnet.yml to insert the following after the name: field:
permissions: contents: read

No changes to external methods, imports, or definitions are required.

.github/workflows/dotnet.yml

+    needs: build
+    steps:
+      - uses: actions/checkout@v5
+
+      - uses: actions/download-artifact@v4
+        with:
+          name: build-output
+
+      - name: Setup .NET
+        uses: actions/setup-dotnet@v5
+        with:
+          dotnet-version: 9.0.x
+
+      - name: Cache NuGet
+        uses: actions/cache@v4
+        with:
+          path: ~/.nuget/packages
+          key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}
+
+      - name: Wait for Orchestrator and Get Tunnel URL
+        shell: bash
+        run: |
+          echo "Waiting for orchestrator to publish tunnel URL..."
+          TUNNEL_URL=""
+          for i in {1..120}; do
+            # Try to download the tunnel URL artifact
+            if gh run download ${{ github.run_id }} -n tunnel-url 2>/dev/null; then
+              TUNNEL_URL=$(cat tunnel-url.txt)
+              echo "✓ Got tunnel URL: $TUNNEL_URL"
+              echo "TUNNEL_URL=$TUNNEL_URL" >> $GITHUB_ENV
+              break
+            fi
+            echo "  Waiting for tunnel URL artifact... ($i/120)"
+            sleep 5
+          done
+
+          if [ -z "$TUNNEL_URL" ]; then
+            echo "✗ Failed to get tunnel URL after 10 minutes"
+            exit 1
+          fi
+        env:
+          GH_TOKEN: ${{ github.token }}
+
+      - name: Run Worker
+        run: dotnet run --framework net9.0 worker "${{ env.TUNNEL_URL }}" "worker-macos"
+        working-directory: src/ModularPipelines.Build
+        timeout-minutes: 25
+
+  worker-linux-2:
+    runs-on: ubuntu-latest


To fix the problem, set a top-level permissions block in your workflow YAML file to restrict the GITHUB_TOKEN's scope. The most minimal, generally safe starting point for typical .NET build/test workflows is contents: read, which permits read-only access to repository contents. You should add this block immediately after the name key and before on: at the root of your workflow. If any jobs or steps in the workflow require escalated permissions, you can override or expand the block later, either per-job or per-step. For now, adding permissions: contents: read at the top ensures that token permissions are not wider than necessary by default.

.github/workflows/dotnet.yml

To fix the issue, an explicit permissions block must be added to the workflow. The best way is to add a permissions block at the root level of .github/workflows/dotnet.yml, which will apply to all jobs unless jobs specify their own permissions. Since most jobs only need access to read repository contents (checkout) and download artifacts, start with minimal permissions: contents: read. If any job later needs additional permissions (such as to write to issues, pull requests, etc.), those jobs can explicitly add job-level permissions. The explicit block should be inserted after the name: and before the on: or env: key.

… modes

Distributed Execution

ad70084

thomhurst had a problem deploying to Pull Requests September 30, 2025 19:12 — with GitHub Actions Failure

thomhurst temporarily deployed to Pull Requests September 30, 2025 19:12 — with GitHub Actions Inactive

github-advanced-security bot found potential problems Sep 30, 2025

View reviewed changes

feat(distributed): implement OS and tool requirements for module exec…

739a34d

…ution

thomhurst had a problem deploying to Pull Requests September 30, 2025 19:29 — with GitHub Actions Failure

thomhurst temporarily deployed to Pull Requests September 30, 2025 19:29 — with GitHub Actions Inactive

feat(distributed): implement distributed execution for unit tests acr…

b44a055

…oss multiple OS

thomhurst marked this pull request as ready for review September 30, 2025 19:52

github-advanced-security bot found potential problems Sep 30, 2025

View reviewed changes

thomhurst and others added 5 commits September 30, 2025 21:03

feat(tunnel): improve tunnel establishment logic and error handling

d510d00

feat(build): update target frameworks to net9.0

22b1153

feat(distributed): specify .NET 9.0 framework for orchestrator and wo…

e69e076

…rker execution

Formatting Markdown

d2b7a1d

feat(distributed): enhance orchestrator and worker setup with tunnel …

0bf1acc

…URL handling

github-advanced-security bot found potential problems Sep 30, 2025

View reviewed changes

feat(distributed): improve console output for orchestrator and worker…

f3d7b20

… modes

@@ -1,5 +1,8 @@
             name: .NET
+            permissions:
+              contents: read
             on:
               push:
                 branches: ["main"]

@@ -18,6 +18,8 @@
                     required: true
                     default: true
+            permissions:
+              contents: read
             env:
               SOLUTIONS: ModularPipelines.sln ModularPipelines.Examples.sln src/ModularPipelines.Azure/ModularPipelines.Azure.sln src/ModularPipelines.AmazonWebServices/ModularPipelines.AmazonWebServices.sln src/ModularPipelines.Google/ModularPipelines.Google.sln

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Distributed Execution #1260

Distributed Execution #1260

Uh oh!

thomhurst commented Sep 30, 2025

Uh oh!

github-advanced-security bot left a comment

Uh oh!

Check warning

Copilot Autofix

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Copilot Autofix

Check warning

Copilot Autofix

Check warning

Copilot Autofix

Check warning

Copilot Autofix

Uh oh!

Uh oh!

Distributed Execution #1260

Are you sure you want to change the base?

Distributed Execution #1260

Uh oh!

Conversation

thomhurst commented Sep 30, 2025

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Check warning

Copilot Autofix

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Copilot Autofix

Check warning

Copilot Autofix

Check warning

Copilot Autofix

Check warning

Copilot Autofix

Uh oh!