Skip to content

79 request add benchmarks and integrate performance testing into cicd #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 47 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
def0b24
Bump crypto-js and oidc-client in /WebSample/ClientApp
dependabot[bot] Oct 25, 2023
e38b367
feat: Add CachingTest project with initial database operations tests
leonibr Jan 21, 2025
3bf2e1a
Merge branch 'master' of https://github.com/leonibr/community-extensi…
leonibr Jul 9, 2025
bf9de6c
Update .gitignore and project files for PostgreSQL caching tests
leonibr Jul 9, 2025
9f3abfe
Update .NET version in CI workflow to 9.0.x
leonibr Jul 9, 2025
b03a088
Update .NET version in CI workflow to support multiple versions
leonibr Jul 9, 2025
2654be6
Refactor exception handling in caching tests to use Npgsql.NpgsqlExce…
leonibr Jul 9, 2025
186be4b
Enhance CI workflow for test coverage reporting
leonibr Jul 9, 2025
f6d7abe
Merge branch 'master' of https://github.com/leonibr/community-extensi…
leonibr Jul 9, 2025
485d8f1
Update CI workflows for improved coverage reporting and NuGet deployment
leonibr Jul 9, 2025
871c524
Merge branch 'dependabot/npm_and_yarn/WebSample/ClientApp/crypto-js-a…
leonibr Jul 9, 2025
2f7c47b
Update README and CI workflow for code coverage enhancements
leonibr Jul 9, 2025
77a7886
Update GitHub Action coverage reporter to use the latest master branch
leonibr Jul 9, 2025
c365093
Update coverage reporting in GitHub Actions workflow
leonibr Jul 9, 2025
01f7d46
Update GitHub Actions workflow to use aGallea/tests-coverage-report a…
leonibr Jul 9, 2025
8f6ef2e
Enhance GitHub Actions workflow for coverage reporting
leonibr Jul 9, 2025
9996b75
feat: Add Azure Key Vault rotation support with reloadable connection…
leonibr Jul 10, 2025
5b4fd3e
chore: Bump version to 5.1.0 for Azure Key Vault rotation feature
leonibr Jul 10, 2025
39e165e
chore: Update GitHub Actions workflow to include 'next' branch for te…
leonibr Jul 10, 2025
711187d
chore: Update version to 5.1.0-next for pre-release
leonibr Jul 10, 2025
f3e46ec
fix: Remove conflicting Program.KeyVault.cs file to resolve build errors
leonibr Jul 10, 2025
7916ecf
Next-review-documentation (#80)
leonibr Jul 11, 2025
d0f9137
Update next from master (#85)
leonibr Jul 12, 2025
421c3ad
Merge branch 'master' of https://github.com/leonibr/community-extensi…
leonibr Jul 12, 2025
e094d44
Add Benchmarks project to solution and create initial Program.cs file
leonibr Jul 12, 2025
21fb9e9
Add performance benchmarks and CI/CD workflows for PostgreSQL caching
leonibr Jul 17, 2025
a24944e
Refactor GitHub Actions workflows to simplify benchmark matrix defini…
leonibr Jul 17, 2025
02eafcb
Update README and GitHub Actions workflow for benchmarks
leonibr Jul 17, 2025
8cce658
Refactor GitHub Actions workflow for benchmarks
leonibr Jul 17, 2025
43c33ec
fix: update performance report template
leonibr Jul 17, 2025
f1454a5
Update setup-benchmark-dashboard.yml
leonibr Jul 17, 2025
c2bbbf9
fix: update setup-benchmark-dashboard.yml
leonibr Jul 17, 2025
96d2c1c
chore: update .gitignore and README for benchmarks
leonibr Jul 17, 2025
308b1bc
chore: update branches in benchmarks-scheduled.yml for CI/CD integration
leonibr Jul 17, 2025
efc6fd0
chore: update README for benchmarks
leonibr Jul 17, 2025
9660989
chore: update benchmark workflows for improved reporting
leonibr Jul 17, 2025
11b2655
fix: update benchmarks-scheduled.yml
leonibr Jul 17, 2025
ace5602
chore: update benchmark output file and descriptions
leonibr Jul 17, 2025
07b27ff
chore: update .gitignore and remove obsolete benchmark report files
leonibr Jul 17, 2025
77bfa13
chore: enhance benchmarks-scheduled.yml with additional logging and o…
leonibr Jul 17, 2025
b2e76da
chore: update benchmark description formatting in Program.cs
leonibr Jul 17, 2025
761ca19
chore: refine benchmarks-scheduled.yml and Program.cs formatting
leonibr Jul 17, 2025
c07e517
chore: improve benchmarks-scheduled.yml and Program.cs formatting
leonibr Jul 17, 2025
a7d0cca
chore: enhance benchmarks-scheduled.yml and Program.cs for improved o…
leonibr Jul 17, 2025
61c6267
chore: streamline benchmarks-scheduled.yml and Program.cs for clarity
leonibr Jul 17, 2025
37f0c61
chore: update benchmarks-scheduled.yml output file path and refine Pr…
leonibr Jul 17, 2025
670cbd9
chore: refine CoreOperationsBenchmark description formatting in Progr…
leonibr Jul 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 250 additions & 0 deletions .github/workflows/benchmarks-pr.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
name: PR Performance Validation

on:
pull_request:
paths:
- 'Extensions.Caching.PostgreSql/**'
- 'Benchmarks/**'
types: [opened, synchronize, labeled]

env:
DOTNET_VERSION: '9.0.x'

jobs:
check-performance-label:
runs-on: ubuntu-latest
outputs:
should-run: ${{ steps.check.outputs.should-run }}
steps:
- name: Check if performance testing is requested
id: check
run: |
# Check if PR has 'performance' label or title contains '[perf]'
LABELS="${{ join(github.event.pull_request.labels.*.name, ' ') }}"
TITLE="${{ github.event.pull_request.title }}"

if [[ "$LABELS" == *"performance"* ]] || [[ "$TITLE" == *"[perf]"* ]]; then
echo "should-run=true" >> $GITHUB_OUTPUT
echo "Performance testing requested via label or title"
else
echo "should-run=false" >> $GITHUB_OUTPUT
echo "Performance testing not requested. Add 'performance' label or '[perf]' in title to run benchmarks."
fi

performance-validation:
needs: check-performance-label
if: needs.check-performance-label.outputs.should-run == 'true'
runs-on: ubuntu-latest
timeout-minutes: 45

permissions:
pull-requests: write
contents: read

steps:
- name: Checkout PR branch
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}

- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: ${{ env.DOTNET_VERSION }}

- name: Restore dependencies
run: dotnet restore Benchmarks/Benchmarks.csproj

- name: Build benchmarks
run: dotnet build Benchmarks/Benchmarks.csproj --configuration Release --no-restore

- name: Run core operations benchmark
run: dotnet run --project Benchmarks/Benchmarks.csproj --configuration Release --no-build -- core

- name: Download previous benchmark data for comparison
uses: dawidd6/action-download-artifact@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
workflow: benchmarks-scheduled.yml
branch: gh-pages
name: github-pages
path: ./previous-data
continue-on-error: true

- name: Store benchmark result with comparison
uses: benchmark-action/github-action-benchmark@v1
with:
name: 'core-benchmark-pr'
tool: 'benchmarkdotnet'
output-file-path: Benchmarks/BenchmarkDotNet.Artifacts/results/Benchmarks.UseCases.CoreOperationsBenchmark-report-full-compressed.json
github-token: ${{ secrets.GITHUB_TOKEN }}
auto-push: false # Don't push PR results to main data
# Show comparison with baseline in PR comments
alert-threshold: '120%' # More sensitive for PRs
comment-on-alert: true
fail-on-alert: false
# Reference main branch data for comparison
external-data-json-path: './previous-data/benchmarks/core-benchmark.json'

- name: Find benchmark results
id: find-results
run: |
GITHUB_MD_FILE=$(find Benchmarks/BenchmarkDotNet.Artifacts/results/ -name "*CoreOperationsBenchmark*github.md" | head -1)
if [ -f "$GITHUB_MD_FILE" ]; then
echo "results-file=$GITHUB_MD_FILE" >> $GITHUB_OUTPUT
echo "results-found=true" >> $GITHUB_OUTPUT
else
echo "results-found=false" >> $GITHUB_OUTPUT
fi

- name: Comment PR with performance results
if: steps.find-results.outputs.results-found == 'true'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const resultsFile = '${{ steps.find-results.outputs.results-file }}';

try {
const results = fs.readFileSync(resultsFile, 'utf8');

const body = `## 🚀 Performance Validation Results

This PR has been tested for performance impact on core cache operations.

**Commit:** \`${{ github.event.pull_request.head.sha }}\`
**Date:** ${new Date().toISOString()}
**Comparison:** vs main branch baseline

### Core Operations Benchmark

${results}

### 📊 Historical Context

- **[View Trends Dashboard](https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/benchmarks/)**
- **[Compare with Baseline](https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/benchmarks/core-benchmark.html)**

### Performance Analysis

⚡ **Regression Detection:** Automatic alerts trigger if performance degrades by >20%
📈 **Trend Analysis:** Compare this PR against historical performance data
🎯 **Baseline Comparison:** Results measured against main branch performance

---

**Note:** These results are from a GitHub Actions runner and should be used for relative comparison only.
For more comprehensive performance testing, consider running the full benchmark suite locally.

**Need more benchmarks?** Add specific benchmark names to your PR description:
- \`datasize\` - Test different payload sizes
- \`expiration\` - Test expiration strategies
- \`concurrency\` - Test concurrent operations
- \`bulk\` - Test bulk operations
`;

// Check if we already commented
const comments = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});

const existingComment = comments.data.find(comment =>
comment.user.login === 'github-actions[bot]' &&
comment.body.includes('Performance Validation Results')
);

if (existingComment) {
// Update existing comment
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existingComment.id,
body: body
});
} else {
// Create new comment
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: body
});
}
} catch (error) {
console.error('Error reading benchmark results:', error);

await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: `## ❌ Performance Validation Failed

Unable to read benchmark results. Check the [workflow logs](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details.

**Commit:** \`${{ github.event.pull_request.head.sha }}\`
**Error:** ${error.message}

**Dashboard:** [View Historical Trends](https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/benchmarks/)
`
});
}

- name: Upload detailed results
if: always()
uses: actions/upload-artifact@v4
with:
name: pr-performance-results-${{ github.event.pull_request.number }}
path: Benchmarks/BenchmarkDotNet.Artifacts/results/
retention-days: 14

performance-guide:
needs: check-performance-label
if: needs.check-performance-label.outputs.should-run == 'false'
runs-on: ubuntu-latest

permissions:
pull-requests: write

steps:
- name: Comment with performance testing guide
uses: actions/github-script@v7
with:
script: |
// Check if we already provided guidance
const comments = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});

const hasGuidance = comments.data.some(comment =>
comment.user.login === 'github-actions[bot]' &&
comment.body.includes('Performance Testing Available')
);

if (!hasGuidance) {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: `## 📊 Performance Testing Available

This PR modifies performance-sensitive code. If you want to validate performance impact:

**Option 1:** Add the \`performance\` label to this PR
**Option 2:** Include \`[perf]\` in your PR title

This will trigger core operations benchmarking with historical comparison to help identify any performance regressions.

**📈 View Current Trends:** [Performance Dashboard](https://${{ github.repository_owner }}.github.io/${{ github.event.repository.name }}/benchmarks/)

**Local testing:** For comprehensive performance analysis, run benchmarks locally:
\`\`\`bash
cd Benchmarks
dotnet run --configuration Release
\`\`\`
`
});
}
Loading
Loading