Skip to content

Enforce release builds for benchmarks and simplify command interface #741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jul 29, 2025

This PR ensures that benchmarks run with release builds by default for meaningful performance measurements, while simplifying the command interface and supporting debug builds for profiling purposes.

Changes Made

Justfile Updates:

  • Removed unnecessary target parameter from bench and bench-ci commands since they only accept release builds
  • Simplified command signatures to just bench [features] and just bench-ci baseline [features]
  • Commands always use --profile=release for consistent performance measurements

GitHub Workflows Updates:

  • Updated Benchmarks.yml and dep_rust.yml to use simplified command signatures
  • Removed redundant target parameter usage while maintaining same functionality

Benchmark Code Changes:

  • Removed compile-time error that prevented running benchmarks with debug symbols
  • Enables profiling capabilities while maintaining release build defaults

Documentation:

  • Updated docs/benchmarking-hyperlight.md to clarify that just bench uses release builds by default
  • Documented that developers can use cargo bench directly for profiling with debug symbols

Behavior

The changes maintain the same core behavior while providing a cleaner interface:

  1. Release builds by default - just bench consistently uses release profile for meaningful results
  2. Simplified commands - No confusing parameters that only accept one value
  3. Profiling flexibility - Developers can compile benchmarks with debug symbols by calling cargo bench directly

Examples

# Clean, simple interface
just bench                    # Run benchmarks with release builds
just bench mshv2             # Run with specific features
just bench-ci baseline       # Compare against baseline

# Advanced usage for profiling
cargo bench                  # Compile with current profile
cargo bench --profile dev    # Debug symbols for profiling

Backward Compatibility

All existing workflows continue to work without modification. The changes are purely simplifications that remove unnecessary complexity while preserving all functionality and adding profiling flexibility.

Fixes #638.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] Make sure that benchmarks do not run with debug builds Enforce release builds for benchmarks across Justfile, source code, and documentation Jul 29, 2025
@Copilot Copilot AI requested a review from jsturtevant July 29, 2025 23:30
Copilot finished work on behalf of jsturtevant July 29, 2025 23:30
@jsturtevant jsturtevant added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Jul 29, 2025
@jsturtevant
Copy link
Contributor

This seems like a much more sane approach than the previous iteration....

@@ -14,6 +14,12 @@ See the License for the specific language governing permissions and
limitations under the License.
*/

// Ensure benchmarks only run with release builds for meaningful performance measurements
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove this compile error? I often run the benchmark with debug symbos when profiling

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the compile-time error in commit 668f0da. Benchmarks can now be compiled with debug symbols for profiling purposes. The Justfile still enforces release builds by default, but you can use cargo bench directly if you need debug symbols.

@Copilot Copilot AI changed the title Enforce release builds for benchmarks across Justfile, source code, and documentation Enforce release builds for benchmarks in Justfile and documentation while allowing debug compilation for profiling Jul 29, 2025
@Copilot Copilot AI requested a review from ludfjig July 29, 2025 23:48
Copilot finished work on behalf of ludfjig July 29, 2025 23:48
Justfile Outdated
cargo bench --profile={{ if target == "debug" { "dev" } else { target } }} {{ if features =="" {''} else { "--features " + features } }} -- --verbose
bench-ci baseline target="release" features="":
@# Benchmarks should only be run with release builds for meaningful results
@if [ "{{ target }}" != "release" ]; then echo "Error: Benchmarks can only be run with release builds. Use 'just bench-ci <baseline> release' instead."; exit 1; fi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this shouldn't be configurable, the parameter should just be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the target parameter from both bench and bench-ci commands in commit c9c9f31. Since the commands only accept release builds, the parameter was indeed unnecessary and confusing. The commands now have cleaner signatures: just bench [features] and just bench-ci baseline [features].

@Copilot Copilot AI changed the title Enforce release builds for benchmarks in Justfile and documentation while allowing debug compilation for profiling Enforce release builds for benchmarks and simplify command interface Jul 30, 2025
@Copilot Copilot AI requested a review from syntactically July 30, 2025 10:28
Copilot finished work on behalf of syntactically July 30, 2025 10:28
Use `just bench [debug/release]` parameter to run benchmarks. Comparing local benchmarks results to github-saved benchmarks doesn't make much sense, since you'd be using different hardware, but you can use `just bench-download os hypervisor [tag] ` to download and extract the GitHub release benchmarks to the correct place folder. You can then run `just bench-ci main` to compare to (and overwrite) the previous release benchmarks. Note that `main` is the name of the baselines stored in GitHub.
Use `just bench` to run benchmarks with release builds (the only supported configuration). Comparing local benchmark results to github-saved benchmarks doesn't make much sense, since you'd be using different hardware, but you can use `just bench-download os hypervisor [tag] ` to download and extract the GitHub release benchmarks to the correct place folder. You can then run `just bench-ci main` to compare to (and overwrite) the previous release benchmarks. Note that `main` is the name of the baselines stored in GitHub.

**Important**: Benchmarks can only be run with release builds to ensure meaningful performance measurements. Debug builds are not supported for benchmarking.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note is no longer applicable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the documentation in commit to reflect that debug builds are now supported for profiling purposes. The note now clarifies that while just bench uses release builds by default, developers can use cargo bench directly for debug symbols.

@Copilot Copilot AI requested a review from ludfjig July 30, 2025 19:42
Copilot finished work on behalf of ludfjig July 30, 2025 19:42
@simongdavies simongdavies marked this pull request as ready for review July 31, 2025 09:43
Copilot AI and others added 5 commits July 31, 2025 12:44
Signed-off-by: Simon Davies <[email protected]>
- Remove target parameter from bench and bench-ci commands since they only accept release builds
- Update GitHub workflows to remove target parameter usage
- Simplify command signatures based on @syntactically's feedback

Co-authored-by: syntactically <[email protected]>
Signed-off-by: Simon Davies <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement For PRs adding features, improving functionality, docs, tests, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make sure that benchmarks do not run with debug builds
4 participants