Skip to content

[Mac] Add support for butler.py to generate Mac ARM64 deployment zip - #5442

Open
JuanMBriones wants to merge 4 commits into
masterfrom
support_mac_arm64
Open

[Mac] Add support for butler.py to generate Mac ARM64 deployment zip#5442
JuanMBriones wants to merge 4 commits into
masterfrom
support_mac_arm64

Conversation

@JuanMBriones

@JuanMBriones JuanMBriones commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

[Mac] Add support for macOS ARM64 deployment packages and bot execution

Problem

ClusterFuzz lacked support for generating macOS ARM64 deployment zip
packages and running bots on Apple Silicon. Additionally, packaging
failed due to mismatched macOS wheel SDK tags in pip download, and
is_supported_cpu_arch_for_job incorrectly parsed string CPU_ARCH values.

Changes

  • butler.py & constants.py: Add macos_arm64 platform target, fallback
    SDK tags, and Python 3.7–3.11 ABIs.
  • common.py: Download platform pip requirements individually to resolve
    differing wheel tags, and support ARM64 ChromeDriver.
  • platform_requirements.txt: Add native C-extension packages (bcrypt,
    cffi, cryptography, google-crc32c, PyYAML, wrapt).
  • environment.py: Add get_cpu_arch() and support arch-specific tool paths
    in get_default_tool_path().
  • update_task.py: Map macOS ARM64 bots to macos_arm64 deployment packages.
  • commands.py: Fix CPU_ARCH parsing in is_supported_cpu_arch_for_job().
  • Tests: Add unit tests for environment arch detection, tool path resolution,
    deployment URLs, and job arch validation.

@JuanMBriones
JuanMBriones requested a review from a team as a code owner August 28, 2026 22:24
@JuanMBriones
JuanMBriones marked this pull request as draft August 28, 2026 22:26
@g-ortuno

g-ortuno commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Looks like All-Tests and Run basic tests are failing. Can you fix those?

g-ortuno
g-ortuno requested changes Aug 29, 2026
View reviewed changes

@g-ortuno g-ortuno left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a description to the PR! It should summarize all the changes we are making and why.

Comment thread src/clusterfuzz/_internal/system/environment.py Outdated
Comment thread src/clusterfuzz/_internal/bot/tasks/commands.py
Comment thread src/clusterfuzz/_internal/bot/tasks/update_task.py Outdated
Comment thread src/clusterfuzz/_internal/bot/tasks/update_task.py Outdated
Comment thread src/clusterfuzz/_internal/system/environment.py
Comment thread src/local/butler/constants.py Outdated
Comment thread src/local/butler/common.py
Comment thread src/platform_requirements.txt
Comment thread butler.py Outdated
Comment thread src/local/butler/common.py Outdated
g-ortuno
g-ortuno reviewed Aug 29, 2026
View reviewed changes
Comment thread src/local/butler/constants.py Outdated
@JuanMBriones

JuanMBriones commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Please add a description to the PR! It should summarize all the changes we are making and why.

Thanks for the review:D I just modified the commit message. Sorry I forgot about this!
Left comments inline and resolved them on a follow-up commit.

@JuanMBriones

JuanMBriones commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Looks like All-Tests and Run basic tests are failing. Can you fix those?

I already fixed failing tests. I saw an error on https://github.com/google/clusterfuzz/pull/5442/checks?check_run_id=99666204114 but it seems theres no correlation with the changes

Step #2 - "Build Ubuntu 24.04": E: Package 'google-cloud-sdk' has no installation candidate
Step #2 - "Build Ubuntu 24.04": The command '/bin/sh -c ... apt-get install -y google-cloud-sdk' returned a non-zero code: 100
ERROR: build step 2 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 100

g-ortuno
g-ortuno requested changes Sep 1, 2026
View reviewed changes

@g-ortuno g-ortuno left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just had a few small comments. Still need to take a look at the tests; I'll do those tomorrow morning.

Comment thread src/clusterfuzz/_internal/system/environment.py
Comment thread src/local/butler/common.py
Comment thread src/local/butler/constants.py Outdated
Comment thread src/local/butler/constants.py Outdated
g-ortuno
g-ortuno requested changes Sep 2, 2026
View reviewed changes
Comment thread src/clusterfuzz/_internal/bot/tasks/update_task.py Outdated
Comment thread src/clusterfuzz/_internal/bot/tasks/update_task.py Outdated
Comment thread src/clusterfuzz/_internal/tests/core/system/environment_test.py
@JuanMBriones
[Mac] Add support for macOS ARM64 deployment packages and bot execution
2850ed5 
### Problem
ClusterFuzz lacked support for generating macOS ARM64 deployment zip
packages and running bots on Apple Silicon. Additionally, packaging
failed due to mismatched macOS wheel SDK tags in pip download, and
is_supported_cpu_arch_for_job incorrectly parsed string CPU_ARCH values.

### Changes
* butler.py & constants.py: Add macos_arm64 platform target, fallback
  SDK tags, and Python 3.7–3.11 ABIs.
* common.py: Download platform pip requirements individually to resolve
  differing wheel tags, and support ARM64 ChromeDriver.
* platform_requirements.txt: Add native C-extension packages (bcrypt,
  cffi, cryptography, google-crc32c, PyYAML, wrapt).
* environment.py: Add get_cpu_arch() and support arch-specific tool paths
  in get_default_tool_path().
* update_task.py: Map macOS ARM64 bots to macos_arm64 deployment packages.
* commands.py: Fix CPU_ARCH parsing in is_supported_cpu_arch_for_job().
* Tests: Add unit tests for environment arch detection, tool path resolution,
  deployment URLs, and job arch validation.

Signed-off-by: Manuel Briones <manuelbriones@google.com>
@JuanMBriones
Address review comments for macOS ARM64 support
ef0a575 
- Rename PLATFORMS to DEPLOYMENT_TARGETS with backward-compatible alias.
- Add macOS version tag comment in constants.py.
- Use tempfile.TemporaryDirectory and add explanatory comments in common.py.
- Remove default platform argument in butler.py package command.
- Add documentation headers to Pipfile and platform_requirements.txt.
- Use platform.machine() for host architecture detection in update_task.py.
- Update CPU_ARCH format comment in commands.py.
- Simplify get_default_tool_path and remove BOT_CPU_ARCH in environment.py.
- Update and add unit tests in environment_test.py and update_task_test.py.

### Tests Executed
* environment_test: 40/40 passed (GetCpuArchTest, GetDefaultToolPathTest, etc.)
* update_task_test: 18/18 passed (GetSourceUrlTest, GetNewerSourceRevisionTest, etc.)
* commands_test: 10/10 passed (IsSupportedCpuArchForJobTest, SetTaskPayloadTest, UpdateEnvironmentForJobTest)
* testcase_manager_test: 5/5 passed (UploadTestcaseOutputTest)
* Linters: pylint passed with 0 errors, yapf passed with 0 diffs.
@JuanMBriones
[Mac] Replace PLATFORMS alias with DEPLOYMENT_TARGETS and add macOS 1…
ca655c9 
…5 target
g-ortuno
g-ortuno reviewed Sep 4, 2026
View reviewed changes
Comment thread src/clusterfuzz/_internal/base/utils.py Outdated
Comment thread src/local/butler/constants.py Outdated
@JuanMBriones
[Mac] Standardize on deployment_target naming and add Android CPU arc…
2f126e5 
…h test

- update_task.py: Rename platform_name to deployment_target and
  platform_mappings to deployment_target_mappings in get_source_url().
- utils.py: Rename get_platform_deployment_filename() to
  get_deployment_target_filename() and update parameter from
  platform to deployment_target (retaining backward-compatible alias).
- package.py: Update call sites to use get_deployment_target_filename().
- environment_test.py: Add unit test verifying get_cpu_arch() delegation
  for Android devices.

Tested:
- core/system/environment_test.py (45/45 passed)
- core/bot/tasks/update_task_test.py (18/18 passed)
- core/local/butler/package_test.py (5/5 passed)
- core/local/butler/deploy_test.py (13/13 passed)
- butler.py lint (clean)

Signed-off-by: Manuel Briones <manuelbriones@google.com>
@JuanMBriones
JuanMBriones requested review from hunsche and vitaliset and removed request for vitaliset September 4, 2026 21:41
@JuanMBriones
JuanMBriones marked this pull request as ready for review September 7, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@g-ortuno g-ortuno g-ortuno requested changes
@hunsche hunsche Awaiting requested review from hunsche
@ViniciustCosta ViniciustCosta Awaiting requested review from ViniciustCosta

Requested changes must be addressed to merge this pull request.

Assignees

@g-ortuno g-ortuno

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@JuanMBriones @g-ortuno