Skip to content

ci: SDK QA & release automation - #73

Open
Dhruwang wants to merge 48 commits into
mainfrom
dhruwang/sdk-qa-release-automation
Open

Dhruwang wants to merge 48 commits into
mainfrom
dhruwang/sdk-qa-release-automation

Conversation

@Dhruwang

@Dhruwang Dhruwang commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

What

Part of the SDK QA & Release Automation project (Linear P-ENG-89). Consistent CI checks (PR Check Summary as the single required check), native E2E against the release Docker image, and a tag-driven release workflow.

Decisions: https://linear.app/formbricks/document/4-engineering-decisions-2de7e803ebdc

Tickets

  • Closes ENG-7212 — iOS: CI parity + fix flaky/slow tests
  • Closes ENG-7213 — iOS: playground app + XCUITest golden journey via tunnel
  • Closes ENG-7214 — iOS: release workflow (verify only)

Changes SDK users will notice

  • Formbricks.cleanup now runs its cleanup on the main thread, and an internal queue moved from .background to .userInitiated QoS. This fixed slow and flaky tests; host apps may see cleanup finish sooner.

Before merging

  • All checks are green, including E2E: the full targeted journey runs against the org secret ENTERPRISE_LICENSE_KEY (Enterprise license with Contacts).
  • After merging, the ruleset's only required check becomes PR Check Summary (source: GitHub Actions).

Runbook: https://app.notion.com/p/419e5de84a5881cd8091c30193db8607

Publishing the GitHub release creates the tag that SPM ships, so the
workflow only checks the result: the tag must be plain semver, must be
visible through git ls-remote, and must resolve through SwiftPM from a
scratch package. RELEASING.md records the accepted exception that iOS
cannot be gated before publication, so the manual QA run comes first.
cleanup(waitForOperations:) and the connectivity check in track() ran on the
.background global queue, the band the scheduler drops first under load. On CI
runners testCleanupWithCompletion took up to 552s and made up most of the ~500s
suite. Both now run at .userInitiated, and the post-wait cleanup runs on main.

The SDK tests also wait on the update queue instead of sleeping 2s per user sync,
and drop 0.5s sleeps after a workspace refresh the mock answers synchronously.
testOverlaySurveyIsPresentedOnTheTopMostViewController timed out on CI because
presenting the real survey starts a WKWebView, and WebKit's first process launch
blocked the main thread for 20+ seconds on a cold simulator. PresentSurveyManager
now builds the survey controller through a swappable factory, the presentation
tests use a plain view controller, and the top-most test waits on the main queue
instead of polling a predicate. It also asserts the survey takes its traits from
the leaf controller, which is the only place the walker's choice is visible.
swift-format (.swift-format) owns layout; SwiftLint (.swiftlint.yml) checks semantics,
with its layout rules disabled so the two never disagree. Both run strict in CI.

Fixes rather than baselines: block comments to line comments, forEach to for-in,
redundant raw values, force casts in APIClient, an over-complex WebView message
handler and an over-long track evaluation split in two, and the compiler warnings
in the test target. APIClient now takes a DataTaskProviding instead of a URLSession
so the test double no longer subclasses URLSession through its deprecated init.

Config, not baselines: AnyCodable is vendored and keeps its upstream shape; test
files skip the length rules; a handful of conventional short names are allowed.
make api-dump writes the public API from the module's symbol graph to
api/FormbricksSDK.api, one sorted line per symbol, conformance and availability, without
doc comments or source locations. make api-check diffs against it. The snapshot matches
the API before this branch.

make package-check resolves the package over git through SPM at a commit into a scratch
consumer and builds it for the iOS Simulator.
… gate

ci.yml runs lint, format, build, unit-tests, api-check, package-check, sonarqube,
dependency-review and workflow-lint on pull_request, merge_group and push to main, each
through the same make target used locally. PR Check Summary is the one check to require:
scripts/pr-check-summary.sh fails it on any result but success, except Sonar and E2E
skipping on fork PRs.

Every job has harden-runner, SHA-pinned actions, a timeout and persist-credentials off;
PR runs cancel superseded runs. Xcode is pinned to 26.3. Sonar now scans on Linux from the
unit-tests coverage artifact, waits for the quality gate, and stays on scan-action v6.
Adds pr-title.yml for semantic PR titles, weekly CodeQL (Swift and Actions) and
Scorecard, and a Dependabot cooldown.
…on' into dhruwang/eng-7212-ios-ci-parity-fix-flakyslow-tests
…shallow clone

unit-tests found no iOS simulator: with only DEVELOPER_DIR set, the simulator service
on macos-15 listed none for Xcode 26.3. The macOS jobs now run xcode-select, as the old
workflow's setup-xcode did.

package-check failed because the shallow CI checkout cannot push to a bare repo; the
bare repo now accepts shallow updates.
…on' into dhruwang/eng-7212-ios-ci-parity-fix-flakyslow-tests
macos-15 has Xcode 26.3 but no iOS 26 simulator runtime for it, so unit-tests found only
placeholder destinations. macos-26 ships the iOS 26.2 runtime Xcode 26.3 uses.

make test no longer hardcodes iPhone 17: scripts/simulator-destination.sh reads
xcodebuild -showdestinations, ignores ineligible runtimes, and picks an iPhone on the
newest one, or fails saying no iOS Simulator runtime is available.
…on' into dhruwang/eng-7212-ios-ci-parity-fix-flakyslow-tests
On a fresh macos-26 runner the simulator list can still be empty when make test asks, so
unit-tests failed with no destination. simulator-destination.sh now polls
simctl list devices available -j with backoff (6 attempts, 5s doubling to 30s), keeps
runtimes no newer than the selected Xcode, prefers iPhone device types, and falls back to
xcodebuild -showdestinations. It fails only after both, listing the runtimes it saw.
…on' into dhruwang/eng-7212-ios-ci-parity-fix-flakyslow-tests
Reuse the Enterprise license the main repo's E2E checks already use,
under the same secret name, instead of a separate QA license secret.
validate-tag.sh now takes the GitHub release's prerelease flag and rejects a
stable tag marked as a pre-release, or a prerelease tag that is not. Both
release jobs check out the tag commit instead of the default branch, so the
scripts that run are the ones that shipped with the release. RELEASING.md
links the Notion runbook.
The PR Check Summary skip rules move out of the ci.yml expression into
scripts/pr-check-summary.sh, with tests: sonarqube and e2e-tests may skip
on fork PRs only, dependency-review on a push only, and every other job must
succeed.
Pushes to main and merge-group runs no longer share one group per ref, so a
later push cannot queue behind or replace an earlier one.
A push has no base to compare against, so the job skips there and the
summary allows that skip. Both events pass explicit base and head commits,
and the default severity and license settings apply.
make script-tests now also runs the E2E harness tests (e2e/*/test), and the
unit-tests job runs it instead of api-check. e2e.yml no longer runs the
harness tests, so fork PRs, which skip E2E, still run them.
Covers scripts/ (CI, release, summary) and the E2E harness under e2e/. The
vendored xccov converter splits the Xcode version on purpose, so that one
warning is disabled with a reason.
The newest pin across the SDK repos. android, js and flutter run v8 behind
harden-runner without the keyserver failure that held this repo at v6.
The unit-tests job failed in 6 of 7 runs. In the failing runs WebKit's GPU
process took 57s and 108s to launch, after the WebView tests in
FormbricksSDKTests. While it launched, the simulator froze the test process
for seconds at a time. A pure JSON-decode test took 5s.

The tests that failed slept for a fixed time and then checked, for example
asyncAfter(0.6) against the 0.5s debounce. That left a 100ms margin. After a
stall the sleep could finish before the timer, or the 1-5s wait timed out. A
block left over from a timed-out test then failed the next test, or crashed
in UpdateQueueTests after tearDown had cleared the mock.

- UpdateQueue gets internal hasScheduledCommit / fireScheduledCommit, so tests
  fire the debounce themselves. One test still waits for the real timer.
- The interaction tests flush through waitForPendingUpdates, which the
  synchronous mock answers inline.
- The sync-timer tests wait for the completion's main-queue hop. They then
  check that the timer is on the main run loop and when it is due, instead
  of waiting to see whether it fires.
- Waits on a real event use a shared 30s cap. A passing test returns as soon
  as the event arrives.

1d30db2 is not the cause: the failures reproduce with FormbricksSDKTests
excluded, so cleanup(waitForOperations:) never runs. No SDK behaviour changes.
Locally the suite drops from about 27s to 6s.
This removes the debug probes from the previous commit.

The two sync-timer tests still failed on CI, both waiting the full 30s for the
mock's completion. The mock ran the completion on DispatchQueue.global(),
and on the CI simulator GCD's global queues can wait a long time for a worker
thread. The probes showed a block on the utility queue starting 91s late,
while a plain Thread started at once. Default-QoS completions were held back
more than 30s in this run and in both earlier failing runs, each time while
WebKit's helper processes were launching.

A Thread with no run loop is the condition the tests check: a timer scheduled
on it would never fire. Unlike the global queue, it starts straight away.
@Dhruwang
Dhruwang marked this pull request as ready for review November 16, 2026 10:26
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: bcadec8e-ddad-4886-a203-813a3bebc0a8
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Dhruwang and others added 6 commits November 16, 2026 16:13
A hung server or tunnel held api() until the job timeout. Requests now
fail after 10s to connect and 60s overall (E2E_HTTP_TIMEOUT_SECONDS).
bootstrap.sql skips rows that already exist, and seed.sh deletes the
previous seed it recorded before creating a new one, so both can run
again on the same stack without unique-key collisions.
The V1 assert and cleanup run in the server job after the journey. If
only they failed, "Re-run failed jobs" re-ran the server job alone, and
it waited 40 minutes for a journey artifact that never came. The server
job now hands its final status back, and the macOS job fails unless it
is success, so the pair always fails and re-runs together.
… Sonar hold

README's own release steps skipped the manual QA run and the pre-release
flag rule that RELEASING.md requires. The Dependabot ignore rule held
sonarqube-scan-action at v6, but ci.yml already pins v8.5.0.
@sonarqubecloud

Copy link
Copy Markdown

@Dhruwang
Dhruwang requested a review from mattinannt November 17, 2026 07:53

@mattinannt mattinannt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Dhruwang Looks good; can you please also fix the 4 sonarqube issues; then it's ready to go :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants