Skip to content

docs: add wide events ADRs for audit stream and events API - #459

Open
adlerhurst wants to merge 6 commits into
mainfrom
cursor/wide-events-adrs-365e
Open

docs: add wide events ADRs for audit stream and events API#459
adlerhurst wants to merge 6 commits into
mainfrom
cursor/wide-events-adrs-365e

Conversation

@adlerhurst

@adlerhurst adlerhurst commented Jul 3, 2026

Copy link
Copy Markdown
Member

Summary

Docs-only ADRs for nextgen audit logging (relational SoT + semantic wide events):

  • ADR 048 — internal events table, categories, Path A (batched request events) + Path B (AllStatements / InsertEvent), deny-by-default PII + x-audit
  • ADR 049 — unified GET /events, time-only retention, first-class managed sinks, project-scoped get

Also updates ADR index, ADR 028 Related, resource-map, api README, and user-schema.md.

Review decisions (latest round)

  • Pre-claim (a): emit/store always; gate list/get/shipper until claim (ADR 046)
  • No resource_scope_index for events; GET /events/{id} is project-scoped
  • Retention: time-only (not sink-gated); events_purged_undelivered metric when aging out undelivered rows
  • Sinks: first-class CRUD, managed sink_… ids; ≤1 deployment stdout + ≤1 deployment webhook + ≤1 project webhook (additive)
  • Path A: bounded buffer, drop new enqueue + metric, retry/backoff, recompute wait
  • x-sensitive: kept complementary to x-audit (still used by OpenAPI/console/config)

Note: #710 also claims ADR 048 (Dev Inbox) — number collision if both merge; coordinate renumber.

Validation

  • Docs-only; no moon ci / Go tests run
  • Grep: no sink-gated purge; pre-claim store+visibility; events not indexed in RSI

Release notes / changeset

Docs-only — no changeset per .changeset/README.md decision table.

Notes

PR title remains conventional docs: (allowed by .github/semantic.yml).

closes #350

Open in Web Open in Cursor 

Formalize the internal wide-event audit model and unified /events API:
- ADR 029: events table schema, categories, v2-statement emission,
  request-wide events for authenticated API calls, PII deny-by-default
- ADR 030: GET /events with category filter, retention, shipped_at export,
  external threat-detection consumer pattern
- Update resource map, ADR index, user-schema x-audit annotation

Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
@vercel

vercel Bot commented Jul 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nextgen Ready Ready Preview Aug 6, 2026 2:19pm
nextgen-docs Ready Ready Preview Aug 6, 2026 2:19pm
nextgen-mock-zitadel Ready Ready Preview Aug 6, 2026 2:19pm

Request Review

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

⚠️ No Changeset found

Latest commit: d4ec053

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/030-events-api-retention-export.md Outdated
Comment thread docs/adrs/030-events-api-retention-export.md Outdated
- Link oxidel ADR-023; drop sequence column (use created_at, id ordering)
- Replace aggregate_* with entity_type/entity_id; remove sdk_* from events table
- Replace shipped_at with per-sink event_deliveries table
- Remove threat-detection section from ADR 030

Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds two new Architecture Decision Records (ADRs) that formalize a “wide events” audit model for nextgen’s relational storage, and updates surrounding design docs to reference the new audit/event concepts (including the x-audit allowlist approach for PII in audit payloads).

Changes:

  • Add ADR 029 (internal wide-event model + emission/PII rules) and ADR 030 (events API surface, retention, and per-sink export tracking).
  • Update ADR index + cross-links (ADR 028, API resource map, API design README).
  • Update Flow Engine user schema annotations to use x-audit (deny-by-default, allowlist for audit payload inclusion).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
docs/design/flowengine/user-schema.md Switches audit-related schema annotation guidance to x-audit and documents deny-by-default payload inclusion.
docs/design/api/resource-map.md Updates the resource map to a unified /events API aligned with the new ADRs.
docs/design/api/README.md Updates API design README to reference ADR 030 for the partially-specified events surface (currently contains a broken relative link).
docs/adrs/README.md Adds ADR 029 and ADR 030 to the ADR index.
docs/adrs/029-wide-events-internal-audit-primitive.md Introduces ADR 029 defining the internal wide-events audit primitive model and emission boundaries.
docs/adrs/030-events-api-retention-export.md Introduces ADR 030 defining /events API semantics, retention strategy, and per-sink export tracking.
docs/adrs/028-storage-v2-statements-and-dialects.md Adds a cross-reference from ADR 028 to the new ADR 029.

Comment thread docs/design/api/README.md Outdated
Comment thread docs/adrs/049-events-api-retention-export.md
Comment thread docs/adrs/030-events-api-retention-export.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/030-events-api-retention-export.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md
Comment thread docs/design/api/README.md Outdated
Comment thread docs/design/flowengine/user-schema.md Outdated
Comment thread docs/adrs/030-events-api-retention-export.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
- Durable sync request-event insert; server-authoritative correlation
- Emit-time team_id + resource_scope_index for GET /events/{id}
- Dialect-specific failure-event TX (SAVEPOINT vs Spanner new TX)
- CASCADE on event_deliveries; executable retention SQL with sink array
- Soften index guidance; restore x-sensitive docs; fix ADR 030 link

Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
Comment thread docs/design/flowengine/user-schema.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/029-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/049-events-api-retention-export.md
cursoragent and others added 2 commits August 5, 2026 07:47
Keep main ADR index 029–047; retain wide-events Related link on ADR 028
(to be renumbered to 048/049 next).

Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
Merge brought main's ADR 029–047; rename wide-events to 048/049.
Sweep obsolete v1/port language; batch request events with
occurred_at = now()-wait; drop x-sensitive deprecation; limit sinks
to stdout + one webhook; align with ADRs 028/033/036/046/047.

Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md Outdated
@adlerhurst
adlerhurst requested a review from fforootd August 5, 2026 08:19
Comment thread docs/design/flowengine/user-schema.md

@fforootd fforootd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with a specific lens: the three data types we distinguish and the write guarantee each should get — (1) stateful (users, domain objects), (2) transient (sessions, tokens), (3) fire-and-forget (HTTP requests and the like). The mapping onto this design works out:

Data type Categories Path Guarantee in this PR
1. Stateful entity, admin B Same-TX co-commit with the mutating statement
2. Transient auth, session B Same-TX co-commit (sessions/tokens are authoritative rows per ADR 010/037, so their lifecycle statements carry the event)
3. Fire-and-forget request A Batched flush (N=100 / T=1s), bounded loss window, drain on shutdown

Types 1 and 2 get something strictly stronger than a dual write: since events live in the same database as the entities, Path B co-commits both rows atomically — no reconciliation problem, no "state changed but no log" race. That's the right call. Type 3 as batching matches the earlier discussion, and the DB-now() / occurred_at split is a clean way to keep one time authority.

The inline comments are all suggestions, aimed at making that contract explicit and closing the few spots where a guarantee is fuzzy or two sections disagree:

  • State the per-category path/durability in the §3 table (also pins down signal, which currently has no declared path).
  • Specify Path A failure semantics — bounded buffer, overflow policy, flush retry — the one place the type-3 guarantee is still soft.
  • Resolve project.created vs the pre-claim policy (ties into the open thread on the pre-claim paragraph).
  • ADR 049: per-project vs server-config sink set for retention (has a real never-purge consequence), occurred_at filterability, plus small nits.

Process note: the branch is behind main; 048/049 are still free today (main tops out at 047), so worth rebasing/updating and landing before the numbers get taken.


Generated by Claude Code

Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md
Comment thread docs/adrs/049-events-api-retention-export.md Outdated
Comment thread docs/adrs/049-events-api-retention-export.md Outdated
Comment thread docs/adrs/049-events-api-retention-export.md Outdated
Comment thread docs/adrs/049-events-api-retention-export.md Outdated
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md Outdated
Comment thread docs/adrs/049-events-api-retention-export.md
Comment thread docs/adrs/048-wide-events-internal-audit-primitive.md
Comment thread docs/adrs/049-events-api-retention-export.md
Pre-claim store+visibility gate; drop events from resource_scope_index;
time-only retention with undelivered metric; managed sink CRUD with
additive delivery; Path A bounded drop+metric buffering; category
durability table and related nits.

Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/adrs/049-events-api-retention-export.md:100

  • ADR 048 defines flow_id (and other correlation dimensions) as nullable (NULL = not applicable). This example uses an empty string for flow_id, which makes the contract ambiguous and diverges from the nullability convention. Prefer null here (or explicitly document a normalization rule) so API and storage semantics match.
  "request_id": "0af7651916cd43dd8448eb211c80319c",
  "session_id": "sess_456",
  "flow_id": "",
  "payload": { "scope": ["openid", "profile"] },
  "metadata": {}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

[ARCH-ADR] Define Architecture for Audit Logging, Observability, and Compliance

6 participants