docs: add wide events ADRs for audit stream and events API - #459
docs: add wide events ADRs for audit stream and events API#459adlerhurst wants to merge 6 commits into
Conversation
Formalize the internal wide-event audit model and unified /events API: - ADR 029: events table schema, categories, v2-statement emission, request-wide events for authenticated API calls, PII deny-by-default - ADR 030: GET /events with category filter, retention, shipped_at export, external threat-detection consumer pattern - Update resource map, ADR index, user-schema x-audit annotation Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
- Link oxidel ADR-023; drop sequence column (use created_at, id ordering) - Replace aggregate_* with entity_type/entity_id; remove sdk_* from events table - Replace shipped_at with per-sink event_deliveries table - Remove threat-detection section from ADR 030 Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR adds two new Architecture Decision Records (ADRs) that formalize a “wide events” audit model for nextgen’s relational storage, and updates surrounding design docs to reference the new audit/event concepts (including the x-audit allowlist approach for PII in audit payloads).
Changes:
- Add ADR 029 (internal wide-event model + emission/PII rules) and ADR 030 (events API surface, retention, and per-sink export tracking).
- Update ADR index + cross-links (ADR 028, API resource map, API design README).
- Update Flow Engine user schema annotations to use
x-audit(deny-by-default, allowlist for audit payload inclusion).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| docs/design/flowengine/user-schema.md | Switches audit-related schema annotation guidance to x-audit and documents deny-by-default payload inclusion. |
| docs/design/api/resource-map.md | Updates the resource map to a unified /events API aligned with the new ADRs. |
| docs/design/api/README.md | Updates API design README to reference ADR 030 for the partially-specified events surface (currently contains a broken relative link). |
| docs/adrs/README.md | Adds ADR 029 and ADR 030 to the ADR index. |
| docs/adrs/029-wide-events-internal-audit-primitive.md | Introduces ADR 029 defining the internal wide-events audit primitive model and emission boundaries. |
| docs/adrs/030-events-api-retention-export.md | Introduces ADR 030 defining /events API semantics, retention strategy, and per-sink export tracking. |
| docs/adrs/028-storage-v2-statements-and-dialects.md | Adds a cross-reference from ADR 028 to the new ADR 029. |
- Durable sync request-event insert; server-authoritative correlation
- Emit-time team_id + resource_scope_index for GET /events/{id}
- Dialect-specific failure-event TX (SAVEPOINT vs Spanner new TX)
- CASCADE on event_deliveries; executable retention SQL with sink array
- Soften index guidance; restore x-sensitive docs; fix ADR 030 link
Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
Keep main ADR index 029–047; retain wide-events Related link on ADR 028 (to be renumbered to 048/049 next). Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
Merge brought main's ADR 029–047; rename wide-events to 048/049. Sweep obsolete v1/port language; batch request events with occurred_at = now()-wait; drop x-sensitive deprecation; limit sinks to stdout + one webhook; align with ADRs 028/033/036/046/047. Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
fforootd
left a comment
There was a problem hiding this comment.
Reviewed with a specific lens: the three data types we distinguish and the write guarantee each should get — (1) stateful (users, domain objects), (2) transient (sessions, tokens), (3) fire-and-forget (HTTP requests and the like). The mapping onto this design works out:
| Data type | Categories | Path | Guarantee in this PR |
|---|---|---|---|
| 1. Stateful | entity, admin |
B | Same-TX co-commit with the mutating statement |
| 2. Transient | auth, session |
B | Same-TX co-commit (sessions/tokens are authoritative rows per ADR 010/037, so their lifecycle statements carry the event) |
| 3. Fire-and-forget | request |
A | Batched flush (N=100 / T=1s), bounded loss window, drain on shutdown |
Types 1 and 2 get something strictly stronger than a dual write: since events live in the same database as the entities, Path B co-commits both rows atomically — no reconciliation problem, no "state changed but no log" race. That's the right call. Type 3 as batching matches the earlier discussion, and the DB-now() / occurred_at split is a clean way to keep one time authority.
The inline comments are all suggestions, aimed at making that contract explicit and closing the few spots where a guarantee is fuzzy or two sections disagree:
- State the per-category path/durability in the §3 table (also pins down
signal, which currently has no declared path). - Specify Path A failure semantics — bounded buffer, overflow policy, flush retry — the one place the type-3 guarantee is still soft.
- Resolve
project.createdvs the pre-claim policy (ties into the open thread on the pre-claim paragraph). - ADR 049: per-project vs server-config sink set for retention (has a real never-purge consequence),
occurred_atfilterability, plus small nits.
Process note: the branch is behind main; 048/049 are still free today (main tops out at 047), so worth rebasing/updating and landing before the numbers get taken.
Generated by Claude Code
Pre-claim store+visibility gate; drop events from resource_scope_index; time-only retention with undelivered metric; managed sink CRUD with additive delivery; Path A bounded drop+metric buffering; category durability table and related nits. Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Suppressed comments (1)
docs/adrs/049-events-api-retention-export.md:100
- ADR 048 defines
flow_id(and other correlation dimensions) as nullable (NULL= not applicable). This example uses an empty string forflow_id, which makes the contract ambiguous and diverges from the nullability convention. Prefernullhere (or explicitly document a normalization rule) so API and storage semantics match.
"request_id": "0af7651916cd43dd8448eb211c80319c",
"session_id": "sess_456",
"flow_id": "",
"payload": { "scope": ["openid", "profile"] },
"metadata": {}
Summary
Docs-only ADRs for nextgen audit logging (relational SoT + semantic wide events):
eventstable, categories, Path A (batched request events) + Path B (AllStatements/InsertEvent), deny-by-default PII +x-auditGET /events, time-only retention, first-class managed sinks, project-scoped getAlso updates ADR index, ADR 028 Related, resource-map, api README, and
user-schema.md.Review decisions (latest round)
resource_scope_indexfor events;GET /events/{id}is project-scopedevents_purged_undeliveredmetric when aging out undelivered rowssink_…ids; ≤1 deployment stdout + ≤1 deployment webhook + ≤1 project webhook (additive)waitx-sensitive: kept complementary tox-audit(still used by OpenAPI/console/config)Note: #710 also claims ADR 048 (Dev Inbox) — number collision if both merge; coordinate renumber.
Validation
moon ci/ Go tests runRelease notes / changeset
Docs-only — no changeset per
.changeset/README.mddecision table.Notes
PR title remains conventional
docs:(allowed by.github/semantic.yml).closes #350