-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Seeding a new subsystem #394
Comments
I think the most straight forward solution is to replay events of interest to X's egress gateway which would make it publish external events that will seed the Y subsystem |
That means you will get a lot of duplicated external events in both X and Y event lakes. Or should we not store external events to the event lakes ? |
No. I have |
Haven't noticed the second question at first. I don't have a definite answer. Right now in John Gilbert's s3 template anything-but-fault is saved. So it means that |
Thanks for you insights. In case you have multiple downstream subsystems, do you still replay events in the egress? We usually have an idempotency table in ingresses that expires records after 30 days to avoid sending duplicates. Should we keep those records forever to be able to replay as much as we want ? |
How would you seed a new subsystem with historical data ?
Example usecase:
Subsys X
has an egress-esg that emits external events and store them to its event lakeSubsys Y
and a routing rule to send external events sent by other subsytems to its event bus.Possible solution:
a script similar to the
aws-lambda-stream-cli
that reads fromSubsys X
's event lake and sends the events toSubsys Y
's event bus. It could work but I fear that it would send duplicates toSubsys Y
's event lake.Has anyone faced this problem before?
The text was updated successfully, but these errors were encountered: