Provide a mechanism for sensible replay of pubsub events

[JamieAP]

There is regularly a need to replay events through our pipelines and, usually, only to a subset of consumers in a given pipeline.

In telecom, we've implemented this using a field in the envelope "replayTargets"[0] that is a list of consumer group names. Any consumer receiving an event where replayTargets is set must compare the contents of that list with their own CG name and consume only if their name is present[1].

Whether that's a good mechanism going forward is up for debate; that's how we've done it in the past.

For this to work at a go-pubsub level, it's hard to see how a standardised event envelope could be avoided.

[0] https://github.com/utilitywarehouse/telecom-contracts/blob/681fafc350b88c8ee147169cc6985fdf235f0c9c/schema/proto/Envelope.proto#L20
[1] https://github.com/utilitywarehouse/freecall-cdr-elasticsearch-indexer/blob/d830d60db88144748045715832099b7444fce332/consumer.go#L113-L115

[mjgarton]

My first thought is that although I agree this is a command and useful thing, I think it belongs in layer(s) above go-pubsub. It may belong in one of the components of a suite of messaging APIs though, that we end up with from the "v2" discussions. (so it's fine for the issue to live here of course)

My second thought is that including knowledge about targets in the produced message means expecting the producers to know about things that they shouldn't know about (especially when these things cross domain boundaries). I'd like to spend some time thinking about alternative approaches that don't need us to do that.

[thinktainer]

I think the assumption of having to bake knowledge into producers is not entirely true. The knowledge is baked into a replay mechanism, which may be completely independent of the original producers. I understand your concern of making imperative calls at the top of the pipeline though, and am interested of what alternative approaches we can come up with.
I think that the inclusion of white / blacklists in some sort of replay mechanism makes a lot of sense, especially when thinking about non idempotent consumers (emails or letters come to mind).

[vnadgir]

Is replay any different to re-consume on a channel/topic with unbounded durability?

[mjgarton]

@thinktainer We still need a way to replay to both idempotent and non-idempotent consumers. Isn't the general thing that you want to replay to everywhere, but tell the consumers not to produce side effects outside of their own state?

Saying that, maybe what is needed first (for me at least) is a list of use cases that we're trying to support here.

[JamieAP]

@mjgarton The goal (achieved by the mechanism I've described) is to be able to exclude some consumers from doing anything at all with the messages, whether that's change their internal state or the state of other services via producing more events/making RPCs.

If you're dealing with a graph of consumer that are all idempotent, you don't need a replay mechanism. At least, not in as far as avoiding bad things happening goes. You may still want services to be aware of unoriginal events for other reasons. However, you can replay events as they originally appeared without the need for a replayTargets like mechanism.

In the case where you have a graph of non-idempotent consumers, that's where you need to be able to instruct subsets of the graph to ignore certain messages.

[thinktainer]

I think this could be scoped to https://github.com/utilitywarehouse/event-envelope-proto, in order to raise awareness on a more modelling focused repo?

Jamie.
That's a good point regarding the sub consumers needing to know the messages are for replay

Did this use case get resolved as I am also facing this use case ?