I am using a publish-subscribe event broker, and am constructing libraries to help people interact with this broker consistently.
One of the things I would like to design is some kind of IEventSource interface that each potential publisher could implement. The broker libraries would then natively understand the IEventSource interface, meaning the application developers would not need to code up all of the plumbing to get events from their system into the broker as publishers.
In trying to think about what this interface should look like, I am running into one fundamental problem: Some event sources will want to publish events in batches (for example, events coming from something like a database with change data capture, or events coming from files full of records). Other systems can natively raise events in a serial fashion as they happen, and would be publishing them one at a time.
So, what does my
IEventSource look like? Does it expose the available events as a collection? Some kind of stream? As individual events via a
I could of course decide that the available events are always exposed as a collection, and if a particular publisher is publishing one at a time, they simply have a collection with a single member. But this seems inelegant and inefficient.
One important consideration is that the
IEventSource will need to have some way of being told that the available events have been successfully received, because the goal is to guarantee successful transmission to the broker, with no events being lost. To use the cdc-enabled-database example again, we might get a batch of 500 events from the source, we then start publishing to the broker, and encounter some kind of issue on the 301st event. Ideally we wouldn’t want to retransmit all 500 messages, just pick up again at 301 and send the last 200. So we’d want to tell the source “you can go ahead and discard the first 300 now”. So in addition to some way to get events from the source (individually or in batches) I would want some way to “acknowledge” the receipt of events back to the source – again, individually or in batches.
I could, of course, simply have an
IEventBatchSource and an
ISingleEventSource, and build two different implementations using those two different interfaces all the way through the plumbing. But for obvious reasons I’d prefer not to write all of the plumbing twice.