I have a service that runs queries on a database and gets (potentially) a lot of lines. I also have services that can export a given set of lines in csv, xlsx or pdf format, etc. I have separate export services to exploit language capabilities, for example, java has a better xlsx export library.
What is the best architecture to propagate the results of my query to these export services in a scalable way horizontally?
I've tried using message queues like
rabbitmq However, the problem is the size of the message. If I send a message in pieces / lines, there is no guarantee that the same operator will receive all the lines due to the roundabout.
All tips / ideas are greatly appreciated