Persistent Messaging/Service Bus - Roll my own or risk the learning curve?

Question

We have a client application that needs to send messages to a server for various notifications. In order that the client can run occasionally connected I'm going to go with a message-queue approach. The queue processing will take messages off the queue and call a web service that will put them on another queue to finally be processed. This question is about the client environment; the server environment is already decided on.

I don't want to use MSMQ because we don't have control over all the client PCs in order to install/configure and secure MSMQ properly, and because support is more challenging due to the quality of tooling for investigating the contents of MSMQ queues. SQL Server 2005 Express is on all the machines, and is used to store data for our application.

I currently have it down to two options:

1. Write a fairly basic persistent message-queue that stores the messages in a table after serialising them, then uses ThreadPool.QueueUserWorkItem to have them processed by handlers configured against each message type. All in a System.Transactions.TransactionScope so they are only removed from the persistent queue if they are successfully processed.
2. Use NServiceBus (this is the service bus we've gone with as a team, so MassTransit etc aren't options) on the client, with a Service Broker transport that uses the local database.

I have little experience with service buses (I still don't really get service bus terminology) so I'm concerned about the learning curve compared to writing something much more simple that meets my requirements in the way I need it to (deployment is a big consideration).

Does anyone have any thoughts?

Answer 1

Well, I don't know that I'm going to suggest MSMQ, but I will suggest that there's a lot of edge cases to think about for 'roll your own'.

Even with a thread pool approach, be aware that there may be ordering issues if you care - two items posted sequentially to thread pools may not be executed in order, due to how the thread pools handle work items.

You'll also need to think about persistence of messages, and how long they should exist, how to detect 'fatal' non-delivery status, and what to do in that case.

There's also a number of potential edge cases in scenarios where your app goes down around the same time as it's queueing a message - for instance, it may not get the acknowledgement that the message was queued, even though it was. Yes, you can ack the queue acknowledgement, but you can endlessly get into ack circles...

Why not just have the app detect when it gets connected, and send all the data at that point?

Answer 2

Having written our own service bus I can tell you it's not a trivial undertaking and you will run into the same edge cases that all other implementers have already run into. kyoryu brings up two very important ones.

Whether you roll your own also depends on the skills you have in-house and will have in the future to maintain the solution.

Another consideration is the eventual scale of the system and it's reliability requirements. Will the in-house solution scale sufficiently?

Our peer-to-peer message bus, Zebus, based on ZeroMq (transport), Cassandra (peer discovery and persistence) and Protobuf (serialisation).

1. No client deployment as on the client it's just a library
2. Message persistence is provided using Cassandra as well and handles many different edge cases (this is regularly tested in our production environment)
3. It performs well and as it's a brokerless solution there is no single performance bottleneck
4. There are no single points of failure as the Directory can be used with an available data store such as Cassandra

It is open source and production tested https://github.com/Abc-Arbitrage/Zebus