Homegrown integration system OR ESB?

Question

I just recently started working for a small company which is currently experiencing growing pains. I am not sure what kind of system it is that I am about to describe here. Essentially we have a potpourri of many different 3rd applications which are talking to each other by a homegrown “integration system” which is a mix of SQL jobs, background services written in .NET, FTP transfers, and SSIS etc.

Here is the bird’s eye view: Our public facing website is an Order Entry System (3rd party Shopping Cart software) hosted offsite by the vendor. We download order information every 4 hours a day. This data then gets massaged by our home grown “integration system” which feeds this information to our Inventory and Warehouse Management System (WMS). It also feeds information to MS Great Plains, Pulse, PayFuse, and third party CMS etc.

As you may have already guessed this architecture is very fragile and a slight mishap (such as FTP failure of SQL job failure) may cause discrepancy in data can have a domino effect. There have been times when due to data related issues or replication issues may cause the whole warehouse to come to stand still and we at times are unable to take orders, process, or ship orders.

My task is to re-architect our systems and remove the tight coupling of the systems to allow business growth. What areas do I need to look into? I have been researching ESB and SOA but am being told that my company cannot afford a half a million dollar undertaking to go with say iWay or Talend.

What options are there? Is in-house development the answer and is it cheaper than ESB implementation? Has anyone gone through similar growing pains and if so how did you handle the integration?

Answer 1

Here's how I would approach this problem.

1. Forget about up front design of a single "perfect" system.
2. Forget about replacing everything all at once.
3. Find something that causes a lot of pain, is relatively easy to replace, and doesn't threaten the existence of the business. Work on that first.

In some ways, the fact that there is a "potpourri of many different 3rd applications" is a good thing. You can leave the better ones in place while concentrating on fixing the ones where there is the most business value.

Seek out your stable business concepts and model those explicitly. Command and Event patterns are your friends here. Group these concepts into "Services" following SOA principles.

From your text, it looks like discussions around SLA have already implicitly begun. Make those SLA discussions explicit, but focus on improvement over time towards a goal rather than overnight transformation.

Hand rolling infrastructure for this transformation is probably not worth the time, but spending 6 or 7 figures on a product before you know where you're going is not wise either. Since you mentioned .Net, I have used NServiceBus and find it to be a pleasant programming experience. You focus on your domain and your business logic and let NSB handle the plumbing/infrastructure. For low message throughput, there is a free option. That allows you to deliver some business value before having a discussion about budget and funding. There is a thriving NServiceBus community to help you get started in addition to the documentation on the website.

There are other options in the .net space as well including MassTransit and EventStore. I have not personally used these and they are not functionally equivalent, so you'd need to look them over and see what meets your needs and your team's capabilities.

Answer 2

We've been working through some similar issues over the past couple of years. Leveraging an ESB will help you chip away at it. Once teams start to appreciate the decoupling this starts to give you it will accelerate the process. I'm most familiar with NServiceBus which we've found to have a really low barrier to entry. Developers pick it up quick and it is low cost to try. I would agree you want to start with the area that is most critical to the business and get that on solid ground first.