What is the lightest solution to creating a multiprocess architecture with state shared across all processes

Question

I have multi-layered application architecture that has 4 parts:

• A networking server/client layer
• An intermediate data layer to handle interactions between processes
• A monitoring layer
• A client layer made up of n number of instances

Client/Server layer:

The client/server layer handles asynchronous network communications with another computer implemented using a custom Layer 2 protocol. Due to design constraints built into the communications, it needs to remain independent and able to poll/push data to the data layer asynchronously.

Intermediate Layer:

The intermediate layer is currently implemented using a database. One table holds all of the possible labels that can be called on (about 120,000). A second table holds an intermediate cache of the first table containing only the values in use, this requires constant updates and gets flushed when a new collection of items is requested. The third table is where collection updates are sent and only contains data when a request is pending.

The Monitor Layer:

The monitor layer is a multi-threaded monolithic application. It spawns n number of client instances based on how many monitors are attached. It manages global state between all client instances because one or more of them may share similar/identical state. It creates a unique listing of values needed, manages sending update requests when the clients need a different set of labels, and manages recurring updates.

Obviously, this isn't ideal. If one instance goes down it can take the rest down with it. What I'd like to do is remove the intermediate layer, replace it with the monitor layer, and make everything spawn as subprocesses of the monitor process so they can be respawned at will if something goes awry (ex. comms heartbeat stops, client crashes, etc).

The database just seems too heavy and not specialized enough to handle the IPC (Inter Process Communications). The program was written under extreme time constraints so utilizing a database was the 'easy solution' with the expectation that it would change in the future. I'm a big fan of the robustness of Google Chrome's multi-process architecture but I know little about how they tie all the processes together (pipes, tcp, ?).

So:

1. Could I expect a significant performance improvement from using IPC over a database for the intermediate layer?

2. What form of IPC would be ideal on a Windows system?

3. Is there a cross platform (read Linux) alternative solution available that could be used in its place if development were moved to Mono?

4. Where can I find resources/examples to help get a start?

Note: I understand that the architecture of this system seems unnecessarily complex but it exists as a front-end for a much larger system. This application is also mission critical so stability trumps efficiency.

Update:

I forgot to mention in the initial question. The database data/index is loaded directly from a ramdisk on boot. The database itself has been indexed for optimal performance. Tables or values that require frequent writes are not indexed but the rest of the data is.

I'm looking for an alternative to measure against because optimization of the db has been taken to its limit and I think there's still a lot of room for improvement.

I will upload a some diagrams of the architecture as soon as I get some time to draw them up.

Answer 1

1. Yes. The database most likely involves the harddrive, and the harddrive is the slowest part of any computer so switching away from using the harddrive will probably have performance benefits.

2. I would go with zeromq / zmq. Its a message oriented framework that supports several communication patterns. For instance PUB/SUB or REQ/REP etc. More examples here

3. zmq is cross platform and its amazingly fast.

4. Some C# examples on github