Which network protocol to use for lightweight notification of remote apps?

Question

I have this situation.... Client-initiated SOAP 1.1 communication between one server and let's say, tens of thousands of clients. Clients are external, coming in through our firewall, authenticated by certificate, https, etc.. They can be anywhere, and usually have their own firewalls, NAT routers, etc... They're truely external, not just remote corporate offices. They could be in a corporate/campus network, DSL/Cable, even Dialup.

Client uses Delphi (2005 + SOAP fixes from 2007), and the server is C#, but from an architecture/design standpoint, that shouldn't matter.

Currently, clients push new data to the server and pull new data from the server on 15-minute polling loop. The server currently does not push data - the client hits the "messagecount" method, to see if there is new data to pull. If 0, it sleeps for another 15 min and checks again.

We're trying to get that down to 7 seconds.

If this were an internal app, with one or just a few dozen clients, we'd write a cilent "listener" soap service, and would push data to it. But since they're external, sit behind their own firewalls, and sometimes private networks behind NAT routers, this is not practical.

So we're left with polling on a much quicker loop. 10K clients, each checking their messagecount every 10 seconds, is going to be 1000/sec messages that will mostly just waste bandwidth, server, firewall, and authenticator resources.

So I'm trying to design something better than what would amount to a self-inflicted DoS attack.

I don't think it's practical to have the server send soap messages to the client (push) as this would require too much configuration at the client end. But I think there are alternatives that I don't know about. Such as:

1) Is there a way for the client to make a request for GetMessageCount() via Soap 1.1, and get the response, and then perhaps, "stay on the line" for perhaps 5-10 minutes to get additional responses in case new data arrives? i.e the server says "0", then a minute later in response to some SQL trigger (the server is C# on Sql Server, btw), knows that this client is still "on the line" and sends the updated message count of "5"?

2) Is there some other protocol that we could use to "ping" the client, using information gathered from their last GetMessageCount() request?

3) I don't even know. I guess I'm looking for some magic protocol where the client can send a GetMessageCount() request, which would include info for "oh by the way, in case the answer changes in the next hour, ping me at this address...".

Also, I'm assuming that any of these "keep the line open" schemes would seriously impact the server sizing, as it would need to keep many thousands of connections open, simultaneously. That would likely impact the firewalls too, I think.

Is there anything out there like that? Or am I pretty much stuck with polling?

TIA,
Chris

UPDATE 4/30/2010:
Having demonstrated that having 7-second notification is neither easy nor cheap, especially without going outside of the corporate standard of HTTPS/SOAP/Firewalls, we're probably going to pitch a two-phase solution. Phase1 will have the clients poll "on-demand" with the GetMessageCount being performed through SOAP, nothing fancy here. There will be a "refresh" button to pull new data (which is reasonable here, as the user will usually have reason to suspect that new data is ready, i.e. they just changed the fabric color in the online system, so they know to click REFRESH before viewing the shipping manifest on the desktop, and now they see the color in the description.) (This is NOT really a garment/fashion app, but you get the idea). The notion of having the two aps always be in sync, with real-time updates pushed from the host, is still on the table, using the technologies discussed here. But I expect that it will be pushed off for another release, as we can deliver 85% of the functionality without having to do this. However, I hope that we get to do a Proof Of Concept, and can demonstrate that it'll work. I'll come back and post future updates. Thanks for everyone's help on this.

Answer 1

Consider "playing" the HTTP protocol a bit to get what you want while still being able to go over all of the proxies and NAT's and firewalls one might have on the client side.

Have every single client do a plain HTTP request for the message count in a way that would inhibit any sort of caching (example: GET http://yourserver.org/getcount/nodeid/timeofday/sequence). In the server-side implementation of the HTTP server delay providing the answer if the "count" is the same it used to be (ie: no new messages).

I've done this for a Ajax-style application that ran in a browser and behaved a bit like a chat application, but your solution can be even faster. I implemented the server side stuff using the TIdHttp server and that allowed me to actually delay providing the answer to the client stuff by simply Sleep()-ing in it's thread. From the client side it looked like an server that's sometimes really slow to give an answer.

Pseudocode for the server-side stuff:

function ClientHasMessages(ClientID:Integer; MaxWait:TDateTime):Boolean;
var MaxTime:TDateTime;
begin
  if ClientActuallyHasMessage(ClientID) then Result := True
  else
    begin
      MaxTime := Now + MaxWait;
      while Now < MaxTime do
      begin
        if ClientActuallyHasMessage(ClientID) then
          begin
            Result := True;
            Exit;
          end
        else
          Sleep(1000);
      end;
      Result := False; // TimeOut
    end;
end;

The idea behind this code: It runs in a thread on your own server, where it can test the message count, presumably, for very little cost:

• It causes no network traffic while waiting.
• It uses no CPU while Sleeping.
• It will let the user know about it's message very quickly.
• It lets the client control how long the wait might be (the client will increase the amount of time the server may delay the answer until it no longer receives the answer, and then step back a bit - that way the protocol adapts to whatever buggy NAT router the client uses).
• You can get away with long periods of no TCP/IP communications and still being able to provide the answer instantly. 30 seconds is easily done and for clients with good NAT routers it can be much longer.

The down size of this would be the requirements on the server, but I'm tempted to say they're doable:

• The server's TCP/IP implementation needs to track quite an number of simultaneous connections (every client will have a HTTP request active at all times). My Linux NAT machine is tracking 15K connections right now and it's basically idle, so it might work.
• The server would have an thread open for every single client HTTP request, at all times: Again, the Server 2008 "Workstation" I'm using to write this (thank you MSDN for allowing me to do such outrageous things) has about 1500 threads active and it's also basically idle...
• Depending on the technology you use for the server-side code MEMORY might be the limiting factor.

Answer 2

I would have a look at kbmMW

I would possibly use a method similar to MS Exchange - connection and authentication via tcp/ip, then notification of update(s) from the server to client via udp, then the client recieves the udp request and downloads the update via tcp/ip.

(At least that's how I understand MS Exchange works)

Answer 3

The two big parties on multi-tier development in Delphi are components4developers (with their kbmMW product described in the answer by Mark Robinson) and RemObjects with their product RemObjects SDK (they have a nice example that might be similar to what you want: Push notifications for iPhone).

In your complex environment, multi-cast UDP might not cut it, but from a overhead perspective it is unbeatable.

If a connection is open, it can be used in a bi-directional way (this is also used by .NET remoting and WCF), but has additional overhead.

You will need to find a balance between keeping connections live (locking resources), and creating new connections (costing time and latency).

--jeroen