deadlock when using WCF Duplex Polling with Silverlight

Question

I have followed Tomek Janczuk's demonstration on silverlight tv to create a chat program that uses WCF Duplex Polling web service. The client subscribes to the server, and then the server initiates notifications to all connected clients to publish events.

The Idea is simple, on the client, there is a button that allows the client to connect. A text box where the client can write a message and publish it, and a bigger text box that presents all the notifications received from the server.

I connected 3 clients (in different browsers - IE, Firefox and Chrome) and it all works nicely. They send messages and receive them smoothly. The problem starts when I close one of the browsers. As soon as one client is out, the other clients get stuck. They stop getting notifications.

I am guessing that the loop in the server that goes through all the clients and sends them the notifications is stuck on the client that is now missing. I tried catching the exception and removing it from the clients list (see code) but it still does not help.

any ideas?

The server code is as follows:

    using System;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.ServiceModel.Activation;
using System.Collections.Generic;
using System.Runtime.Remoting.Channels;

namespace ChatDemo.Web
{
    [ServiceContract]
    public interface IChatNotification 
    {
        // this will be used as a callback method, therefore it must be one way
        [OperationContract(IsOneWay=true)]
        void Notify(string message);

        [OperationContract(IsOneWay = true)]
        void Subscribed();
    }

    // define this as a callback contract - to allow push
    [ServiceContract(Namespace="", CallbackContract=typeof(IChatNotification))]
    [AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
    [ServiceBehavior(InstanceContextMode=InstanceContextMode.Single)]
    public class ChatService
    {
        SynchronizedCollection<IChatNotification> clients = new SynchronizedCollection<IChatNotification>();

        [OperationContract(IsOneWay=true)]
        public void Subscribe()
        {
            IChatNotification cli = OperationContext.Current.GetCallbackChannel<IChatNotification>();
            this.clients.Add(cli);
            // inform the client it is now subscribed
            cli.Subscribed();

            Publish("New Client Connected: " + cli.GetHashCode());

        }

        [OperationContract(IsOneWay = true)]
        public void Publish(string message)
        {
            SynchronizedCollection<IChatNotification> toRemove = new SynchronizedCollection<IChatNotification>();

            foreach (IChatNotification channel in this.clients)
            {
                try
                {
                    channel.Notify(message);
                }
                catch
                {
                    toRemove.Add(channel);
                }
            }

            // now remove all the dead channels
            foreach (IChatNotification chnl in toRemove)
            {
                this.clients.Remove(chnl);
            }
        }
    }
}

The client code is as follows:

void client_NotifyReceived(object sender, ChatServiceProxy.NotifyReceivedEventArgs e)
{
    this.Messages.Text += string.Format("{0}\n\n", e.Error != null ? e.Error.ToString() : e.message);
}

private void MyMessage_KeyDown(object sender, KeyEventArgs e)
{
    if (e.Key == Key.Enter)
    {
        this.client.PublishAsync(this.MyMessage.Text);
        this.MyMessage.Text = "";
    }
}

private void Button_Click(object sender, RoutedEventArgs e)
{
    this.client = new ChatServiceProxy.ChatServiceClient(new PollingDuplexHttpBinding { DuplexMode = PollingDuplexMode.MultipleMessagesPerPoll }, new EndpointAddress("../ChatService.svc"));

    // listen for server events
    this.client.NotifyReceived += new EventHandler<ChatServiceProxy.NotifyReceivedEventArgs>(client_NotifyReceived);

    this.client.SubscribedReceived += new EventHandler<System.ComponentModel.AsyncCompletedEventArgs>(client_SubscribedReceived);

    // subscribe for the server events
    this.client.SubscribeAsync();

}

void client_SubscribedReceived(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
    try
    {
        Messages.Text += "Connected!\n\n";
        gsConnect.Color = Colors.Green;
    }
    catch
    {
        Messages.Text += "Failed to Connect!\n\n";

    }
}

And the web config is as follows:

  <system.serviceModel>
    <extensions>
      <bindingExtensions>
        <add name="pollingDuplex" type="System.ServiceModel.Configuration.PollingDuplexHttpBindingCollectionElement, System.ServiceModel.PollingDuplex, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
      </bindingExtensions>
    </extensions>
    <behaviors>
      <serviceBehaviors>
        <behavior name="">
          <serviceMetadata httpGetEnabled="true"/>
          <serviceDebug includeExceptionDetailInFaults="false"/>
        </behavior>
      </serviceBehaviors>
    </behaviors>
    <bindings>
      <pollingDuplex>        
        <binding name="myPollingDuplex" duplexMode="MultipleMessagesPerPoll"/>
      </pollingDuplex>
    </bindings>
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true"/>
    <services>
      <service name="ChatDemo.Web.ChatService">
        <endpoint address="" binding="pollingDuplex" bindingConfiguration="myPollingDuplex" contract="ChatDemo.Web.ChatService"/>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
      </service>
    </services>
  </system.serviceModel>

Answer 1

Try to set inactivityTimeout. Had the same problem before. Worked out for me. pollingDuplex inactivityTimeout="02:00:00" serverPollTimeout="00:05:00" maxPendingMessagesPerSession="2147483647" maxPendingSessions="2147483647" duplexMode="SingleMessagePerPoll"

Answer 2

OK, I finally found a solution. Its kind of a dirty patch, but it works and its stable, so that's what I'll use.

First, I want to clarify the situation itself. I thought this was a deadlock, but it wasn't. It was actually a combination of 2 different problems that made me think that the clients are all waiting while the server is stuck on something. The server was not stuck, it was just in the middle of a very lengthy process. The thing is, that the IE client had a problem of its own, which made it seem like it was waiting forever.

I eventually managed to isolate the 2 problems and then gave each problem its own solution.

Problem number 1: The server hangs for a long time while trying to send a notification to a client that was disconnected.

Since this was done in a loop, other clients had to wait as well:

 foreach (IChatNotification channel in this.clients)
            {
                try
                {
                    channel.Notify(message); // if this channel is dead, the next iteration will be delayed
                }
                catch
                {
                    toRemove.Add(channel);
                }
            }

So, to solve this problem, I made the loop start a distinct thread for each client, so the notifications to the clients become independent. Here is the final code:

[OperationContract(IsOneWay = true)]
public void Publish(string message)
{
    lock (this.clients)
    {
        foreach (IChatNotification channel in this.clients)
        {
            Thread t = new Thread(new ParameterizedThreadStart(this.notifyClient));
            t.Start(new Notification{ Client = channel, Message = message });
        }
    }

}

public void notifyClient(Object n)
{
    Notification notif = (Notification)n;
    try
    {
        notif.Client.Notify(notif.Message);
    }
    catch
    {
        lock (this.clients)
        {
            this.clients.Remove(notif.Client);
        }
    }
}

Note that there is one thread to handle each client notification. The thread also discards the client, if it failed to send the notification.

Problem number 2: The client kills the connection after 10 idle seconds.

This problem, surprisingly, only happened in explorer... I can't really explain it, but after doing some research in Google I found that I was not the only one to notice it, but could not find any clean solution except the obvious - "just ping the server every 9 seconds". Which is exactly what I did.

So I extended the contract interface to include a server Ping method, which instantly calls a client's Pong method:

[OperationContract(IsOneWay = true)]
public void Ping()
{
    IChatNotification cli = OperationContext.Current.GetCallbackChannel<IChatNotification>();
    cli.Pong();
}

the client's Pong event handler creates a thread that sleeps for 9 seconds and then calls the ping method again:

void client_PongReceived(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
    // create a thread that will send ping in 9 seconds
    Thread t = new Thread(new ThreadStart(this.sendPing));
    t.Start();
}

void sendPing()
{
    Thread.Sleep(9000);
    this.client.PingAsync();
}

And that was it. I tested it with multiple clients, removed some clients by closing their browsers, then re launched them, it all worked. And the lost clients were eventually cleaned by the server.

One more note - Since the client connection proved to be unreliable, I surrounded it with a try - catch exception so I can respond to cases where the connection spontaneously dies:

        try
        {
            this.client.PublishAsync(this.MyMessage.Text);
            this.MyMessage.Text = "";
        }
        catch
        {
            this.Messages.Text += "Was disconnected!";
            this.client = null;
        }

This, of course, does not help, since the "PublishAsync" returns instantly, and successfully, while the code that was automatically generated (in Reference.cs) does the actual work of sending the message to the server, in another thread. The only way I could think of to catch this exception is by updating the automatically generated proxy... which is a very bad idea... but I could not find any other way. (Ideas will be appreciated).

That's all. If anybody knows of an easier way to work around this issue, I will be more than happy to hear.

Cheers,

Kobi

Answer 3

A better way to solve problem #1 is to set up the callback using the async pattern:

    [OperationContract(IsOneWay = true, AsyncPattern = true)]
    IAsyncResult BeginNotification(string message, AsyncCallback callback, object state);
    void EndNotification(IAsyncResult result);

When the server notifies the remaining clients it issues the first half:

    channel.BeginNotification(message, NotificationCompletedAsyncCallback, channel);

This way the remaining clients get notified without having to wait for the time-out on the client that has dropped off.

Now set up the static completed method as

    private static void NotificationCompleted(IAsyncResult result)

In this completed method call the remaining half of the call like this:

    IChatNotification channel = (IChatNotification)(result.AsyncState);
    channel.EndNotification(result);