Atmosphere broadcasts don't reach client when running in Tomcat

Question

Substantial update below (October 15, 2014)

========== Original posting ===================

I have a notification service webapp using Atmosphere and Jersey. I am trying to run it in Tomcat on Linux, and have run into a problem. (Note that this works fine when running in Tomcat-embedded-within-Eclipse, so it's possible it's a Tomcat configuration problem.)

The problem is that broadcast messages seem to be queued up, or buffered, and are not actually delivered to the client until I shut down Tomcat. At that point they are all received by the client at once.

I have tried a number of things that I found in various postings to try to solve this, but my efforts are scattershot because I don't know what is causing the problem:

• I have tried explicitly using Tomcat's NIO protocol.
• I have tried explicitly disabling text and binary buffering, and I have also queued up enough responses to trigger a flush if buffering were the cause.
• I tried running in Tomcat 8 instead (but that had different unrelated problems.)
• I have added the atmosphere-runtime-native, atmosphere-compat-jbossweb, and atmosphere-compat-tomcat dependencies.

None of these has changed the behavior of the service. And, again, it seems to work fine when running in Eclipse.

Environment: Ubuntu 14.04 LTS. Tomcat 7.0.54. Atmosphere 2.1.5. For a client, I'm just using telnet at the moment.

(This code is a bit more complicated than the samples because I have a number of different routes I want to support, and I'm using the method proposed by this answer, which seems to be working nicely. Also I'm returning notification objects rather than Strings. This also seems to be working fine. I've stripped out most of the authentication stuff for clarity.)

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.clearcaptial</groupId>
  <artifactId>notifications</artifactId>
  <packaging>war</packaging>
  <version>0.0.1-SNAPSHOT</version>
  <name>notifications Maven Webapp</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>org.atmosphere</groupId>
      <artifactId>atmosphere-jersey</artifactId>
      <version>2.1.5</version>
    </dependency>
    <dependency>
      <groupId>com.sun.jersey</groupId>
      <artifactId>jersey-json</artifactId>
      <version>1.17.1</version>
    </dependency>
    <dependency>
      <groupId>org.atmosphere</groupId>
      <artifactId>atmosphere-runtime-native</artifactId>
      <version>2.1.5</version>
    </dependency>
    <dependency>
      <groupId>org.atmosphere</groupId>
      <artifactId>atmosphere-compat-jbossweb</artifactId>
      <version>2.0.1</version>
    </dependency>
    <dependency>
      <groupId>org.atmosphere</groupId>
      <artifactId>atmosphere-compat-tomcat</artifactId>
      <version>2.0.1</version>
    </dependency>
  </dependencies>
  <build>
    <finalName>notifications</finalName>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.1</version>
        <configuration>
          <source>1.7</source>
          <target>1.7</target>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

web.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!-- This web.xml file is not required when using Servlet 3.0 container, 
    see implementation details http://jersey.java.net/nonav/documentation/latest/jax-rs.html -->
<web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee"
    xmlns:j2ee="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_3.0.xsd">

    <servlet>
    <servlet-name>AtmosphereServlet</servlet-name>
    <servlet-class>org.atmosphere.cpr.AtmosphereServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
    <async-supported>true</async-supported>
    <init-param>
      <param-name>com.sun.jersey.config.property.packages</param-name>
      <param-value>com.clearcapital.notifications</param-value>
    </init-param>
    <init-param>
      <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
      <param-value>true</param-value>
    </init-param>
    <init-param>
      <param-name>org.atmosphere.useWebSocketAndServlet3</param-name>
      <param-value>true</param-value>
    </init-param>
  </servlet>
  <servlet-mapping>
    <servlet-name>AtmosphereServlet</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

from ValidationsRouter.java:

@Path("/validations")
public class ValidationsRouter {

  @Context
  private AtmosphereResource atmosphereResource;

  @GET
  public SuspendResponse<ValidationNotification> subscribe(@Context UriInfo uri) throws URISyntaxException, NamingException, IOException {
    return new ValidationsResource().subscribe(uri, atmosphereResource);
  }

  @Broadcast
  @POST
  @Consumes(MediaType.APPLICATION_JSON)
  @Produces(MediaType.APPLICATION_JSON)
  public Broadcastable broadcast(ValidationNotification message, @Context UriInfo uri) {
    return new ValidationsResource().broadcast(message, uri);
  }
}

from ValidationsResource.java:

public class ValidationsResource extends ResourceBase<ValidationNotification>  {
  // a bunch of authentication stuff omitted for clarity.
}

from ResourceBase.java:

public abstract class ResourceBase<T> {

  public SuspendResponse<T> subscribe(UriInfo uri, AtmosphereResource atmosphereResource) throws URISyntaxException, NamingException, IOException {

    Broadcaster myBroadcaster = BroadcasterFactory.getDefault().lookup(uri.getPath(), true);
    return new SuspendResponse.SuspendResponseBuilder<T>().broadcaster(myBroadcaster).outputComments(true)
                .addListener(new EventsLogger()).type(MediaType.APPLICATION_JSON_TYPE).build();
  }

  public Broadcastable broadcast(T message, UriInfo uri) {
    Broadcaster myBroadcaster = BroadcasterFactory.getDefault().lookup(uri.getPath(), true);
    return new Broadcastable(message, "", myBroadcaster);
  }
}

============ End of Original Post ======================

Updated October 15, 2014:

I've come back to this problem after some time on a different project. After updating the project with a new version of Atmosphere (2.2.3), I find no difference in behavior. It is still the case that, when running the service on a different computer than the client is running on, the atmosphere broadcasts are not received by the client until Tomcat shuts down. When running the client and the service on the same computer, everything works as expected.

So, to determine where the problem lies, I have started just using one of the Atmosphere sample apps in my testing. I am using the jersey-pubsub sample app (with minor updates to get it to compile), because that's the one that's closest to what I'm trying to do. I have created a brand new server environment to run the sample app, to minimize any variables based on Linux configuration. Here are the new steps -- maybe someone can replicate the behavior.

To get a new server with Tomcat7 installed (I'm using Vagrant and VirtualBox for these tests):

host$ vagrant init ubuntu/trusty64
host$ nano Vagrantfile

change network config to a known IP: config.vm.network :private_network, ip: "10.1.1.2"

host$ vagrant box add https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/1/providers/virtualbox.box
host$ vagrant up
------------------
vagrant$ sudo apt-get update
vagrant$ sudo apt-get upgrade
vagrant$ sudo apt-get install tomcat7

Following these steps, I ended up with Tomcat 7.0.52, and Ubuntu 14.04

I have a copy of just the jersey-pubsub project (without parent project etc.) that I build using

host$ mvn clean install

and then deploy by

host$ cp target/jersey-pubsub.war ../../vagrants/trusty64/
vagrant$ sudo cp /vagrant/jersey-pubsub.war /var/lib/tomcat7/webapps

I can then test this using telnet, for example:

host$ telnet 10.1.1.2 8080 <return>
GET /jersey-pubsub/pubsub/foo HTTP/1.0 <return>
<return>

At this point, telnet continues to wait for data. In the tomcat server log, I can see an entry:

20:33:55.805 [http-bio-8080-exec-1] INFO  o.a.samples.pubsub.EventsLogger - onSuspend(): 10.1.1.1:59357

I then send a POST request to the server. I'm using POSTman for this, but I don't think it matters.

POST http://10.1.1.2:8080/jersey-pubsub/pubsub/foo
Accept: text/html
Content-Type: application/x-www-form-urlencoded
message: <html><body><p>This is a test!</p></body></html>

In the server log I can see that the server believes it broadcast the message:

20:34:06.990 [Atmosphere-Shared-AsyncOp-0] INFO  o.a.samples.pubsub.EventsLogger - onBroadcast(): <html><body><p>This is a test!</p></body></html>

but nothing shows up in the telnet client. I can repeat ad infinitum and see no result, until I shut down Tomcat, at which point all of the broadcast messages are received by Telnet immediately.

In contrast, if I deploy the service on a tomcat instance running on the host machine, or if I run telnet within the vagrant instance, everything works as expected:

vagrant$ telnet localhost 8080 <return>
GET /jersey-pubsub/pubsub/foo HTTP/1.0 <return>
<return>

If I then send the same POST request, I immediately receive the broadcast message in telnet. The behavior seems to be that if the service and the client are not on the same computer, then the network traffic is held up while Tomcat is running.

Here's the source code for my copy of the jersey-pubsub sample, with just enough changes to get it to compile and build as a standalone project.

EventsLogger.java: (Unchanged except to remove @Override in 2 places to get Eclipse to stop complaining)

FileResource.java: (Unchanged)

JerseyPubSub.java: (Unchanged)

web.xml: (Unchanged)

pom.xml (modified to build as a standalone, without reference to parent pom):

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.atmosphere.samples</groupId>
    <artifactId>atmosphere-jersey-pubsub</artifactId>
    <packaging>war</packaging>
    <version>0.0.1-SNAPSHOT</version>
    <name>atmosphere-jersey-pubsub</name>
    <url>http://maven.apache.org</url>
    <properties>
        <atmosphere-version>2.2.3</atmosphere-version>
        <client-version>2.2.3</client-version>
        <logback-version>1.0.13</logback-version>
    </properties>
    <repositories>
        <repository>
            <id>oss.sonatype.org</id>
            <url>http://oss.sonatype.org/content/repositories/releases</url>
        </repository>
        <repository>
            <id>oss.sonatype.org-snapshot</id>
            <url>http://oss.sonatype.org/content/repositories/snapshots</url>
        </repository>
        <!-- <repository> <id>scala-tools.org</id> <name>Scala-Tools Maven2 Repository</name> 
            <url>http://scala-tools.org/repo-releases</url> </repository> -->
        <repository>
            <id>jboss</id>
            <url>https://repository.jboss.org/nexus/content/groups/public/</url>
        </repository>
        <repository>
            <id>codehaus</id>
            <name>repository.codehaus.org</name>
            <url>http://repository.codehaus.org</url>
        </repository>
        <repository>
            <id>codehaus-snapshots</id>
            <url>http://snapshots.repository.codehaus.org</url>
        </repository>
        <repository>
            <id>maven.java.net</id>
            <url>https://maven.java.net/content/groups/public/</url>
        </repository>
    </repositories>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.atmosphere</groupId>
                <artifactId>atmosphere-runtime</artifactId>
                <version>${atmosphere-version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.geronimo.specs</groupId>
                <artifactId>geronimo-servlet_3.0_spec</artifactId>
                <version>1.0</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-classic</artifactId>
                <version>${logback-version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.atmosphere.client</groupId>
            <artifactId>javascript</artifactId>
            <version>${client-version}</version>
            <type>war</type>
        </dependency>
        <dependency>
            <groupId>org.atmosphere</groupId>
            <artifactId>atmosphere-jersey</artifactId>
            <version>${atmosphere-version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.geronimo.specs</groupId>
            <artifactId>geronimo-servlet_3.0_spec</artifactId>
        </dependency>

        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
        </dependency>

        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-core</artifactId>
            <version>${logback-version}</version>
        </dependency>
    </dependencies>
    <build>
        <finalName>jersey-pubsub</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Answer 1

1. Are you running tomcat behind an apache proxy?
2. Is apache running HTTP 1.1?
3. Some more apache things to try here

Answer 2

After returning to this problem after several months, I was able to figure out why I was seeing this behavior: it was specifically the built-in Mac telnet client that was behaving badly. I confirmed that the responses were logged immediately when using telnet from Windows or Linux, and I also tested with a different Mac telnet client, and it behaved as expected, as well. Although I worked with it for a while, I was not able to figure out if there were telnet options that I could specify on the Mac that would make it behave in a fashion comparable to the Linux ones I was testing with.