java.util.Scanner hangs on hasNext()

Question

I'm encountering a very strange issue with the Scanner class. I'm using Scanner to read messages from a Socket with a special EOF token. Everything works fine if the client writes all requests at once, or the requests have data, but the blocking hasNext() operation hangs on the server, and in turn the client, when the messages are written in chunks and the next token should be an empty string.

What would cause this? How can I go about avoiding this?

Here's a simplified version of what I'm trying to do, \n is being used for testing purposes, assume that the delimiter could be any string.

Server Code:

ServerSocketChannel serverChannel = null;
try {
    serverChannel = ServerSocketChannel.open();

    ServerSocket serverSocket = serverChannel.socket();
    serverSocket.bind(new InetSocketAddress(9081));

    SocketChannel channel = serverChannel.accept();
    Socket socket = channel.socket();

    InputStream is = socket.getInputStream();
    Reader reader = new InputStreamReader(is);
    Scanner scanner = new Scanner(reader);
    scanner.useDelimiter("\n");

    OutputStream os = socket.getOutputStream();
    Writer writer = new OutputStreamWriter(os);

    while (scanner.hasNext()) {
        String msg = scanner.next();
        writer.write(msg);
        writer.write('\n');
        writer.flush();
    }
} catch (IOException e) {
    e.printStackTrace();
} finally {
    if (serverChannel != null) {
        try {
            serverChannel.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Working Client:

Socket socket = new Socket();
try {
    socket.connect(new InetSocketAddress("localhost", 9081));

    InputStream is = socket.getInputStream();
    Reader reader = new InputStreamReader(is);
    Scanner scanner = new Scanner(reader);
    scanner.useDelimiter("\n");

    OutputStream os = socket.getOutputStream();
    Writer writer = new OutputStreamWriter(os);

    writer.write("foo\n\nbar\n");
    writer.flush();
    System.out.println(scanner.next());
    System.out.println(scanner.next());
    System.out.println(scanner.next());

} catch (IOException e) {
    e.printStackTrace();
} finally {
    try {
        socket.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

Hanging Client:

Socket socket = new Socket();
try {
    socket.connect(new InetSocketAddress("localhost", 9081));

    InputStream is = socket.getInputStream();
    Reader reader = new InputStreamReader(is);
    Scanner scanner = new Scanner(reader);
    scanner.useDelimiter("\n");

    OutputStream os = socket.getOutputStream();
    Writer writer = new OutputStreamWriter(os);

    writer.write("foo\n");
    writer.flush();
    System.out.println(scanner.next());

    writer.write("\n");
    writer.flush();
    System.out.println(scanner.next());

    writer.write("bar\n");
    writer.flush();
    System.out.println(scanner.next());

} catch (IOException e) {
    e.printStackTrace();
} finally {
    try {
        socket.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

Answer 1

I spent some time tracing the code and the problem is most definitely a defect in the Scanner class.

public boolean hasNext() {
    ensureOpen();
    saveState();
    while (!sourceClosed) {
        if (hasTokenInBuffer())
            return revertState(true);
        readInput();
    }
    boolean result = hasTokenInBuffer();
    return revertState(result);
}

hasNext() calls hasTokenInBuffer()

private boolean hasTokenInBuffer() {
    matchValid = false;
    matcher.usePattern(delimPattern);
    matcher.region(position, buf.limit());

    // Skip delims first
    if (matcher.lookingAt())
        position = matcher.end();

    // If we are sitting at the end, no more tokens in buffer
    if (position == buf.limit())
        return false;

    return true;
}

hasTokenInBuffer() always skips the first delimiter if it exists as explained in the javadoc.

The next() and hasNext() methods and their primitive-type companion methods (such as nextInt() and hasNextInt()) first skip any input that matches the delimiter pattern, and then attempt to return the next token. Both hasNext and next methods may block waiting for further input. Whether a hasNext method blocks has no connection to whether or not its associated next method will block.

First we skip the token that was still in the buffer from the last request, then we notice we don't have any new data in our buffer so we call readInput(), in this case just \n, then we loop back around to hasTokenInBuffer() which skips our delimiter again!

At this point the Server is waiting for more input, and the Client is waiting for a response. Deadlock.

This can easily be avoided if we check if we skipped the last token...

private boolean skippedLast = false;

private boolean hasTokenInBuffer() {
    matchValid = false;
    matcher.usePattern(delimPattern);
    matcher.region(position, buf.limit());

    // Skip delims first
    if (!skippedLast && matcher.lookingAt()) {
        skippedLast = true;
        position = matcher.end();
    } else {
        skippedLast = false;
    }

    // If we are sitting at the end, no more tokens in buffer
    if (position == buf.limit())
        return false;

    return true;
}