Java Scanner ignores first token if it is empty

Question

I'm trying to read an InputStream of String tokens with a Scanner. Every token ends with a comma ,. An empty string "" is also a valid token. In that case the whole token is just the comma that ends it.

The InputStream is slowly read from another process, and any tokens should be handled as soon as they have been fully read. Therefore reading the whole InputStream to a String is out of the question.

An example input could look like this:

ab,,cde,fg,

If I set the delimiter of the Scanner to a comma, it seems to handle the job just fine.

InputStream input = slowlyArrivingStreamWithValues("ab,,cde,fg,");

Scanner scan = new Scanner(input);
scan.useDelimiter(Pattern.quote(","));
while (scan.hasNext()) {
    System.out.println(scan.next());
}

output:

ab

cde
fg

However the problems appear when the stream begins with an empty token. For some reason Scanner just ignores the first token if it is empty.

/* begins with empty token */
InputStream input = slowlyArrivingStreamWithValues(",ab,,cde,fg,");
...

output:

ab

cde
fg

Why does Scanner ignore the first token? How can I include it?

Answer 1

Try using a lookbehind as the pattern:

(?<=,)

and then replace comma with empty string with each token that you match. Consider the following code:

String input = ",ab,,cde,fg,";
Scanner scan = new Scanner(input);
scan.useDelimiter("(?<=,)");
while (scan.hasNext()) {
    System.out.println(scan.next().replaceAll(",", ""));
}

This outputs the following:

(empty line)
ab

cde
fg

Demo