StringTokenizer split at "<br/>"

Question

Maybe I am stupid but I don't understand why the behaviour of StringTokenizer here:

import static org.apache.commons.lang.StringEscapeUtils.escapeHtml;

String object = (String) value;
String escaped = escapeHtml(object);
StringTokenizer tokenizer = new StringTokenizer(escaped, escapeHtml("<br/>"));

If fx. value is

Hej<br/>$user.get(0).name Har vundet<br/><table border='1'><tr><th>Name</th><th>Played</th><th>Brewed</th></tr>#foreach( $u in $user )<tr><td>$u.name</td> <td>$u.played</td> <td>$u.brewed</td></tr>#end</table><br/>

Then the result is

Hej
$use
.
e
(0).name Ha
 vunde
a
e 
o
de
='1'
h
Name
h
h
P
ayed
h
h
B
ewed
h
#fo
each( $u in $use
 )
d
$u.name
d

d
$u.p
ayed
d

d
$u.
ewed
d
#end
a
e

It makes no sense to me.

How can I make it behave as I expect to.

Answer 1

From the documentation:

The characters in the delim argument are the delimiters for separating tokens. Delimiter characters themselves will not be treated as tokens.

In other words, the characters that tell StringTokenizer when to separate the string are:

• <
• b
• r
• /
• >

When it matches any of those characters in the string (the variable escaped in your code), the StringTokenizer instance will split the result and drop the token. You can confirm this by noting that the letter r does not occur in the output.

Use String.split, instead, as others suggest.