jsp utf encoding

Question

I'm having a hard time figuring out how to handle this problem:

I'm developing a web tool for an Italian university, and I have to display words with accents (such as è, ù, ...); sometimes I get these words from a PostgreSql table (UTF8-encoded), but mostly I have to read long passages from a file. These files are encoded as utf-8 xml, and display fine in Smultron or any utf-8 editor (they were created parsing in python old files with entities such as è instead of "è").

I wrote a java class which extracts the relevant segments from the xml file, which works like this:

String s = parseText(filename, position)

if I write the returned String to a file, everything looks fine; the problem is that if I do

out.write(s)

in the jsp page, I get strange characters. By the way, I use

String s = getWordFromPostgresql(...)

out.write(s)

in the very same jsp and it displays OK.

Any hint?

Thanks Nicola

---

@krosenvold

Thanks for your response, however that directive is already in the page, but it doesn't work (actually it "works" but only for the strings I get from the database). I think there's something about reading from the files, but I can't understand ... they work in "java" but not in "jsp" (can't think about a better explanation ...)

here's a basic example extracted from the actual code: the method to read from the files return a Map, from a Mark (an object representing a position in the text) to a String (containing the text):

this is in the .jsp page (with the utf-directive cited in the posts above)

    // ...
    Map<Mark, String> map = TestoMarkParser.parseMarks(...);
    out.write(map.get(m));

and this is the result:

"Fu però così in uso il Genere Enharmonico, che quelli quali vi si esercitavano,"

if I put the same code in a java class, and substitute out.write with System.out.println, the result is this:

"Fu però così in uso il Genere Enharmonico, che quelli quali vi si esercitavano,"

---

I've been doing some analysis with an hex editor, here it is:

original string: "fu però così "

ò in xml file: C3 B2

ò as rendered by out.write() in the jsp file: E2 88 9A E2 89 A4

ò as written to file via:

FileWriter w = new FileWriter(new File("out.txt"));
w.write(s);     // s is the parsed string
w.close();

C3 B2

printing the values of each character as an int

0: 70 = F
1: 117 = u
2: 32 =  
3: 112 = p
4: 101 = e
5: 114 = r
6: 8730 = � 
7: 8804 = � 
8: 32 =  
9: 99 = c
10: 111 = o
11: 115 = s
12: 8730 = �
13: 168 = �
14: 10 = `

Answer 1

In the jsp page directive you should try setting your content-type to utf-8, which will set the pageEncoding to utf-8 also.

<%@page contentType="text/html;charset=UTF-8"%>

UTF-8 is not default content type in jsp, and there are all sorts of interesting problems that arise from this. The problem is that the underlying stream is interpreted as an ISO-8859-1 stream by default. If you write some unicode bytes to this stream, they will be interpreted as ISO-8859-1. I find that setting the encoding to utf-8 is the best solution.

Edit: Furthermore, a string variable in java should always be unicode. So you should always be able to say

System.out.println(myString) 

and see the proper character set coming in the console window of your web-server (or just stop in the debugger and examine it). I suspect that you'll be seeing incorrect characters when you do this, which leads me to believe you have an encoding problem when constructing the string.