Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What are the options to convert ISO-8859-1 / Latin-1 to a String (UTF-8)?

I scanned the Rust documentation for some way to convert between character encodings but did not find anything. Did I miss something?

Is it supported (directly or indirectly) by the Rust language and its standard libraries or even planned to be in the near future?

As one of the answers suggested that there is an easy solution because u8 can be cast to (Unicode) chars. With Unicode being a superset of the codepoints in ISO-8859-1, thats a 1:1 mapping which encodes to multiple bytes in UTF-8 which is the internal encoding of Strings in Rust.

fn main() {
    println!("{}", 196u8 as char);
    println!("{}", (196u8 as char) as u8);
    println!("{}", 'Ä' as u8);
    println!("{:?}", 'Ä'.to_string().as_bytes());
    println!("{:?}", "Ä".as_bytes());
    println!("{}",'Ä' == 196u8 as char);
}

gives:

Ä
196
196
[195, 132]
[195, 132]
true

Which I had not even considered to work!

like image 317
OderWat Avatar asked Jan 27 '15 11:01

OderWat


2 Answers

Strings in Rust are unicode (UTF-8), and unicode codepoints are a superset of iso-8859-1 characters. This specific conversion is actually trivial.

fn latin1_to_string(s: &[u8]) -> String {
    s.iter().map(|&c| c as char).collect()
}

We interpret each byte as a unicode codepoint and then build a String from these codepoints.

like image 65
barjak Avatar answered Sep 19 '22 12:09

barjak


Standard library does not have any API to deal with encodings. Encodings, like date and time, are difficult to do right and need a lot of work, so they are not present in the std.

The crate to deal with encodings as of now is rust-encoding. You will almost certainly find everything you need there.

like image 41
Vladimir Matveev Avatar answered Sep 18 '22 12:09

Vladimir Matveev