Some information:
What happens when I run it
I get the expected string "abcd" repeated until it hits the position of 4094 characters. After that all it outputs is this sign "?" until the end of the file.
What do I think about this?
I think this is not the expected behavior and that it must be a bug somewhere.
Code you can test with:
#include <iostream>
#include <fstream>
#include <locale>
#include <codecvt>
void createTestFile() {
std::ofstream file ("utf16le.txt", std::ofstream::binary);
if (file.is_open()) {
uint16_t bom = 0xFEFF; // UTF-16 little endian BOM
uint64_t abcd = 0x0064006300620061; // UTF-16 "abcd" string
file.write((char*)&bom,2);
for (size_t i=0; i<2000; i++) {
file.write((char*)&abcd,8);
}
file.close();
}
}
int main() {
//createTestFile(); // uncomment to make the test file
std::wifstream file;
std::wstring line;
file.open("utf16le.txt");
file.imbue(std::locale(file.getloc(), new std::codecvt_utf16<wchar_t, 0x10ffff, std::consume_header>));
if (file.is_open()) {
while (getline(file,line)) {
std::wcout << line << std::endl;
}
}
}
This looks like a library bug to me. Stepping through the sample program as compiled by gcc 7.1.1 using gdb
:
(gdb) n
28 while (getline(file,line)) {
(gdb) n
29 std::wcout << line << std::endl;
(gdb) p line.size()
$1 = 8000
8000 characters read, as expected. But then:
(gdb) p line[4092]
$18 = (__gnu_cxx::__alloc_traits<std::allocator<wchar_t> >::value_type &) @0x628240: 97 L'a'
(gdb) p line[4093]
$19 = (__gnu_cxx::__alloc_traits<std::allocator<wchar_t> >::value_type &) @0x628244: 98 L'b'
(gdb) p line[4094]
$20 = (__gnu_cxx::__alloc_traits<std::allocator<wchar_t> >::value_type &) @0x628248: 25344 L'挀'
(gdb) p line[4095]
$21 = (__gnu_cxx::__alloc_traits<std::allocator<wchar_t> >::value_type &) @0x62824c: 25600 L'搀'
(gdb) p line[4096]
$22 = (__gnu_cxx::__alloc_traits<std::allocator<wchar_t> >::value_type &) @0x628250: 24832 L'愀'
line[4092]
and line[4093]
look ok. But then, I see line[4094]
, line[4095]
, and line[4096]
, containing 6300
, 6400
and 6500
, instead of 0063
, 0064
, and 0065
.
So, this is getting messed up starting with character 4094, and not 4096, actually. I dumped the binary UTF-16 file, and it looks correct to me. The BOM marker is followed by consistent endian-ness for the entire contents of the file.
The only thing that's puzzling is why both clang and gcc are supposedly affected, but a quick Google search indicates that clang also uses gcc's libstdc++, at least up until recently. So, this looks like a libstdc++ bug to me.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With