Using std::endl on an ostream makes my file binary

Question

I am working on a project that uses libzip. I'm working in c++14 and I wrote a tiny wrapper around libzip to make my life easier.

I have an std::ostream object built around custom class that inherits std::streambuf. This streambuf uses the libzip functions to write in a file in the archive.

Everything works great until I use std::endl. When I do this, the output file is read as binary by all my text readers (only writing strings).

My text reader detect that its binary because in the place I used std::endl there a NUL byte, any file with a NUL byte inside of it is seen as binary.

So my question is : Is this normal ? Is there a way for me to use std::endl ?

My code (extracted so it may not be exactly the same).

source.hpp

// my attributes
std::unique_ptr<zip_source_t, std::function<void(zip_source_t*)>> _source;
std::unique_ptr<std::ostream> _stream;
std::unique_ptr<_ZipBuffer> _buffer;

class _ZipBuffer : public std::streambuf {
    private:
        zip_source_t* _source;

        std::streamsize xsputn(char const* s, std::streamsize n) override;
        int overflow(int c) override;

    public:
        _ZipBuffer(zip_source_t* file);
};

source.cpp

// create the streambuf and send it to the ostream
_buffer.reset(new _ZipBuffer(_source.get()));
_stream.reset(new std::ostream(_buffer.get()));

// the implementation of _ZipBuffer
Zip::Source::_ZipBuffer::_ZipBuffer(zip_source_t* source) {
    _source = source;
}

std::streamsize Zip::Source::_ZipBuffer::xsputn(char const* s, std::streamsize n) {
    return zip_source_write(_source, s, n * sizeof(char));
}

int Zip::Source::_ZipBuffer::overflow(int c) {
    return zip_source_write(_source, &c, sizeof(int));
}

main.cpp

Zip::Source src;

src << "Some text and a number : " << 2.5 << std::endl;
src << "another line !";

// zip is an object of class Zip that takes my source and write it in the archive
zip.addFile("test.txt", src);

If I remove the std::endl in my main, the text file is recognized as a text file. If I add it, it it recognized as binary file.

The binary file is a valid utf-8 output (except for the NUL byte) :

496c 2065 7374 2070 6f73 7369 626c 6520
6427 c3a9 6372 6972 6520 6465 7320 6e6f
6d62 7265 7320 c3a0 2076 6972 6775 6c65
203a 2032 2e35 0a00 0000 736f 6d65 7468
696e 6720 656c 7365 

Thanks!

Answer 1

You implemented overflow() as follows:

int Zip::Source::_ZipBuffer::overflow(int c) {
   return zip_source_write(_source, &c, sizeof(int));
}

Your C++ library apparently implements std::endl, by calling overflow() with '\n' getting passed as a parameter.

This is perfectly compliant with C++ spec. Your implementation of overflow() has a bug.

The parameter to overflow() is a single character, passed as an int. Your implementation writes an entire binary int to the output file, which is exactly what you're seeing. Your sizeof(int) is, apparently, 4, so you see 0x0a and three more null bytes written to the output.

Answer 2

Ok, just found the problem...

In the std::streambuf::overflow overload I wrote, I'm writing an integer when I'm receiving a char. So the leading 0 in my int was written in the file.

I had to cast my int to char and the problem disappeared :)