Why are null bytes appearing? Even after "sanitizing" the stream

Question

I've been tying to figure why null bytes are appearing in certain strings. Example below.

{"gender":"fema\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000le"}

I essentially I wrap an io.Reader from an HTTP request and decode into a struct. See below

func bodyToStruct(res *http.Request, v gojay.UnmarshalerJSONObject) error {
    var reader io.ReadCloser
    var err error
    switch res.Header.Get("Content-Encoding") {
    case "gzip":
        reader, err = pool.Gzip.GetReader(res.Body)
        if err != nil {
            return err
        }
        defer pool.Gzip.PutReader(reader)
    case "deflate":
        reader = flate.NewReader(res.Body)
        defer reader.Close()
    default:
        reader = res.Body
    }

    decoder := gojay.BorrowDecoder(streams.NewNullByteRemoverStream(reader)) //wrapped in NewNullByteRemoverStream
    defer decoder.Release()

    return decoder.DecodeObject(v)
}

I've tried numerous ways to try and remove the null bytes, I assume they are coming in the request from Android clients.

From help on an earlier stack thread, I was able to deploy the below implementation into production in an attempt too remove the null bytes.

package streams

import (
    "io"
)

// NullByte is a stream wrapper that should remove null bytes from the byte stream as well as reject any and all control bytes
type NullByte struct {
    Reader io.Reader
}

// NewNullByteRemoverStream creates a new NullByte reader which passes passes the parent stream through and remove null bytes
func NewNullByteRemoverStream(reader io.ReadCloser) *NullByte {
    return &NullByte{
        Reader: reader,
    }
}

func (s *NullByte) Read(p []byte) (n int, err error) {
    n, err = s.Reader.Read(p)
    var nn int
    for i := 0; i < n; i++ {
        if p[i] >= 32 && p[i] <= 126 {
            p[nn] = p[i]
            nn++
        } 
    }
    return nn, err
}

I even went as far to attempt to remove the string literal of \u0000 as seen here (also tested in production for a bit)

package streams

import (
    "io"
)

const _unicodeCodePointLength = 6

var (
    _sControlByte   = byte(92)
    _sNullByteBlock = []byte{92, 117, 48, 48, 48, 48}
)

// NullByte is a stream wrapper that should remove null bytes from the byte stream as well as reject any and all control bytes
type NullByte struct {
    Reader io.Reader
    state  int
}

// NewNullByteRemoverStream creates a new NullByte reader which passes passes the parent stream through and remove null bytes
// as well as \u0000 as a string representation
func NewNullByteRemoverStream(reader io.ReadCloser) *NullByte {
    return &NullByte{
        Reader: reader,
    }
}

func (s *NullByte) Read(p []byte) (n int, err error) {
    n, err = s.Reader.Read(p)

    var nn, i int
    for i < n {
        if p[i] == _sControlByte {
            s.state = 0
        }

        if p[i] == _sControlByte || s.state > 0 {
            var broke bool
            if p[i] == _sControlByte {
                stop := 0
                for j := i; j < n; j++ {
                    if stop == _unicodeCodePointLength {
                        break
                    }
                    if p[j] != _sNullByteBlock[stop] {
                        broke = true
                        break
                    }
                    stop++
                }

                if broke {
                    p[nn] = p[i]
                    i++
                    nn++
                    s.state = 0
                    continue
                }
            }

            if s.state < _unicodeCodePointLength {
                i++
                s.state++
                continue
            }
        }

        if p[i] != 0 {
            p[nn] = p[i]
            nn++
        }
        i++
    }
    return nn, err
}

unfortunately both versions do not stop the issue. I can see in production logs \u0000 appearing in a percentage of logs. I thought that by wrapping the io.Reader responses in the Sanitizers above that the issue would stop. I can see from tests that null bytes 0 and \u0000 get removed... but the issues persists in production. I suspect that the issue is with the request from the clients still. This is because the issue only appears with a particular client version. Other app versions and platforms aren't triggering null bytes to appear in the strings and all clients communicate to the same centralized servers. I'm out of ideas. I have no idea why the sanitizers above don't remove the null bytes before the JSON decoder loads the data into the strut. Does anyone have any insights?

Answer 1

Edit: this is incorrect, even though it may have coincidentally solved the problem. Null bytes should be removed whether a buffer is used or not.

Hard to say why the null bytes are appearing. But the issue with the stream readers not dropping nulls might be because they lack their own buffer. Here's an example of a null dropping reader with it's own buffer (playground):

type DropReader struct {
    buf    []byte
    reader io.Reader
    nulls  int
    reads  int
}

func (dr *DropReader) Read(data []byte) (int, error) {

    n, err := dr.reader.Read(dr.buf)

    dr.reads++

    j := 0
    for i := 0; i < n; i++ {

        c := dr.buf[i]

        if c == 0 {
            dr.nulls++
            continue
        }

        data[j] = c
        j++

    }

    return j, err
}