Read lines from a file with variable line endings in Go

Question

How can I read lines from a file where the line endings are carriage return (CR), newline (NL), or both?

The PDF specification allows lines to end with CR, LF, or CRLF.

• bufio.Reader.ReadString() and bufio.Reader.ReadBytes() allow a single delimiter byte.

• bufio.Scanner.Scan() handles \n optionally preceded by \r, but not a lone \r.

  The end-of-line marker is one optional carriage return followed by one mandatory newline.

Do I need to write my own function that uses bufio.Reader.ReadByte()?

Answer 1

You can write custom bufio.SplitFunc for bufio.Scanner. E.g:

// Mostly bufio.ScanLines code:
func ScanPDFLines(data []byte, atEOF bool) (advance int, token []byte, err error) {
    if atEOF && len(data) == 0 {
        return 0, nil, nil
    }
    if i := bytes.IndexAny(data, "\r\n"); i >= 0 {
        if data[i] == '\n' {
            // We have a line terminated by single newline.
            return i + 1, data[0:i], nil
        }
        advance = i + 1
        if len(data) > i+1 && data[i+1] == '\n' {
            advance += 1
        }
        return advance, data[0:i], nil
    }
    // If we're at EOF, we have a final, non-terminated line. Return it.
    if atEOF {
        return len(data), data, nil
    }
    // Request more data.
    return 0, nil, nil
}

And use it like:

scan := bufio.NewScanner(r)
scan.Split(ScanPDFLines)