Golang - How to overcome Scan() buffer limit from bufio?

Question

I am using the bufio scanner's .Scan() method to read text file lines. However when I reach a certain size in the file lines, the scanner no longer allows me to read, it just returns an empty line.

How should I configure the buffer to take in larger amounts of data?

bigfile.txt is just a file with many integers in a line separated by a space. For example, 40000 integers in one line. (note that it works for file lines of 10000 integers and less, but not 40000 for example)

234 544 765 45 34 67 67 87 98 43 [... n = 40000 ]

func main() {
    readInputFile("bigfile.txt")
}

func readInputFile(name string) {
    inFile, _ := os.Open(name)
    defer inFile.Close()

    scanner := bufio.NewScanner(inFile)

    for scanner.Scan() {
        line := scanner.Text()
        fmt.Printf(line)
    }
}

Answer 1

According to documentation and corresponding source code, by default Scanner uses internal buffer in which the capacity is 64K. In your case, the internal buffer is not sufficient for storing 40000 integers. Set the buffer to be used by scanner before calling Scan, i.e.

scanner := bufio.NewScanner(inFile)

//adjust the capacity to your need (max characters in line)
const maxCapacity = 512*1024  
buf := make([]byte, maxCapacity)
scanner.Buffer(buf, maxCapacity)