Menu
I have a very large file and i need to process each line (each line of file is independent). How can I use goroutines (or should I not use them?) to read the file in the fastest way?
212
The simplest way of reading a text or binary file in Go is to use the ReadFile () function from the os package. This function reads the entire content of the file into a byte slice, so you should be careful when trying to read a large file - in this case, you should read the file line by line or in chunks.
Read the entire file in GoLang In Go, Reading an entire file content/text is to use ReadFile () function from the ioutil/os package. This function reads the entire content of the file into a byte slice. The ioutil package should not be used in reading a large file thus the function is quite sufficient for small files.
First, we need to create an instance with the NewScanner function which is a very familiar way for constructing the structs in golang. this function accepts a Reader interface as input, and the good news is os.File implemented this interface. It means, we can open a file and pass the pointer of the file to bufio.NewScanner. Let’s see it in action.
In Go, the ReadFile () function of the ioutil library is used to read data from a file. Using ioutil makes file reading easier as you don’t have to worry about closing files or using buffers. 1. Reading an entire file into memory To read an entire file into a variable, use the ReadFile function from the ioutil library.
As long as your hard disk is orders of magnitude slower than your CPU, which is still a quite common situation, then you cannot magically make the file reading (domain: from a single HD) any faster by throwing more CPU cycles onto it. (Assuming cold file caches and/or file size much bigger then all available file cache memory).
180
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
