Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

NodeJS: reading a big csv file

So I have a 70mb .csv file that I wanna parse and convert into a json, trying to do the json convert in a 500kb test csv I found an easy solution with regex.
The problem was when I put my actual data, I can't use fs.readFileSync anymore, so I need to work with streams.
My problem is: how can I work with streams and regex? Supose that my stream cuts the buffer in the middle of a possible regex match, I think if that happens I will lose that data. Also, the data isn't structured so I don't realise other way to parse it than regexs.
Please let me know if I wasn't clear with my explanation, english isn't my main language but also I know the english community is the biggest also the fastest and more reliable.

Thanks in advance.

like image 489
Lucas Janon Avatar asked Feb 13 '26 22:02

Lucas Janon


1 Answers

there is a stable readline core module

and you can do this

let lineReader = require('readline').createInterface({
  input: require('fs').createReadStream('file.csv')
})

lineReader.on('line', (line) => {
  // do regexs with line
})
like image 90
raksa Avatar answered Feb 15 '26 21:02

raksa



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!