Node.js: remove specific columns from CSV file

Question

I have a CSV file can contain around million records, how can I remove columns starting with _ and generate a resulting csv

For the sake of simplicity, consider i have the below csv

Sr.No Col1 Col2 _Col3   Col4 _Col5
1     txt  png  676766  win  8787
2     jpg  pdf  565657  lin  8787
3     pdf  jpg  786786  lin  9898

---

I would want the output to be

Sr.No Col1 Col2 Col4
1     txt  png  win 
2     jpg  pdf  lin 
3     pdf  jpg  lin

Do i need to read the entire file to achive this or is there a better approach to do this.

const csv = require('csv-parser');
const fs = require('fs');

fs.createReadStream('data.csv')
  .pipe(csv())
  .on('data', (row) => {
    // generate a new csv with removing specific column
  })
  .on('end', () => {
    console.log('CSV file successfully processed');
  });

Any help on how can i achieve this would be helpful.

Thanks.

Answer 1

To anyone who stumbles on the post

I was able to transform the csv's using below code using fs and csv modules.

await fs.createReadStream(m.path)
      .pipe(csv.parse({delimiter: '\t', columns: true}))
      .pipe(csv.transform((input) => {
        delete input['_Col3'];
        console.log(input);
        return input;
      }))
      .pipe(csv.stringify({header: true}))
      .pipe(fs.createWriteStream(transformedPath))
      .on('finish', () => {
        console.log('finish....');
      }).on('error', () => {
        console.log('error.....');
      });

Source: https://gist.github.com/donmccurdy/6cbcd8cee74301f92b4400b376efda1d

Answer 2

Try this with csv lib

const csv = require('csv');
const fs = require('fs');

const csvString=`col1,col2
               value1,value2`

csv.parse(csvString, {columns: true})
   .pipe(csv.transform(({col1,col2}) => ({col1}))) // remove col2
   .pipe(csv.stringify({header:true}))
   .pipe(fs.createWriteStream('./file.csv'))