Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Remove a line of a csv or ignore it

I'm using fast-csv to read csv files but the format of my files aren't exactly as expected for fast-csv:

First line with some details (not important)
Second line with header
Third line with data
Fourth line with data
...

And how i read it:

const csv = require('fast-csv');
const stream = fs.createReadStream('myfile.csv');
const csvStream = csv
    .fromStream(stream,
        {
            headers: true,
            delimiter: ',',
            rowDelimiter: '\n',
            quoteHeaders: false,
            quoteColumns: false
        })
    .on("data", data => {
       // i do something with datas

    })
    .on("error", error => {
        console.log("CSV is invalid !", error);
    })
    .on("end", data => {
        console.log("End of parsing");
        console.log(data);
    });

The problem here is

headers: true,

will looks for first line as headers that's why I want to remove first line or ignore it. How to do that ? (I know I can read the file and write the file without the first line but I don't think that's the proper way to do that...)

Versions: node v10.0.0 & fast-csv 5.6.0


Edit: Test with @Anders Carstensen example:

const file = 'myfile.csv';
const fs = require("fs");
const stream = fs.createReadStream(file, {
    encoding: 'utf8'
});

stream.on('readable', () => {
    // Read through the stream until we encounter a new line
    let chunk;
    while (null !== (chunk = stream.read(1))) {

        if (chunk == '\n'){
            console.log('line break');
            break;
        }
    }
    console.log('test');


    // CSV parsing
    const csvStream = csv.fromStream(stream,
    {
        renameHeaders: false,
        headers: true,
        delimiter: ',',
        rowDelimiter: '\n',
        quoteHeaders: false,
        quoteColumns: false
    })
    .on("data", data => {
        console.log(data);

    })
    .on("error", error => {
        console.log("CSV invalid !", error);
    })
    .on("end", data => {
        console.log("End of parsing");
        console.log(data);
    });
});

Output:

line break
test
line break
test

Edit2: Check @Anders Carstensen answer for the solution (problem was using .on and not .once)

like image 464
Emerica Avatar asked Jul 23 '18 09:07

Emerica


People also ask

How do I remove a new line character from a CSV file?

Press and hold the Alt key and then enter “010” from your keyboard's 10-keypad part. (Note: The numbers in the top row of the keyboard won't work.) If '010' doesn't work try '013' because the data imported from a different source might have line breaks represented by a different code.

How do I ignore headers when reading CSV?

To read CSV file without header, use the header parameter and set it to “None” in the read_csv() method.


1 Answers

First, you should chew through the part of the stream you don't want the parser to look at. Here I read until I have encountered the first new line (\n).

var fs = require('fs'),
    csv = require('fast-csv');

var stream = fs.createReadStream('c:\\temp\\test.csv', {
  encoding: 'utf8'
});

stream.once('readable', function () {
  // Read through the stream until we encounter a new line
  var chunk;
  while (null !== (chunk = stream.read(1))) {
    if (chunk === '\n')
      break;
  }

  // Then do the CSV parsing
  const csvStream = csv
    .fromStream(stream,
      {
        headers: true,
        delimiter: ',',
        rowDelimiter: '\n',
        quoteHeaders: false,
        quoteColumns: false
      })
    .on("data", data => {
      // i do something with datas
      console.log('data', data);
    })
    .on("data-invalid", data => {
      console.log('invalid data', data);
    })
    .on("error", error => {
      console.log("Le fichier CSV est invalide !", error);
    })
    .on("end", data => {
      console.log("End of parsing");
      console.log(data);
    });
});
like image 143
Anders Carstensen Avatar answered Oct 31 '22 22:10

Anders Carstensen