Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Remove a single row from a csv without copying files

Tags:

There are multiple SO questions addressing some form of this topic, but they all seem terribly inefficient for removing only a single row from a csv file (usually they involve copying the entire file). If I have a csv formatted like so:

fname,lname,age,sex John,Doe,28,m Sarah,Smith,27,f Xavier,Moore,19,m 

What is the most efficient way to remove Sarah's row? If possible, I would like to avoid copying the entire file.

like image 858
SamBG Avatar asked Apr 09 '18 12:04

SamBG


People also ask

How do I delete a row in CSV?

Method 3: Removing using Conditions Select required data. Find the row that specifies the specified condition. Use drop() method and pass the index of the fetched row as a parameter in the drop method.

How do I skip a row in CSV reader?

This way, we can ignore the header row from the csv while reading the data. Line 1: We import the Pandas library as a pd. Line 2: We read the csv file using the pandas read_csv module, and in that, we mentioned the skiprows=[0], which means skip the first line while reading the csv file data.

How do you delete a row in a DataSet?

When using a DataSet or DataTable in conjunction with a DataAdapter and a relational data source, use the Delete method of the DataRow to remove the row. The Delete method marks the row as Deleted in the DataSet or DataTable but does not remove it.


1 Answers

You have a fundamental problem here. No current filesystem (that I am aware of) provides a facility to remove a bunch of bytes from the middle of a file. You can overwrite existing bytes, or write a new file. So, your options are:

  • Create a copy of the file without the offending line, delete the old one, and rename the new file in place. (This is the option you want to avoid).
  • Overwrite the bytes of the line with something that will be ignored. Depending on exactly what is going to read the file, a comment character might work, or spaces might work (or possibly even \0). If you want to be completely generic though, this is not an option with CSV files, because there is no defined comment character.
  • As a last desperate measure, you could:
    • read up to the line you want to remove
    • read the rest of the file into memory
    • and overwrite the line and all subsequent lines with the data you want to keep.
    • truncate the file as the final position (filesystems usually allow this).

The last option obviously doesn't help much if you are trying to remove the first line (but it is handy if you want to remove a line near the end). It is also horribly vulnerable to crashing in the middle of the process.

like image 198
Martin Bonner supports Monica Avatar answered Oct 28 '22 05:10

Martin Bonner supports Monica