Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Fastest way to read a csv file

I'm looking for a very fast method to read a csv file. My data structure looks like this:

timestamp ,float     , string    ,ip          ,string
1318190061,1640851625, lore ipsum,84.169.42.48,appname

and I'm using fgetcsv to read this data into arrays.

The problem: Performance. On a regular basis the script has to read (and process) more than 10,000 entries.

My first attempt is very simple:

//Performance: 0,141 seconds / 13.5 MB

while(!feof($statisticsfile)) 
    {
    $temp = fgetcsv($statisticsfile);
    $timestamp[] = $temp[0];
    $value[] = $temp[1];
    $text[] = $temp[2];
    $ip[] = $temp[3];
    $app[] = $temp[4];
    }

My second attempt:

//Performance: 0,125 seconds / 10.8 MB

while (($userinfo = fgetcsv($statisticsfile)) !== FALSE) {
   list ($timestamp[], $value[], $text, $ip, $app) = $userinfo;
}
  1. Is there any way to improve performance even further, or is my method as fast as it could get?
  2. Probably more important: Is there any way to define what columns are read, e.g. sometimes only the timestamp, float columns are needed. Is there any better way than my way (have a look at my second attempt :)

Thanks :)

like image 469
Nyoman Avatar asked Oct 09 '11 21:10

Nyoman


1 Answers

How long is the longest line? Pass that as the second parameter to fgetcsv() and you'll see the greatest improvement.

like image 110
Robin Millette Avatar answered Sep 28 '22 18:09

Robin Millette