Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Which one is best CSV or JSON in order to import big data (PHP) [closed]

I'm trying to get a big number of data (about 3M rows) and I have only two options to do that.

  1. Call an API then recover the 3M JSON objects.
  2. Import a CSV file containing the 3M rows.

I didn't test any of these solutions yet to tell which one is best in terms of speed.

like image 872
teeyo Avatar asked Oct 02 '14 08:10

teeyo


People also ask

Which is faster to read JSON or CSV?

CSV should generally be the fastest to write, JSON the easiest for a human to understand and Parquet the fastest to read. CSV is the defacto standard of a lot of data and for fair reasons; it's [relatively] easy to comprehend for both users and computers and made more accessible via Microsoft Excel.

Are JSON files larger than CSV?

1 Answer. Show activity on this post. This is for a simple reason that the JSON has a lot more than the CSV. If you look at the generated JSON you will see that it has a lot of keys like field1, field2 etc.


1 Answers

If you want to retrieve simple data as lists or rows with some columns the option #2 is the good one, you can read below a set of advantages and disadvantages:

Pros

  • Less bandwidth needed because JSON needs more syntax characters to keep the format while CSV is as simple as use a character separator
  • Process data is faster because only needs to split by the separator character while JSON needs to interpret the syntax
  • Big data technology as Hadoop has an integrated parse for CSV format while needs a specific function for parse JSON (for example using Hive language).

Cons

  • Unstructured data and more difficult to be read by humans
  • You have to take care as separator character cannot appear in data fields.

If the data will contain complex data as tuples, arrays and structures JSON are better because:

  • Keeps a clear and structured format
  • Doesn't repeat data to reference it because one label could contain multiple data.
like image 192
Miguel Avatar answered Sep 20 '22 17:09

Miguel