Unity: Reading and updating from large CSV file

Question

I have a very large CSV file (1 million+ rows) with four columns of data time, id, x and y. Here is a sample:

t   id  x   y
434 84  0   0
435 84  28.22   -4.5
435 611 1895.13 755.17
435 872 2401.08 159.12
435 65  0   226.39
436 84  50.44   -4.5
436 611 1890.63 732.5
436 872 2373.9  151.04
436 990 2614.97 372.74
...

In my simulation, as time elapses I need to do one of three things:

1. If it's the first time an id has appeared, create an object with that id at the x,y coordinates

2. If an object with an id already exists update that object's x,y coordinates

3. If an id does not appear anymore delete that object

I'm guessing it's very intensive to keep a running timer, check the CSV every second, locate all rows with the current time and execute one of the above steps. Is there a more efficient way of dealing with time-series data in Unity simulations?

Answer 1

With files that large, you should start looking for alternatives. Here's some ideas, but the best option depends on what specifically you're doing.

• Can the component updating this CSV instead communicate directly with Unity3d? Using, e.g. a socket connection would avoid the need to save and read this information to and from the disk constantly. However, this depends on how this CSV data is being created, obviously
• Could you split the csv into smaller files? One for each timestamp, for instance? This way, there's less overhead to update the simulation at each step.
• Can you reduce the frequency of your updates? Is it essential to update every second?
• Or, can you only read from the CSV every, say, 10 seconds, load in all the data for the next 10 seconds (all timestamps in that range, for instance), store them in memory, and then for the next 10 seconds only use the info from memory to update instead of reading the file again? This will reduce your calls to the disk.