55 gig - comma delimited file - reading into mysql DB

Question

any ideas on how to open this file?

i have no choice whatsoever on the size of this file.

its regular text data

the reason i cannot break up the file is because it is stored remotely and the ONLY thing i can do is run sql statements on it

please help!

Answer 1

for windows there is a really old application called "thegun" that opens any file with any size. under linux/mac you have less/head/tail and you can even use "dd".

Answer 2

What protocols do you have available for opening the file?

A 55G file doesn't sound too bad, can you just download it and handle it locally?

Clearly it would be better if the data providers made more smaller files; a very large file is difficult to handle.

I might suggest using rsync (if you can) as that will enable resuming if the transfer fails (it's going to take a couple of hours I guess, if it's from a remote site).

How often do these files get generated? Can you get the party generating them to create deltas instead? That would probably be a better way to handle it (assuming the data aren't entirely new each time)