Multiple insert speed up

Question

i'm parsing large log files (5+GB) and extracting ad-hoc placed profiling lines (call name and execution time). I want to insert those lines into a MySql db.

My question is: should I execute the insert statement every time I get the line while parsing or there's some best practice to speed up everything?

Answer 1

If there is any way that you could do a bulk insert, that would help a lot (or at least send your data to the database in batches, instead of making separate calls each time).

Edit

LOAD DATA INFILE sounds even faster ;o)

https://web.archive.org/web/20150413042140/http://jeffrick.com/2010/03/23/bulk-insert-into-a-mysql-database