Is there a max file size hard limit for a .csv file?

Question

Somehow one of our batches failed last night, the file size that would have been generated is 500mb with around 500,000 records (rows) in it.

Just wonderingly, is there any file size hard limit for .csv files?

I understand that Excel application has 60k row hard limit, but thats opening.

Answer 1

The maximum file size of any file on a filesystem is determined by the filesystem itself - not by the file type or filename suffix. So the answer is no.

But, as you said, the application you are using to process the file might have limitations.

Answer 2

Depending on the way you are handling the CSV file, you might run into memory and/or CPU bottlenecks with large files.

Also, on 32 bit Linux systems there exists a 2GB file size limit which will limit your maximum usable CSV size regardless of your CPU power and memory. Having said that, CSV files that big are a good sign that you should consider a more efficient and robust solution for handling your data such as a database system.