Java File IO vs Local database

Question

I am working on a project that involves parsing through a LARGE amount of data rapidly. Currently this data is on disk and broken down into a directory hierarchy:

(Folder: DataSource) -> (Files: Day1, Day2, Day3...Day1000...)
(Folder: DataSource2) -> (Files: Day1, Day2, Day3...Day1000...) 
...
(Folder: DataSource1000) -> ...
...

Each Day file consists of entries that need to be accessed very quickly.

My initial plans were to use traditional FileIO in java to access these files, but upon further reading, I began to fear that this might be too slow.

In short, what is the fastest way I can selectively load entries from my filesystem from varying DataSources and Days?

Answer 1

The issue could be solved both ways but it depends on few factors

go for FileIO.

1. if the volume is < millons of rows
2. if your dont do a complicated query like Jon Skeet said
3. if your referance for fetching the row is by using hte Folder Name: "DataSource" as the key

go for DB

1. if you see your program reading through millions of records
2. you can do complicated selection, even multiple rows using a single select.
3. if you have knowledge of creating a basic table structure for DB