Logo Questions Linux Laravel Mysql Ubuntu Git Menu

directory monitoring





What is the best way for me to check for new files added to a directory, I dont think the filesystemwatcher would be suitable as this is not an always on service but a method that runs when my program starts up.

there are over 20,000 files in the folder structure I am monitoring, at present I am checking each file individually to see if the filepath is in my database table, however this is taking around ten minutes and I would like to speed it up is possible,

I can store the date the folder was last checked - is it easy to get all files with createddate > last checked date.

anyone got any Ideas?



like image 410
foz1284 Avatar asked May 08 '10 08:05


1 Answers

Your approach is the only feasible (i.e. file system watcher allows you to see changes, not check on start).

Find out what takes so long. 20.000 checks should not take 10 minutes - maybe 1 maximum. Your program is written slowly. How do you test it?

Hint: do not ask the database, get a list of all files into memory, a list of all filesi n the database, check in memory. 20.000 SQL statements to the database are too slow, this way you need ONE to get the list.

like image 118
TomTom Avatar answered Oct 05 '22 20:10