C# GetFiles with Date Filter

Question

Is there a more efficient way to populate a list of file names from a directory with a date filter?

Currently, I'm doing this:

foreach (FileInfo flInfo in directory.GetFiles())
{
    DateTime yesterday = DateTime.Today.AddDays(-1);
    String name = flInfo.Name.Substring(3,4);
    DateTime creationTime = flInfo.CreationTime;
    if (creationTime.Date == yesterday.Date)
       yesterdaysList.Add(name);
}

This goes through every file in the folder, and I feel like there should be a more efficient way.

Answer 1

First Solution:

You can use LINQ:

List<string> yesterdaysList = directory.GetFiles().Where(x => x.CreationTime.Date == DateTime.Today.AddDays(-1))
                                                  .Select(x => x.Name)
                                                  .ToList();

Then you can use directly this list of names.

Second Solution:

Another solution to make it faster could be:

DateTime yesterday = DateTime.Today.AddDays(-1); //initialize this variable only one time

foreach (FileInfo flInfo in directory.GetFiles()){
    if (flInfo.CreationTime.Date == yesterday.Date) //use directly flInfo.CreationTime and flInfo.Name without create another variable 
       yesterdaysList.Add(flInfo.Name.Substring(3,4));
}

Benchmark:

I did a benchmark by using this code:

class Program {
    static void Main( string[ ] args ) {
        DirectoryInfo directory = new DirectoryInfo( @"D:\Films" );
        Stopwatch timer = new Stopwatch( );
        timer.Start( );

        for ( int i = 0; i < 100000; i++ ) {
            List<string> yesterdaysList = directory.GetFiles( ).Where( x => x.CreationTime.Date == DateTime.Today.AddDays( -1 ) )
                                              .Select( x => x.Name )
                                              .ToList( );
        }

        timer.Stop( );
        TimeSpan elapsedtime = timer.Elapsed;
        Console.WriteLine( string.Format( "{0:00}:{1:00}:{2:00}", elapsedtime.Minutes, elapsedtime.Seconds, elapsedtime.Milliseconds / 10 ) );
        timer.Restart( );

        DateTime yesterday = DateTime.Today.AddDays( -1 ); //initialize this variable only one time
        for ( int i = 0; i < 100000; i++ ) {
            List<string> yesterdaysList = new List<string>( );

            foreach ( FileInfo flInfo in directory.GetFiles( ) ) {
                if ( flInfo.CreationTime.Date == yesterday.Date ) //use directly flInfo.CreationTime and flInfo.Name without create another variable 
                    yesterdaysList.Add( flInfo.Name.Substring( 3, 4 ) );
            }
        }


        timer.Stop( );
        elapsedtime = timer.Elapsed;
        Console.WriteLine( string.Format("{0:00}:{1:00}:{2:00}", elapsedtime.Minutes, elapsedtime.Seconds, elapsedtime.Milliseconds / 10));
        timer.Restart( );

        for ( int i = 0; i < 100000; i++ ) {
            List<string> list = new List<string>( );

            foreach ( FileInfo flInfo in directory.GetFiles( ) ) {
                DateTime _yesterday = DateTime.Today.AddDays( -1 );
                String name = flInfo.Name.Substring( 3, 4 );
                DateTime creationTime = flInfo.CreationTime;
                if ( creationTime.Date == _yesterday.Date )
                    list.Add( name );
            }
        }

        elapsedtime = timer.Elapsed;
        Console.WriteLine( string.Format( "{0:00}:{1:00}:{2:00}", elapsedtime.Minutes, elapsedtime.Seconds, elapsedtime.Milliseconds / 10 ) );
    }
}

Results:

First solution: 00:19:84
Second solution: 00:17:64
Third solution: 00:19:91 //Your solution

Answer 2

I didn't feel like creating enough files with the correct creation date to do a decent benchmark, so I did a more general version that takes a start and end time and gives out the names of files that match. Making it give a particular substring of files created yesterday follows naturally from that.

The quickest single-threaded pure .NET answer I came up with was:

private static IEnumerable<string> FilesWithinDates(string directory, DateTime minCreated, DateTime maxCreated)
{
    foreach(FileInfo fi in new DirectoryInfo(directory).GetFiles())
        if(fi.CreationTime >= minCreated && fi.CreationTime <= maxCreated)
            yield return fi.Name;
}

I would have expected EnumerateFiles() to be slightly faster, but it turned out slightly slower (might do better if you're going over a network, but I didn't test that).

There's a slight gain with:

private static ParallelQuery<string> FilesWithinDates(string directory, DateTime minCreated, DateTime maxCreated)
{
    return new DirectoryInfo(directory).GetFiles().AsParallel()
        .Where(fi => fi.CreationTime >= minCreated && fi.CreationTime <= maxCreated)
        .Select(fi => fi.Name);
}

But not much since it doesn't help the actual call to GetFiles(). If you don't have the cores to use, or there isn't a big enough result from GetFiles() then it'll just make things worse (the overheads of AsParallel() being greater than the benefit of doing the filtering in parallel). On the other hand, if you can do your next steps of processing also in parallel, then the overall application speed could improve.

There seems to be no point doing this with EnumerateFiles() because it doesn't seem to parallelise well, because it's based on the same approach I'm coming to last, and that's inherently serial - needing the previous result to produce the next.

The fastest I got was:

public const int MAX_PATH = 260;
public const int MAX_ALTERNATE = 14;

[StructLayoutAttribute(LayoutKind.Sequential)]
public struct FILETIME
{
    public uint dwLowDateTime;
    public uint dwHighDateTime;
    public static implicit operator long(FILETIME ft)
    {
        return (((long)ft.dwHighDateTime) << 32) | ft.dwLowDateTime;
    }
};

[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)]
public struct WIN32_FIND_DATA
{
    public FileAttributes dwFileAttributes;
    public FILETIME ftCreationTime;
    public FILETIME ftLastAccessTime;
    public FILETIME ftLastWriteTime;
    public uint nFileSizeHigh;
    public uint nFileSizeLow;
    public uint dwReserved0;
    public uint dwReserved1;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst=MAX_PATH)]
    public string cFileName;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst=MAX_ALTERNATE)]
    public string cAlternate;
}

[DllImport("kernel32", CharSet=CharSet.Unicode)]
public static extern IntPtr FindFirstFile(string lpFileName, out WIN32_FIND_DATA lpFindFileData);

[DllImport("kernel32", CharSet=CharSet.Unicode)]
public static extern bool FindNextFile(IntPtr hFindFile, out WIN32_FIND_DATA lpFindFileData);

[DllImport("kernel32.dll")]
public static extern bool FindClose(IntPtr hFindFile);

private static IEnumerable<string> FilesWithinDates(string directory, DateTime minCreated, DateTime maxCreated)
{
    long startFrom = minCreated.ToFileTimeUtc();
    long endAt = maxCreated.ToFileTimeUtc();
    WIN32_FIND_DATA findData;
    IntPtr findHandle = FindFirstFile(@"\\?\" + directory + @"\*", out findData);
    if(findHandle != new IntPtr(-1))
    {
        do
        {
            if(
                (findData.dwFileAttributes & FileAttributes.Directory) == 0
                &&
                findData.ftCreationTime >= startFrom
                &&
                findData.ftCreationTime <= endAt
            )
            {
                yield return findData.cFileName;
            }
        }
        while(FindNextFile(findHandle, out findData));
        FindClose(findHandle);
    }
}

It's dicey not having that FindClose() promised by an IDisposable, and a hand-rolled implementation of IEnumerator<string> should not only make that easier to do (serious reason for doing it) but also hopefully shave off like 3 nanoseconds or something (not a serious reason for doing it), but the above shows the basic idea.

Answer 3

I think you are after getting more efficiency at the file system level, not at the C# level. If that is the case the answer is no: There is no way to tell the file system to filter by date. It will needlessly return everything.

If you are after CPU efficiency: This is pointless becauseadding items to a listbox is so incredibly more expensive than filtering on date. Optimizing your code will yield no results.