How to use boost::fs to only load 30 newest files and not the entire directory?

Question

New to using boost. Using it to load a collection of images. The issue is that the images will continue to grow in number in the folder and I will eventually not want to add all of them to my display program. I am on OS X and using C++.

How can I adjust this example code to only load say only 30 images from the top or bottom of the directory? Loading only the newest files would be awesome, but I would settle for just changing this. Unfortunately just saying (it <30) in my loop doesn't work because it needs to be equivalent to fs::directory_iterator.

Example code:

fs::path pPhoto( photobooth_texture_path );
for ( fs::directory_iterator it( pPhoto ); it != fs::directory_iterator(); ++it )
{
    if ( fs::is_regular_file( *it ) )
    {
        // -- Perhaps there is a better way to ignore hidden files
        string photoFileName = it->path().filename().string();
        if( !( photoFileName.compare( ".DS_Store" ) == 0 ) )
        {
            photoboothTex.push_back( gl::Texture( loadImage( photobooth_texture_path + photoFileName ), mipFmt) );
            cout << "Loaded: " << photoFileName <<endl;
        }
    }
}

EDIT: This is how I ended up doing it. Sort of a hybrid of the two methods, but i needed to sort backwards, even though it won't necessarily be predictably going backwards...taking my chances. Not the cleanest thing in the world, but i had to translate their ideas to a flavor of C++ I understood

    vector<string> fileList;
int count = 0;

photoboothTex.clear();//clear this out to make way for new photos

fs::path pPhoto( photobooth_texture_path );
for ( fs::directory_iterator it( pPhoto ); it != fs::directory_iterator(); ++it )    {
    if ( fs::is_regular_file( *it ) )
    {
        // -- Perhaps there is a better way to ignore hidden files
        string photoFileName = it->path().filename().string();

        if( !( photoFileName.compare( ".DS_Store" ) == 0 ) )
        {
            fileList.push_back(photoFileName);
        }
    }
}
for (int i=(fileList.size()-1); i!=0; i--) {

        photoboothTex.push_back( gl::Texture( loadImage( photobooth_texture_path + fileList[i%fileList.size()] )) );
        cout << "Loaded Photobooth: " << fileList[i%fileList.size()] <<endl;
        if(++count ==40) break; //loads a maximum of 40 images
}

Answer 1

Here's a working example that uses boost::filter_iterator with directory_iterator to store paths to regular files in a vector. I sorted the vector based on last_write_time(). I also ommited error checking for brevity - this example will crash if there's less than 30 files in the directory.

#include <iostream>
#include <vector>
#include <algorithm>
#include <iterator>
#include <boost/filesystem.hpp>
#include <boost/iterator/filter_iterator.hpp>
namespace fs = boost::filesystem;

int main()
{
    fs::path p("My image directory");
    fs::directory_iterator dir_first(p), dir_last;
    std::vector<fs::path> files;

    auto pred = [](const fs::directory_entry& p)
    {
        return fs::is_regular_file(p);
    };

    std::copy(boost::make_filter_iterator(pred, dir_first, dir_last),
              boost::make_filter_iterator(pred, dir_last, dir_last),
              std::back_inserter(files)
              );

    std::sort(files.begin(), files.end(),
              [](const fs::path& p1, const fs::path& p2)
              {
                  return fs::last_write_time(p1) < fs::last_write_time(p2);
              });

    std::copy_n(files.begin(), 30, std::ostream_iterator<fs::path>(std::cout, "\n"));
}

To make your example work, you could structure the for-loop like this:

fs::path pPhoto( photobooth_texture_path );
fs::directory_iterator it( pPhoto );

for ( size_t i = 0;  i < 30 && it != fs::directory_iterator(); ++it )
{
    if ( fs::is_regular_file( *it ) )
    {
        // load the image
        ++i;
    }
}

Answer 2

Obviously you can't just say it < 30 because 30 isn't a directory_iterator.

And, even if you could, that would only count the first 30 files period, not the first 30 non-hidden files, which I suspect isn't what you want (especially since the usual *nix rule for "hidden" is "starts with '.'", and those files tend to come first).

But you can easily keep track of the count yourself:

int count = 0;

fs::path pPhoto( photobooth_texture_path );
for ( fs::directory_iterator it( pPhoto ); it != fs::directory_iterator(); ++it )
{
    if ( fs::is_regular_file( *it ) )
    {
        // -- Perhaps there is a better way to ignore hidden files
        string photoFileName = it->path().filename().string();
        if( !( photoFileName.compare( ".DS_Store" ) == 0 ) )
        {
            photoboothTex.push_back( gl::Texture( loadImage( photobooth_texture_path + photoFileName ), mipFmt) );
            cout << "Loaded: " << photoFileName <<endl;
            if (++count == 30) break;
        }
    }
}

That's it.

Loading only the newest files would be awesome, but I would settle for just changing this.

This does not get the newest 30, just "some 30". boost::filesystem iterates "as if by calling POSIX readdir_r()", and readdir_r iterates over a directory stream which is specified as "an ordered sequence of all the directory entries in a particular directory", but there's no way to tell it what ordering you want for that sequence.

Of course you can add ordering yourself by reading in the whole list, then sorting however you want. See jrok's answer above for that. But there are some down-sides to this:

• It's not as simple.
• It's going to be much slower if you have large directories (because you have to read in, and possibly stat, all 3000 entries to sort them, instead of just reading 30).
• It's going to take much more memory.

Ultimately, it's a tradeoff.

While it's not as simple, someone (jrok) has already written the code, and understanding his code is a worthwhile learning experience anyway. While it's "much slower", it may still be "more than fast enough". While it takes "much more memory", it's likely still just a drop in the bucket. But you have to evaluate those factors and decide for yourself.

I'll mention two other quick things:

First, if speed is not an issue but memory is (very unlikely, but not completely impossible), you can make the code a bit more complex by just keeping the last 30 files found so far, instead of all of them. (For example, stick them in a set instead of a vector; for each new value, if it's older than the oldest value in the set, ignore it; otherwise, insert it into the set and remove the oldest value.)

Second, if you don't care about portability, and are willing to trade from boost::filesystem to some ugly, platform-specific, C-based API, your platform might have a way to read directory entries in sorted order. But I wouldn't pursue this unless you really do need both ordering and efficiency, so much that you're willing to completely sacrifice portability and simplicity.