How to use Files.walk()... to get a graph of files based on conditions?

Question

I have the following directory structures:

/path/to/stuff/org/foo/bar/
/path/to/stuff/org/foo/bar/1.2.3/
/path/to/stuff/org/foo/bar/1.2.3/myfile.ext
/path/to/stuff/org/foo/bar/1.2.4/
/path/to/stuff/org/foo/bar/1.2.4/myfile.ext
/path/to/stuff/org/foo/bar/blah/
/path/to/stuff/org/foo/bar/blah/2.1/
/path/to/stuff/org/foo/bar/blah/2.1/myfile.ext
/path/to/stuff/org/foo/bar/blah/2.2/
/path/to/stuff/org/foo/bar/blah/2.2/myfile.ext

I would like to get the following output:

/path/to/stuff/org/foo/bar/
/path/to/stuff/org/foo/bar/blah/

I have the following code (below), which is inefficient, as it prints out:

/path/to/stuff/org/foo/bar/
/path/to/stuff/org/foo/bar/
/path/to/stuff/org/foo/bar/blah/
/path/to/stuff/org/foo/bar/blah/

Here is the Java code:

public class LocatorTest
{

    @Test
    public void testLocateDirectories()
            throws IOException
    {
        long startTime = System.currentTimeMillis();

        Files.walk(Paths.get("/path/to/stuff/"))
             .filter(Files::isDirectory)
             .forEach(Foo::printIfArtifactVersionDirectory);

        long endTime = System.currentTimeMillis();

        System.out.println("Executed in " + (endTime - startTime) + " ms.");
    }

    static class Foo
    {

        static void printIfArtifactVersionDirectory(Path path)
        {
            File f = path.toAbsolutePath().toFile();
            List<String> filePaths = Arrays.asList(f.list(new MyExtFilenameFilter()));

            if (!filePaths.isEmpty())
            {
                System.out.println(path.getParent());
            }
        }

    }

}

The filter:

public class MyExtFilenameFilter
        implements FilenameFilter
{

    @Override
    public boolean accept(File dir, String name)
    {
        return name.endsWith(".ext");
    }

}

Answer 1

Files.walk(Paths.get("/path/to/stuff/"))
     .filter(p -> p.toString().endsWith(".ext"))
     .map(p -> p.getParent().getParent())
     .distinct()
     .forEach(System.out::println);

This filters all files that have the extension and gets the parent path of their directory. distinct ensures that every path is used only once.

Answer 2

You are invoking the method printIfArtifactVersionDirectory for all visited directories. I did a little change to make it obvious:

static void printIfArtifactVersionDirectory(Path path) {
    System.out.println("--- " + path);
    ...
}

With that additional output you will get:

--- C:\Projects\stuff
--- C:\Projects\stuff\org
--- C:\Projects\stuff\org\foo
--- C:\Projects\stuff\org\foo\bar
--- C:\Projects\stuff\org\foo\bar\1.2.3
C:\Projects\stuff\org\foo\bar
--- C:\Projects\stuff\org\foo\bar\1.2.4
C:\Projects\stuff\org\foo\bar
--- C:\Projects\stuff\org\foo\bar\blah
--- C:\Projects\stuff\org\foo\bar\blah\2.1
C:\Projects\stuff\org\foo\bar\blah
--- C:\Projects\stuff\org\foo\bar\blah\2.2
C:\Projects\stuff\org\foo\bar\blah

So you get the output as often as you have artifact version directories. If you want to remember that you already did the output for one directory, you must make store this information somewhere. One quick implementation could be:

static class Foo {
    private static final Set<Path> visited = new HashSet<>();

    static void printIfArtifactVersionDirectory(Path path) {
        ...
        Path parent = path.getParent();
        if (!filePaths.isEmpty() && !visited.contains(parent)) {
            visited.add(parent);
            System.out.println(parent);
        }
    }
}

With this you get the expected output:

C:\Projects\stuff\org\foo\bar
C:\Projects\stuff\org\foo\bar\blah

A better solution would be to use the set for storing the visited parents and only print them after visiting them all:

static class PathStore {
    private final Set<Path> store = new HashSet<>();

    void visit(Path path) {
        File f = path.toAbsolutePath().toFile();
        List<String> filePaths = Arrays.asList(f.list(new MyExtFilenameFilter()));
        if (!filePaths.isEmpty()) {
            store.add(path.getParent());
        }
    }

    void print() {
        store.forEach(System.out::println);
    }
}

Usage:

PathStore pathStore = new PathStore();
Files.walk(Paths.get("/path/to/stuff/"))
        .filter(Files::isDirectory)
        .forEach(pathStore::visit);
pathStore.print();

Answer 3

Adding to @a-better-oliver's answer, here is what you can do if your action declares IOExceptions.

You can treat the filtered stream as an Iterable, and then do your action in a regular for-each loop. This way, you don't have to handle exceptions inside a lambda.

try (Stream<Path> pathStream = Files
        .walk(Paths.get("/path/to/stuff/"))
        .filter(p -> p.toString().endsWith(".ext"))
        .map(p -> p.getParent().getParent())
        .distinct()) {

    for (Path file : (Iterable<Path>) pathStream::iterator) {
        // something that throws IOException
        Files.copy(file, System.out);
    }
}

Found that trick here: https://stackoverflow.com/a/32668807/1207791