How to retrieve a recursive directory and file list from PowerShell excluding some files and folders?

Question

I want to write a PowerShell script that will recursively search a directory, but exclude specified files (for example, *.log, and myFile.txt), and also exclude specified directories, and their contents (for example, myDir and all files and folders below myDir).

I have been working with the Get-ChildItem CmdLet, and the Where-Object CmdLet, but I cannot seem to get this exact behavior.

Answer 1

I like Keith Hill's answer except it has a bug that prevents it from recursing past two levels. These commands manifest the bug:

New-Item level1/level2/level3/level4/foobar.txt -Force -ItemType file cd level1 GetFiles . xyz | % { $_.fullname } 

With Hill's original code you get this:

...\level1\level2 ...\level1\level2\level3 

Here is a corrected, and slightly refactored, version:

function GetFiles($path = $pwd, [string[]]$exclude) {     foreach ($item in Get-ChildItem $path)     {         if ($exclude | Where {$item -like $_}) { continue }          $item         if (Test-Path $item.FullName -PathType Container)         {             GetFiles $item.FullName $exclude         }     } }  

With that bug fix in place you get this corrected output:

...\level1\level2 ...\level1\level2\level3 ...\level1\level2\level3\level4 ...\level1\level2\level3\level4\foobar.txt 

I also like ajk's answer for conciseness though, as he points out, it is less efficient. The reason it is less efficient, by the way, is because Hill's algorithm stops traversing a subtree when it finds a prune target while ajk's continues. But ajk's answer also suffers from a flaw, one I call the ancestor trap. Consider a path such as this that includes the same path component (i.e. subdir2) twice:

\usr\testdir\subdir2\child\grandchild\subdir2\doc 

Set your location somewhere in between, e.g. cd \usr\testdir\subdir2\child, then run ajk's algorithm to filter out the lower subdir2 and you will get no output at all, i.e. it filters out everything because of the presence of subdir2 higher in the path. This is a corner case, though, and not likely to be hit often, so I would not rule out ajk's solution due to this one issue.

Nonetheless, I offer here a third alternative, one that does not have either of the above two bugs. Here is the basic algorithm, complete with a convenience definition for the path or paths to prune--you need only modify $excludeList to your own set of targets to use it:

$excludeList = @("stuff","bin","obj*") Get-ChildItem -Recurse | % {     $pathParts = $_.FullName.substring($pwd.path.Length + 1).split("\");     if ( ! ($excludeList | where { $pathParts -like $_ } ) ) { $_ } } 

My algorithm is reasonably concise but, like ajk's, it is less efficient than Hill's (for the same reason: it does not stop traversing subtrees at prune targets). However, my code has an important advantage over Hill's--it can pipeline! It is therefore amenable to fit into a filter chain to make a custom version of Get-ChildItem while Hill's recursive algorithm, through no fault of its own, cannot. ajk's algorithm can be adapted to pipeline use as well, but specifying the item or items to exclude is not as clean, being embedded in a regular expression rather than a simple list of items that I have used.

I have packaged my tree pruning code into an enhanced version of Get-ChildItem. Aside from my rather unimaginative name--Get-EnhancedChildItem--I am excited about it and have included it in my open source Powershell library. It includes several other new capabilities besides tree pruning. Furthermore, the code is designed to be extensible: if you want to add a new filtering capability, it is straightforward to do. Essentially, Get-ChildItem is called first, and pipelined into each successive filter that you activate via command parameters. Thus something like this...

Get-EnhancedChildItem –Recurse –Force –Svn     –Exclude *.txt –ExcludeTree doc*,man -FullName -Verbose  

... is converted internally into this:

Get-ChildItem | FilterExcludeTree | FilterSvn | FilterFullName 

Each filter must conform to certain rules: accepting FileInfo and DirectoryInfo objects as inputs, generating the same as outputs, and using stdin and stdout so it may be inserted in a pipeline. Here is the same code refactored to fit these rules:

filter FilterExcludeTree() {   $target = $_   Coalesce-Args $Path "." | % {     $canonicalPath = (Get-Item $_).FullName     if ($target.FullName.StartsWith($canonicalPath)) {       $pathParts = $target.FullName.substring($canonicalPath.Length + 1).split("\");       if ( ! ($excludeList | where { $pathParts -like $_ } ) ) { $target }     }   } }  

The only additional piece here is the Coalesce-Args function (found in this post by Keith Dahlby), which merely sends the current directory down the pipe in the event that the invocation did not specify any paths.

Because this answer is getting somewhat lengthy, rather than go into further detail about this filter, I refer the interested reader to my recently published article on Simple-Talk.com entitled Practical PowerShell: Pruning File Trees and Extending Cmdlets where I discuss Get-EnhancedChildItem at even greater length. One last thing I will mention, though, is another function in my open source library, New-FileTree, that lets you generate a dummy file tree for testing purposes so you can exercise any of the above algorithms. And when you are experimenting with any of these, I recommend piping to % { $_.fullname } as I did in the very first code fragment for more useful output to examine.