Why is there a need for a separate item in my MSBuild file?

Question

There are many articles (like this and this) that show how to add files to be published, and they all say to add something like this to the publish profile (.pubxml):

<Target Name="CustomCollectFiles">
  <ItemGroup>
    <_CustomFiles Include="..\Extra Files\**\*" />
    <FilesForPackagingFromProject  Include="%(_CustomFiles.Identity)">
      <DestinationRelativePath>Extra Files\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
    </FilesForPackagingFromProject>
  </ItemGroup>
</Target>

Why is there a need for the new _CustomFiles item? Why not simply <FilesForPackagingFromProject Include="..\Extra Files\**\*">? I tried it, and for some reason this causes every file in the project to end up in the deployed Extra Files folder. Can someone explain me this behaviour please?

Answer 1

Since you are asking about why this is required, I will have to dive deep into what this code means to explain what you're seeing. <Message /> is our friend!

The meaning of %

Let's first look at what % means by using it in a <Message> task:

<ItemGroup>
  <_CustomFiles Include="..\Extra Files\**\*" />
</ItemGroup>

<Message Text="File: %(_CustomFiles.Identity)" />

When you run this, you'll get the following output:

File: ..\Extra Files\file1.txt
File: ..\Extra Files\file2.txt
File: ..\Extra Files\file3.txt
...
File: ..\Extra Files\etc.txt

Basically, the Message task runs once for each item in the item group, because we used %.

What's in the item group?

Let's take a peek at the item group before we even make any changes to it. When this task begins, FilesForPackagingFromProject already has all of the files in them, with various metadata properties, including DestinationRelativePath. Let's see it by adding just this to our task:

<Message Text="File: %(FilesForPackagingFromProject.Identity) -> %(FilesForPackagingFromProject.DestinationRelativePath)" />

This outputs:

File: ..\obj\TempBuildDir\PrecompiledApp.config -> PrecompiledApp.config
File: ..\obj\TempBuildDir\Web.config -> Web.config
File: ..\obj\TempBuildDir\App_Themes\theme.css -> App_Themes\theme.css
...

It's important to realise that this item group is not empty to begin with. You are trying to add items to it.

The working code

When you have sub-elements in an element that has %, they apply once to each iteration, so let's now look at the working code:

<FilesForPackagingFromProject Include="%(_CustomFiles.Identity)">
  <DestinationRelativePath>Extra Files\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</FilesForPackagingFromProject>

<Message Text="File: %(FilesForPackagingFromProject.Identity) -> %(FilesForPackagingFromProject.DestinationRelativePath)" />

For each item in _CustomFiles, we include it into the FilesForPackagingFromProject item group and set the DestinationRelativePath metadata property to the appropriate RecursiveDir/Filename values - basically the ones that apply for the current element being looked at. Let's look at what this outputs:

File: ..\obj\TempBuildDir\PrecompiledApp.config -> PrecompiledApp.config
File: ..\obj\TempBuildDir\Web.config -> Web.config
File: ..\obj\TempBuildDir\App_Themes\theme.css -> App_Themes\theme.css
...
File: ..\Extra Files\file1.txt -> Extra Files\file1.txt
File: ..\Extra Files\file2.txt -> Extra Files\file2.txt
File: ..\Extra Files\file3.txt -> Extra Files\file3.txt
...
File: ..\Extra Files\etc.txt -> Extra Files\etc.txt

Including just a single file

If you wanted to include just a single file, you can do so as follows:

<FilesForPackagingFromProject Include="..\Extra Files\file1.txt">
  <DestinationRelativePath>Extra Files\file1.txt</DestinationRelativePath>
</FilesForPackagingFromProject>

This has no % to expand anywhere, so it does exactly what you would expect: it includes a single file into the output.

The broken code

Now let's try to include a single file, but without hard-coding the path and instead using the % expression from the original code:

<FilesForPackagingFromProject Include="..\Extra Files\file1.txt">
  <DestinationRelativePath>Extra Files\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</FilesForPackagingFromProject>

<Message Text="File: %(FilesForPackagingFromProject.Identity) -> %(FilesForPackagingFromProject.DestinationRelativePath)" />

There are % here so things get expanded, but because this doesn't have a % in the item group element, the expansion works differently and things get pear-shaped:

File: ..\obj\TempBuildDir\PrecompiledApp.config -> PrecompiledApp.config
File: ..\obj\TempBuildDir\Web.config -> Web.config
File: ..\obj\TempBuildDir\App_Themes\theme.css -> App_Themes\theme.css
...
File: ..\Extra Files\file1.txt -> Extra Files\PrecompiledApp.config
File: ..\Extra Files\file1.txt -> Extra Files\Web.config
File: ..\Extra Files\file1.txt -> Extra Files\theme.css

So instead of adding file1.txt to the item group once, it iterates over the entire collection and adds file1.txt once for each file already in it. RecursiveDir is not set in this context, while Filename/Extension are the original filename of each file in the group.

Hopefully you can see now that this will create a file for each file in your entire deployment, but in a flat tree, and notably, the contents will be that of file1.txt rather than the original file.

When you include a wildcard instead of just one file, the same thing happens for every file matched by the wildcard.

How to fix this

Stick with the %(_CustomFiles) fix. Hopefully you will now see why it's necessary and how it does what it does. I do believe this is how you are supposed to do this: here's another question about it, with an answer that recommends this approach.