Speed up msbuild with fsutil hardlink create instead of slow copy tasks

Question

I'm trying to do speed up our build (csharp, msbuild, .net 3.5). Replace copy with fsutil hardlink create.

Previously, I almost got it with running a script over sln files and making the dll references private = false, then having a post build event create the hardlinks. Problem is transitive dependencies are not included. So, I think I need to reference the ResolveAssemblyReference task in msbuild to get the transitive dependencies I need to hardlink.

Any ideas?

This person tried the same thing, but did not post an final solution.

To be clear: What I want is to keep separate bin directories, but instead of copying file from one to another to create a hard link from a source (of a reference or a dependency) to the destination (the current project's bin). Which is much faster, and gives approximately the same effect as copying.

Answer 1

This is supported in VS 2010. But not 2008. See the UseHardLinksIfPossible option to Copy in _CopyFilesMarkedCopyLocal.

See also http://social.msdn.microsoft.com/Forums/en/tfsbuild/thread/9382a3d8-4632-4826-ad15-d5e845080981, http://msdn.microsoft.com/en-us/library/ms171466(v=VS.90).aspx for context.

Override the _CopyFilesMarkedCopyLocal target. We add something like this to the csproj files at the bottom: <Import Project="..\..\..\..\..\\CommonBuild\TW.Override.Microsoft.Common.targets" /> (This is auto added to the file with a nant task on every full nant build, so every project benefits).

Our new targets file is:

<Project ToolsVersion="3.5" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
     <UsingTask TaskName="CopyWithHardlinkOption" AssemblyFile="..\lib\TWBuildOptimization\TW.Hardlinker.dll" />
  <!--
    ============================================================
                                        _CopyFilesMarkedCopyLocal
    Overridden in order to allow hardlinking with our custom Copy Task.                                         

    Hardlinking is a major performance improvement. Sometimes 50% of the time compared to copying.

    Copy references that are marked as "CopyLocal" and their dependencies, including .pdbs, .xmls and satellites.
    ============================================================
  -->
    <Target
        Name="_CopyFilesMarkedCopyLocal">
        <CopyWithHardlinkOption
            SourceFiles="@(ReferenceCopyLocalPaths)"
            DestinationFiles="@(ReferenceCopyLocalPaths->'$(OutDir)%(DestinationSubDirectory)%(Filename)%(Extension)')"
            SkipUnchangedFiles="true"
              UseHardlinksIfPossible="true"
            OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)">
            <Output TaskParameter="DestinationFiles" ItemName="FileWritesShareable"/>
        </CopyWithHardlinkOption>
    </Target>
</Project>

See the msbuild 4 Copy UseHardlinksIfPossible option. I backported that to 3.5 through decompiling and reimplementing. The relevant logic in CopyFileWithLogging was:

      // The port from 4.0's task that allows hardlinking
        bool hardlinkSucceeded = false;
        if (UseHardlinksIfPossible)
        {
            if (File.Exists(destinationFile))
            {
                FileUtilities.DeleteNoThrow(destinationFile);
            }
            if (!TwNativeMethods.CreateHardLink(destinationFile, sourceFile, IntPtr.Zero))
            {
                var win32Exception = new Win32Exception(Marshal.GetLastWin32Error());
                Log.LogMessage(MessageImportance.High, "Hardlinking had a problem {0}, will retry copying. {1}", new object[] {win32Exception.Message, win32Exception});
            }
            hardlinkSucceeded = true;
        }
        if (!hardlinkSucceeded)
        {
            Log.LogMessageFromResources(MessageImportance.Normal, "Copy.FileComment", new object[] { sourceFile, destinationFile });
            Log.LogMessageFromResources(MessageImportance.Low, "Shared.ExecCommand", new object[0]);
            Log.LogCommandLine(MessageImportance.Low, "copy /y \"" + sourceFile + "\" \"" + destinationFile + "\"");
            File.Copy(sourceFile, destinationFile, true);
        }
        // end port

Also had to add these:

// decompiled from 4.0
internal static class TwNativeMethods
{
    [DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)]
    internal static extern bool CreateHardLink(string newFileName, string exitingFileName, IntPtr securityAttributes);
}

// decompiled from 4.0
internal static class FileUtilities
{
    internal static void DeleteNoThrow(string path)
    {
        try
        {
            File.Delete(path);
        }
        catch (Exception exception)
        {
            if (ExceptionHandling.NotExpectedException(exception))
            {
                throw;
            }
        }
    }
}

// decompiled from 4.0
internal static class ExceptionHandling
{
    // Methods
    internal static bool IsCriticalException(Exception e)
    {
        return (((e is StackOverflowException) || (e is OutOfMemoryException)) || ((e is ExecutionEngineException) || (e is AccessViolationException)));
    }

    internal static bool NotExpectedException(Exception e)
    {
        return (((!(e is UnauthorizedAccessException) && !(e is ArgumentNullException)) && (!(e is PathTooLongException) && !(e is DirectoryNotFoundException))) && ((!(e is NotSupportedException) && !(e is ArgumentException)) && (!(e is SecurityException) && !(e is IOException))));
    }

    internal static bool NotExpectedReflectionException(Exception e)
    {
        return ((((!(e is TypeLoadException) && !(e is MethodAccessException)) && (!(e is MissingMethodException) && !(e is MemberAccessException))) && ((!(e is BadImageFormatException) && !(e is ReflectionTypeLoadException)) && (!(e is CustomAttributeFormatException) && !(e is TargetParameterCountException)))) && (((!(e is InvalidCastException) && !(e is AmbiguousMatchException)) && (!(e is InvalidFilterCriteriaException) && !(e is TargetException))) && (!(e is MissingFieldException) && NotExpectedException(e))));
    }
}