Azure DevOps Pipeline - dotnet restore Package Content Hash Validation Fails

Question

I have setup a build pipeline in Azure DevOps for my Function App that takes advantage of nuget caching and thus the package.lock.json file. However, I keep running into package validation hash issues such as these:

Package content hash validation failed for Microsoft.Extensions.DependencyInjection.2.2.0. Expected: ASF77AJjnyi9hL7IJU1KCAvnCTgI3JEwkU+D4gnKd53nFIYpibVjR6SW8tdTkkuZ+QkmIx2rPvKdTMNVPfVU9A== Actual: MZtBIwfDFork5vfjpJdG5g8wuJFt7d/y3LOSVVtDK/76wlbtz6cjltfKHqLx2TKVqTj5/c41t77m1+h20zqtPA==
Package content hash validation failed for Microsoft.Extensions.DependencyInjection.Abstractions.2.2.0. Expected: 2xMk9LHz1EY+7gVG0lG4qBvkUiVjg8QNPqd2HYmEP5+PL7Ayo96EhBieAhd++Gx4yM+xN8kNqmhZdFMBHeG0HQ== Actual: f9hstgjVmr6rmrfGSpfsVOl2irKAgr1QjrSi3FgnS7kulxband50f2brRLwySAQTADPZeTdow0mpSMcoAdadCw==
Package content hash validation failed for runtime.fedora.24-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0. Expected: c3YNH1GQJbfIPJeCnr4avseugSqPrxwIqzthYyZDN6EuOyNOzq+y2KSUfRcXauya1sF4foESTgwM5e1A8arAKw== Actual: LdIvj7Bi2jiaNTqY/ezZGVXHe1KI5fjLSI026O1TjVzsmdgTP/zTF+f3nwHCjwttyhsPBEiswv0PekimPWZwWg==
Package content hash validation failed for runtime.native.System.IO.Compression.4.3.0. Expected: INBPonS5QPEgn7naufQFXJEp3zX6L4bwHgJ/ZH78aBTpeNfQMtf7C6VrAFhlq2xxWBveIOWyFzQjJ8XzHMhdOQ== Actual: b+V9JC/Ii3sR659flBeaBJww111425tgjcDS1k+hqV4sGh9FALRDBvJnDtQ895gAzpPTUOFDHdqaZ2Et7BpZMg==
Package content hash validation failed for runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.Apple.4.3.0. Expected: kVXCuMTrTlxq4XOOMAysuNwsXWpYeboGddNGpIgNSZmv1b6r/s/DPk0fYMB7Q5Qo4bY68o48jt4T4y5BVecbCQ== Actual: Kh9W4agE0r/hK8AX1LvyQI2NrKHBL8pO0gRoDTdDb0LL6Ta1Z2OtFx3lOaAE0ZpCUc/dt9Wzs3rA7a3IsKdOVA==
Package content hash validation failed for runtime.ubuntu.14.04-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0. Expected: ytoewC6wGorL7KoCAvRfsgoJPJbNq+64k2SqW6JcOAebWsFUvCCYgfzQMrnpvPiEl4OrblUlhF2ji+Q1+SVLrQ== Actual: JGc0pAWRE8lB4Ucygk2pYSKbUPLlAIq6Bczf5/WF2D/VKJEPtYlVUMxk8fbl1zRfTWzSHi+VcFZlaPlWiNxeKg==
Package content hash validation failed for System.Collections.Specialized.4.3.0. Expected: Epx8PoVZR0iuOnJJDzp7pWvdfMMOAvpUo95pC4ScH2mJuXkKA2Y4aR3cG9qt2klHgSons1WFh4kcGW7cSXvrxg== Actual: NoPBj0ykejqAWW4p4gGtrrL+3c84ZLSvGnHgq422ew1Rj4WKj1FA8/BCybqC111EtgcqUl6ZJNFYYS22HLgbjA==
Package content hash validation failed for System.ComponentModel.Annotations.4.4.0. Expected: wohleA9W059afFBm49G4cNZJiPK5KShuC+fWxMp3wiugD/aYL7n9zmtvv8wQlh8brOca0GGROSBnz77dtwJbXQ== Actual: 29K3DQ+IGU7LBaMjTo7SI7T7X/tsMtLvz1p56LJ556Iu0Dw3pKZw5g8yCYCWMRxrOF0Hr0FU0FwW0o42y2sb3A==
Package content hash validation failed for System.Runtime.Serialization.Json.4.3.0. Expected: Ma/DVHfRcOcgQFHVGafUrT7hT1IitsnmUjpNZG5xJCYrI/8wfaYKGYNZycxQyl9Nk+9IAJiMJE6RFuavRQ2WEg== Actual: CpVfOH0M/uZ5PH+M9+Gu56K0j9lJw3M+PKRegTkcrY/stOIvRUeonggxNrfBYLA5WOHL2j15KNJuTuld3x4o9w==
Package content hash validation failed for System.Runtime.Serialization.Primitives.4.3.0. Expected: Wz+0KOukJGAlXjtKr+5Xpuxf8+c8739RI1C+A2BoQZT+wMCCoMDDdO8/4IRHfaVINqL78GO8dW8G2lW/e45Mcw== Actual: 2Z5t70a2SwMsfQDp9KOclaZNyQhfIga2gppq9lIUDM1A4ohTshn4JqT7ir8bvIhXgorWKYDAr6rPzEbi/nTGKg==
Package content hash validation failed for System.Security.Cryptography.Csp.4.3.0. Expected: X4s/FCkEUnRGnwR3aSfVIkldBmtURMhmexALNTwpjklzxWU7yjMk7GHLKOZTNkgnWnE0q7+BCf9N2LVRWxewaA== Actual: yO2k5o+Z+DiFRBvvB9vdRRAGHi6bm02M9OWXfCqQ8K0UxD3Woc3svQheZfb7PoTEFs0kGacO0IzzMWsb6Mkeow==
Package content hash validation failed for System.Security.Cryptography.OpenSsl.4.3.0. Expected: h4CEgOgv5PKVF/HwaHzJRiVboL2THYCou97zpmhjghx5frc7fIvlkY1jL+lnIQyChrJDMNEXS6r7byGif8Cy4w== Actual: vOYy7Jv9KsG3ld2hLt0GoERd82SZi4BelrbXLwI9yFBYX7kpbvUCWYo4eyevk47cuJXZ9ZLVAryANcc7iY71aA==

Conceptually, I know they are happening because the content hashes my machine is putting in the lock file when I add/update packages are different than the build agent (windows-latest) is calculating. But why are they different?

How do I go about resolving this problem beyond c/p the hashes the build agent is looking for into the lock file? I currently have the latest VS 2019 Enterprise installed (16.5.5).

In addition, here is the yml file I'm using if it helps/

name: $(Date:yyyy.MMdd)$(Rev:.r)

trigger:
  batch: true
  branches:
    include:
      - deploy/ConfigurationAPI
  paths:
    include:
      - server/functions/ConfigurationAPI/*

variables:
  NUGET_PACKAGES: $(Pipeline.Workspace)/.nuget/packages
  ArtifactName: BuildResult
  AzureSubscription: StellaNovaAzureResources
  FunctionName: stellanovaconfigurationapi
  ResourceGroupName: Shared-ResourceGroup-WestUS
  SlotName: deployment

stages:
  - stage: Build
    jobs:
      - job: BuildApp
        displayName: Build ConfigurationAPI
        pool:
          vmImage: 'windows-latest'

        steps:
          - task: Cache@2
            displayName: Cache NuGet packages
            inputs:
              path: $(NUGET_PACKAGES)
              key: 'nuget | "$(Agent.OS)" | **/packages.lock.json,!**/bin/**'
              restoreKeys: nuget | "$(Agent.OS)"
              cacheHitVar: CACHE_RESTORED

          - task: DotNetCoreCLI@2
            displayName: Restore NuGet packages
            inputs:
              command: restore
              arguments: '--locked-mode --packages $(NUGET_PACKAGES)'
              projects: 'server/functions/ConfigurationAPI/ConfigurationAPI.csproj'

          - task: DotNetCoreCLI@2
            displayName: Build Function
            inputs:
              command: publish
              arguments: '--configuration Release --no-restore --output buildoutput'
              projects: 'server/functions/ConfigurationAPI/ConfigurationAPI.csproj'
              publishWebProjects: false
              modifyOutputPath: false
              zipAfterPublish: false

          - task: PublishSymbols@2
            displayName: Publish Symbols
            inputs:
              SearchPattern: '**/bin/**/*.pdb'
              SymbolServerType: 'TeamServices'
              SymbolsProduct: 'StellaNova.ConfigurationAPI'
              DetailedLog: false

          - task: ArchiveFiles@2
            inputs:
              rootFolderOrFile: '$(System.DefaultWorkingDirectory)/buildoutput'
              includeRootFolder: false
              archiveFile: '$(System.DefaultWorkingDirectory)/$(Build.BuildNumber).zip'

          - publish: $(System.DefaultWorkingDirectory)/$(Build.BuildNumber).zip
            artifact: $(ArtifactName)

  - stage: Release
    condition: ne(variables['Build.Reason'], 'PullRequest')
    jobs:
      - job: DeployApp
        displayName: Deploy ConfigurationAPI
        pool:
          vmImage: 'windows-latest'

        steps:
          - download: current
            artifact: BuildResult

          - task: AzureFunctionApp@1
            displayName: Deploy to slot
            inputs:
              azureSubscription: '$(AzureSubscription)'
              resourceGroupName: '$(ResourceGroupName)'
              appType: functionApp
              appName: '$(FunctionName)'
              deployToSlotOrASE: true
              slotName: '$(SlotName)'
              package: '$(Pipeline.Workspace)/$(ArtifactName)/$(Build.BuildNumber).zip'
              appSettings: -RefreshSentinelKey RefreshSentinel -AppEnvironment DEV

Answer 1

This is because of different implementation of hash function on different OSes.

Please check out this proposed solution on GitHub.

You can also read about this in my blog. I decided to create lock json file on host agent publish it and checked in into source control. I know that this is a plenty of work. But I really want to have the same file not generated each time like it is presented in GitHub solution.

Answer 2

This can also happen when the source of the package is actually different. In my case it happened when switching from a local NuGet package source to one hosted in Azure DevOps. My local machine was still using the old source, while Azure DevOps builds were using the new source.

Removing the local source from Visual Studio and clearing the NuGet cache solved the issue (after checking in the newly generated packages.lock.json files to source control).

Hope this helps someone in the future.