Could not load file or assembly Newtonsoft.Json when running app from the dotnet publish output folder

Question

I am finding a problem with Newtonsoft.Json library throwing a

System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified

when running an app as a Docker container and I'd like to know why this happens and why dependency management does not work smoothly.

I use .NET 5.

I have a library MyLibrary.A that explicitly uses Newtonsoft.Json 13.0.1 for serializing and deserializing json.

I have a different library MyLibrary.B that wraps a MassTransit.AmazonSQS library. This MassTransit library also uses Newtonsoft.Json, but probably a different version. If I don't do anything explicit, it seems the MassTransit dependency shows the Newtonsoft.Json 11.0.2. If at MyLibrary.B I explicitly add Newtonsoft.Json 13.0.1, even though I don't explicitly use it, then MassTransit seems to be happy with using this newest Newtonsoft.Json 13.0.1

Now, I have a web app MyApp that uses MyLibrary.A and MyLibrary.B. It works fine locally, but I use a CI/CD server to generate a Docker image.

Now I spin up this Docker image as a container locally (as a Docker Compose) and I get the error

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.

It complains about a version that does not even exist. There is no 13.0.0.0, This library seems to go from 12.0.3 to 13.0.1.

I am now going through all my libraries and making sure they all use Newtonsoft.Json 13.0.1 explicitly. And when I detect that some of them use some third party that relies on Newtonsoft.Json I add explicitly the very same version, so that I can get everywhere the 13.0.1 version.

---

UPDATE 1: My workaround didn't work. I don't know what else to try.

I have even added the Newtonsoft.Json 13.0.1 explicitly to my webapp so I was hoping that at least at runtime it has it available.

Also, if I run my web up locally as a standard kestrel AspNetCore app (.NET 5) it launches properly. What is going on? Why is my docker container complaining about Newtonsoft.Json 13.0.0.0 not being found?

These are the traces when attempting to run as a Docker container

docker run -p 8080:80 \
>     -e ASPNETCORE_ENVIRONMENT=Production \
>     registry.gitlab.com/sample/foo-integration-service:latest
Unable to find image 'registry.gitlab.com/sample/foo-integration-service:latest' locally
latest: Pulling from sample/foo-integration-service
07aded7c29c6: Pull complete 
97aff7269a5a: Pull complete 
633b89d569a5: Pull complete 
bd0e639a2ac9: Pull complete 
a9a5571a369e: Pull complete 
9569d825ee3a: Pull complete 
Digest: sha256:5499b40392512f1731890ccf1ee13507769b733ee2f30c95d281f0550f7a892e
Status: Downloaded newer image for registry.gitlab.com/sample/foo-integration-service:latest
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.

File name: 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at foo.ItgService.Program.Main(String[] args) in /builds/sample/foo-integration-service/src/foo.ItgService/Program.cs:line 10

---

UPDATE 2: I decided to match MassTransit dependency Newtonsoft.Json 11.0.2 everywhere in my libraries. The problem remains. The error is now

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=11.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.

I don't get it. All my dependencies of Newtonsoft.Json are now 11.0.2 and still it complains. I'll add this version explicitly (even though I don't directly need it) at the web app main assembly and see if it still complains about it.

---

UPDATE 3: Still same problem after adding Newtonsoft.Json 11.0.2 to all my libraries and after adding that very same dependency to my web app assembly as a dependency.

As per Chris' comment I have now a .dockerignore.

bin/
obj/

The way I build the image is, with GitLab, with a standard dotnet build, dotnet publish and then copying all the contents of publish folder into the Docker image like this

FROM mcr.microsoft.com/dotnet/aspnet:5.0
COPY publish/ .
EXPOSE 80
ENTRYPOINT ["dotnet", "MyCompany.ItgService.dll"]

More specific, I use Kaniko and this is my .gitlab-ci.yml

image: mcr.microsoft.com/dotnet/sdk:5.0

variables:
  GIT_DEPTH: 1000
  PUBLISH_OUTPUT_DIR: publish
  ENTRYPOINT_DLL: ReplaceMe.dll
  CLUSTER_NAME: ReplaceMe
  SERVICE_NAME: ReplaceMe

stages:
  - build
  - test
  - publish
  - delivery

build:
  stage: build
  script:
    - dotnet restore --no-cache --force
    - dotnet build --configuration Release --no-restore
  artifacts:
    paths:
    - test
    expire_in: 8 hour
  rules:
    - if: $CI_COMMIT_TAG
      when: never  
    - when: always

test:
  stage: test
  services:
    - name: localstack/localstack:0.12.17.5
      alias: localstack
  variables:
    # Localstack with SNS and SQS
    AWS_DEFAULT_REGION: "us-east-1"
    EDGE_PORT: "4566"
    SERVICES: "sns,sqs"
  before_script:
    - rounds=10;
      while [ $rounds -gt 0 ]; do
        curl http://localstack:4566 && echo OK && break || echo FAIL
        rounds=$rounds - 1;
        sleep 5;
      done;
  script: dotnet test --blame --configuration Release
  rules:
    - if: $CI_COMMIT_TAG
      when: never  
    - exists:
      - test/**/*Tests.csproj

publish:
  stage: publish
  before_script:
    - export PATH=$PATH:/root/.dotnet/tools
    - dotnet tool install --global GitVersion.Tool --version 5.7.0
    - dotnet gitversion
    - SEMVER=$(dotnet gitversion -showvariable semver)
    - mkdir version
    - echo "${SEMVER}" > ./version/semver
    - APP_VERSION=$(cat ./version/semver)
  script:
    - dotnet publish -c Release -o $PUBLISH_OUTPUT_DIR -p:Version=$APP_VERSION
  artifacts:
    paths: 
      - $PUBLISH_OUTPUT_DIR/
      - version/
    expire_in: 8 hour
  rules:
    - if: $CI_COMMIT_TAG
      when: never  
    - when: always

container_registry:
  stage: delivery
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  before_script:
    - IMAGE_TAG=$(cat ./version/semver)
    - echo "bin/" > $CI_PROJECT_DIR/.dockerignore
    - echo "obj/" > $CI_PROJECT_DIR/.dockerignore
    - echo "FROM mcr.microsoft.com/dotnet/aspnet:5.0" > $CI_PROJECT_DIR/Dockerfile
    - echo "COPY $PUBLISH_OUTPUT_DIR/ ." >> $CI_PROJECT_DIR/Dockerfile
    - echo "EXPOSE 80" >> $CI_PROJECT_DIR/Dockerfile
    - echo "ENTRYPOINT [\"dotnet\", \"$ENTRYPOINT_DLL\"]" >> $CI_PROJECT_DIR/Dockerfile
    - cat $CI_PROJECT_DIR/Dockerfile
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - cat /kaniko/.docker/config.json
  script:
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:latest --destination $CI_REGISTRY_IMAGE:$IMAGE_TAG
  rules:
    - if: $CI_COMMIT_TAG
      when: never
    - when: always

I don't know if the dotnet publish is messing something up for docker or whether there is some flaw in my process (it works fine for all other services). What might be the problem?

I can see Newtonsoft.Json.dll among the files produced by dotnet publish along with MassTransit.AmazonSqsTransport and all others that are supposedly being copied properly to the Docker image.

Also, If I open the .deps.json I can see all references to Newtonsoft.Json are 11.0.2, so no version conflict anymore (I think!).

I'm out of ideas.

---

UPDATE 4 I've just cleared all my local nuget packages (using Ubuntu)

dotnet nuget locals all --clear
Clearing NuGet HTTP cache: /home/diegosasw/.local/share/NuGet/v3-cache
Clearing NuGet global packages folder: /home/diegosasw/.nuget/packages/
Clearing NuGet Temp cache: /tmp/NuGetScratch
Clearing NuGet plugins cache: /home/diegosasw/.local/share/NuGet/plugins-cache
Local resources cleared.

An interesting thing is that when I restore dependencies after that on my project

dotnet restore

I can see at /home/diegosasw/.nuget/packages/newtonsoft.json that there are versions 11.0.2 and 9.0.1

So I guess somewhere somehow a sub-dependency is using Newtonsoft 9.0.1, even though there is no trace of a Newtonsoft.Json 9.0.1 in my the *.deps.json that my dotnet publish produces, and I'm wondering whether this is related to my problem, maybe because that's the assembly being loaded and the 11 is being ignored?

---

UPDATE 5 I've just seen that the Newtonsoft.Json 9.0.1 is being used by some test project, because I can trace it at coverlet.collector.deps.json under coverlet.core 1.0.0 and the Microsoft.Extensions.DependencyModel 2.1.0

I guess it's not the cause of my problem then.

Also I've verified with docker export $(docker ps -lq) -o foo.tar that the container has the Newtonsoft.Json.dll.

I would like to understand why is this happening and learn how to better troubleshoot these kind of things.

---

UPDATE 6 (4 Oct)

I don't think the problem is with Docker. I think the problem is likely with dotnet publish or something I'm missing or doing wrong.

I have left Docker outside since I didn't see anything wrong there. I tried to simply do a

dotnet publish -c Release -o publish

and execute the app in that publish folder with dotnet MyCompany.ItgService.dll to reproduce the exception.

But before publishing, when I run the application with a

dotnet run -c Release --project src/Rubiko.ItgService

I don't get that exception.

See https://github.com/dotnet/sdk/issues/21716 for full details, traces, tree structure, etc.

Summary

The questions are:

• Why dotnet publish does not seem to produce everything my app needs to run?

• Why does it complain at runtime about an assembly that is there?

  $ ls publish/ | grep Newtonsoft
  Newtonsoft.Json.Bson.dll*
  Newtonsoft.Json.dll*
  

---

FINAL UPDATE: Problem solved. See my own response where info on how to properly troubleshoot these kind of issues and how I solved it by ensuring my test projects (that use different version of the library) don't publish artifacts and overwrite the desired dependency assembly.

Answer 1

I solved the mystery. It had nothing to do with Docker. It kind of had something to do with dotnet publish but the SDK works well.

The problem was, as initially suspected, with a version conflict. When publishing with dotnet publish -c Release -o publish I could see the Newtonsoft.Json.dll there. But the following made me suspicious

ls publish/ -al | grep Newtonsoft
-rwxrw-r-- 1 diegosasw  89K mar 22  2017 Newtonsoft.Json.Bson.dll*
-rwxrw-r-- 1 diegosasw 641K mar 24  2018 Newtonsoft.Json.dll*

2018 seems a bit old for that version. What if.. that Newtonsoft.Json assembly being published wasn't the version 11.0.2 after all?

I updated all my libraries to match MassTransit dependency on Newtonsoft 11.0.2 but my findings on Update 4 made me think there was some other project depending indirectly on Newtonsoft.Json 9.0.1, and that's why I could see that nuget package being cached locally.

If, somehow, the assembly being published is not the expected Newtonsoft.Json 11.0.2 but the Newtonsoft.Json 9.0.1, the error would make sense when complaining about not finding the assembly Newtonsoft.Json 11.0.2

Bingo!

I installed exiftool in my Ubuntu to check dll and exe versions.

sudo apt install libimage-exiftool-perl

I run the following

$ exiftool publish/Newtonsoft.Json.dll
ExifTool Version Number         : 11.88
File Name                       : Newtonsoft.Json.dll
Directory                       : publish
File Size                       : 465 kB
File Modification Date/Time     : 2021:07:19 19:52:18+02:00
File Access Date/Time           : 2021:10:04 12:53:14+02:00
File Inode Change Date/Time     : 2021:10:04 12:44:39+02:00
File Permissions                : rwxrw-r--
File Type                       : Win32 DLL
File Type Extension             : dll
MIME Type                       : application/octet-stream
Machine Type                    : Intel 386 or later, and compatibles
Time Stamp                      : 2016:06:13 13:05:00+02:00
Image File Characteristics      : Executable, Large address aware, DLL
PE Type                         : PE32
Linker Version                  : 48.0
Code Size                       : 465920
Initialized Data Size           : 2048
Uninitialized Data Size         : 0
Entry Point                     : 0x738b6
OS Version                      : 4.0
Image Version                   : 0.0
Subsystem Version               : 4.0
Subsystem                       : Windows command line
File Version Number             : 9.0.1.19813
Product Version Number          : 9.0.1.0
File Flags Mask                 : 0x003f
File Flags                      : (none)
File OS                         : Win32
Object File Type                : Dynamic link library
File Subtype                    : 0
Language Code                   : Neutral
Character Set                   : Unicode
Comments                        : Json.NET is a popular high-performance JSON framework for .NET
Company Name                    : Newtonsoft
File Description                : Json.NET .NET Standard 1.0
File Version                    : 9.0.1.19813
Internal Name                   : Newtonsoft.Json.dll
Legal Copyright                 : Copyright © James Newton-King 2008
Legal Trademarks                : 
Original File Name              : Newtonsoft.Json.dll
Product Name                    : Json.NET
Product Version                 : 9.0.1
Assembly Version                : 9.0.0.0

and as you can see, the published assembly is Newtonsoft.Json 9.0.1. A mix of relief invades me.

I went to my test projects and added the following to the *.csproj

<IsPublishable>false</IsPublishable>

and

rm -rd publish
dotnet publish -c Release -o publish

Moment of truth

$ exiftool publish/Newtonsoft.Json.dll
ExifTool Version Number         : 11.88
File Name                       : Newtonsoft.Json.dll
Directory                       : publish
File Size                       : 641 kB
File Modification Date/Time     : 2018:03:24 18:44:14+01:00
File Access Date/Time           : 2021:10:04 12:44:38+02:00
File Inode Change Date/Time     : 2021:10:04 12:57:29+02:00
File Permissions                : rwxrw-r--
File Type                       : Win32 DLL
File Type Extension             : dll
MIME Type                       : application/octet-stream
Machine Type                    : Intel 386 or later, and compatibles
Time Stamp                      : 2098:12:14 20:33:48+01:00
Image File Characteristics      : Executable, Large address aware, DLL
PE Type                         : PE32
Linker Version                  : 48.0
Code Size                       : 653824
Initialized Data Size           : 2048
Uninitialized Data Size         : 0
Entry Point                     : 0xa16b6
OS Version                      : 4.0
Image Version                   : 0.0
Subsystem Version               : 4.0
Subsystem                       : Windows command line
File Version Number             : 11.0.2.21924
Product Version Number          : 11.0.2.0
File Flags Mask                 : 0x003f
File Flags                      : (none)
File OS                         : Win32
Object File Type                : Dynamic link library
File Subtype                    : 0
Language Code                   : Neutral
Character Set                   : Unicode
Comments                        : Json.NET is a popular high-performance JSON framework for .NET
Company Name                    : Newtonsoft
File Description                : Json.NET .NET Standard 2.0
File Version                    : 11.0.2.21924
Internal Name                   : Newtonsoft.Json.dll
Legal Copyright                 : Copyright © James Newton-King 2008
Legal Trademarks                : 
Original File Name              : Newtonsoft.Json.dll
Product Name                    : Json.NET
Product Version                 : 11.0.2
Assembly Version                : 11.0.0.0

Now the assembly published is the expected 11.0.2.

I verify that running my app from the publish folder now works fine!

cd publish
dotnet MyCompany.ItgService.dll

Answer 2

Just use the version that MassTransit depends upon, which is much earlier than v13. Upgrading past that without the proper assembly redirects is likely causing your issue.