Deploy a C# Stateful Service Fabric application from Visual Studio to Linux

Question

EDIT 04/06/18 => Updated question with last status

---

So I have this working .Net 4.6 Stateful Service that currently run on my Windows Service Fabric cluster deployed on Azure.

Starting from 09/2017, I should be able to move to Linux: https://blogs.msdn.microsoft.com/azureservicefabric/2017/09/25/service-fabric-6-0-release/

So I'm trying to deploy it on Linux so I can save costs.

1. First things first, I've migrated all my code from .Net 4.6 to .Net Core 2.0. Now I can compile my binaries without issues. I've basically created new .Net Core projects and then moved all my source code from .Net 4.6 projects to the new .Net Core ones.

2. Then I've updated my Service Fabric application. I removed my previous SF services from my sfproj, then I've added my new .Net Core ones.

Looks like there is a warning (nothing on the output window though), but it's here anyway if I try to create a new empty Statful service using .Net core 2.0 through the template provided by Service Fabric Tools 2.0 (beta):

So I'm going to live with it.

3. On my dev machine, I've modified the 2 csproj projects that contain my Stateful services so they can run locally as Windows executables. I've used the win7-x64 runtimeIdentifier.

Running my SF cluster locally on my Windows machine is fine.

4. Then I've slightly changed the previous csproj files for Linux. I used the ubuntu.16.10-x64 runtimeIdentifier.

Also I've changed the ServiceManifest.xml file to target the linux-compatible binary:

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.9.6">
    <EntryPoint>
      <ExeHost>
        <Program>entryPoint.sh</Program>
      </ExeHost>
    </EntryPoint>
  </CodePackage>

entryPoint.sh is a basic script that eventually executes:

dotnet $DIR/MyService.dll

5. Then I've successfully deployed to my secured SF Linux cluster from Visual Studio. Unfortunately I have the following errors for both my stateful services:

Error event: SourceId='System.Hosting', Property='CodePackageActivation:Code:EntryPoint'. There was an error during CodePackage activation.The service host terminated with exit code:134

Looks like my binary crashes when starting. So here are my questions:

• Is the approach right to deploy a C# .Net Core SF stateful service on Linux from Visual Studio?

EDIT: looking inside the LinuxsyslogVer2v0 table, I get the following error:

starthost.sh[100041]: Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'System.Threading.Thread, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

I found the following bug report: https://github.com/dotnet/sdk/issues/1502 Unfortunately, I still get the error without using MSBuild (using dotnet deploy).

EDIT: further clarification:

• My boss want me to run on Linux because starting from D1v2 machines, it's half the price compared to Windows machines (no license etc.)
• My .NET Core 2.0 services successfully run on Windows. So the .NET Core port should be fine.

Answer 1

So, this was a real pain in the ass to get it working properly. But it works. Well, kind of.

---

First, Reliable Services are still in preview on Linux: https://github.com/Microsoft/service-fabric/issues/71

Full Linux support should come very soon (actually it should be available already according to the previous link...).

Now for the details about how to procede, here is some information to help others, because there is just nothing about that on Microsoft documentation and I literally lost 3 days trying to make it work.

1. Do use .NET Core 2.0 for your projects.

It is supported on Linux. On preview for now, but it works.

2. Do use the right RID for your projects.

As of today (April 2018), the right RID to use is ubuntu.16.04-x64. Edit the csproj files of your Reliable Service projects and set the RID like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <IsServiceFabricServiceProject>True</IsServiceFabricServiceProject>
    <RuntimeIdentifier>ubuntu.16.04-x64</RuntimeIdentifier>
    <Platforms>AnyCPU;x64</Platforms>
  </PropertyGroup>

The fun part is, you should be able to provide multiple RIDs using the RuntimeIdentifiers parameter (with a S at the end) like that:

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <IsServiceFabricServiceProject>True</IsServiceFabricServiceProject>
    <RuntimeIdentifiers>win7x64;ubuntu.16.04-x64</RuntimeIdentifiers>
    <Platforms>AnyCPU;x64</Platforms>
  </PropertyGroup>

So you could build Windows binaries and Linux binaries at the same time. But it simply doesn't work. When building the project from Visual Studio, I end up with the following directory only:

bin/Debug/netcoreapp2.0/

Only DLLs, no valid entry point. No win7-x64 folder, no ubuntu.16.04-x64, no nothing. This is a bug, supposed to be fixed, but it's not (I use Visual Studio 15.6.2 all up-to-date as of today). See https://github.com/dotnet/core/issues/1039

3. You need a valid Entry Point for your service.

On Windows it's an executable file (*.exe). On Linux it's not. I ended up getting the Linux C# example and copied/pasted the entry point. https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-create-your-first-linux-application-with-csharp

So basically I now have on my ServiceManifest.xml file of each Reliable Service the following EntryPoint :

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="XXXX"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <!-- This is the name of your ServiceType.
         This name must match the string used in RegisterServiceType call in Program.cs. -->
    <StatefulServiceType ServiceTypeName="YYY" HasPersistedState="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ExeHost>
        <Program>entryPoint.sh</Program>
      </ExeHost>
    </EntryPoint>
  </CodePackage>

entryPoint.sh is as follows:

#!/usr/bin/env bash
check_errs()
{
  # Function. Parameter 1 is the return code
  if [ "${1}" -ne "0" ]; then
    # make our script exit with the right error code.
    exit ${1}
  fi
}

DIR=`dirname $0`
echo 0x3f > /proc/self/coredump_filter
source $DIR/dotnet-include.sh
dotnet $DIR/NAME_OF_YOUR_SERVICE_DLL.dll $@
check_errs $?

dotnet-include.sh is as follows:

#!/bin/bash
. /etc/os-release
linuxDistrib=$ID
if [ $linuxDistrib = "rhel" ]; then
  source scl_source enable rh-dotnet20
  exitCode=$?
  if [ $exitCode != 0 ]; then
    echo "Failed: source scl_source enable rh-dotnet20 : ExitCode: $exitCode"
    exit $exitCode
  fi
fi

Both are inside the PackageRoot folder. I specified for both their properties so the Build Action is "Content" and the Copy to Output Directory is "Copy always".

4. Do NOT build using MSBuild !!

Yeah it is supposed to build Linux packages too, or at least it seems so, because MSBuild is able to produce the following files when you right click on your project and click "Build":

Don't trust the apparent success of the operation, it will miserably FAIL to properly execute when deployed. Some *.so files missing and other issues. MSBuild is buggy as hell and misbehaves regarding dependencies.

See for instance this bug report: https://github.com/dotnet/sdk/issues/1502 Still not fixed after almost a year...

Or https://github.com/dotnet/core/issues/977 (got this one, too).

5. Do write some PowerShell script to build the stuff by yourself.

I ended up reinventing the wheel using the following script to build my package:

# Creating binaries for service 1
cd DIRECTORY_OF_MY_SERVICE_1
dotnet publish -c Release -r ubuntu.16.04-x64

# Creating binaries for service 2
cd ..\DIRECTORY_OF_MY_SERVICE_2
dotnet publish -c Release -r ubuntu.16.04-x64

# Creating binaries for service 3
cd ..\DIRECTORY_OF_MY_SERVICE_3
dotnet publish -c Release -r ubuntu.16.04-x64

# Copying ApplicationManifest.xml
cd ..
mkdir PKG\ServiceFabricApplication
echo F|xcopy "ServiceFabricApplication\ApplicationPackageRoot\ApplicationManifest.xml" "PKG\ServiceFabricApplication\ApplicationManifest.xml" /sy

# Copying Service1 files
mkdir "PKG\ServiceFabricApplication\Service1Pkg"
mkdir "PKG\ServiceFabricApplication\Service1Pkg\Code"
xcopy "Service1\PackageRoot\*" "PKG\ServiceFabricApplication\Service1Pkg" /sy /D
xcopy "Service1\bin\Release\netcoreapp2.0\ubuntu.16.04-x64\publish\*" "PKG\ServiceFabricApplication\Service1Pkg\Code" /sy

# Copying Service2 files
mkdir "PKG\ServiceFabricApplication\Service2Pkg"
mkdir "PKG\ServiceFabricApplication\Service2Pkg\Code"
xcopy "Service2\PackageRoot\*" "PKG\ServiceFabricApplication\Service2Pkg" /sy /D
xcopy "Service2\bin\Release\netcoreapp2.0\ubuntu.16.04-x64\publish\*" "PKG\ServiceFabricApplication\Service2Pkg\Code" /sy

# Copying Service3 files
mkdir "PKG\ServiceFabricApplication\Service3Pkg"
mkdir "PKG\ServiceFabricApplication\Service3Pkg\Code"
xcopy "Service3\PackageRoot\*" "PKG\ServiceFabricApplication\Service3Pkg" /sy /D
xcopy "Service3\bin\Release\netcoreapp2.0\ubuntu.16.04-x64\publish\*" "PKG\ServiceFabricApplication\Service3Pkg\Code" /sy

# Compresses the package
Write-host "Compressing package..."
Copy-ServiceFabricApplicationPackage -ApplicationPackagePath .\PKG\ServiceFabricApplication -CompressPackage -SkipCopy

sfproj file is a Visual Studio / MSBuild related project, so you need to build everything by yourself. The script above produces the same content as the pkg folder created by MSBuild when building your sfproj using Visual Studio. It copies everything on a PKG folder at the root of your solution.

The package structure is detailed here: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/service-fabric/service-fabric-package-apps.md

6. Now it's time to deploy!

At this point I didn't trusted Visual Studio anymore, so I built my own PowerShell script:

. .\ServiceFabricApplication\Scripts\Deploy-FabricApplication.ps1 -ApplicationPackagePath '.\PKG\ServiceFabricApplication' -PublishProfileFile '.\ServiceFabricApplication\PublishProfiles\Cloud.xml' -DeployOnly:$false -ApplicationParameter:@{} -UnregisterUnusedApplicationVersionsAfterUpgrade $false -OverrideUpgradeBehavior 'None' -OverwriteBehavior 'SameAppTypeAndVersion' -SkipPackageValidation:$false -ErrorAction Stop

It reuses the Deploy-FabricApplication.ps1 script provided by the Service Fabric project template inside the sfproj project. This script parses the Cloud.xml PublishProfile and deploys to your service fabric cluster.

So you specifies the rights values on both PublishProfiles/Cloud.xml and ApplicationParameters/Cloud.xml then execute the script.

It only works if you have the certificate used to secure the cluster installed on your machine, of course. Do note the first dot '.' is important, because if you don't use it, you'll have the following error:

Get-ServiceFabricClusterManifest : Cluster connection instance is null

See https://stackoverflow.com/a/38104087/870604

Oh, and as there are bugs on the Service Fabric SDK too, you might want to shutdown your local SF cluster too... https://github.com/Azure/service-fabric-issues/issues/821

7. Now it's time for another deception.

It simply doesn't work, the service crashes on startup. After searching hours inside the LinuxsyslogVer2v0 Azure Storage table (the log table for Linux, located in one of the two Azure Storage Accounts created automatically with the SF cluster), I found that Microsoft own Nuget Packages were buggy too.

Specifically, the Nuget package Microsoft.Azure.Devices doesn't work on version 1.6.0. An issue with a reference of a dll not found or whatever. I rollbacked to a previous version, namely 1.5.1, and it was fixed.

At this point I didn't had anymore energy to create another Github issue about that. Sorry MS, I'm not your QA team, I'm getting tired.

8. Build again using the first PowerShell script, deploy using the second PowerShell script, and you're done.

You've finally deployed C# Reliable Services using .NET Core 2.0 from Visual Studio (kind of, as it's buggy and I used PowerShell) on Windows to a Linux SF Cluster.

Now I still have issues with my ASP.NET Core service, but it will be a story for another day.

---

Conclusion: TL;DR

The whole thing is a mess. Bugs everywhere. In the SDK, in the tools, in some of Microsoft Nuget Packages. Awful experience. But it is supported (in preview for now) and you can make it work. Hope this post will help...

Answer 2

I was having similar issues, but I believe this is the issue:

In this release, .NET Core 2.0 services are only supported on Service Fabric for Windows. Full cross-platform support for .NET Core 2.0 services on Windows and Linux is coming soon.

From the Service Fabric 6.1 Release Notes So no Linux as long as you are targeting .net core 2.0.