Cannot connect to secured Azure Service Fabric Cluster via Powershell or Visual Studio

Question

I've created a Service Fabric Application currently consisting of two Reliable Services and a Reliable Actor. For development, I created an SQL Server and database in Azure, and hardcoded the connection string into my application, which I was running on my local SF cluster. This worked fine, and I was able to run my application locally whilst manipulating the database in the cloud.

I now want to publish my service to the cloud, and run it all remotely (so that I can set up and test the Web API is exposes), and this is where the problems start.

Following Azure docs:

1. Create a Service Fabric cluster in Azure using Azure Resource Manager
2. Connect to a secure cluster
3. Configure secure connections to a Service Fabric cluster from Visual Studio
4. Service Fabric cluster security scenarios
5. Publish an application to a remote cluster by using Visual Studio
6. Add or remove certificates for a Service Fabric cluster in Azure

I have taken the following steps:

1. Used Powershell (with ServiceFabricRPHelpers cmdlets) to create a KeyVault resource group, and within that a KeyVault.

2. Used New-SelfSignedCertificate with -DnsName set to api.mydomain.co.uk, which I have already purchased and created a CNAME record for api leading to mycluster.northeurope.cloudapp.azure.com:19000 (though of course it doesn't exist at this stage of the process), followed by Export-PfxCertificate to create the .pfx file. The .pfx was then imported to cert:\CurrentUser\TrustedPeople and cert:\CurrentUser\My.

3. Called Invoke-AddCertToKeyVault to add the newly generated certificate to my KeyVault.

4. Used the SetupApplications.ps1 script to configure AAD.

5. Placed all resulting strings etc. into azuredeploy.json and azuredeploy.parameters.json, resolved errors (some of which seemed to contradict the documentation..), and successfully deployed the cluster. It is now visible on my Azure Portal.

6. Assigned User Roles (admin to myself) from the classic portal.

7. Used Invoke-AddCertToKeyVault to (this time create and) add a second, "admin client" certificate to the cluster (as opposed to the first which was a cluster certificate).

So, with all of that done, I believe I should have done everything I need to in order to be able to connect to the cluster to publish via VS2015, and access the management interface from api.mydomain.co.uk:19080. Alas, that doesn't happen...

Connection to the database within the resource group my cluster still works from VS via the SQL Server Explorer using SQL authentication, however, any attempt to communicate with the server itself using AAD or X509 based authentication results in a wait while it tries to connect, and then failure. A few examples:

Trying to connect to the management console says it's blocked, which implies to me it is there, but all the documentation ends before telling me how to access it.

Attempting to connect using Connect-ServiceFabricCluster also fails, and searching the error messages hasn't given me any indication of what to do.

After spending two days absorbing all of this and trying to get it working, I'm all out of ideas on what to try and change. Can anyone find a problem in what I have done, or suggest anything I could try? If you need more details from me then please just ask!

Answer 1

I too had a nightmare attempting to deploy a secure cluster, using much of the same documentation you too have tried to consume. After spending days getting my hands dirty I managed to finally get it working.

Here is my own helper and template: SecureCluster

The key things to watch are:

• Make sure your client and cluster certificates are both in your key vault and referenced within your ARM template under the OSProfile of the VM scale set (I noticed in your example that you were adding the client admin certificate after modifying the ARM template):

  
  "osProfile": {
          "adminUsername": "[parameters('adminUsername')]",
          "adminPassword": "[parameters('adminPassword')]",
          "computernamePrefix": "[parameters('vmNodeType0Name')]",
          "secrets": [
                          {
                              "sourceVault": {
                                  "id": "[parameters('sourceVault')]"
                              },
                              "vaultCertificates": [
                                  {
                                      "certificateStore": "My",
                                      "certificateUrl": "[parameters('clusterCertificateUrl')]"
                                  },
                                  {
                                      "certificateStore": "My",
                                      "certificateUrl": "[parameters('adminCertificateUrl')]"
                                  }
                              ]
                          }
                      ]
        },
  

This will make sure all your certificates are installed onto each node within the cluster.

Next is to make sure that the Service Fabric extension within the scale set also has your certificate:

"extensions": [
              {
                "name": "[concat(parameters('vmNodeType0Name'),'_ServiceFabricNode')]",
                "properties": {
                  "type": "ServiceFabricNode",
                  "autoUpgradeMinorVersion": false,
                  "protectedSettings": {
                    "StorageAccountKey1":
                      "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('supportLogStorageAccountName')),'2015-05-01-preview').key1]",
                    "StorageAccountKey2":
                      "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('supportLogStorageAccountName')),'2015-05-01-preview').key2]"
                  },
                  "publisher": "Microsoft.Azure.ServiceFabric",
                  "settings": {
                    "clusterEndpoint": "[reference(parameters('clusterName')).clusterEndpoint]",
                    "nodeTypeRef": "[parameters('vmNodeType0Name')]",
                    "dataPath": "D:\\\\SvcFab",
                    "durabilityLevel": "Bronze",
                    "certificate": {
                        "thumbprint": "[parameters('clusterCertificateThumbPrint')]",
                        "x509StoreName": "My"
                    }
                  },
                  "typeHandlerVersion": "1.0"
                }
              },

Finally, under the Service Fabric resource section within the ARM template make sure you specify which certificates to use for node to node security and which is for client to node security.

certificate": {
            "thumbprint": "[parameters('clusterCertificateThumbPrint')]",
            "x509StoreName": "My"
        },
        "clientCertificateCommonNames": [],
        "clientCertificateThumbprints": [{
                    "CertificateThumbprint": "[parameters('adminCertificateThumbPrint')]",
                    "IsAdmin": true
                }],

You should then be able to securely connect to the cluster in the way you are attempting to. Although one thing I have found is that the URL shouldbn't be prefixed with "http" within the publish profile and when trying you browse to the explorer you will need the URL to be https://[n]:19080/Explorer/index.html

Hopefully you will find this of some help.