How to sign code built using Azure Pipelines using a certificate/key in Azure Key Vault?

Question

We're in the process of moving from on-premise build servers to Azure Pipelines. We produce "shrink-wrap" desktop software so clearly we need to sign all our binaries before releasing. Our current build infrastructure does this using a USB hardware token from GlobalSign, but clearly that isn't going to work when we're doing cloud builds - sadly, clouds are not equipped with USB ports :D

Now, GlobalSign has recently started advertising Azure Key Vault as a key storage option, and they're perfectly happy to sell this to us, but I'm not sure how we'd actually integrate that with our build pipelines (or indeed whether that's even possible).

Has anyone actually made this work?

Answer 1

Code Signing

I've been battling with Azure Key Vault and Azure Pipelines to get our code signed, and succeeded. So here's what I found out.

Critically, Extended Validation (EV) certificates used for code signing are very different animals to 'normal' SSL certificates. The standard ones can be exported as much as you like, which means you can upload it to Azure Pipelines and use it with the standard Microsoft Sign Tool.

However, once an EV certificate is in Azure Key Vault, it isn't coming out in any usual fashion. You must call it from Pipelines using the excellent Azure Sign Tool as discovered by Anodyne above

Get your certificate into Key Vault. You can use any certificate authority you like to generate the certificate, as long as they understand that you'll need an EV certificate, and critically one that has a hardware security module (HSM), and not one with a physical USB key. Any cloud based system like Key Vault will need an HSM version.

To get the permissions to access this certificate externally you can follow this page but beware it misses a step. So read that document first, then these summarised steps, to get the Key Vault set up:

1. Open the Azure portal, go to the Azure Active Directory area, and create an App registration: put in a memorable name, ignore the Redirect URI, and save it.
2. Go to your specific Key Vault, then Access control (IAM), then Add role assignment. Type the name of the app you just created into the select input box. Also choose a Role, I suggest Reader and then save.
3. The Missing Part: Still in the Key Vault, click the Access policies menu item. Click Add Access Policy and add your application. The Certificate Permissions need to have the Get ticked. And the Key Permissions, despite the fact that you may not have any keys at all in this vault, need to have Get and Sign. You would have thought these two would be in the certificate perms...
4. Go back to the application you just created. Select the Certificates & secrets, and either choose to upload a certificate (a new one purely for accessing the Key Vault remotely) or create a client secret. If the latter, keep a copy of the password, you won't see it again!
5. In the Overview section of the app will be the Application (client) ID. This, and the password or certificate, is what will be fed to the Azure Sign Tool later on in a Pipelines task.

Handling the actual code signing from Azure requires a number of steps. The following applies to Microsoft hosted agents, although similar issues will affect any private agents that you have.

1. The Azure Sign Tool needs the .NET Core SDK to be installed, but a version that's at least version 2.x, and since the latest .NET Core SDK is always used, this means as long as the version of Windows is current enough, you don't need to install it yourself. And you can see which version of the SDK is shipped with which Windows agent.

   The current Hosted OS version in Azure Pipelines, also called Default Hosted, is, at the time of writing, Windows Server 2012 R2. Which isn't up to date enough. Installing a newer .NET Core SDK to overcome this is a time drag on every build, and although the installation works, calling the Azure Sign Tool may not work. It seems to be finding only older versions of the SDK, and throws this error: Unable to find an entry point named 'SignerSignEx3' in DLL 'mssign32'.

   So the easiest thing to do is change your build to use a later OS image. Windows 2019 works like a charm. And there is no need to install any version of .NET Core.

2. Then create a command line task to install the Azure Sign Tool. You can use a .NET Core CLI task as well, but there is no need. In the task, type this:

   set DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true dotnet tool install --global AzureSignTool --version 2.0.17 

   Naturally using whichever version that you want.

   The DOTNET_SKIP_FIRST_TIME_EXPERIENCE environment variable isn't strictly necessary, but setting it speeds things up quite a bit (see here for an explanation).

3. Finally, create another command line task and type in the Azure Sign Tool command that you wish to run with. On Windows this would be something like below, note with ^ not / as a line continuation marker. Naturally, see here for more parameter information:

    AzureSignTool.exe sign -du "MY-URL" ^    -kvu https://MY-VAULT-NAME.vault.azure.net ^    -kvi CLIENT-ID-BIG-GUID ^    -kvs CLIENT-PASSWORD ^    -kvc MY-CERTIFICATE-NAME ^    -tr http://timestamp.digicert.com ^    -v ^    $(System.DefaultWorkingDirectory)/Path/To/My/Setup/Exe 

And in theory, you should have success! The output of the sign tool is rather good, and usually nails where the problem is.

Re-issuing Certificates

If you need to re-issue a certificate, the situation is quite different.

1. In Azure, go to the certificate and click on it, opening a page showing the versions of that certificate, both current and older versions.

2. Click the 'New Version' button, probably accepting the defaults (depending on the choices you wish to make) and click 'Create'.

3. This takes you back to the Versions page, and there will be a message box stating 'The creation of certificate XXXX is currently pending'. Click there (or on the 'Certificate Operation' button) to open the 'Certificate Operation' side page. Once there, download the CSR (certificate signing request).

4. In GlobalSign, follow their instructions to re-issue the existing certificate. Once it has been re-issued, they will send an email describing how to download it.

5. Log into GlobalSign again, and after entering the temporary password, open the CSR and copy the whole text (which starts with -----BEGIN CERTIFICATE REQUEST-----) into GlobalSign. Submit it.

6. Download using the 'Install My Certificate' button. Then in the Azure 'Certificate Operation' side page - use the 'Merge Signed Request' button that to upload the .CER file to Azure. This creates the new version of the certificate.

7. Disable the old version of the certificate.

Answer 2

Yes, it's able to do this in Azure DevOps Service Build pipeline.

For normal situation, we usually use SignTool.exe commands to sign files. There is also an extension Code Signing in marketplace, which could sign a single file, you could use script to run SignTool.exe commands for multiple files.

So you can export your codesigning certificate to a pfx file, which you then upload as a secure file to Azure Devops secure file storage which makes it available to your builds.

Azure DevOps could store secure files. Check this link for details: Secure files

Azure Key Vault instance is kind of more complicated. We also have an Azure Key Vault task.

Use this task in a build or release pipeline to download secrets such as authentication keys, storage account keys, data encryption keys, .PFX files, and passwords from an Azure Key Vault instance.

The task can be used to fetch the latest values of all or a subset of secrets from the vault, and set them as variables that can be used in subsequent tasks of a pipeline.

Not sure how GlobalSign will integrate code sign with your environment. Theoretically, it's able to do this. For the detail parts and implementation, you may need to discuss with their pre-sales. Hope this helps.