Why can't I encrypt web.config/appSettings using a custom configProtectionProvider on Azure (Windows Server 2012)?

Question

I have an MVC application which is deployed to a Windows Azure hosted service running on a Windows Server 2012 virtual machine. In the web.config file, I have 3 sections encrypted using the PKCS12ProtectedConfigurationProvider: connectionStrings, dataCacheClients, and system.net/mailSettings/smtp. Here is what the relevant sections look like:

<configuration>
  ...
  <configProtectedData>
    <providers>
      <add name="CustomProvider" thumbprint="[this is secret]"
          type="Pkcs12ProtectedConfigurationProvider.Pkcs12ProtectedConfigurationProvider, PKCS12ProtectedConfigurationProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=34da007ac91f901d" />
    </providers>
  </configProtectedData>
  ...
  <connectionStrings configProtectionProvider="CustomProvider">
    <EncryptedData ... ommitted for brevity
  </connectionStrings>
  ...
  <system.net>
    <mailSettings>
      <smtp configProtectionProvider="CustomProvider">
        <EncryptedData ommitted for brevity
      </smtp>
    </mailSettings>
  </system.net>
  ...
  <dataCacheClients configProtectionProvider="CustomProvider">
    <EncryptedData ommitted for brevity
  </dataCacheClients>
  ...
</configuration>

All of the above works perfectly fine. When deployed to Azure, the connection strings, SMTP mail, and data cache all work. The PKCS12ProtectedConfiguration provider uses my custom certificate to decrypt the sections and all is well.

However I can't seem to use the same approach to encrypt web.config/appSettings. When I try deploying something like the following to Azure...

<configuration>
  ...
  <appSettings configProtectionProvider="CustomProvider">
    <EncryptedData ommitted for brevity
  </appSettings>
  ...
</configuration>

...then I get the following exception:

Server Error in '/' Application.

Configuration Error

Description: An error occurred during the processing of a configuration file
required to service this request. Please review the specific error details
below and modify your configuration file appropriately.

Parser Error Message: An error occurred loading a configuration file: Could not
load file or assembly 'PKCS12ProtectedConfigurationProvider, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=34da007ac91f901d' or one of its dependencies.
The system cannot find the file specified.

Source Error: 

Line 41:     </EncryptedData>
Line 42:   </connectionStrings>
Line 43:   <appSettings configProtectionProvider="CustomProvider">
Line 44:     <EncryptedData ...>
Line 45:       <EncryptionMethod .../>

Source File: E:\sitesroot\0\web.config    Line: 43 

Assembly Load Trace: The following information can be helpful to determine why
the assembly 'PKCS12ProtectedConfigurationProvider, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=34da007ac91f901d' could not be loaded.

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value
[HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure
logging.
To turn this feature off, remove the registry value
[HKLM\Software\Microsoft\Fusion!EnableLog].

However I know the PKCS12ProtectedConfigurationProvider.dll is in the /bin folder because:

1. I remoted into the VM and saw it in both approot/bin and siteroot/bin
2. When I deploy with the appSettings unencrypted, the other 3 sections are successfully decrypted using this dll.

It's almost as if the assembly loader can't look for the PKCS12ProtectedConfigurationProvider.dll file until after it parses and loads the appSettings section. I have tried omitting the Version, Culture, and PublicKeyToken parts of the provider config section, but then the error just changes to this:

Parser Error Message: An error occurred loading a configuration file: Could not 
load file or assembly 'PKCS12ProtectedConfigurationProvider' or one of its 
dependencies. The system cannot find the file specified.

Is it possible to encrypt web.config/appSettings using a custom configProtectionProvider when deploying to a Windows Azure hosted service running on a Windows Server 2012 virtual machine? If so, what am I missing here?

Update:

After posting this I turned on the Fusion!EnableLog registry key and now I get this additional information in the exception:

Assembly Load Trace: The following information can be helpful to determine why
the assembly 'PKCS12ProtectedConfigurationProvider, Version=1.0.0.0, 
Culture=neutral, PublicKeyToken=34da007ac91f901d' could not be loaded.

=== Pre-bind state information ===
LOG: User = NT AUTHORITY\NETWORK SERVICE
LOG: DisplayName = PKCS12ProtectedConfigurationProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=34da007ac91f901d
 (Fully-specified)
LOG: Appbase = file:///d:/windows/system32/inetsrv/
LOG: Initial PrivatePath = NULL
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: No application configuration file found.
LOG: Using host configuration file: D:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet.config
LOG: Using machine configuration file from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: PKCS12ProtectedConfigurationProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=34da007ac91f901d
LOG: Attempting download of new URL file:///d:/windows/system32/inetsrv/PKCS12ProtectedConfigurationProvider.DLL.
LOG: Attempting download of new URL file:///d:/windows/system32/inetsrv/PKCS12ProtectedConfigurationProvider/PKCS12ProtectedConfigurationProvider.DLL.
LOG: Attempting download of new URL file:///d:/windows/system32/inetsrv/PKCS12ProtectedConfigurationProvider.EXE.
LOG: Attempting download of new URL file:///d:/windows/system32/inetsrv/PKCS12ProtectedConfigurationProvider/PKCS12ProtectedConfigurationProvider.EXE.

So here's another question: Why is IIS looking in the inetsrv path for this assembly instead of looking for it in the app's /bin folder? I see in the log "No application configuration file found." Does this mean the assembly binder has to find a web.config with an unencrypted appSettings section in order to parse and load it?

Answer 1

Did you attempt installing the assembly in the GAC yet on role start?

Information on adding assemblies to the gac on start up http://blogs.infosupport.com/adding-assemblies-to-the-gac-in-windows-azure/