"Unable to validate data" using FormsAuthentication between different web apps

Question

I have two .NET web applications running on the same server - sentinel (hosted at https://sentinel.mydomain.com/) and fortknox (at http://www.mydomain.com/fortknox)

Sentinel is an authentication 'portal'. FortKnox is a 'proof of concept' app that uses forms authentication but has the loginUrl set to https://sentinel.mydomain.com/login (along with a special Application_EndRequest handler to qualify the ReturnUrl). Sentinel is written in .NET 4.0 using MVC 4 and Razor; FortKnox is ASP.NET MVC 2 using .NET 2.0.

I'm using ASP.NET FormsAuthentication with the cookie domain set to .mydomain.com so that cookies set by sentinel.mydomain.com will be sent with requests to www.mydomain.com, and vice versa. The cookie part is working perfectly - both applications are getting the same .ASPXAUTH encrypted forms ticket.

The problem is that, on our production servers, fortknox can't decrypt cookies created by sentinel - even though they have identical machine keys. Even when both apps are running on the same physical box, it doesn't work.

A user hits fortknox, they're redirected to sentinel, they log in, the cookie is set, they're redirected back to fortknox, and then I get "Unable to validate data":

Exception: Unable to validate data.
  at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, IVType ivType, Boolean useValidationSymAlgo, Boolean signData)
  at System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, IVType ivType, Boolean useValidationSymAlgo)
  at System.Web.Security.FormsAuthentication.Decrypt(String encryptedTicket)
  at FortKnox.Web.MvcApplication.Application_BeginRequest() 

The machine keys are identical - I've gone as far as including this chunk of (nasty!) code in the mark-up of each page:

try {
    var cookie = Request.Cookies[".ASPXAUTH"].Value;
    Response.Write("Cookie: " + cookie + Environment.NewLine);
    var ticket = FormsAuthentication.Decrypt(cookie);
    Response.Write("Ticket name: " + ticket.Name + Environment.NewLine);
} catch (Exception x) {
    Response.Write("Exception: " + x.Message + Environment.NewLine);
    Response.Write(x.StackTrace);
}
Response.Write("<hr /></pre>");
var machineConfigMachineKey = (MachineKeySection)WebConfigurationManager.OpenMachineConfiguration().SectionGroups["system.web"].Sections["machineKey"];
var webConfigMachineKey = (MachineKeySection)WebConfigurationManager.OpenWebConfiguration("").SectionGroups["system.web"].Sections["machineKey"];
Response.Write("<pre>");
Response.Write("<b>machine.config decrypt:  </b>" + machineConfigMachineKey.DecryptionKey + "<br />");
Response.Write("<b>web.config decrypt:      </b>" + webConfigMachineKey.DecryptionKey + "<br />");
Response.Write("<br />");
Response.Write("<b>machine.config validate: </b>" + machineConfigMachineKey.ValidationKey + "<br />");
Response.Write("<b>web.config validate:     </b>" + webConfigMachineKey.ValidationKey + "<br />");
Response.Write("</pre>");
Response.Write("<hr />");

and verified that the machine keys being used at runtime are exactly the same.

What's especially frustrating is that this has been working on our development and staging servers, and has only failed in production. The only difference between the servers is that the production boxes have Windows Updates installed frequently whilst our dev/staging boxes are potentially missing some updates; they're otherwise identical (cloned from the same image and created using the same setup scripts)

So... same server; same machine key. ASP.NET 4 sets a FormsAuthentication cookie. ASP.NET 2 app can't decrypt it. Bug only happening on certain servers; on others, it's working. At this point, I'm completely stuck... any ideas?

EDIT: Live server has been brought right up to the latest patch level. I have tried applying

<add key="aspnet:UseLegacyEncryption" value="true" />

as both true AND false, to both the login app and the fortknox app. Still no luck...

Answer 1

Any chance it has something to do with the old 2010 padding oracle security patch - http://weblogs.asp.net/scottgu/archive/2010/09/28/asp-net-security-update-now-available.aspx? Try setting

<add key="aspnet:UseLegacyEncryption" value="true" />

to force the patched servers to act like they used to before the patch?

(or, you know... patch your servers. Your choice.)