How to make sure a piece of code never leaves the CPU cache (L3)?

Question

The latest Intel's XEON processors have 30MB of L3 memory which is enough to fit a thin type 1 Hypervisor.

I'm interested in understanding how to keep such an Hypervisor within the CPU, i.e. prevented from being flushed to RAM or, at least, encrypt data before being sent to memory/disk.

Assumes we are running on bare metal and we can bootstrap this using DRTM (Late Launch), e.g. we load from untrusted memory/disk but we can only load the real operating system if we can unseal() a secret which is used to decrypt the Operating System and which take place after having set the proper rules to make sure anything sent to RAM is encrypted.

p.s. I know TXT's ACEA aka ACRAM (Authenticated Code Execution Area aka Authentication Code RAM) is said to have such guarantee (i.e. it is restrain to the CPU cache) so I wonder if some trickery could be done around this.

p.p.s. It seems like this is beyond current research so I'm actually not quite sure an answer is possible to this point.

Answer 1

Your question is a bit vague, but it seems to broil down to whether you can put cache lines in lockdown on a Xeon. The answer appears to be no because there's no mention of such a feature in Intel docs for Intel 64 or IA-32... at least for the publicly available models. If you can throw a few million $ at Intel, you can probably get a customized Xeon with such a feature. Intel is into the customized processors business now.

Cache lockdown is typically available on embedded processors. The Intel XScale does have this feature, as do many ARM processors etc.

Do note however that cache lockdown does not mean that the cached data/instructions are never found in RAM. What you seem to want is a form of secure private memory (not cache), possibly at the microcode level. But that is not a cache, because it contradicts the definition of cache... As you probably know, every Intel CPU made in the past decade has updatable microcode, which is stored fairly securely inside the cpu, but you need to have the right cryptographic signing keys to produce code that is accepted by the cpu (via microcode update). What you seem want is the equivalent of that, but at x86/x64 instruction level rather than at microcode level. If this is your goal, then licensing an x86/x64-compatible IP core and adding crypto-protected EEPROM to that is the way to go.

The future Intel Software Guard Extensions (SGX), which you mention in your further comments (after your question, via the Invisible Things Lab link), does not solve the issue of your hypervisor code never being stored in clear in RAM. And that is by design in SGX, so the code can be scanned for viruses etc. before being enclaved.

Finally, I cannot really comment on privatecore's tech because I can't find a real technological description of what they do. Twitter comments and news articles on start-up oriented sites don't provide that and neither does their site. Their business model comes down to "trust us, we know what we do" right now. We might see a real security description/analysis of their stuff some day, but I can't find it now. Their claims of being "PRISM proof" are probably making someone inside the NSA chuckle...

Important update: it's apparently possible to actually disable the (whole) cache from writing back to RAM in the x86 world. These are officially undocumented modes known as "cache-as-RAM mode" in AMD land an "no-fill mode" in Intel's. More at https://www.youtube.com/watch?v=EHkUaiomxfE Being undocumented stuff, Intel (at least) reserves the right to break that "feature" in strange ways as discussed at https://software.intel.com/en-us/forums/topic/392495 for example.

Update 2: A 2011 Lenovo patent http://www.google.com/patents/US8037292 discusses using the newer (?) No-Eviction mode (NEM) on Intel CPUs for loading the BIOS in the CPU's cache. The method can probably be used for other type of code, including supervisors. There's a big caveat though. Code other than the already cached stuff will run very slowly, so I don't see this as really usable outside the boot procedure. There's some coreboot code showing how to enable NEM (https://chromium.googlesource.com/chromiumos/third_party/coreboot/+/84defb44fabf2e81498c689d1b0713a479162fae/src/soc/intel/baytrail/romstage/cache_as_ram.inc)