why can't Javascript shellcode exploits be fixed via "data execution prevention"?

Question

The "heap spraying" wikipedia article suggests that many javascript exploits involve positioning a shellcode somewhere in the script's executable code or data space memory and then having interpreter jump there and execute it. What I don't understand is, why can't the interpreter's entire heap be marked as "data" so that interpreter would be prevented from executing the shellcode by DEP? Meanwhile the execution of javascript derived bytecode would be done by virtual machine that would not allow it to modify memory belonging to the interpreter (this wouldn't work on V8 that seems to execute machine code, but probably would work on Firefox that uses some kind of bytecode).

I guess the above sounds trivial and probably something a lot like that is in fact being done. So, I am trying to understand where is the flaw in the reasoning, or the flaw in existing interpreter implementations. E.g. does the interpreter rely on system's memory allocation instead of implementing its own internal allocation when javascript asks for memory, hence making it unduly hard to separate memory belonging to interpreter and to javascript? Or why is it that the DEP based methods cannot completely eliminate shellcodes?

Answer 1

To answer your question we first need to define, Data Execution Prevention, Just In Time Compilation and JIT Spraying.

Data Execution Prevention is a security feature that prohibits the execution of code from a non-executable memory area. DEP can be implemented by hardware mechanisms such the NX bit and/or by software mechanism by adding runtime checks.

Just In Time (JIT) compilers are dynamic compilers that translate byte codes during run time to machine code. The goal is to combine the advantages of interpreted code and the speed of compiled code. It should compile methods only if the extra time spent in compilation can be amortized by the performance gain expected from the compiled code. [1]

JIT spraying is the process of coercing the JIT engine to write many executable pages with embedded shellcode.

[....]

For example, a Javascript statement such as “var x = 0x41414141 + 0x42424242;” might be compiled to contain two 4 byte constants in the executable image (for example, “mov eax, 0x41414141; mov ecx, 0x42424242; add eax, ecx”). By starting execution in the middle of these constants, a completely different instructions stream is revealed.

[....]

The key insight is that the JIT is predictable and must copy some constants to the executable page. Given a uniform statement (such as a long sum or any repeating pattern), those constants can encode small instructions and then control flow to the next constant's location. [2]

Advanced techniques, beyond the scope of this answer, must then be used to find the address of the JIT sprayed block and trigger the exploit.

It should now be clear that

If the attacker’s code is generated by JIT engine it will also reside in the executable area. In other words, DEP is not involved in the protection of code emitted by the JIT compiler. [3]

References

[1] A Dynamic Optimization Framework for a Java Just-in-Time Compiler

[2] Interpreter Exploitation: Pointer Inference and JIT Spraying

[3] JIT spraying and mitigations