gcc - how to detect pointer-based memory access

Question

I'm focusing on micropython, specifically the branch dynamic-native-modules.

This feature will, in the future, allow you to compile a C/C++ function into a native .obj and package it together with a .py interface for a huge speed boost.

Awesome! But the issue is that if you're using a RTOS, which doesn't have virtual memory, then any executing native code can access any part of the address space including peripherals, the RTOS' state etc.

You don't want the user to be able to do something like this:

void user_func()
{
/* point to arbitrary memory, potentially the reset registers, flash erase . . . you get the point */
  int * a = (int*)0x1234;  
  *a = 0x10110000; // DESTROY!!!
}

Even the following should be disallowed:

void user_func()
{
  int a;
  (int*)(&a-1000) = 0x10010111;
}

SOLUTIONS?

• Create own version of gcc (for each binary format)
• Decompile .obj files and detect use of pointers (for each binary binary format)

FEEDBACK TO COMMENTS

I get that it may be impossible to stop a malicious user but that's not the #1 worry. We want to stop well-meaning but accidental code. If it's not possible to stop every, single case that's ok.

If we can prohibit/detect explicit pointer accesses and simply provide warnings regarding array use, that is still very valuable.

WARNING: YOU'RE USING AN ARRAY! MAKE SURE YOU DON'T GO OUT-OF-BOUNDS

Answer 1

Your best chance is a GCC plugin which looks at the frontend-generated GENERIC or GIMPLE IRs and implements the policies you want. Depending on the policies and the source code you want to accept, this could be a lot of work and very difficult.

If you want a purely syntax-based or type-based approach (simply rejecting all pointer arithmetic), Clang with its ASTs is easier to work with than GCC.