Is there a way to disable inline assembler in GCC?

Question

I'm developing an online judge system for programming contests like LeetCode, Codeforces, etc. As for most programming contests, inline assembler is not allowed in C/C++, so I would like to add the same restriction to my system.

I would like to let GCC and G++ produce an error when compiling a C/C++ program containing inline assembler, so that any code containing inline assembler will result in compilation error.

Is there a way to achieve that? Should I pass some command line arguments to GCC/G++?

Note: disabling inline assembler is just for obeying the rules, not for security concerns.

Answer 1

Is there a way to disable inline assembler in GCC?

Yes there are a couple of methods.

Turn off assembly in the compiler

To do it in compilation phase, use the parameter -fno-asm. However, keep in mind that this will only affect asm and not __asm__.

Documentation:

-fno-asm

Do not recognize "asm", "inline" or "typeof" as a keyword, so that code can use these words as identifiers. You can use the keywords "__asm__", "__inline__" and "__typeof__" instead. -ansi implies -fno-asm.

In C++ , this switch only affects the "typeof" keyword, since "asm" and "inline" are standard keywords. You may want to use the -fno-gnu-keywords flag instead, which has the same effect. In C99 mode (-std=c99 or -std=gnu99), this switch only affects the "asm" and "typeof" keywords, since "inline" is a standard keyword in ISO C99.

Define a macro

You can use the parameters -Dasm=error -D__asm__=error

Note that this construction is generic. What it does is to create macros. It works pretty much like a #define. The documentation says:

-D name=definition

The contents of definition are tokenized and processed as if they appeared during translation phase three in a #define directive. In particular, the definition will be truncated by embedded newline characters.

...

So what it does is simply to change occurrences of asm or __asm__ to error. This is done in the preprocessor phase. You don't have to use error. Just pick anything that will not compile.

Use a macro that fires during compilation

A way to solve it in compilation phase by using a macro, as suggested in comments by zwol, you can use -D'asm(...)=_Static_assert(0,"inline assembly not allowed")'. This will also solve the problem if there exist an identifier called error.

Note: This method requires -std=c11 or higher.

Using grep before using gcc

Yet another way that may be the solution to your problem is to just do a grep in the root of the source tree before compiling:

grep -nr "asm"

This will also catch __asm__ but it may give false positives, for instance is you have a string literal, identifier or comment containing the substring "asm". But in your case you could solve this problem by also forbidding any occurrence of that string anywhere in the source code. Just change the rules.

Possible unexpected problems

Note that disabling assembly can cause other problems. For instance, I could not use stdio.h with this option. It is common that system headers contains inline assembly code.

A way to cheat above methods

It is possible to execute strings as machine code. See this answer for an example: https://stackoverflow.com/a/18477070/6699433

A piece of the code from the link above:

/* our machine code */
char code[] = {0x55,0x48,0x89,0xe5,0x89,0x7d,0xfc,0x48,
0x89,0x75,0xf0,0xb8,0x2a,0x00,0x00,0x00,0xc9,0xc3,0x00};

/* copy code to executable buffer */    
void *buf = mmap (0,sizeof(code),PROT_READ|PROT_WRITE|PROT_EXEC,
            MAP_PRIVATE|MAP_ANON,-1,0);
memcpy (buf, code, sizeof(code));

/* run code */
int i = ((int (*) (void))buf)();

The code above is only intended to give a quick idea of how to trick the rules OP has stated. It is not intended to be a good example of how to actually perform it in reality. Furthermore, the code is not mine. It is just a short code quote from the link I supplied. If you have ideas about how to improve it, then please comment on 4pie0:s original post instead.