How does Linux execute a file? [closed]

Question

I want to know how does Linux operating systems execute files. So from my searches across the web I understood that every file which has the runnable bit set on can be executed. But then I learned that there is an ELF called format which is the Linux standard for executables.

So what I want to know is what is necessary to a file which has permission to run (runnable bit is on), in order to execute code in the system? Can I just create a new file with hex editor and write 90 inside (NOP opcode) and expect it to be executed? Or does Linux requires some kind of standard format, like ELF format or Bash format?

Answer 1

Or does linux requires some kind of standard format, like ELF format or bash format?

Yes, linux requires file to be in some supported (registered) format and execute bit set in order to execute it. Most files in Linux has either ELF format, or "shebang" format (two first symbols of them are #! and then path to interpreter is written, used by bash, perl, python and most other scripts). Sometimes text files are allowed to execute as shell scripts, e.g. when you do ./script from bash (handled not by kernel, but by bash shell).

More details are available in fs/exec.c file from linux kernel, beginning from do_execve function.

There is kernel subsystem "binfmt" to register other executable formats. For example, binfmt_misc allows you to define and register own binary format via /proc/sys/fs/binfmt_misc special file. The execution is handled via user-defined "interpreter", the program which can read, load and execute target executable. For example, Windows PE binaries may be started with help of wine not-an-emulator.

We can see several builtin binfmt modules in fs directory of kernel sources. Most common are: binfmt_elf.c (ELF binary format) and binfmt_script.c (which detects "shebang" and starts the interpreter). There is simple binary format "a.out" from AT&T, handled by binfmt_aout.c, which can be easier to generate than ELF.

binfmt_aout.c   11374 bytes
binfmt_elf.c    58415 bytes
binfmt_elf_fdpic.c  48256 bytes
binfmt_em86.c   2710 bytes
binfmt_flat.c   27054 bytes
binfmt_misc.c   15175 bytes
binfmt_script.c 2768 bytes
binfmt_som.c    7315 bytes

If the file you try to execute is not of supported format, exec* syscalls will return error:

$ hexdump -C asd
00000000  07 01 09 00 11 12 13 14  0a                       |.........|
00000009
$ strace ./asd
execve("./asd", ["./asd"], [/* 179 vars */]) = -1 ENOEXEC (Exec format error)
....

According to execve man page, the return code means:

ENOEXEC

An executable is not in a recognized format, is for the wrong architecture, or has some other format error that means it cannot be executed.