How do I check if an object file is in COFF or ELF format from within C?

Question

I would like to be able to check the format of an object file from within my C code so that I can use different functions for reading the file based on whether it is COFF or ELF format.

Is this possible, and if so, how would I do it?

This is in Linux btw if that makes any difference.

Answer 1

Read the first four bytes. If they are equal to \x7fELF, it's an ELF file. Otherwise, you should parse it as COFF and see if it makes sense. (Note that COFF magic is a lot more complicated; I get no less than 42 magic entries in /usr/share/file/magic for it).