best practice for delivering a C API hiding internal functions [closed]

Question

I have written a C library which consists in a few .h files and .c files. I compile it as a .a static library.

I would like to expose only certain functions to the user and keep the rest as "obscure" as possible to make reverse engineering reasonably difficult.

Ideally my library would consist of:

1- one .h file with only the functions exposed to the user

2- myLibrary.a: as un-reversengineerable as possible

What are the best practices for that? Where should I look, is there a good tutorial/book somewhere?

More specifically:

for - 1

I already have all my .h and .c working and I would like to avoid changing them around, moving function declarations from .h to .c and go into circular references potential pbs. Is That possible?

For instance is it a good idea to create a new .h file which I would use only for distributing with my .a? That .h would contain copies of the functions I want to expose and forward declarations of types I use. Is that a good idea?

for - 2

a) what gcc flags (or xcode) shall I be aware of (for stripping, not having debug symbols etc) b) a good pointer to learn about how to do code obfuscation?

Any thought will help,

Thanks, baba

Answer 1

The usual practice is to make sure that every function and global variable that is for use only internal to some module is declared static in that module. That limits exposure of internal implementation details from a single module.

If you need internal implementation details that cross between modules, but which are not for public consumption, then declare one or more .h files that are kept private and not delivered to end users. The names of objects defined in that way will still be visible to the linker (and to tools such as objdump and nm) but their detailed signatures will not be.

If you have data structures that are delivered to the end user, but which are opaque, then consider having the API deliver them as pointers to a struct that is declared by not defined in the public API .h file. That will preserve type safety, while concealing the implementation details. Naturally, the complete struct definition is in a private .h file.

With care, you can keep a partially documented publicly known struct that is a type-pun for the real definition but which only exposes the public members. This is more difficult to keep up to date, and if you do it, I would make certain that there are some strong test cases to validate that the public version is in fact equivalent to the private version in all ways that matter.

Naturally, use strip to remove the debug segments so that the internal details are not leaked that way.

There are tools out there that can obfuscate all the names that are intended to be only internal use. If run as part of the build process, you can work with an internal debug build that has sensible names for everything, and ship a build that has named all the internal functions and global variables with names that only a linker can love.

Finally, get used to the fact that anyone that can use your library will be able to reverse engineer your library to some extent. There are anti-debugger measures that can be taken, but IMHO that way lies madness and frustration.