Loading ELF shared library and custom binfmt executable into same Linux address space

Question

I am working on a project to load and run a custom binary format executable (PE, in my case) on a Linux platform. I've done this pretty successfully so far by first loading the executable and then loading a small ELF shared library that calls the start address of the executable and then exits safely.

I would really like not doing the ELF loading myself for a few reasons, though. First, the shared library I use is written in assembly (I can't use anything else because I'm not linking to libc, etc.), which will be very platform-specific, and I'd like to move away from that and use C so I can compile for any platform. Also, it will be easier and safer to use Linux's native ELF loader instead of my own simplified version.

I'm wondering if there is a way to use my binfmt handler, an installed kernel module, to load my executable and then ask Linux to load my shared library (and its dependencies) into the same address space without overwriting my executable code. I first thought that the uselib syscall might be useful, but the description on the man page is unclear about whether or not this will serve my purposes:

From libc 4.4.4 on only the library "/lib/ld.so" is loaded, so that this dynamic library can load the remaining libraries needed (again using this call). This is also the state of affairs in libc5.

glibc2 does not use this call.

I've also never seen an example of its use, and I'm always wary of using syscalls that I don't understand.

Is there a good way to achieve what I've described? Can I use Linux's existing capabilities to load a shared library (written in C) into an address space already containing executable code, and, if so, how can I use that library without knowing where it has been loaded?

Answer 1

there is already a project like this called binfmt_pe (by me!) which is a kernel module and will have it's own linker (similar to /lib/ld). check it out here.

As for your question about making modules and the loader/linker, there are links below. I also included links with info about ELF and PE executables.

I hope this helps. :)

Useful information for making a Linux Kernel Module

• The Linux Kernel Module Programming Guide
• Writing Your Own Loadable Kernel Module
• Linux Data Structures
• The Linux kernel: The kernel source
• Kernel Support for miscellaneous Binary Formats
• binfmt_elf.c
• binfmt_misc.c

Information About Dynamnic Loading/Linking

• Understanding ld-linux.so.2
• ld.so : Dynamic-Link Library support
• How To Write Shared Libraries - PDF
• ld-linux(8) - Linux man page
• rpath
• Listing Shared Library Dependencies

Information About ELF and PE Formats

• OSRC: Executable File Formats
• EXE Format
• How Windows NT Recognizes MS-DOS - Based Applications
• Common Object File Format (COFF)
• Peering Inside the PE: A Tour of the Win32 Portable Executable File Format
• Executable and Linkable Format
• An In-Depth Look into the Win32 Portable Executable File Format
• An In-Depth Look into the Win32 Portable Executable File Format, Part 2
• Injective Code Inside an Import Table
• The PE Format | Hackers Library
• IMAGE_NT_HEADERS structure (Windows)
• x86 Disassembly/Windows Executable Files
• the Portable Executable Format on Windows