How can I use NASM as a library? [duplicate]

Question

I would like to include NASM itself (the assembler) in a C++ project. Can I compile NASM as a shared library? If not, is there another assembler that works as a C or C++ library?

I checked libyasm but couldn't understand how I can use it to assemble my code.

Answer 1

Woah, this exploded when I was away.

I had solved this problem by tampering with the YASM source code, and totally forgot about the question in SO as it received absolutely no attention 8 months ago. Below are the details, followed by a better suggestion.

For the project that I had in mind, I needed to use YASM as a library, and I was in a hurry because I was doing this for a company. Back then there were no good libraries that I was aware of; and I had concluded that getting used to the LLVM framework was an overkill for the task (because all I wanted was to assemble singular x86 - x86_64 instructions and receive the bytes).

So I downloaded the source code for YASM.

Upon meddling with the code for a while, I noticed that the executable receives the file paths for input and output files; and passes these two strings along. I wanted char arrays in memory for the input and output; not files. So I figured, maybe if I could find all FILE pointers that are passed around, I can convert them to char pointers, and change every file read/write to array operations.

This turned out to be even more cumbersome than it sounds. Apparently YASM does not open input/output files once and uses the same FILE pointers; instead it passes around copies of the filepath strings. I needed a script that could make all the necessary changes for me, this wasn't good for me.

Eventually, I found all fopen/fclose calls in the program with a script, and replaced them with my_fopen/my_fclose. For each file that I made these replacements, I included my header file in which I implemented these two functions.

In both of these functions, I checked the incoming string, compared it with "fake_file". If they are equal, I passed a 'fake' FILE pointer pointing to two portions of memory, obtained from the function calls fmemopen and open_memstream. Otherwise I simply called the actual fopen/fclose functions. In other words, I redirected these two calls (only for a given filename) to a memory file. Then, I called the library with the filename parameter set to 'fake_file'.

Since I have had limited myself to Linux at that point, this approach worked for me. I also found out (using Valgrind) that there was a memory leak in the library version, so I wrote a very primitive garbage collector for it. Basically I wrapped malloc's etc. to keep track of all allocations that are not freed, and clean them after each execution.

This approach also allowed me to automate these changes using a script. Unfortunately I did all these in a company so I cannot leak any actual code.

Better suggestion: As of May 31, 2016; you can use Keystone Engine instead. It is "based on LLVM, but it goes much further with a lot more to offer." The disassembly engine Capstone and this are a near perfect couple for assembly and disassembly. If you need either of these components, I suggest these instead of doing the hacks I described. Both of these engines are currently being developed; and even though Keystone has some small bugs, Capstone is very robust at the moment.

TL;DR: Use keystone.