How to write your own code generator backend for gcc?

Question

I have created my very own (very simple) byte code language, and a virtual machine to execute it. It works fine, but now I'd like to use gcc (or any other freely available compiler) to generate byte code for this machine from a normal c program. So the question is, how do I modify or extend gcc so that it can output my own byte code? Note that I do NOT want to compile my byte code to machine code, I want to "compile" c-code to (my own) byte code.

I realize that this is a potentially large question, and it is possible that the best answer is "go look at the gcc source code". I just need some help with how to get started with this. I figure that there must be some articles or books on this subject that could describe the process to add a custom generator to gcc, but I haven't found anything by googling.

Answer 1

I am busy porting gcc to an 8-bit processor we design earlier. I is kind of a difficult task for our machine because it is 8-bit and we have only one accumulator, but if you have more resources it can became easy. This is how we are trying to manage it with gcc 4.9 and using cygwin:

1. Download gcc 4.9 source
2. Add your architecture name to config.sub around line 250 look for # Decode aliases for certain CPU-COMPANY combinations. In that list add | my_processor \
3. In that same file look for # Recognize the basic CPU types with company name. add yourself to the list: | my_processor-* \

4. Search for the file gcc/config.gcc, in the file look for case ${target} it is around line 880, add yourself in the following way:

     ;; my_processor*-*-*)   c_target_objs="my_processor-c.o"   cxx_target_objs="my_processor-c.o"   target_has_targetm_common=no   tmake_file="${tmake_file} my_processor/t-my_processor"   ;; 

5. Create a folder gcc-4.9.0\gcc\config\my_processor
6. Copy files from an existing project and just edit it, or create your own from scratch. In our project we had copied all the files from the msp430 project and edited it all
7. You should have the following files (not all files are mandatory):
   • my_processor.c
   • my_processor.h
   • my_processor.md
   • my_processor.opt
   • my_processor-c.c
   • my_processor.def
   • my_processor-protos.h
   • constraints.md
   • predicates.md
   • README.txt
   • t-my_processor
8. create a path gcc-4.9.0/build/object
9. run ../../configure --target=my_processor --prefix=path for my compiler --enable-languages="c"
10. make
11. make install
12. Do a lot of research and debugging.
13. Have fun.

Answer 2

It is hard work.

For example I also design my own "architecture" with my own byte code and wanted to generate C/C++ code with GCC for it. This is the way how I make it:

1. At first you should read everything about porting in the manual of GCC.
2. Also not forget too read GCC Internals.
3. Read many things about Compilers.
4. Also look at this question and the answers here.
5. Google for more information.
6. Ask yourself if you are really ready.
7. Be sure to have a very good cafe machine... you will need it.
8. Start to add machine dependet files to gcc.
9. Compile gcc in a cross host-target way.
10. Check the code results in the Hex-Editor.
11. Do more tests.
12. Now have fun with your own architecture :D

When you are finished you can use c or c++ only without os-dependet libraries (you have currently no running OS on your architecture) and you should now (if you need it) compile many other libraries with your cross compiler to have a good framework.

PS: LLVM (Clang) is easier to port... maybe you want to start there?