Exit from an application in an OS without memory separation

Question

I am writing a monolithic OS(It is a joke to call it an OS but it does have very minimal, school level functionalists). When I say monolithic, I meant, it is compiled as a single binary blob and no support for file system etc. Currently I just have a rudimentary simple user space which is nothing but infinite while loop.

I am planning to make my OS little more useful and want to able to write user apps which can terminate like regular apps on a full blown OS.

I don't have glibc or equivalent. My current library in the user space is code which I have written. Now my problem is how to add a framework for user space apps, which would let them terminate in a fix point.

I do know how programs get compiled on regular systems and what happens when a program terminates. However, in my case I don't have luxury to compile programs against libraries and if a program terminates then my instruction pointer just goes on a wild detour.

Currently I am making all apps to make to do a "return call" and I am pre-populating the app stack with a fix address(during launch). Is there any better way to handle the problem?

Along with the answer, I would be more than happy to get clarity on some of OS concepts.

I am working on x86 emulator-platform and compiling my binary statically. (I do have virtual memory support)

Answer 1

Hand crafting the first stack frame with a return into whatever process cleanup code you need to run seems like a perfectly reasonable method. If your OS has "syscalls" then user-space process cleanup code (maybe called exit()) probably ends with a call to the _exit() syscall. You still need to handle the case where the program tries to execute code in ''la-la land'' because that can still happen (however doing that before you have a page-protection system might be a hard problem).