What's the point of using Busybox in a low ram embedded system

Question

I'm working on bringing linux to a custom Cortex-M7 board with 16 Mb of SDRAM and 64 Mb of Flash. The platform has no-MMU, no shared libraries, FLAT executables.

I'm having problems booting a Busybox system with very simple init.d shell scripts. The system is running out of memory by executing simple shell commands like "[" or "printf". It turns out that everytime one of these commands are executed the system needs to load the FULL, one and only busybox executable (650 Kb on my system).

So the question is: if the system always have to load in memory a huge executable for each command implemented inside busybox, then how is this convenient? I don't get the point of saving some megabytes of cheap and abundant storage while going out of ram extremely fast, but maybe I'm overlooking something.

Is my platform an use case for Busybox? if not, is there anything to conveniently build linux system utilities each on their own executable?

Thanks in advance!

EDIT:

Busybox, according to themselves "has been written with size-optimization and limited resources in mind" and so became a sort of unquestioned de-facto standard in embedded systems. But how their statement relates to the forementioned issues on RAM (not storage) constrained systems? I believe this worths some clarification.

Follow up, system details:

The kernel is already compiled for XIP, executing from the 64 Mb external flash. The entire read/write ext3 root filesystem (including busybox binary) now resides on a micro SD card. Busybox executable is using the FLAT format ("bFLT") with load to RAM bit enabled, that bit seems to be causing a new load on a different memory block each time it runs a concurrent command until it exhausts the fitting blocks. The advice to put busybox (the entire /bin, /sbin) on a XIP filesytem is brilliant and it will surely improve the execution speed (of course, this new filesystem will need to reside on the 64 Mb external flash). I never tried a "bFLT" to be executed in place on such a filesystem (nor have an idea if it works) but I'll do my research/testing about that.

Answer 1

TL-DR; Linux has a huge infrastructure and variety of rootfs or boot file systems available. The choice is due to accommodation of different system constraints and end user functions. Busybox is a good choice for the target system, but any software can be misused if a system engineer doesn't spend time to understand it.

---

Is my platform an use case for Busybox?

It is if you take time to minimize the kernel size and busybox itself. It is unlikely you need all features in your current busybox.

if not, is there anything to conveniently build linux system utilities each on their own executable?

See klibc information below. You can also build dash with musl, with buildroot and busybox. Many filesystem builders support shared libraries or static binaries. However, there are many goals such as package management, and live updates, that a filesystem builder may target.

More Details

You can configure features out of busybox. The idea is that all of the configured features are needed. Therefore you need them all in memory. With busybox, ls, mkdir, printf, etc are all the same binary. So if you run a shell script the one code load is all code loads. The other way, you have many separate binaries and each will take extra memory. You need to minimize Linux to get more RAM and you can take features out of busybox to make it smaller. Busybox is like a giant shared library; or more accurately a shared process. All code pages are the same.

a custom Cortex-M7 board with 16 Mb of SDRAM and 64 Mb of Flash

...

one and only busybox executable (650 Kb on my system)

Obviously 650KB is far less than 16MB. You don't say what the other RAM is used for. For another good alternative look at the klibc toolsuite. What is not clear is whether the FLASH is NAND/NOR and if you have XIP enabled. Generally, busybox would be better with XIP flash and klibc would be better (and more limited) for SDRAM only, with some filesystem in flash.

See: Memory used by relocatable code, PIC, and static linking in the Busybox FAQ. It is designed to run from Read-only memory which can be a big gain depending on system structure. It probably provides a more rich feature set than klibc as the goal with that project is just to boot some other mount device (a hard drive, SSD etc).

Klibc does not have as much documentation as busybox. It can be either a shared library or statically linked. Each binary will only use the RAM needed for that task with static linking, but this will take more flash space. The binary with klibc are,

 1. dash    2. chroot     3. dd      4.  dmesg  5.  mkdir  6.  mkfifo
 7. mknode  8. pivot_root 9. unmount 10. true   11. false  12. sleep
 13. ln    14. ls        15. mv      16. nuke   17. minips 18. cat
 19. uname 20. halt      21. kill    22. cpio   23. sync   24. readlink
 25. gzip  26. losetup

and that is IT! No networking, no media players, etc. You can write code to use klibc, but it is a very constrained library and may not have features that you require. Generally it would be limit to disk only tasks. It is great to probe USB for external device to boot from for example.

Busybox can do a lot more. Most klibc static binaries will be under 100kB; with 10-30kB typical. Dash and gzip are larger. However, I think you need to remove configuration items from your kernel as 650KB << 16MB and busybox would be a fine choice for this system even without XIP.

I should also be noted that Linux does 'demand page loading' for code with an MMU system. Even if you don't have swap, code can be kicked out of RAM and reloaded later with a page fault. Your system is no-MMU, so busybox will not perform as well in this case. With an mmu and 'demand page loading' it will do much better.

For severe constraints, you can always code a completely library free binary. This avoids libgcc startup and support infrastructure which you might not need. Generally, this is only good to test a kernel vs. initrd issue and for script/binary that must run in many different library environments.

See also:

• AXFS - xip read-only file system.
• CramFs - another xip file system.
• XIP kernel - the kernel can be huge. Get it out of RAM if possible. Configure with EMBEDDED option if not.
• nommu.org - some information on github
• elf2flt - Mike Frysingers updates to binutils 2.27-2.31.1
• fdpic gcc - notes from 2016 by mickael-guene.

XIP can only work with ROM, NOR flash and possibly SPI-NOR MTDs.

Answer 2

The point of BusyBox is that you need to have only one (shared) executable in memory.

You wrote:

It turns out that everytime one of these commands are executed the system needs to load the FULL, one and only busybox executable (650 Kb on my system).

That's not entirely true. For the first command the executable is loaded in memory indeed. But if you're running multiple commands (i.e. multiple BusyBox instances), the executable is not loaded into memory multiple times. A large part of the binary (simply said: all read-only data, such as the executable code and constant data) will be reused. Each additional process only requires some additional memory for its own tasks.

So if a single instance of BusyBox consumes 640 kB of memory, it's possible that 10 instances together only use 1 MB of memory, while 10 unique executables of 200 kB would use 2 MB.

I would recommend to do some realistic tests on your system and check the actual memory consumption. But note that tools such as ps or top can be a bit misleading or difficult to understand. A lot has been written about that already, and if you would like to know more about that, a good starting point would be here.