Identical (almost) ELF headers but executables won't run on each other's system

Question

I'm trying to compile a couple of programs for a little embedded device I own. It's a Little-endian MIPS (mipsel) processor. I retrieved this executable from it via telnet and the builtin ftp client:

root@debian-mipsel:/home/user/wansview/devel# readelf -h unzip1 
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           MIPS R3000
  Version:                           0x1
  Entry point address:               0x401cc0
  Start of program headers:          52 (bytes into file)
  Start of section headers:          169960 (bytes into file)
  Flags:                             0x10001007, noreorder, pic, cpic, o32, mips2
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         6
  Size of section headers:           40 (bytes)
  Number of section headers:         24
  Section header string table index: 23

root@debian-mipsel:/home/user/wansview/devel# file unzip1
unzip1: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), dynamically linked (uses shared libs), stripped

I then downloaded the MIPSEL version of Debian and I'm running it in QEMU. When I run the retrieved program above I get:

root@debian-mipsel:/home/user/wansview/devel# ./unzip1
-bash: ./unzip1: No such file or directory

Which I understand means it's not the right platform. Stubbornly I compiled a little hello world nonetheless to compare the ELF and file info. My hello world runs fine in Debian MIPSEL but also returns No such file or directory on the embedded device. It's readelf and file output is strikingly similar though:

root@debian-mipsel:/home/user/wansview/devel# readelf -h hello
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           MIPS R3000
  Version:                           0x1
  Entry point address:               0x400740
  Start of program headers:          52 (bytes into file)
  Start of section headers:          3652 (bytes into file)
  Flags:                             0x10001005, noreorder, cpic, o32, mips2
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         10
  Size of section headers:           40 (bytes)
  Number of section headers:         36
  Section header string table index: 35

root@debian-mipsel:/home/user/wansview/devel# file hello
hello: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0xeb3877062337a3dfd15cc09305691685ac0e8c57, with unknown capability 0xf41 = 0x756e6700, with unknown capability 0x70100 = 0x1040000, stripped

I'm trying to better understand how my two systems differ and why the executables won't run on both. Are there any flags I could add to gcc to successfully compile for the embedded device?

More info about the device

# cat /proc/cpuinfo
system type             : Ralink SoC
processor               : 0
cpu model               : MIPS 24K V4.12
BogoMIPS                : 239.10
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 32
extra interrupt vector  : yes
hardware watchpoint     : yes
ASEs implemented        : mips16 dsp
VCED exceptions         : not available
VCEI exceptions         : not available

More info about Debian MIPSEL

(Binaries compiled on debian-mipsel won't run on the target embedded device)

root@debian-mipsel:/home/user/wansview/devel# cat /proc/cpuinfo 
system type     : MIPS Malta
processor       : 0
cpu model       : MIPS 24Kc V0.0  FPU V0.0
BogoMIPS        : 1038.33
wait instruction    : yes
microsecond timers  : yes
tlb_entries     : 16
extra interrupt vector  : yes
hardware watchpoint : yes, count: 1, address/irw mask: [0x0ff8]
ASEs implemented    : mips16
shadow register sets    : 1
kscratch registers  : 0
core            : 0
VCED exceptions     : not available
VCEI exceptions     : not available

More info about Aboriginal Linux Mipsel

(Binaries compiled on Aboriginal Linux will run on the embedded device, and it can run binaries retrieved from the device. I'm not happy with it as it doesn't have make and other tools I need for compiling larger applications)

(mipsel:1) /home/wansview # cat /proc/cpuinfo 
system type     : MIPS Malta
machine         : Unknown
processor       : 0
cpu model       : MIPS 24Kc V0.0  FPU V0.0
BogoMIPS        : 1013.76
wait instruction    : yes
microsecond timers  : yes
tlb_entries     : 16
extra interrupt vector  : yes
hardware watchpoint : yes, count: 1, address/irw mask: [0x0ff8]
isa         : mips1 mips2 mips32r1 mips32r2
ASEs implemented    : mips16
shadow register sets    : 1
kscratch registers  : 0
core            : 0
VCED exceptions     : not available
VCEI exceptions     : not available

LDD

Here's a screenshot with ldd ran against my hello world and against unzip1 on both aboriginal linux and debian mipsel. Aboriginal Linux runs applications retrieved from the device just fine, and if I compile under Aboriginal Linux I can run the resulting binary on the embedded device. The reason I'm not content with Aboriginal is that doesn't have GNU make and other useful tools for larger applications, and no easy way to get them there.

Answer 1

You clearly need a different toolchain. On your Debian-mipsel, your toolchain uses glibc while your target uses uClibc.

So, maybe you would like to generate it by yourself using Buildroot:

wget http://buildroot.uclibc.org/downloads/buildroot-2014.11.tar.gz
tar zxf http://buildroot.uclibc.org/downloads/buildroot-2014.11.tar.gz
cd buildroot-2014.11

A trick to preconfigure for mipsel, R1 without soft-float (my will, check yours):

cat <<_EOF > .config
BR2_HAVE_DOT_CONFIG=y
BR2_mipsel=y
BR2_ARCH="mipsel"
BR2_ENDIAN="LITTLE"
BR2_GCC_TARGET_ARCH="mips32"
BR2_GCC_TARGET_ABI="32"
BR2_ARCH_HAS_ATOMICS=y
BR2_mips_32=y
# BR2_MIPS_SOFT_FLOAT is not set
BR2_MIPS_OABI32=y
_EOF

Finish your choice in Buildroot menuconfig, but you can also keep it like this with save and exit.

make menuconfig   # tweak options at your will,
make -j8          # takes 8 minutes on my machine

Then, your compiler can be found in ./output/host/usr/bin

A real example:

echo '#include <stdio.h>                                                                                                                                                                   
 int main(int argc, char* argv[]) {
     printf("Hello World.\n");
     return 0;
 }' > hello.c

And compile it with your brand-new uClibc GCC compiler

output/host/usr/bin/mipsel-buildroot-linux-uclibc-gcc -o hello hello.c

A glimpse into the hello program: (didn't had time to fix my ldd...)

$ file hello
hello: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), dynamically linked (uses shared libs), not stripped
$ strings hello | grep "lib.*so*"
/lib/ld-uClibc.so.0
libgcc_s.so.1
libc.so.0

It's done with the toolchain and compile your program. Now that you have time :-) see what Buildroot offers : a complete distribution (in output/target/) for embedded systems in many architectures.

EDIT: Better chance to execute

You can statically link your program in order to maximize the chances to run your code on any target.

$ output/host/usr/bin/mipsel-linux-gcc -Wall -o hello -static hello.c
$ file ./hello
./hello: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), dynamically linked (uses shared libs), not stripped

And now, as that static version doesn't depend anymore on any external lib (only uClibc, here), this MIPS executable can even run on my x86_64 machine (thanks to binfmt and Qemu) :

$ uname -mo   
x86_64 GNU/Linux
$ ./hello
Hello World.

Cheers.