"Illegal instruction" when cross-compiling Qt 4.7

Question

I'm struggling with this problem for more than a week now, and still can't find a solution...

I'm trying to cross-compile Qt 4.7 embedded open-source version for an ARM device. The build process itself completes without problems, but the generated binaries seem to contain instructions that the processor does not understand.

• Build host is Debian 5 (Etch) on i386 (running on a virtual PC)
• The device is a Trimble Nomad handheld with an ARM processor (see full cpuinfo and kernel configuration)
• I use the original build toolchain that was made for the device and that worked fine to date (even could build Gnash successfully) - see compiler settings and version
• I'm using a custom qmake.conf based on linux-arm-gnueabi-g++ and adapted to use the correct toolchain - see source code here
• I had a partial improvement by adding -msoft-float -D__GCC_FLOAT_NOT_NEEDED to the compiler flags but I still get "Illegal instruction" errors in some situations (but at least this was a big improvement)
• The binaries themselves basically work, but in certain situations the program crashes with the "Illegal instruction" error. I believe this happens during certain floating point operations while doing graphics stuff.
• Adding -mcpu=xscale, -march=armv4, -O0, -march=armv4, -mtune=arm920t (not all at the same time) did not help in any way.
• Building Qt with the --debug flag appears to resolve all problems but adding the -O2 flag reintroduces them. Strangely the -O0 setting without --debug does not help.
• The compilete configure and make output can be seen here. There are lots of alignment warnings but they are said to be false warnings of the compiler.
• there must have been some change in Qt 4.7.2 because earlier versions (4.7.1, 4.7.0) do run fine.

configure settings:

  ./configure \
    -embedded arm \
    -xplatform qws/linux-arm-angstrom-gnueabi-g++ \
    -debug \
    -no-largefile \
    -no-multimedia \
    -no-audio-backend \
    -no-phonon \
    -no-phonon-backend \
    -webkit \
    -javascript-jit \
    -no-xshape \
    -no-xvideo \
    -no-xsync \
    -no-xinerama \
    -no-xcursor \
    -no-xfixes \
    -no-xrandr \
    -no-xrender \
    -no-xinput \
    -no-xkb \
    -no-opengl \
    -nomake docs \
    -nomake examples \
    -nomake tools \
    -nomake demos \
    -nomake translations \
    -opensource \
    -qt-mouse-tslib \
    -qt-libjpeg \
    -qt-gif 

strace before the crash:

$ LD_LIBRARY_PATH=. QT_QWS_FONTDIR=$PWD/fonts QT_PLUGIN_PATH=$PWD/plugins QWS_MOUSE_PROTO=tslib:/dev/input/touchscreen0 strace ./digitalclock  -qws test.htm
...
lseek(15, 0, SEEK_END)                  = 16998
write(15, "\t\n\f\0\367\t", 6)          = 6
write(15, "\0\0+\234\325\343\306{\3\0\0\0\0J\370\377\351\301\336\377"..., 120) = 120
lseek(15, 0, SEEK_END)                  = 17124
write(15, "\10\10\10\0\371\10", 6)      = 6
write(15, "\0\6j\251\260\201\27\0\2\276\377\351\334\377\346\32K\377"..., 64) = 64
lseek(15, 0, SEEK_END)                  = 17194
write(15, "\7\10\10\0\371\7", 6)        = 6
write(15, "\0\4c\245\263\224 \0\1\271\377\367\315\356P\0I\377\364"..., 64) = 64
lseek(15, 0, SEEK_END)                  = 17264
write(15, "\10\n\10\1\366\10", 6)       = 6
write(15, "\37 \3\0\0\0\0\0\374\377\34\0\0\0\0\0\374\377\34\0\0\0"..., 80) = 80
fcntl64(15, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
lseek(15, 0, SEEK_END)                  = 17350
mremap(0x415f5000, 16552, 17350, MREMAP_MAYMOVE) = 0x415f5000
--- SIGILL (Illegal instruction) @ 0 (0) ---
rt_sigaction(SIGILL, {SIG_DFL}, {0x401b7d34, [ILL], SA_RESTART|0x4000000}, 8) = 0
socket_subcall(0x1f8004, 0, 0x100, 0, 0, 0x18844, 0x18840, 0x12c) = 0
ioctl(12, KDSKBMODE, 0x2)               = 0
ioctl(12, SNDCTL_TMR_START or TCSETS, {B38400 -opost -isig -icanon -echo ...}) = 0
close(12)                               = 0
ioctl(10, KDSETMODE, 0x1)               = 0
write(10, "\33[9;15]\33[?33h\33[?25h\33[?0c\0", 25) = 25
close(10)                               = 0
statfs64(umovestr: Input/output error
0x6d4f, 27983, {???})          = 0
sigreturn()                             = ? (mask now [ILL ABRT BUS FPE USR1 SEGV USR2 PIPE STKFLT CHLD CONT STOP TTOU URG XCPU VTALRM PROF WINCH IO PWR RTMIN])
--- SIGILL (Illegal instruction) @ 0 (0) ---
+++ killed by SIGILL +++
Process 27983 detached

gdb backtrace of the crash (I'm missing debug symbols since compiling with debug information resolves the problem):

(gdb) run -qws
Starting program: /home/.qt-test2/digitalclock -qws

Program received signal SIGILL, Illegal instruction.
0x4130268c in __sigsetjmp () from /lib/libc.so.6
(gdb) bt
#0  0x4130268c in __sigsetjmp () from /lib/libc.so.6
#1  0x4046ee5c in ?? () from ./libQtGui.so.4
(gdb)

Note the device comes with Qtopia 4.3 preinstalled and the vendor can't explain the problem with my build either.

---

Update

With help from Igor Skochinsky I could find the exact assembler instruction that is causing the SIGILL. For some reason the instruction works fine for 47 times before causing the error. See gdb output below (note I'm not familiar with ARM assembler at all ):

$ LD_LIBRARY_PATH=. QT_QWS_FONTDIR=$PWD/fonts QT_PLUGIN_PATH=$PWD/plugins QWS_MOUSE_PROTO=tslib:/dev/input/touchscreen0 gdb ./digitalclock
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "arm-angstrom-linux-gnueabi"...
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) start -qws
Breakpoint 1 at 0xaa58: file main.cpp, line 47.
Starting program: /home/.qt-test2/digitalclock -qws
[Thread debugging using libthread_db enabled]
[New Thread 1073870720 (LWP 2799)]
[Switching to Thread 1073870720 (LWP 2799)]
main (argc=2, argv=0xbea17d04) at main.cpp:47
47      main.cpp: No such file or directory.
        in main.cpp
(gdb) display/i $pc
1: x/i $pc  0xaa58 <main+24>:   sub     r3, r11, #28    ; 0x1c
(gdb) display/x $r2
2: /x $r2 = 0xbea17d10
(gdb) display/x $f2
3: /x $f2 = 0x0
(gdb) b *0x41302684
Breakpoint 2 at 0x41302684
(gdb) continue
Continuing.

---> no problem here:

Breakpoint 2, 0x41302684 in __sigsetjmp () from /lib/libc.so.6
3: /x $f2 = 0x0
2: /x $r2 = 0x293
1: x/i $pc  0x41302684 <__sigsetjmp+52>:        beq     0x413026a0 <Lno_iwmmxt>
(gdb) si
0x41302688 in __sigsetjmp () from /lib/libc.so.6
3: /x $f2 = 0x0
2: /x $r2 = 0x293
1: x/i $pc  0x41302688 <__sigsetjmp+56>:        stfp    f2, [r12], #8
(gdb) si
0x4130268c in __sigsetjmp () from /lib/libc.so.6
3: /x $f2 = 0x0
2: /x $r2 = 0x293
1: x/i $pc  0x4130268c <__sigsetjmp+60>:        stfp    f3, [r12], #8
(gdb) si
0x41302690 in __sigsetjmp () from /lib/libc.so.6
3: /x $f2 = 0x0
2: /x $r2 = 0x293
1: x/i $pc  0x41302690 <__sigsetjmp+64>:        stfp    f4, [r12], #8
(gdb) continue
Continuing.

Breakpoint 2, 0x41302684 in __sigsetjmp () from /lib/libc.so.6
3: /x $f2 = 0x0
2: /x $r2 = 0x293
1: x/i $pc  0x41302684 <__sigsetjmp+52>:        beq     0x413026a0 <Lno_iwmmxt>
(gdb) continue 46
Will ignore next 45 crossings of breakpoint 2.  Continuing.


---> __sigsetjmp still working fine, but then:


Breakpoint 2, 0x41302684 in __sigsetjmp () from /lib/libc.so.6
3: /x $f2 = 0x0
2: /x $r2 = 0x293
1: x/i $pc  0x41302684 <__sigsetjmp+52>:        beq     0x413026a0 <Lno_iwmmxt>
(gdb) si
0x41302688 in __sigsetjmp () from /lib/libc.so.6
3: /x $f2 = 0x0
2: /x $r2 = 0x293
1: x/i $pc  0x41302688 <__sigsetjmp+56>:        stfp    f2, [r12], #8
(gdb) si

Program received signal SIGILL, Illegal instruction.
0x4130268c in __sigsetjmp () from /lib/libc.so.6
3: /x $f2 = 0x0
2: /x $r2 = 0x293
1: x/i $pc  0x4130268c <__sigsetjmp+60>:        stfp    f3, [r12], #8

Any suggestions what I could try next?

Answer 1

The posted disassembly is quite interesting.

 0x41302678 <__sigsetjmp+40>:    fmrx    r2, fpscr
 0x4130267c <__sigsetjmp+44>:    str     r2, [r12], #4
 0x41302680 <__sigsetjmp+48>:    tst     r2, #512        ; 0x200
 0x41302684 <__sigsetjmp+52>:    beq     0x413026a0 <__sigsetjmp+80>
 0x41302688 <__sigsetjmp+56>:    stfp    f2, [r12], #8
*0x4130268c <__sigsetjmp+60>:    stfp    f3, [r12], #8*
 0x41302690 <__sigsetjmp+64>:    stfp    f4, [r12], #8
 0x41302694 <__sigsetjmp+68>:    stfp    f5, [r12], #8
 0x41302698 <__sigsetjmp+72>:    stfp    f6, [r12], #8
 0x4130269c <__sigsetjmp+76>:    stfp    f7, [r12], #8

The code checks for bit 9 in fpscr, and, if set, tries to save registers f2-f7. What are those? I've never seen them in recent processors, but I think those are FPA ("Floating Point Accelerator") registers, implemented in a few old cores, and used for soft FP before VFP appeared.

So, here's what I think happens:

1. The libc on your device was compiled with FPA support, probably by mistake.
2. In FPA processors bit 9 meant "FPA enabled" or something similar
3. In the debug version of Qt the bit 9 of FPSCR (DZE = Division by Zero exception enable bit) is not set, so they don't try to save FPA registers. However, it gets set in the release version.

I see here two options:

1. Rebuild libc without FPA support
2. Find where DZE gets set in the release ver (not sure how to do that)

---

Update: I was wrong. The gdb disassembly confused me. I found the source of setjmp.S, here's the relevant part:

        tst     a3, #HWCAP_ARM_VFP
        beq     Lno_vfp

        /* Store the VFP registers.  */
        /* Following instruction is fstmiax ip!, {d8-d15}.  */
        stc     p11, cr8, [r12], #68
        /* Store the floating-point status register.  */
        /* Following instruction is fmrx r2, fpscr.  */
        mrc     p10, 7, r2, cr1, cr0, 0
        str     r2, [ip], #4
Lno_vfp:

        tst     a3, #HWCAP_ARM_IWMMXT
        beq     Lno_iwmmxt

        /* Save the call-preserved iWMMXt registers.  */
        /* Following instructions are wstrd wr10, [ip], #8 (etc.)  */
        stcl    p1, cr10, [r12], #8
        stcl    p1, cr11, [r12], #8
        stcl    p1, cr12, [r12], #8
        stcl    p1, cr13, [r12], #8
        stcl    p1, cr14, [r12], #8
        stcl    p1, cr15, [r12], #8
Lno_iwmmxt:

So, it's trying to store WMMXt registers, not FPA. However, there is a bug here. It's using r2 to temporarily store fpscr, but that ovewrites the previously loaded hwcap value in a3 (a3 is the APCS name for r2). Maybe the author meant to use a2, not r2, or maybe the two parts were done by different people. In either case, somehow the release version of Qt changes FPSCR (which is most likely emulated by the kernel) and the code storing iwmmxt regs is triggered.

Still, that's not the whole story. The hwcaps you pasted claim that the CPU does support iWMMXt, so I'm not sure why those instructions would be giving trouble. Maybe the reported PC value is wrong somehow. I think you should try putting breakpoint on __sigsetjmp and stepping through it by instruction (stepi), to see where exactly it crashes.