legacy gcc compiler issues

Question

We are using a legacy compiler, based on gcc 2.6.0, to cross compile for an old imbedded processor we are still using (yes, it is still in use since 1994!). The engineer that did the gcc port for this chip has long since moved on. Although we might be able to recover the gcc 2.6.0 source from somewhere on the web, the change set for this chip has disappeared in the halls of corporate history. We have muddled along until recently as the compiler still ran and produced workable executables, but as of linux kernel 2.6.25 (and also 2.6.26) it fails with the message gcc: virtual memory exhausted... even when run with no parameters or with only -v. I have rebooted my development system (from 2.6.26) using the 2.6.24 kernel and the compiler works again (rebooting with 2.6.25 does not).

We have one system that we are keeping at 2.6.24 just for the purpose of doing builds for this chip, but are feeling a bit exposed in case the linux world moves on to the point that we cannot any longer rebuild a system that will run the compiler (i.e. our 2.6.24 system dies and we cannot get 2.6.24 to install and run on a new system because some of the software parts are no longer available).

Does anyone have any ideas for what we might be able to do to a more modern installation to get this legacy compiler to run?

Edit:

To answer some of the comments...

Sadly it is the source code changes that are specific to our chip that are lost. This loss occurred over two major company reorgs and several sysadmins (a couple of which really left a mess). We now use configuration control, but that is closing the barn door too late for this problem.

The use of a VM is a good idea, and may be what we end up doing. Thank you for that idea.

Finally, I tried strace as ephemient suggested and found that the last system call was brk() which returned an error on the new system (2.6.26 kernel) and returned success on the old system (2.6.24 kernel). This would indicate that I really am running out of virtual memory, except that tcsh "limit" returns the same values on old and new systems, and /proc/meminfo shows the new systems has slightly more memory and quite a bit more swap space. Maybe it is a problem of fragmentation or where the program is being loaded?

I did some further research and "brk randomization" was added in kernel 2.6.25, however CONFIG_COMPAT_BRK is supposedly enabled by default (which disables brk randomization).

Edit:

OK, more info: It really looks like brk randomization is the culprit, the legacy gcc is calling brk() to change the end of the data segment and that now fails, causing the legacy gcc to report "virtual memory exhausted". There are a few documented ways to disable brk randomization:

• sudo echo 0 > /proc/sys/kernel/randomize_va_space

• sudo sysctl -w kernel.randomize_va_space=0

• starting a new shell with setarch i386 -R tcsh (or "-R -L")

I have tried them and they do seem to have an effect in that the brk() return value is different (and always the same) than without them (tried on both kernel 2.6.25 and 2.6.26), but the brk() still fails so the legacy gcc still fails :-(.

In addition I have set vm.legacy_va_layout=1 and vm.overcommit_memory=2 with no change, and I have rebooted with the vm.legacy_va_layout=1 and kernel.randomize_va_space=0 settings saved in /etc/sysctl.conf. Still no change.

Edit:

Using kernel.randomize_va_space=0 on kernel 2.6.26 (and 2.6.25) results in the following brk() call being reported by strace legacy-gcc:

brk(0x80556d4) = 0x8056000

This indicates the brk() failed, but it looks like it failed because the the data segment already ends beyond what was requested. Using objdump, I can see the data segment should end at 0x805518c whereas the failed brk() indicates that the data segment currently ends at 0x8056000:

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .interp       00000013  080480d4  080480d4  000000d4  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .hash         000001a0  080480e8  080480e8  000000e8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .dynsym       00000410  08048288  08048288  00000288  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .dynstr       0000020e  08048698  08048698  00000698  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .rel.bss      00000038  080488a8  080488a8  000008a8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .rel.plt      00000158  080488e0  080488e0  000008e0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .init         00000008  08048a40  08048a40  00000a40  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  7 .plt          000002c0  08048a48  08048a48  00000a48  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  8 .text         000086cc  08048d10  08048d10  00000d10  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  9 .fini         00000008  080513e0  080513e0  000093e0  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 10 .rodata       000027d0  080513e8  080513e8  000093e8  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 11 .data         000005d4  08054bb8  08054bb8  0000bbb8  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 12 .ctors        00000008  0805518c  0805518c  0000c18c  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 13 .dtors        00000008  08055194  08055194  0000c194  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 14 .got          000000b8  0805519c  0805519c  0000c19c  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 15 .dynamic      00000088  08055254  08055254  0000c254  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 16 .bss          000003b8  080552dc  080552dc  0000c2dc  2**3
                  ALLOC
 17 .note         00000064  00000000  00000000  0000c2dc  2**0
                  CONTENTS, READONLY
 18 .comment      00000062  00000000  00000000  0000c340  2**0
                  CONTENTS, READONLY
SYMBOL TABLE:
no symbols

Edit:

To echo ephemient's comment below: "So strange to treat GCC as a binary without source"!

So, using strace, objdump, gdb and my limited understanding of 386 assembler and architecture I have traced the problem to the 1st malloc call in the legacy code. The legacy gcc calls malloc, which returns NULL, which results in the "virtual memory exhausted" message on stderr. This malloc is in libc.so.5, and it calls getenv a bunch of times and ends up calling brk()... I guess to increase the heap... which fails.

From this I can only surmise that the problem is more than brk randomization, or I have not fully disabled brk randomization, despite the randomize_va_space=0 and legacy_va_layout=1 sysctl settings.

Answer 1

Install linux + the old gcc onto a virtual machine.

Answer 2

Do you have the sources for this custom compiler? If you can recover the 2.6.0 baseline (and that should be relatively easy), then diff and patch should recover your change set.

What's I'd then recommend is using that change set to build a new version against up to date gcc. AND THEN PUT IT UNDER CONFIGURATION CONTROL.

Sorry, don't mean to shout. It's just I've been saying the same thing for most of 30 years.