GnuCOBOL failing to find dynamic symbols, only on recent Ubuntu

Question

Something changed recently, I think.

GnuCOBOL relies on dynamic linking, symbols looked up with dlsym at run-time. This CALL run-time support code has been in OpenCOBOL for some 7 years now. It no longer works on Ubuntu 14.04, but does under Fedora 19/20.

ldd no longer shows any libraries listed using -l

For instance as a test, Ubuntu 14.04.1

The following COBOL program

   identification division.
   program-id. simple.

   procedure division.
   call "gtk_init" using
       by value 0
       by reference null
     returning omitted 
   end-call
   goback.
   end program simple.


$ cobc -x -v -lgtk-3 simple.cob
preprocessing simple.cob into /tmp/cob710_0.cob
parsing /tmp/cob710_0.cob (simple.cob)
Return status:  0
translating /tmp/cob710_0.cob into /tmp/cob710_0.c (simple.cob)
gcc -pipe -c -I/usr/local/include   -Wno-unused -fsigned-char -Wno-pointer-sign  -o "/tmp/cob710_0.o" "/tmp/cob710_0.c"
gcc -pipe  -Wl,--export-dynamic -o simple /tmp/cob710_0.o  -L/usr/local/lib -lcob -lm -lgmp -lncurses -ldb -ldl -l"gtk-3"

The binary has NO indication that libgtk-3.so is in the mix.

./simple
libcob: Cannot find module 'gtk_init'

$ ldd simple
    linux-vdso.so.1 =>  (0x00007fff2c9fe000)
    libcob.so.1 => /usr/local/lib/libcob.so.1 (0x00007f2549b06000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2549740000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2549439000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f25491c5000)
    libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007f2548fa2000)
    libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f2548d78000)
    libdb-5.3.so => /usr/lib/x86_64-linux-gnu/libdb-5.3.so (0x00007f25489d6000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f25487d2000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f2549d56000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f25485b3000)

and then on Fedora 20, same version of the compiler (built slightly differently, finding ncursesw instead of ncurses - assuming this is not part of the issue at hand)

$ cobc -x -v -lgtk-3 simple.cob
Command line:   cobc -x -v -lgtk-3 simple.cob 
Preprocessing:  simple.cob -> /tmp/cob20658_0.cob
Return status:  0
Parsing:        /tmp/cob20658_0.cob (simple.cob)
Return status:  0
Translating:    /tmp/cob20658_0.cob -> /tmp/cob20658_0.c (simple.cob)
Executing:      gcc -std=gnu99 -c -I/usr/local/include -pipe -Wno-unused
                -fsigned-char -Wno-pointer-sign -o "/tmp/cob20658_0.o"
                "/tmp/cob20658_0.c"
Return status:  0
Executing:      gcc -std=gnu99 -Wl,--export-dynamic -o "simple"
                "/tmp/cob20658_0.o" -L/usr/local/lib -lcob -lm -lgmp
                -lncursesw -ldb -ldl -l"gtk-3"
Return status:  0

$ ldd simple
    linux-vdso.so.1 =>  (0x00007fffae9cf000)
    libcob.so.4 => /usr/local/lib/libcob.so.4 (0x00007f4ff2548000)
    libm.so.6 => /lib64/libm.so.6 (0x0000003e5ae00000)
    libgmp.so.10 => /lib64/libgmp.so.10 (0x0000003e7a200000)
    libncursesw.so.5 => /lib64/libncursesw.so.5 (0x0000003e5d200000)
    libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003e69800000)
    libdb-5.3.so => /lib64/libdb-5.3.so (0x0000003e6ac00000)
    libdl.so.2 => /lib64/libdl.so.2 (0x0000003e5b200000)
    libgtk-3.so.0 => /lib64/libgtk-3.so.0 (0x0000003e6ba00000)
    libc.so.6 => /lib64/libc.so.6 (0x0000003e5aa00000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003e5b600000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003e5a600000)
    libgdk-3.so.0 => /lib64/libgdk-3.so.0 (0x0000003e6a800000)
    libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x0000003e65600000)
    libpangocairo-1.0.so.0 => /lib64/libpangocairo-1.0.so.0 (0x0000003e75200000)
    libX11.so.6 => /lib64/libX11.so.6 (0x00007f4ff2206000)
    libXi.so.6 => /lib64/libXi.so.6 (0x0000003e62600000)
    libXfixes.so.3 => /lib64/libXfixes.so.3 (0x0000003e5fe00000)
    libcairo-gobject.so.2 => /lib64/libcairo-gobject.so.2 (0x0000003e6a400000)
    libcairo.so.2 => /lib64/libcairo.so.2 (0x0000003e71000000)
    libgdk_pixbuf-2.0.so.0 => /lib64/libgdk_pixbuf-2.0.so.0 (0x0000003e6e000000)
    libatk-1.0.so.0 => /lib64/libatk-1.0.so.0 (0x0000003e75600000)
    libatk-bridge-2.0.so.0 => /lib64/libatk-bridge-2.0.so.0 (0x0000003e6c600000)
    libpangoft2-1.0.so.0 => /lib64/libpangoft2-1.0.so.0 (0x0000003e71c00000)
    libpango-1.0.so.0 => /lib64/libpango-1.0.so.0 (0x0000003e73600000)
    libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x0000003e61600000)
    libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x0000003e66600000)
    libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x0000003e5fa00000)
    libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x0000003e5e600000)
    libXinerama.so.1 => /lib64/libXinerama.so.1 (0x0000003e61e00000)
    libXrandr.so.2 => /lib64/libXrandr.so.2 (0x0000003e62200000)
    libXcursor.so.1 => /lib64/libXcursor.so.1 (0x0000003e62e00000)
    libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x0000003e74e00000)
    libXdamage.so.1 => /lib64/libXdamage.so.1 (0x0000003e67e00000)
    libwayland-client.so.0 => /lib64/libwayland-client.so.0 (0x0000003e6ec00000)
    libxkbcommon.so.0 => /lib64/libxkbcommon.so.0 (0x0000003e6b000000)
    libwayland-cursor.so.0 => /lib64/libwayland-cursor.so.0 (0x0000003e69c00000)
    libXext.so.6 => /lib64/libXext.so.6 (0x0000003e5ea00000)
    librt.so.1 => /lib64/librt.so.1 (0x0000003e5ce00000)
    libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x0000003e61a00000)
    libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x0000003e6f000000)
    libfreetype.so.6 => /lib64/libfreetype.so.6 (0x0000003e60e00000)
    libxcb.so.1 => /lib64/libxcb.so.1 (0x0000003e5da00000)
    libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x0000003e6f800000)
    libEGL.so.1 => /lib64/libEGL.so.1 (0x0000003e73200000)
    libpng16.so.16 => /lib64/libpng16.so.16 (0x0000003e5f600000)
    libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x0000003e6e800000)
    libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x0000003e70800000)
    libXrender.so.1 => /lib64/libXrender.so.1 (0x0000003e61200000)
    libz.so.1 => /lib64/libz.so.1 (0x0000003e5ba00000)
    libGL.so.1 => /lib64/libGL.so.1 (0x0000003e68200000)
    libatspi.so.0 => /lib64/libatspi.so.0 (0x0000003e6c200000)
    libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x0000003e62a00000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x0000003e60a00000)
    libffi.so.6 => /lib64/libffi.so.6 (0x0000003e5ee00000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003e5ca00000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003e5e200000)
    libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x0000003e6fc00000)
    libXau.so.6 => /lib64/libXau.so.6 (0x0000003e5d600000)
    libX11-xcb.so.1 => /lib64/libX11-xcb.so.1 (0x0000003e65e00000)
    libxcb-dri2.so.0 => /lib64/libxcb-dri2.so.0 (0x0000003e67200000)
    libxcb-xfixes.so.0 => /lib64/libxcb-xfixes.so.0 (0x0000003e70400000)
    libxcb-shape.so.0 => /lib64/libxcb-shape.so.0 (0x0000003e72a00000)
    libgbm.so.1 => /lib64/libgbm.so.1 (0x0000003e70c00000)
    libudev.so.1 => /lib64/libudev.so.1 (0x0000003e63200000)
    libwayland-server.so.0 => /lib64/libwayland-server.so.0 (0x0000003e74a00000)
    libglapi.so.0 => /lib64/libglapi.so.0 (0x0000003e67600000)
    libdrm.so.2 => /lib64/libdrm.so.2 (0x0000003e67a00000)
    libxcb-glx.so.0 => /lib64/libxcb-glx.so.0 (0x0000003e66e00000)
    libXxf86vm.so.1 => /lib64/libXxf86vm.so.1 (0x0000003e66200000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x0000003e5c600000)
    liblzma.so.5 => /lib64/liblzma.so.5 (0x0000003e5c200000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003e5be00000)

The generated C code (GnuCOBOL uses C intermediates) on Ubuntu

/* Line: 4         : CALL               : simple.cob */
cob_procedure_params[0] = (cob_field *)&c_1;
cob_procedure_params[1] = NULL;
cob_glob_ptr->cob_call_params = 2;
if (unlikely(call_gtk_init.funcvoid == NULL)) {
  call_gtk_init.funcvoid = cob_resolve_cobol ("gtk_init", 0, 1);
}
call_gtk_init.funcnull ((cob_s32_t)0LL, NULL);
b_1 = 0;

And Fedora

/* Line: 4         : CALL               : simple.cob */
cob_procedure_params[0] = (cob_field *)&c_1;
cob_procedure_params[1] = NULL;
cob_glob_ptr->cob_call_params = 2;
if (unlikely(call_gtk_init.funcvoid == NULL)) {
  call_gtk_init.funcvoid = cob_resolve_cobol ("gtk_init", 0, 1);
}
call_gtk_init.funcnull ((cob_s32_t)0LL, NULL);
b_1 = 0;

I get good results on Ubuntu (full ELF linkage hints) when gtk_init is called from C, not as a string passed to cob_resolve. Tested with gcc -o simple simple-gtk.c -lgtk-3

So what changed with the assumptions surrounding -llibname? It does not seem to be included in ELF data for dlopen to even bother trying looking for libgtk-3.so

More information: Erroneous compile lines on Ubuntu really make it look like this should be working. (Misspelling gtk)

$ cobc -x -v -lgkt-3 simple.cob
Command line:   cobc -x -v -lgkt-3 simple.cob 
Preprocessing:  simple.cob -> /tmp/cob13556_0.cob
Return status:  0
Parsing:        /tmp/cob13556_0.cob (simple.cob)  
Return status:  0
Translating:    /tmp/cob13556_0.cob -> /tmp/cob13556_0.c (simple.cob)
Executing:      gcc -std=gnu99 -c -I/usr/local/include -pipe -Wno-unused
                -fsigned-char -Wno-pointer-sign -o "/tmp/cob13556_0.o"
                "/tmp/cob13556_0.c"
Return status:  0
Executing:      gcc -std=gnu99 -Wl,--export-dynamic -o "simple"
                "/tmp/cob13556_0.o" -L/usr/local/lib -lcob -lm -lgmp -lncurses
                -ldb -ldl -l"gkt-3"
/usr/bin/ld: cannot find -lgkt-3
collect2: error: ld returned 1 exit status
Return status:  256

And yet, with the correct compile line, the ELF is showing no linkage hints to gtk-3

I've been head scratching on this on and off for a while now. Looking for a hint as to what assumptions changed with Ubuntu and gcc and/or ld and/or dlopen dlsym.

The open-cobol package has been working in Debian and Ubuntu repositories for quite a few years now. Even older versions of GnuCOBOL (GNU Cobol, and or OpenCOBOL) on Ubuntu all fail now. Something changed and we didn't get the memo. More than willing to change the compiler sources, but looking for friendly StackOverflow insights first.

This does not seem to be a local environment issue, as this Ubuntu problem is showing up for others as well. This also feels like one of those DOH! simple to fix problems.

More than willing to add more compile logs, LD_DEBUG=all dumps, or straces etc.

Answer 1

This looks to be a change made to the compiler driver for ubuntu - it's adding the --as-needed option to the compile line when sending the code to collect2 aka the linker.

To understand what's happening we need to disassemble the execution of cobc more than it's being shown:

cobc -x -v simple.cob -lgtk-3
preprocessing simple.cob into /tmp/cob2743_0.cob
translating /tmp/cob2743_0.cob into /tmp/cob2743_0.c
gcc -pipe -c  -Wno-unused -fsigned-char -Wno-pointer-sign  -o /tmp/cob2743_0.o /tmp/cob2743_0.c
gcc -pipe  -Wl,--export-dynamic -o simple /tmp/cob2743_0.o  -L/usr/lib -lcob -lm -lgmp -lncurses -ldb -ldl -lgtk-3

if we break it into produce C code, and then compile it we get:

cobc -C -x -v simple.cob
gcc -pipe -c  -Wno-unused -fsigned-char -Wno-pointer-sign -o simple.o simple.c
gcc -pipe -Wl,--export-dynamic -o simple simple.o  -L/usr/lib -lcob -lm -lgmp -lncurses -ldb -ldl -lgtk-3

We need to further disassemble the last gcc line into:

gcc -### -pipe -Wl,--export-dynamic -o simple simple.o  -L/usr/lib -lcob -lm -lgmp -lncurses -ldb -ldl -lgtk-3

which yields as output:

 /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 "--sysroot=/" --build-id --eh-frame-hdr -m elf_x86_64 "--hash-style=gnu" --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -z relro -o simple /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o -L/usr/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. --export-dynamic simple.o -lcob -lm -lgmp -lncurses -ldb -ldl -lgtk-3 -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o

The problem is the presence of the first --as-needed option - it overturns all the explicit -l options on the command line - if there's nothing found in the .o's that make up the file that depend on the library, it won't link the library in - this is the exact dynamic loading case.

This seems to be the case since Oneiric.

This simplest workaround is adding:

COB_LDFLAGS=-Wl,--no-as-needed

to your environment which should fix the linking issue.

Answer 2

It appears from the output that the C code that dynamically looks up gtk_init doesn't know about loading the actual libgtk-3.so shared object at runtime.

You could compile with cobc using the -fstatic-call option. This will call your library functions like gtk_init directly, and not via calls to dlopen/dlsym . More information on this option can be found in the info pages via the command info opencobol:

With the compiler options -fstatic-call, more efficient code will be generated like this:

subr(X);

Note that this option is effective only when the called program name is a literal (like CALL "subr".'). With a data name (likeCALL SUBR.'), the program is still called dynamically.

If you want to go the dynamic route I can only suggest telling your Cobol application exactly what dynamic objects you wish to load at runtime. To do this you can set the environment variable COB_PRE_LOAD prior to launching your application. Information on this environment variable can be found in this Open Cobol documentation. In particular it says:

COB_PRE_LOAD is an environment variable that controls what dynamic link modules are included in a run.

$ cobc occurl.c
$ cobc occgi.c
$ cobc -x myprog.cob
$ export COB_PRE_LOAD=occurl:occgi
$ ./myprog

That will allow the OpenCOBOL runtime link resolver to find the entry point for CALL “CBL_OC_CURL_INIT” in the occurl.so module. Note: the modules listed in the COB_PRE_LOAD environment variable DO NOT have extensions. OpenCOBOL will do the right thing on the various platforms.

You can do it a couple ways (using bash):

COB_PRE_LOAD=libgtk-3 ./simple

This will set COB_PRE_LOAD to use libgtk-3.so (you leave off the .so) launch your application, and when finished reset COB_PRE_LOAD back to what it was. You can also use export to set the environment variable for the duration of your session with:

export COB_PRE_LOAD=libgtk-3 
./simple

You can specify multiple shared objects using COB_PRE_LOAD by separating each with a colon. So if you needed libgtk-3 and libgmp for example, you could do this:

COB_PRE_LOAD=libgtk-3:libgmp ./simple