debugging a fastcgi application

Question

I have deployed a fastcgi application on my server. When I try to run it on the server, it crashes (more of this later).

First some facts on the local (dev) machine:

1. The application is C/C++ application
2. I developed the app on Ubuntu 10.0.4 LTS (same OS on server)
3. During development, I used gdb to step through code to fix initial bugs
4. I run ldd on the app and all required shared libs are found
5. I can test it on my local machine, using curl http://path/to/cgiapp

After getting the code tested and working on my local machine, I built a release version and packaged it, using Netbeans 6.9 IDE

I installed the package on the server and run the following tests

1. First run ldd on the installed app, I noticed that some files were not found. I fixed this by copying the files on the dev machine into the location that the app was searching for them (not ideal perhaps) - BUT, when I run ldd again, all libraries were found

2. I then tested the app by running curl http://path/to/cgiapp at the console

Instead of the expected message, there was what appeared to be a core dump (i.e. binary stream dump) onto the console.

My questions are:

1. Can anyone explain why ldd indicates all files (loaded?) correctly - and yet app crashes
2. What could be the most likely reason why the app runs on my dev machine but fails to run on the server?
3. Since the app crashes before starting, I am not sure how to use gdb to debug the application - obviously, I will copy the debug version to the server before I attempt to debug.

There are further complication regarding point 3 above. The server is a lean and mean, headless server, so it does not have gdb installed.

So even if I found a way to debug the app (before it crashes), I don't know how I can attach to a remote process.

Any ideas on how to start getting to the bottom of this would be much appreciated.

Answer 1

Your question is based on way too many misunderstandings.

Can anyone explain why ldd indicates all files (loaded?) correctly - and yet app crashes

For the same reason some applications run 100s of times without any problems, then crash "randomly".

Not all crashes are related to the dynamic linker, in fact majority of crashes are not. Other bugs, such as heap corruption, buffer overflows, off-by-one errors, etc. etc. are much more frequent cause.

What could be the most likely reason why the app runs on my dev machine but fails to run on the server?

You have not supplied sufficient data to answer that question.

What are the symptoms? Have you looked at apache error log for clues? What is the crash stack trace? Which libraries were missing on the server and were copied over?

Since the app crashes before starting, I am not sure how to use gdb to debug the application - obviously, I will copy the debug version to the server before I attempt to debug.

You don't need to copy debug binary to debug it (it may or may not help, depending on where the crash actually happens).