How to profile a continuously running server running on FreeBSD [duplicate]

Question

Possible Duplicate:
Saving gmon.out before killing a process

I'm trying to profile a server (source code available to me. c-code) on Linux environment. This server runs continuously like a web server. I'm trying to use gprof to profile the server. If the server exits by itself, gmon.out file is generated. I can use gprof and gmon.out to understand profiled data. Now the problem I have is, this server is running continuously, waiting for incoming socket connections, requests etc. If I kill this server, gmon.out is not generated. At this point I see the following options.

1. change the source code to profile itself and log this information after receiving SIGKILL signal. This is by far the ugliest solution and may introduce noise in the measurement.
2. Maybe there is a way to profile this server using gprof while the server is still running.
3. Other tools to try?

EDIT: The server is multi-process server. running on FreeBSD 7.2

I'm sure, people have solved these kind of problems before. I failed to find useful information on SO or outside.

I appreciate any thoughts/solutions people have.

Thanks a bunch.

UPDATE 1:

1. gprof doesn't seem to work with multi-process server. When I manage to get gmon.out after executing my server, only parent process is instrumented which actually doesnt do real work!.
2. oProfile doesn't support FreeBSD which is what my server is running on. For various reasons I can't(not allowed to) change OS.
3. Valgrind website doesnt have a port for FreeBSD. But there are some references to a port to FreeBSD. I failed to find FreeBSD port source.

Somehow I managed to get ports for valgrind. When I run make I get the following errors.

=> valgrind-stable-352.tar.gz doesn't seem to exist in /usr/obj/ports/distfiles/.
=> Attempting to fetch from ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/.
fetch: ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/distfiles/valgrind-stable-352.tar.gz: File unavailable (e.g., file not found, no access)
=> Attempting to fetch from http://www.rabson.org/.
fetch: http://www.rabson.org/valgrind-stable-352.tar.gz: No address record
=> Couldn't fetch it - please try to retrieve this
=> port manually into /usr/obj/ports/distfiles/ and try again.
*** Error code 1

I tried to find valgrind-stable-352.tar.gz on web. All of the links I found are dead.

4. I got pstack installed on my freebsd and the realised pstack gives only stack trace. reference : http://sourceforge.net/projects/bsd-pstack/

5. My understanding is that systemtap is only for kernel-space events, instrumentation etc.

I could be wrong or have insufficient information. Please correct me and give your thoughts. I really appreciate your assistance.

UPDATE 2: I think it will be helpful to give some details about the server that I'm trying to profile.

1. it is multi-server program. I/O bound, to be specific mysql database.
2. No threads involved. Each child-server-process handles only one request. configurable number of processes are created when the server starts.
3. I would like to find time spent in each function and its frequency. function codes are a mix of CPU-bound and IO bound (I believe more IO).
4. it is running on FreeBSD 7.2
5. written in c. number of reads is much greater than writes to the database via this server.

Answer 1

While you certainly should take your precations on profiling critical production systems, use oprofile or/and systemtap , They're likely included with your distro already.

Answer 2

Even if you get gprof to serve you, there are problems.

• It is blind to any system calls or I/O. It is based on the assumption that you will never do an unnecessary hang. It only looks at CPU-bound issues.

• If there is any recursion, it just can't handle it.

• The times it gives you are based on shaky assumptions, such as that every call to a routine takes about the same amount of time. It gives you no line-level information.

Measuring is one thing, but if you want to find "bottlenecks" that are doing unnecessary things, whether CPU or I/O, a very rough but effective tool is lsstack (which I think is on SourceForge).

Also, take a look at Zoom. It is a wall-time stack-sampler for Linux. It gives line-level percents, and I believe it can be attached and detached from a running process.