Writing a DTrace consumer in C

Question

I'd like to write a C program on FreeBSD 10.1 that implements a DTrace consumer using libdtrace.

I know that I need to start with a call to dtrace_open() - e.g. I have found this old presentation, but I can't get even started, since there is no dtrace.h installed (only in the system source tree).

The shared library is installed, e.g. the /usr/sbin/dtrace tool included with FreeBSD can act as a DTrace consumer and this tool links to /lib/libdtrace.so.2 (which is also pointed to via a symbolic link from /usr/lib/libdtrace.so).

Any basic example, including build instructions (FreeBSD 10.1 / clang) will help me a lot.

---

The actual goal of writing a custom consumer is creating a CFFI based wrapper usable in Python and PyPy. Means: above C program is just to get started, learn and then proceed.

CFFI is the recommended, modern, high-performance way of interfacing PyPy with shared libraries.

CFFI can be used at the ABI and API level. The latter requires a header file to include, the former requires to declare the stuff used from a lib.

---

Adapted from Adam's answer, here is a complete example that works on FreeBSD 10.1.

Makefile:

all:
        cc \
         -I /usr/src/cddl/compat/opensolaris/include \
         -I /usr/src/cddl/contrib/opensolaris/lib/libdtrace/common/ \
         -I /usr/src/sys/cddl/compat/opensolaris \
         -I /usr/src/sys/cddl/contrib/opensolaris/uts/common/ \
         hello_dtrace.c \
         -l dtrace -l proc -l ctf -l elf -l z -l rtld_db -l pthread -l util \
         -o hello_dtrace

hello_dtrace.c:

#include <dtrace.h>
#include <signal.h>
#include <stdio.h>

static dtrace_hdl_t* g_dtp;

static int chewrec (const dtrace_probedata_t *data, const dtrace_recdesc_t *rec, void *arg) {
   printf("chewing dtrace record ..\n");
   // A NULL rec indicates that we've processed the last record.
   if (rec == NULL) {
      return (DTRACE_CONSUME_NEXT);
   }
   return (DTRACE_CONSUME_THIS);
}

static const char* g_prog = "BEGIN { printf(\"hello from dtrace\\n\"); }";
//static const char* g_prog = "syscall::open*:entry { printf(\"%s %s\\n\", execname, copyinstr(arg0)); }";

static int g_intr;
static int g_exited;

static void intr (int signo) {
   g_intr = 1;
}


int main (int argc, char** argv) {
   int err;

   if ((g_dtp = dtrace_open(DTRACE_VERSION, 0, &err)) == NULL) {
      fprintf(stderr, "failed to initialize dtrace: %s\n", dtrace_errmsg(NULL, err));
      return -1;
   }
   printf("Dtrace initialized\n");

   (void) dtrace_setopt(g_dtp, "bufsize", "4m");
   (void) dtrace_setopt(g_dtp, "aggsize", "4m");
   printf("dtrace options set\n");

   dtrace_prog_t* prog;
   if ((prog = dtrace_program_strcompile(g_dtp, g_prog, DTRACE_PROBESPEC_NAME, 0, 0, NULL)) == NULL) {
      fprintf(stderr, "failed to compile dtrace program\n");
      return -1;
   } else {
      printf("dtrace program compiled\n");
   }

   dtrace_proginfo_t info;
   if (dtrace_program_exec(g_dtp, prog, &info) == -1) {
      fprintf(stderr, "failed to enable dtrace probes\n");
      return -1;
   } else {
      printf("dtrace probes enabled\n");
   }

   struct sigaction act;
   (void) sigemptyset(&act.sa_mask);
   act.sa_flags = 0;
   act.sa_handler = intr;
   (void) sigaction(SIGINT, &act, NULL);
   (void) sigaction(SIGTERM, &act, NULL);

   if (dtrace_go(g_dtp) != 0) {
      fprintf(stderr, "could not start instrumentation\n");
      return -1;
   } else {
      printf("instrumentation started ..\n");
   }

   int done = 0;
   do {
      if (!g_intr && !done) {
         dtrace_sleep(g_dtp);
      }

      if (done || g_intr || g_exited) {
         done = 1;
         if (dtrace_stop(g_dtp) == -1) {
            fprintf(stderr, "could not stop tracing\n");
            return -1;
         }
      }

      switch (dtrace_work(g_dtp, stdout, NULL, chewrec, NULL)) {
         case DTRACE_WORKSTATUS_DONE:
            done = 1;
            break;
         case DTRACE_WORKSTATUS_OKAY:
            break;
         default:
            fprintf(stderr, "processing aborted");
            return -1;
      }
   } while (!done);

   printf("closing dtrace\n");
   dtrace_close(g_dtp);

   return 0;
}

To run:

[oberstet@brummer2 ~/hello_dtrace]$ make; sudo ./hello_dtrace
cc  -I /usr/src/cddl/contrib/opensolaris/lib/libdtrace/common/  -I /usr/src/sys/cddl/compat/opensolaris  -I /usr/src/sys/cddl/contrib/opensolaris/uts/common/  hello_dtrace.c  -l dtrace  -l proc  -l ctf  -l elf  -l rtld_db  -l z  -l pthread  -l util  -o hello_dtrace
Dtrace initialized
dtrace options set
dtrace program compiled
dtrace probes enabled
instrumentation started ..
chewing dtrace record ..
hello from dtrace
chewing dtrace record ..
^Cclosing dtrace

Answer 1

The libdtrace API isn't necessarily intended for stable consumers, but it's pretty easy to learn from some existing consumers to get started. The simplest and most modern is plockstat, the illumos utility for user-land locking statistics.

Here's the basic flow of a simple DTrace consumer program:

dtrace_open() to get a dtrace_hdl_t, a handle for other libdtrace interactions
dtrace_setopt() to configure options (-x flag to dtrace(1M))
dtrace_strcompile() to compile a string of your D program
dtrace_program_exec() to send that program to the kernel
dtrace_go() to instrument the system and start recording data
dtrace_close() to clean up at the end

For the main data collection loop (between dtrace_go() and dtrace_close()) do the following:

dtrace_sleep() to pause according to the DTrace options (switchrate and aggrate)
dtrace_work() to process traced data
dtrace_stop() to abort

See the main loop in plockstat.c for more.

For other simple DTrace consumers, check out intrstat and lockstat. For the kitchen sink, check out the code the the dtrace(1M) command-line utility.

Answer 2

dtrace is considered a system internal interface, and writing a custom consumer is not something that is supported.

The best you can do is check out a copy of the FreeBSD source tree and build your code from there. This actually isn't terribly difficult. Obviously, dtrace(1) is the canonical dtrace consumer, so you can look at its implementation for examples of how to use libdtrace. Additionally dtrace.h has a huge amount of comments explaining data structures and internals.

Building the files in the context of the source tree is pretty easy; the FreeBSD source tree layout is simple and writing Makefiles to build binaries in a self-contained manner is basically given to you "for free".

Pertinent points:

• Clone the dtrace(1) Makefile into a new directory in your source tree checkout. Modify the Makefile such that .PATH is correct, and set PROG and SRCS to include the set of sources comprising your custom consumer.

• Clone dtrace.c to your source directory (whereever you pointed .PATH to) and make changes as needed. Although the source is nearly 2,000 lines, much of it is support code. If you're just looking to clone a subset of the functionality, you'll find that most of the options to the binary are implemented in self-contained functions, and it should therefore be fairly easy to trim dtrace.c down to a bare-minimum form.

Without knowing what specifically you're needing to do with your custom consumer, it's difficult to tell you what else you'll need to call into. I'm assuming you'll probably want compile_file in there as well as exec_prog. Of course, your needs may differ.

The dtrace.h you will be using has a number of comments about the various interfaces provided.

Hopefully this is enough to get you started; unfortunately, there's not a way to do what you want from a vanilla install. You could, of course, hack up the relevant Makefiles to install the necessary header files and create your own internal distribution. But that seems like more pain than it's worth.