
## Introducing DTrace

This chapter introduces the dynamic tracing (DTrace) facility,
available on Linux.
You can use DTrace to examine the behavior of the operating system
and of user-space programs, particularly those that have been
instrumented with DTrace probes.
DTrace has been implemented on Linux using eBPF,
allowing DTrace to run as a user-space tool,
albeit as root for most uses.

### About This Tutorial

This tutorial includes a variety of DTrace scripts and describes
different ways in which you can use DTrace.
Several examples have additional exercises that offer further
practice in using DTrace.
You should already have a good understanding of Linux administration
and system programming, and broad experience using a programming language,
such as C or C++, and a scripting language, such as Python.
If you are not familiar with terms such as *system call*, *type*,
*cast*, *signal*, *struct*, or *pointer*,
you might have difficulty in understanding some of the examples
or completing some of the exercises in this tutorial.
However, each exercise provides a sample solution in case you do get stuck.
You are encouraged to experiment with the examples to develop your
skills at creating DTrace scripts.

>Caution:
>
>To run the examples and perform the exercises in this tutorial,
you need to have `root` access to a system.
Only the `root` user or a user with `sudo` access to run commands
as `root` can use the `dtrace` utility.
As `root`, you have total power over a
system and so have total responsibility for that system.
Although DTrace is designed so that you can use it safely
without needing to worry about corrupting the operating system
or other processes,
there are ways to circumvent some of the default, built-in safety measures.
>
>To minimize risk, perform the examples and exercises in this
tutorial on a system other than a production system.
>
>Also, make sure that `/usr/sbin` is before `/usr/bin` in the root path,
or specify `/usr/sbin/dtrace` explicitly.

The examples in this tutorial demonstrate the different ways that
you can perform dynamic tracing of your system: by entering a
simple D program as an argument to `dtrace` on
the command line, by using the `dtrace` command
to run a script that contains a D program, or by using an
executable D script that contains a *hashbang*
(`#!` or *shebang*) invocation of `dtrace`.
When you create your own D programs,
you can choose which method best suits your needs.

### About DTrace

DTrace is a comprehensive dynamic tracing facility that was first
developed for use on the Solaris operating system (now known as
Oracle Solaris) and subsequently ported to Oracle Linux.
You can use
DTrace to explore the operation of your system to better
understand how it works, to track down performance problems across
many layers of software, or to locate the causes of aberrant
behavior.

Using DTrace, you can record data at previously instrumented
places of interest, which are referred to as
*probes*, in kernel and user-space programs.
A probe is a location to which DTrace can bind a request to perform
a set of actions, such as recording a stack trace, a timestamp, or
the argument to a function.
Probes function like programmable
sensors that record information.
When a probe is triggered, DTrace
gathers data that you have designated in a D script and reports
this data back to you.

Using DTrace's D programming language, you can query the system
probes to provide immediate and concise answers to any number of
questions that you might formulate.

A D program describes the actions that occur if one or more
specified probes is triggered.
A probe is uniquely specified by
the name of the DTrace provider that publishes the probe, the name
of the module, library, or user-space program in which the probe
is located, the name of the function in which the probe is
located, and the name of the probe itself, which usually describes
some operation or functionality that you can trace.
Because you do
not need to specify probes exactly, this allows DTrace to perform
the same action for a number of different probes.
Full and
explicit representation of a single probe in the D language takes
the form *provider*:*module*:*function*:*name*.

When you use the `dtrace` command to run a D
program, you invoke the compiler for the D language.
The compiled code, eBPF instructions, is loaded into the kernel
and attached to the appropriate probes.
DTrace handles any runtime errors that might occur
during your D program's execution, including dividing by zero,
dereferencing invalid memory, and so on, and reports them to you.

The *module* is the kernel module in which a probe appears or,
in the case of user code, the load object.  Some probes, such
as clock-triggered profiling interrupts, have no module associated
with them.

Unless you explicitly permit DTrace to perform potentially
destructive actions, you cannot construct an unsafe program that
would cause DTrace to inadvertently damage either the operating
system or any process that is running on your system. These safety
features enable you to use DTrace in a production environment
without worrying about crashing or corrupting your system. If you
make a programming mistake, DTrace reports the error and
deactivates your program's probes. You can then correct your
program and try again.

For more information about using DTrace, see the
[DTrace User Guide](../userguide/index.md).

### About DTrace Providers

Here are the providers in the Oracle Linux implementation of DTrace:

```
           kernel       user space
     +-----------------------------+
     |             dtrace          |  I want to control my D program.
     +----------------+------------+
     |                |  syscall   |  I want to see how I call the kernel.
     +----------------+------------+
     |          cpc profile        |  I want to look at resource usage.
     +----------------+------------+
     | ip tcp udp     |            |
     | io lockstat    |            |  I know the semantics of the code.
     | sdt rawtp      | usdt       | (I know what probes are in the code.)
     | proc sched     |            |
     +----------------+------------+
     | fbt rawfbt     | pid        |  I know the functions of the code.
     +----------------+------------+
```

That is, these providers provide:

- `dtrace`: probes that relate to DTrace itself,
    such as `BEGIN`, `ERROR`, and `END`.
    You can use these probes to initialize DTrace's state before
    tracing begins, process its state after tracing has completed,
    and handle unexpected execution errors in other probes.
- `syscall`: probes that fire at the entry to and return from every system call.
    Because system calls are the primary interface between user-level
    applications and the operating system kernel, these probes can
    offer you an insight into the interaction between applications and
    the system.
- `cpc`: probes that fire when hardware resource counters (or
    software emulation thereof) trip, giving you an idea of
    where statistically hardware events are triggered.
- `profile`: probes that fire at fixed and specified time intervals.
    You can use these probes to sample some aspect of a system's state.
- `ip`: probes that fire for important steps in the IP protocol,
    for both IPv4 and IPv6.
- `tcp`: probes that fire for important steps in the TCP protocol,
   for both IPv4 and IPv6.
- `udp`: probes that fire for important steps in the the UDP protocol,
    for both IPv4 and IPv6.
- `io`: probes that relate to data input and output.
    The `io` provider enables quick exploration of behavior
    observed through I/O monitoring.
- `lockstat`: probes that fire for important steps in lock handling,
    including for mutexes, read-write locks, and spin locks.
- `sdt`: probes that fire when statically defined tracepoints in the Linux kernel
    are executed.
- `rawtp`: probes that fire when statically defined tracepoints in the Linux kernel
    are executed, but report raw arguments.
- `proc`: probes that fire for process creation and termination, LWP creation
    and termination, execution of new programs, and signal handling.
- `sched`: probes that fire for important steps in CPU scheduling.
    Because CPUs are the one resource that all threads must consume,
    the `sched` provider is very useful for understanding systemic behavior.
- `usdt`: probes that fire when user-space statically defined trace points
    are encountered.
- `fbt`: probes that fire for function boundary tracing (FBT) -- that is,
    when kernel functions are entered or return.
- `rawfbt`: probes that are like `fbt`, but are implemented with `kprobes`,
    include synthetic functions (such as compiler-generated optimized
    variants of functions with `.` suffixes.), and have unconverted and
    untyped probe arguments.
- `pid`: probes that fire for the specified process id (`pid`) upon
    entry into or return from a function.

See [DTrace Providers](../userguide/reference/dtrace_providers.md) in
the [DTrace User Guide](../userguide/index.md)
for more information about providers and their probes.

### Preparing to Install and Configure DTrace

DTrace is available for Linux from
[github](https://github.com/oracle/dtrace-utils/tree/stable).
Or, there are
[DTrace packages built for Oracle Linux](https://www.oracle.com/linux/downloads/linux-dtrace.html).

#### Example: Displaying Probes for a Provider

The following example shows how you would display the probes
for a provider, such as `proc`, by using the `dtrace` command.

```
# dtrace -l -P proc
   ID   PROVIDER    MODULE          FUNCTION NAME
   53       proc   vmlinux                   start
   52       proc   vmlinux                   signal-send
   51       proc   vmlinux                   signal-handle
   50       proc   vmlinux                   signal-discard
   49       proc   vmlinux                   signal-clear
   48       proc   vmlinux                   lwp-start
   47       proc   vmlinux                   lwp-exit
   46       proc   vmlinux                   lwp-create
   45       proc   vmlinux                   exit
   44       proc   vmlinux                   exec-success
   43       proc   vmlinux                   exec-failure
   42       proc   vmlinux                   exec
   41       proc   vmlinux                   create
```

These probes enable you to monitor how the system creates
processes, executes programs, and handles signals.

The output shows the numeric identifier of the probe, the name
of the probe provider, the name of the probe module, the name
of the function that contains the probe (none for this provider),
and the name of the probe itself.

The full name of a probe is *provider*:*module*:*function*:*name*,
for example, `proc:vmlinux::create`.

When probes are listed, a missing field indicates there is no
corresponding value.
When you specify a probe, omitting a field is equivalent to using
the wildcard `*`.

#### Exercise: Enabling and Listing DTrace Probes

Try listing the probes of the `syscall` provider.
Notice that both `entry` and `return` probes are provided for each system call.

#### Solution to Exercise: Enabling and Listing DTrace Probes

```
# dtrace -l -P syscall
   ID  PROVIDER     MODULE          FUNCTION NAME
 5055   syscall    vmlinux            socket entry
 5054   syscall    vmlinux            socket return
 5053   syscall    vmlinux        socketpair entry
 5052   syscall    vmlinux        socketpair return
 5051   syscall    vmlinux              bind entry
 5050   syscall    vmlinux              bind return
 5049   syscall    vmlinux            listen entry
 5048   syscall    vmlinux            listen return
 ...
 4375   syscall    vmlinux            ioperm entry
 4374   syscall    vmlinux            ioperm return
 4373   syscall    vmlinux              iopl entry
 4372   syscall    vmlinux              iopl return
 4371   syscall    vmlinux      rt_sigreturn entry
 4370   syscall    vmlinux      rt_sigreturn return
 4369   syscall    vmlinux        arch_prctl entry
 4368   syscall    vmlinux        arch_prctl return
```

The probe ID numbers can differ from run to run.

### Running a Simple DTrace Program

The following example shows how you could use a simple D program
that is in a file called `hello.d`.

#### Example: Simple D Program That Uses the BEGIN Probe (hello.d)

```
/* hello.d -- A simple D program that uses the BEGIN probe */

BEGIN
{
    /* This is a C-style comment */
    trace("hello, world");
    exit(0);
}
```

A D program consists of a series of clauses, where each clause
describes one or more probes to enable, and an optional set of actions
to perform when the probe fires. The actions are listed as a series
of statements enclosed in braces `{}` following the probe name.
Each statement ends with a semicolon (`;`).

In this example, the function `trace` directs DTrace to record the
specified argument, the string ”hello, world”, when the `BEGIN` probe fires,
and then print it out.
The function `exit()` tells DTrace to cease tracing and exit the `dtrace`
command.

The full name of the `BEGIN` probe is `dtrace:::BEGIN`.
The `dtrace` provider provides three probes: `dtrace:::BEGIN`,
`dtrace:::END`, and `dtrace:::ERROR`.
Because these probe names are unique to the `dtrace` provider,
their names can be shortened to `BEGIN`, `END`, and `ERROR`.

When you have saved your program, you can run it by using the
`dtrace` command with the `-s` option,
which specifies the name of the file that contains the D program:

```
# dtrace -s hello.d
dtrace: script 'hello.d' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0      1                           :BEGIN   hello, world
```

DTrace interprets and runs the script. You will notice
that in addition to the string `"hello,world"`,
the default behavior of DTrace is to display information about
the CPU on which the script was running when a probe fired,
the ID of the probe, the name of the function that contains
the probe, and the name of the probe itself.
The function name is displayed as blank for `BEGIN`,
as DTrace provides this probe.

You can suppress the probe information in a number of different
ways, for example, by specifying the `-q` quiet option:

```
# dtrace -q -s hello.d
hello, world
```

#### Exercise: Using the END Probe

Copy the `hello.d` program to the file `goodbye.d`.
Edit this file so that it traces the string "goodbye, world"
and uses the `END` probe instead of `BEGIN`.
When you run this new script, you need to type `Ctrl-C` to cause the
probe to fire and exit `dtrace`.

#### Solution to Exercise and Example: Using the END Probe

The following is an example of a simple D program that
demonstrates the use of the `END` probe:

```
/* goodbye.d -- Simple D program that demonstrates the END probe */

END
{
  trace("goodbye, world");
}
```

The default output is:

```
# dtrace -s goodbye.d
dtrace: script 'goodbye.d' matched 1 probe
^C
CPU     ID                    FUNCTION:NAME
  3      2                             :END   goodbye, world
```

Or, with the `-q` quiet option:

```
# dtrace -q -s ./goodbye.d
^C
goodbye, world
```
