
## Tracing User-Space Applications

This chapter provides information about how to trace a user-space
application and includes examples of D programs that you can use
to investigate what is happening in an example user-space program.

### Sample Application

This section provides a sample application to be used in
subsequent exercises and examples in this chapter. The example,
which illustrates a simple program, favors brevity and probing
opportunity rather than completeness or efficiency.

>Note:
>
>The following simple program is provided for example purposes
*only* and is not intended to efficiently solve a practical
problem nor exhibit preferred coding methods.

The sample program finds the lowest factor of a number, which you input.
The program is comprised of the following four files:
`makefile`, `primelib.h`, `primelib.c`, and `primain.c`,
which are stored in the same working directory.

#### Description and Format of the makefile File

The following example shows the contents of the `makefile` file.

>Note:
>
>A `makefile` must use tabs for indentation so
that the `make` command can function properly.
Also, be sure that tabs are retained if the
file is copied and then used.

```
default: prime

# compile the library primelib first
primelib.o: primelib.c
	gcc -c primelib.c

# compile the main program
primain.o: primain.c
	gcc -c primain.c

# link and create executable file "prime"
prime: primelib.o primain.o
	gcc primain.o primelib.o -o prime -lm

clean:
	-rm -f *.o
	-rm -f prime
```

#### Description of the primelib.h Source File

The following example shows the contents of the `primelib.h` file.

```
int findMaxCheck( int inValue );
int seekFactorA( int input, int maxtry );
int seekFactorB( int input );
```

#### Description of the primelib.c Source File

The following example shows the contents of the `primelib.c` file.

```
#include <stdio.h>
#include <math.h>

/*
 * utility functions which are called from the main source code
 */

// Find and return our highest value to check -- which is the square root
int findMaxCheck( int inValue )  {
  float sqRoot;
  sqRoot = sqrt( inValue );
  printf("Square root of %d is %lf\n", inValue, sqRoot);
  return floor( sqRoot );
}

int debugFlag = 0;

// Search for a factor to the input value, proving prime on return of zero
int seekFactorA( int input, int maxtry )  {
  int divisor, factor = 0;
  for( divisor=2; divisor<=maxtry; ++divisor ) {
    if( 0 == input%divisor ) {
      factor = divisor;
      break;
    }
    else if ( debugFlag != 0 )
      printf( "modulo %d yields: %d\n", divisor, input%divisor );
  }
  return factor;
}

// Search for a factor to the input value, proving prime on return of zero
// This is a different method than "A", using one argument
int seekFactorB( int input )  {
  int divisor, factor = 0;
  if( 0 == input%2 ) return 2;
  for( divisor=3; divisor<=input/2; divisor+=2 ) {
    if( 0 == input%divisor ) {
      factor = divisor;
      break;
    }
  }
  return factor;
}
```

#### Description of the primain.c Source File

The following example shows the contents of the `primain.c` file.

```
#include <stdio.h>
#include "primelib.h"

/*
 * Nominal C program churning to provide a code base we might want to
 * instrument with D
 */

// Search for a divisor -- thereby proving composite value of the input.
int main()  {
  int targVal, divisor, factorA=0, factorB=0;

  printf( "Enter a positive target integer to test for prime status: " );
  scanf( "%d", &targVal );

  // Check that the user input is valid
  if( targVal < 2 ) {
    printf( "Invalid input value -- exiting now\n" );
    return -2;
  }

  // Search for a divisor using method and function A
  int lastCheck;
  lastCheck = findMaxCheck( targVal );
  printf( "%d highest value to check as divisor\n", lastCheck );
  factorA = seekFactorA( targVal, lastCheck );

  // Search for a divisor using method and function B
  factorB = seekFactorB( targVal );

  // Warn if the methods give different results
  if (factorA != factorB)
    printf( "%d does not equal %d! How can this be?\n", factorA, factorB );

  // Print results
  if( !factorA )
    printf( "%d is a prime number\n", targVal );
  else
    printf( "%d is not prime because there is a factor %d\n",
	    targVal, factorA );
  return 0;
}
```

#### Compiling the Program and Running the prime Executable

With the four files previously described located in the
same working directory, compile the program by using the
`make` command as follows:

```
# make
gcc -c primelib.c
gcc -c primain.c
gcc primain.o primelib.o -o prime -lm
```

Running the `make` command creates an executable named `prime`,
which can be run to find the lowest prime value of the input,
as shown in the following two examples:

```
# ./prime
Enter a positive target integer to test for prime status: 5099
Square root of 5099 is 71.407280
71 highest value to check as divisor
5099 is a prime number
```
```
# ./prime
Enter a positive target integer to test for prime status: 95099
Square root of 95099 is 308.381256
308 highest value to check as divisor
95099 is not prime because there is a factor 61
```

### Using the `pid` Provider

So far, we have built our application without
taking DTrace at all into account.
Nevertheless, we can still study execution of this application,
using the `pid` provider to trace entry into and return from user functions.

For example, from one window, start the application
but do not yet provide it with any input,
thereby leaving the application paused and waiting:

```
$ ./prime 
Enter a positive target integer to test for prime status: 
```

In another window, we can list the probes supplied by the `pid` provider.
Actually, there is a different such provider for each process,
meaning that we must append the process ID to specify the process to trace.
For example, we can run `pgrep prime` to get the process ID.
In our case, it is 2889188.
Then, we can list the `pid` probes associated with this process in either of two ways:

```
# dtrace -ln pid2889188:a.out::
# dtrace -p 2889188 -ln pid'$target':a.out::
```

In the first case, we append the process ID
explicitly to form the provider name `pid2889188`.
In the second case, the provider name refers to the process ID symbolically,
using the macro `$target`,
and the traced process 2889188 is specified using a
`dtrace` command-line option `-p`.
This second form is useful when the probe specification is contained in
a D script that we do not want to modify each time we use it.

We specify a module of the application executable.
It could be the load object of the executable, here referred to as `a.out`,
or it could be a shared library.
The probe function and name are omitted, equivalent to the wildcard `*`.

In our case, this probe specification results in 187 probes.
We show a few of them here:

```
   ID   PROVIDER            MODULE                          FUNCTION NAME
 5433 pid2889188             prime                       seekFactorB 56
 5431 pid2889188             prime                       seekFactorB 55
 5429 pid2889188             prime                       seekFactorB 52
 [...]
 5375 pid2889188             prime                       seekFactorB 4
 5373 pid2889188             prime                       seekFactorB 1
 5371 pid2889188             prime                       seekFactorB 0
 5370 pid2889188             prime                       seekFactorB entry
 5368 pid2889188             prime                       seekFactorB return
 5366 pid2889188             prime                       seekFactorA 68
 5364 pid2889188             prime                       seekFactorA 67
 5362 pid2889188             prime                       seekFactorA 64
 [...]
 5302 pid2889188             prime                       seekFactorA 4
 5300 pid2889188             prime                       seekFactorA 1
 5298 pid2889188             prime                       seekFactorA 0
 5297 pid2889188             prime                       seekFactorA entry
 5295 pid2889188             prime                       seekFactorA return
 5293 pid2889188             prime                      findMaxCheck 70
 5291 pid2889188             prime                      findMaxCheck 6f
 5289 pid2889188             prime                      findMaxCheck 6b
 [...]
 5243 pid2889188             prime                      findMaxCheck 4
 5241 pid2889188             prime                      findMaxCheck 1
 5239 pid2889188             prime                      findMaxCheck 0
 5238 pid2889188             prime                      findMaxCheck entry
 5236 pid2889188             prime                      findMaxCheck return
 5234 pid2889188             prime                              main ef
 5232 pid2889188             prime                              main ee
 5230 pid2889188             prime                              main e9
 [...]
 5110 pid2889188             prime                              main 4
 5108 pid2889188             prime                              main 1
 5106 pid2889188             prime                              main 0
 5105 pid2889188             prime                              main entry
 5103 pid2889188             prime                              main return
 [...]
```

The module, which we specified by the nickname `a.out`,
is listed by its executable name `prime`.
The functions of this module appear:  `seekFactorB`, `seekFactorA`, and so on.
For each function, there is a probe for entry, return, and each instruction offset.

This listing shows us what probes we might specify to trace this
user program without having added any DTrace-specific instrumentation.
For more information on how to use the `pid` provider to trace
user function entry and return or on specific instruction offsets,
see
[Pid Provider](../userguide/reference/dtrace_providers_pid.md)
in the
[Oracle Linux: DTrace Reference Guide](../userguide/index.md).

### Adding USDT Probes to an Application

In this section, we practice adding USDT probes to an application.
For background information and other details, see
[Adding USDT Probes to Application Code](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdt_probe_add_prov)
in the
[Oracle Linux: DTrace Reference Guide](../userguide/index.md).

To get started, you will need to create a `.d` file, as described in
[Defining USDT Providers and Probes](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdtprobes_prov)
in the
[Oracle Linux: DTrace Reference Guide](../userguide/index.md).

>Note:
>
>This `.d` file is not a script that is run in the same
way that is shown in previous examples in this tutorial,
but is rather the `.d` source file that you use when
compiling and linking your application.

In this `.d` file, you define the probes you will
place in the `primain.c` source file.
These probes mark the sequence of operations that are
used after the user entry is completed and checked:

| Description                              | Probe                       |
| :-----------                             | :-----------                |
| User entry complete and checked          | `userentry( int )`          |
| Return from `findMaxCheck()`             | `maxcheckval( int, int )`   |
| Return from `seekFactorA()`              | `factorreturnA( int, int )` |
| Return from `seekFactorB()`              | `factorreturnB( int, int )` |
| Immediately prior to the program exiting | `final()`                   |

#### Exercise: Creating a dprime.d File

Declare the above probes in a file `dprime.d` and store the
file in the same working directory as the other source files.

#### Solution to Exercise: Creating a dprime.d File

```
provider primeget
{
  probe query__userentry( int );
  probe query__maxcheckval( int, int );
  probe query__factorreturnA( int, int );
  probe query__factorreturnB( int, int );
  probe query__final();
};
```

#### Example: Creating a .h File From a dprime.d File

The next step is to create a `.h` header file to use in your
C source program from the `dprime.d` file, as shown here:

```
# dtrace -h -s dprime.d
```

The `dprime.h` file that is created contains a reference to each
of the probe points that are defined in the `dprime.d` file.

Next, in the application source file, `primain.c`, we add
`#include "dprime.h"` and the appropriate probe macros at the proper
locations.

In the resulting `primain.c` file, the probe macros are easy
to recognize, as they appear in uppercase letters:

```
#include <stdio.h>
#include "primelib.h"
#include "dprime.h"

/*
 * Nominal C program churning to provide a code base we might want to
 * instrument with D
*/

// Search for a divisor -- thereby proving composite value of the input.
int main()  {
  int targVal, divisor, factorA=0, factorB=0;

  printf( "Enter a positive target integer to test for prime status: " );
  scanf( "%d", &targVal );

  // Check that the user input is valid
  if( targVal < 2 ) {
    printf( "Invalid input value -- exiting now\n" );
    return -2;
  }
  if (PRIMEGET_QUERY_USERENTRY_ENABLED())
    PRIMEGET_QUERY_USERENTRY(targVal);

  // Search for a divisor using method and function A
  int lastCheck;
  lastCheck = findMaxCheck( targVal );
  printf( "%d highest value to check as divisor\n", lastCheck );
  if (PRIMEGET_QUERY_MAXCHECKVAL_ENABLED())
    PRIMEGET_QUERY_MAXCHECKVAL(lastCheck, targVal);

  factorA = seekFactorA( targVal, lastCheck );
  if (PRIMEGET_QUERY_FACTORRETURNA_ENABLED())
    PRIMEGET_QUERY_FACTORRETURNA(factorA, targVal);

  // Search for a divisor using method and function B
  factorB = seekFactorB( targVal );
 if (PRIMEGET_QUERY_FACTORRETURNB_ENABLED())
    PRIMEGET_QUERY_FACTORRETURNB(factorB, targVal);

  // Warn if the methods give different results
  if (factorA != factorB)
    printf( "%d does not equal %d! How can this be?\n", factorA, factorB );

  // Print results
  if( !factorA )
    printf( "%d is a prime number\n", targVal );
  else
    printf( "%d is not prime because there is a factor %d\n",
	    targVal, factorA );
  if (PRIMEGET_QUERY_FINAL_ENABLED())
    PRIMEGET_QUERY_FINAL();

  return 0;
}
```

Any `*_ENABLED()` probe will translate into a truth value if
the associated probe is enabled (some consumer is using it),
and a false value if the associated probe is not enabled.

Next, you will need to modify the `makefile` file.
For step-by-step instructions, See
[Building Applications With USDT Probes](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdt_build_prov)
in the
[Oracle Linux: DTrace Reference Guide](../userguide/index.md).

#### Exercise: Directing makefile to Re-Create the dprime.h File

Add a target that instructs `dtrace` to re-create the `dprime.h` file
in the event that changes are subsequently made to the `dprime.d` file.
This step ensures that you do not have to manually run the
`dtrace -h -s dprime.d` command if any changes are made.

This exercise also has you direct `dtrace` to create a `prime.o` file.

#### Solution to Exercise: Directing makefile to Re-Create the dprime.h File

```
default: prime

# re-create new dprime.h if dprime.d file has been changed
dprime.h: dprime.d
	dtrace -h -s dprime.d

# compile the library primelib first
primelib.o: primelib.c
	gcc -c primelib.c

# compile the main program
primain.o: primain.c dprime.h
	gcc -c -I/usr/lib64/dtrace/include primain.c

# have dtrace post-process the object files
prime.o: dprime.d primelib.o primain.o
	dtrace -G -s dprime.d primelib.o primain.o -o prime.o

# link and create executable file "prime"
prime: prime.o
	gcc -Wl,--export-dynamic,--strip-all -o prime prime.o primelib.o primain.o -lm

clean:
	-rm -f *.o
	-rm -f prime
	-rm -f dprime.h
```

Notice that the `primain.c` file now includes the generated
`dprime.h` file, which includes `<sys/usdt.h>`.
Therefore, when we compile `gcc -c primain.c`,
we include a `-I` option to find `<sys/usdt.h>`.
The path to use on your system is given by:

```
pkg-config --cflags dtrace_sdt
```

which is `/usr/lib64/dtrace/include` in the example above.
In the DTrace source code directory, the file is in `uts/common`.

In the link stage, the `-Wl,--export-dynamic` link options to `gcc`
are required for symbol lookup in a stripped executable at runtime,
for example, when you use the D function `ustack()`.

#### Example: Testing the Program

After creating a fresh build, test that the executable is still
working as expected:

```
# make clean
rm -f *.o
rm -f prime
rm -f dprime.h
# make
gcc -c primelib.c
dtrace -h -s dprime.d
gcc -c primain.c
dtrace -G -s dprime.d primelib.o primain.o -o prime.o
gcc -Wl,--export-dynamic,--strip-all -o prime prime.o primelib.o primain.o dprime.h -lm
```

```
# ./prime
Enter a positive target integer to test for prime status: 6799
Square root of 6799 is 82.456047
82 highest value to check as divisor
6799 is not prime because there is a factor 13
```

### Using USDT Probes

This section provides some practice in the nominal use of the USDT
probes that were created in
[Adding USDT Probes to an Application](#adding-usdt-probes-to-an-application).

Initially, the probes are not visible because the application is
not running with the probes, as shown in the following output:

```
# dtrace -l -P 'prime*'
  ID   PROVIDER            MODULE                          FUNCTION NAME
dtrace: failed to match prime*:::: No probe matches description
```

Start the application, but do not enter any value yet:

```
# ./prime
Enter a positive target integer to test for prime status:
```

From another command line, issue a probe listing:

```
# dtrace -l -P 'prime*'
   ID      PROVIDER            MODULE                          FUNCTION NAME
 2475 primeget26556             prime                              main query-factorreturnA
 2476 primeget26556             prime                              main query-factorreturnB
 2477 primeget26556             prime                              main query-final
 2478 primeget26556             prime                              main query-maxcheckval
 2479 primeget26556             prime                              main query-userentry
```

Note that the provider name is a combination of the defined `provider primeget`,
from the `dprime.d` file, and the PID of the running application `prime`.
The output of the following command displays the PID of prime:

```
# ps aux | grep prime
root 26556 0.0 0.0 7404 1692 pts/0 S+ 21:50 0:00 ./prime
```

At this point, we can kill the application or provide an input
value so that the application runs to completion.

#### Example: Using simpleTimeProbe.d to Show the Elapsed Time Between Two Probes

The following example shows how you would create a simple script
that measures the time elapsed between the first probe and the
second probe (`query-userentry` to `query-maxcheckval`).

```
/* simpleTimeProbe.d */

/* Show how much time elapses between two probes */

primeget*:::query-userentry
{
  self->t = timestamp; /* Initialize a thread-local variable with the time */
}

primeget*:::query-maxcheckval
/self->t != 0/
{
  this->timeNow = timestamp;
  /* Divide by 1000 for microseconds */
  printf("%s (pid=%d) spent %d microseconds between userentry & maxcheckval\n",
         execname, pid, ((this->timeNow - self->t)/1000));

  self->t = 0; /* Reset the variable */
}
```

Since `timeNow` is used only within this clause,
we use `this->` to denote its clause-local scope.

Once again, start the execution of the target application:

```
# ./prime
Enter a positive target integer to test for prime status:
```

Then, run the DTrace script from another window:

```
# dtrace -q -s simpleTimeProbe.d
```

As the application is running, the output of the script is also
running in parallel:

```
# ./prime
Enter a positive target integer to test for prime status: 7921
Square root of 7921 is 89.000000
89 highest value to check as divisor
7921 is not prime because there is a factor 89
# ./prime
Enter a positive target integer to test for prime status: 995099
Square root of 995099 is 997.546509
997 highest value to check as divisor
995099 is not prime because there is a factor 7
# ./prime
Enter a positive target integer to test for prime status: 7921
Square root of 7921 is 89.000000
89 highest value to check as divisor
7921 is not prime because there is a factor 89
```

On the command line where the script is being run, you should
see output similar to the following:

```
# dtrace -q -s simpleTimeProbe.d
prime (pid=2328) spent 45 microseconds between userentry & maxcheckval
prime (pid=2330) spent 41 microseconds between userentry & maxcheckval
prime (pid=2331) spent 89 microseconds between userentry & maxcheckval
^C
```

Another way of running the D script is with the `-Z` option.
Previously, we started the application first so that some
`primeget*` probes would be found.
With the `-Z` option, `dtrace` will start up even if
zero matching probes are found.
Then, it will wait patiently until such probes are discovered
and those probes fire.  That is, in one window, run:

```
# dtrace -q -Z -s simpleTimeProbe.d
```

Note the addition of the `-Z` option.
In other window, run the application:

```
# ./prime
Enter a positive target integer to test for prime status: 7921
Square root of 7921 is 89.000000
89 highest value to check as divisor
7921 is not prime because there is a factor 89
#
```

The first window should show the `dtrace` output
and you can terminate the script.

```
# dtrace -q -Z -s simpleTimeProbe.d
prime (pid=4334) spent 85 microseconds between userentry & maxcheckval
^C
```

And yet one more way of launching both the D script and the
application, from the same window, is to use the `-c` option
to `dtrace` to launch a command to trace:

```
# dtrace -q -c ./prime -s simpleTimeProbe.d
Enter a positive target integer to test for prime status: 579
Square root of 579 is 24.062418
24 highest value to check as divisor
579 is not prime because there is a factor 3
prime (pid=2884516) spent 98 microseconds between userentry & maxcheckval
```

Here, `dtrace` is launched, it starts the command `./prime`,
and when `dtrace` or the application terminates, so does the other.

#### Example: Using timeTweenprobes.d to Show the Elapsed Time Between Each Probe

You can broaden the script to monitor all of the following
probes in the application:

- `query-userentry`
- `query-maxcheckval`
- `query-factorreturnA`
- `query-factorreturnB`
- `query-final`

```
/* timeTweenProbes.d */

/* show how much time elapses between each probe */

BEGIN
{
  iterationCount = 0;
}

primeget*:::query-userentry
{
  printf("%s (pid=%d) running\n", execname, pid);
  self->t = timestamp; /* Initialize a thread-local variable with time */
}

primeget*:::query-maxcheckval
/self->t != 0/
{
  this->timeNow = timestamp;
  printf(" maxcheckval spent %d microseconds since userentry\n",
         ((this->timeNow - self->t)/1000));  /* Divide by 1000 for microseconds */
  self->t = this->timeNow; /* set the time to recent sample */
}

primeget*:::query-factorreturnA
/self->t != 0/
{
  this->timeNow = timestamp;
  printf(" factorreturnA spent %d microseconds since maxcheckval\n",
         ((this->timeNow - self->t)/1000));  /* Divide by 1000 for microseconds */
  self->t = this->timeNow; /* set the time to recent sample */
}

primeget*:::query-factorreturnB
/self->t != 0/
{
  this->timeNow = timestamp;
  printf(" factorreturnB spent %d microseconds since factorreturnA\n",
         ((this->timeNow - self->t)/1000));  /* Divide by 1000 for microseconds */
  self->t = this->timeNow; /* set the time to recent sample */
}

primeget*:::query-final
/self->t != 0/
{
  printf(" prime spent %d microseconds from factorreturnB until ending\n",
         ((timestamp - self->t)/1000));
  self->t = 0; /* Reset the variable */
  iterationCount++;
}

END
{
  trace(iterationCount);
}
```

Again, start the execution of the target application first,
without yet providing input.
Then from another window, start the D script.
Back in the first window, you can run the application multiple times.
Finally, in the second window, you can terminate the D script.
The first window might look something like this:

```
# ./prime
Enter a positive target integer to test for prime status: 995099
Square root of 995099 is 997.546509
997 highest value to check as divisor
995099 is not prime because there is a factor 7
# ./prime
Enter a positive target integer to test for prime status: 7921
Square root of 7921 is 89.000000
89 highest value to check as divisor
7921 is not prime because there is a factor 89
# ./prime
Enter a positive target integer to test for prime status: 95099
Square root of 95099 is 308.381256
308 highest value to check as divisor
95099 is not prime because there is a factor 61
# ./prime
Enter a positive target integer to test for prime status: 95099
Square root of 95099 is 308.381256
308 highest value to check as divisor
95099 is not prime because there is a factor 61
# ./prime
Enter a positive target integer to test for prime status: 5099
Square root of 5099 is 71.407280
71 highest value to check as divisor
5099 is a prime number
```

The corresponding output from the script is similar to the
following:

```
# dtrace -q -s ./timeTweenProbes.d
prime (pid=2437) running
 maxcheckval spent 96 microseconds since userentry
 factorreturnA spent 9 microseconds since maxcheckval
 factorreturnB spent 6 microseconds since factorreturnA
 prime spent 9 microseconds from factorreturnB until ending
prime (pid=2439) running
 maxcheckval spent 45 microseconds since userentry
 factorreturnA spent 10 microseconds since maxcheckval
 factorreturnB spent 7 microseconds since factorreturnA
 prime spent 9 microseconds from factorreturnB until ending
prime (pid=2440) running
 maxcheckval spent 43 microseconds since userentry
 factorreturnA spent 11 microseconds since maxcheckval
 factorreturnB spent 8 microseconds since factorreturnA
 prime spent 10 microseconds from factorreturnB until ending
prime (pid=2441) running
 maxcheckval spent 53 microseconds since userentry
 factorreturnA spent 10 microseconds since maxcheckval
 factorreturnB spent 7 microseconds since factorreturnA
 prime spent 10 microseconds from factorreturnB until ending
prime (pid=2442) running
 maxcheckval spent 40 microseconds since userentry
 factorreturnA spent 9 microseconds since maxcheckval
 factorreturnB spent 48 microseconds since factorreturnA
 prime spent 10 microseconds from factorreturnB until ending

^C
5
```

As is observed in the previous example, there is now a set of
DTrace features that can be used with the probes that were
created.
