LTTNG(1)

NAME
lttng — LTTng 2.1.x tracer control command line tool

SYNOPSIS

lttng [OPTIONS] <COMMAND>

DESCRIPTION

The LTTng project aims at providing highly efficient tracing tools for Linux. It's tracers help tracking down performance issues and debugging problems involving multiple concurrent processes and threads. Tracing across multiple systems is also possible.

The lttng command line tool from the lttng-tools package is used to control both kernel and user-space tracing. Every interactions with the tracer should be done by this tool or by the liblttng-ctl provided with the lttng-tools package.

LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry, which allows you to interact with multiple tracers (kernel and user-space) inside the same container, a tracing session. Traces can be gathered from the kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading those traces is done using the babeltrace(1) text viewer.

We introduce the notion of tracing domains which is essentially a type of tracer (kernel or user space for now). In the future, we could see a third tracer being for instance an hypervisor. For some commands, you'll need to specify on which domain the command applies (-u or -k). For instance, enabling a kernel event, you must specify the kernel domain to the command so we know for which tracer this event is for.

In order to trace the kernel, the session daemon needs to be running as root. LTTng provides the use of a tracing group (default: tracing). Whomever is in that group can interact with the root session daemon and thus trace the kernel. Session daemons can co-exist meaning that you can have a session daemon running as Alice that can be used to trace her applications along side with a root daemon or even a Bob daemon. We highly recommend to start the session daemon at boot time for stable and long term tracing.

Every user-space applications instrumented with lttng-ust(3), will automatically register to the session daemon. This feature gives you the ability to list available traceable applications and tracepoints on a per user basis. (See list command).

OPTIONS

This program follow the usual GNU command line syntax with long options starting with two dashes. Below is a summary of the available options.

-h, --help
    Show summary of possible options and commands.

-v, --verbose
    Increase verbosity. Three levels of verbosity are available which are triggered by putting additional v to the option (-vv or -vvv)

-q, --quiet
    Suppress all messages (even errors).

-g, --group NAME
    Set unix tracing group name. (default: tracing)

-n, --no-sessiond
    Don't automatically spawn a session daemon.

--sessiond-path PATH
    Set session daemon full binary path.

--list-options
    Simple listing of lttng options.

--list-commands
    Simple listing of lttng commands.

COMMANDS

add-context
    Add context to event(s) and/or channel(s).

    A context is basically extra information appended to a channel. For instance,
    you could ask the tracer to add the PID information for all events in a
    channel. You can also add performance monitoring unit counters (perf PMU) using
    the perf kernel API).

    For example, this command will add the context information 'prio' and two perf
    counters (hardware branch misses and cache misses), to all events in the trace
    data output:

    # lttng add-context -k -t prio -t perf:branch-misses -t perf:cache-misses

    Please take a look at the help (-h/--help) for a detailed list of available
    contexts.

    If no channel is given (-c), the context is added to all channels. Otherwise
    the context will be added only to the given channel (-c).

    If -s, --session is omitted, the session name is taken from the .lttngrc
    file.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    -s, --session NAME
            Apply on session name.
    -c, --channel NAME
            Apply on channel name.
    -k, --kernel
            Apply for the kernel tracer
    -u, --userspace
            Apply for the user-space tracer
    -t, --type TYPE
            Context type. You can repeat this option on the command line. Please
            use "lttng add-context -h" to list all available types.

calibrate
    Quantify LTTng overhead

    The LTTng calibrate command can be used to find out the combined average
    overhead of the LTTng tracer and the instrumentation mechanisms used. This
    overhead can be calibrated in terms of time or using any of the PMU performance
    counter available on the system.

    For now, the only calibration implemented is that of the kernel function
    instrumentation (kretprobes).

    * Calibrate kernel function instrumentation

    Let's use an example to show this calibration. We use an i7 processor with 4
    general-purpose PMU registers. This information is available by issuing dmesg,
    looking for "generic registers".

    This sequence of commands will gather a trace executing a kretprobe hooked on
    an empty function, gathering PMU counters LLC (Last Level Cache) misses
    information (see lttng add-context --help to see the list of available PMU
    counters).

    # lttng create calibrate-function
    # lttng enable-event calibrate --kernel --function lttng_calibrate_kretprobe
    # lttng add-context --kernel -t perf:LLC-load-misses -t perf:LLC-store-misses \
                      -t perf:LLC-prefetch-misses
    # lttng start
    # for a in $(seq 1 10); do \
            lttng calibrate --kernel --function;
      done
    # lttng destroy
    # babeltrace $(ls -1drt ~/lttng-traces/calibrate-function-* | tail -n 1)

    The output from babeltrace can be saved to a text file and opened in a
    spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
    consecutive "calibrate_entry" and "calibrate_return" events. Note that these
    counters are per-CPU, so scheduling events would need to be present to account
    for migration between CPU. Therefore, for calibration purposes, only events
    staying on the same CPU must be considered.

    The average result, for the i7, on 10 samples:

                              Average     Std.Dev.
    perf_LLC_load_misses:       5.0       0.577
    perf_LLC_store_misses:      1.6       0.516
    perf_LLC_prefetch_misses:   9.0      14.742

    As we can notice, the load and store misses are relatively stable across runs
    (their standard deviation is relatively low) compared to the prefetch misses.
    We can conclude from this information that LLC load and store misses can be
    accounted for quite precisely, but prefetches within a function seems to behave
    too erratically (not much causality link between the code executed and the CPU
    prefetch activity) to be accounted for.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    -k, --kernel
            Apply for the kernel tracer
    -u, --userspace
            Apply for the user-space tracer
    --function
            Dynamic function entry/return probe (default)

create [NAME] [OPTIONS]
    Create tracing session.

    A tracing session contains channel(s) which contains event(s). It is domain
    agnostic meaning that you can enable channels and events for either the
    user-space tracer and/or the kernel tracer. It acts like a container
    aggregating multiple tracing sources.

    On creation, a .lttngrc file is created in your $HOME directory
    containing the current session name. If NAME is omitted, a session name is
    automatically created having this form: 'auto-yyyymmdd-hhmmss'.

    If no -o, --output is specified, the traces will be written in
    $HOME/lttng-traces.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options
    -o, --output PATH
            Specify output path for traces

    Using these options, each API call can be controlled individually. For
    instance, -C does not enable the consumer automatically. You'll need the -e
    option for that.

    -U, --set-uri=URL
            Set URL for the enable-consumer destination. It is persistent for the
            session lifetime. Redo the command to change it. This will set both
            data and control URL for network.
    -C, --ctrl-url=URL
            Set control path URL. (Must use -D also)
    -D, --data-url=URL
            Set data path URL. (Must use -C also)
         --no-consumer
            Don't activate a consumer for this session.
         --disable-consumer
            Disable consumer for this session.

    See enable-consumer command below for the supported URL format.

    EXAMPLES:

    # lttng create -U net://192.168.1.42
    Uses TCP and default ports for the given destination.

    # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
    Uses TCP, default ports and IPv6.

    # lttng create s1 -U net://myhost.com:3229
    Create session s1 and set its consumer to myhost.com on port 3229 for control.

destroy [OPTIONS] [NAME]
    Teardown tracing session

    Free memory on the session daemon and tracer side. It's gone!

    If NAME is omitted, the session name is taken from the .lttngrc file.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    -a, --all
            Destroy all sessions
    --list-options
            Simple listing of options

enable-channel NAME[,NAME2,...] [-k|-u] [OPTIONS]
    Enable tracing channel

    To enable event, you must first enable a channel which contains event(s).

    If -s, --session is omitted, the session name is taken from the .lttngrc
    file.

    OPTIONS:

    -h, --help
            Show this help
    --list-options
            Simple listing of options
    -s, --session NAME
            Apply on session name
    -k, --kernel
            Apply to the kernel tracer
    -u, --userspace
            Apply to the user-space tracer

    --discard
            Discard event when subbuffers are full (default)
    --overwrite
            Flight recorder mode : overwrites events when subbuffers are full
    --subbuf-size SIZE
            Subbuffer size in bytes (default: 4096, kernel default: 262144)
    --num-subbuf NUM
            Number of subbuffers (default: 4)
            Needs to be a power of 2 for kernel and ust tracers
    --switch-timer USEC
            Switch subbuffer timer interval in usec (default: 0)
            Needs to be a power of 2 for kernel and ust tracers
    --read-timer USEC
            Read timer interval in usec (default: 200)
    --output TYPE
            Channel output type. Possible values: mmap, splice

enable-consumer [-u|-k] [URL] [OPTIONS]
    Enable a consumer for the tracing session and domain.

    By default, every tracing session has a consumer attached to it using the local
    filesystem as output. The trace is written in $HOME/lttng-traces. This command
    allows the user to specify a specific URL after the session was created for a
    specific domain. If no domain is specified, the consumer is applied on all
    domains.

    Without options, the behavior is to enable a consumer to the current URL. The
    default URL is the local filesystem at the path of the session mentioned above.

    The enable-consumer feature supports both local and network transport. You must
    have a running lttng-relayd(8) for network transmission or any other daemon
    that can understand the streaming protocol of LTTng.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options
    -s, --session NAME
            Apply on session name
    -k, --kernel
            Apply for the kernel tracer
    -u, --userspace
            Apply for the user-space tracer

    Using these options, each API call can be controlled individually. For
    instance, -C does not enable the consumer automatically. You'll need the -e
    option for that.

    -U, --set-uri=URL
            Set URL for the enable-consumer destination. It is persistent for the
            session lifetime. Redo the command to change it. This will set both
            data and control URL for network.
    -C, --ctrl-url=URL
            Set control path URL. (Must use -D also)
    -D, --data-url=URL
            Set data path URL. (Must use -C also)
    -e, --enable
            Enable consumer

    URL FORMAT:

    proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]

    Supported protocols are (proto):
    > file://...
            Local filesystem full path.

    > net://...
            This will use the default network transport layer which is TCP for both
            control (PORT1) and data port (PORT2). The default ports are
            respectively 5342 and 5343. Note that net[6]:// is not yet supported.

    > tcp[6]://...
            Can only be used with -C and -D together

    NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)

    EXAMPLES:

    $ lttng enable-consumer -u net://192.168.1.42

    Uses TCP and default ports for user space tracing (-u) where the IP address
    above is the destination machine where the traces will be streamed and a
    lttng-relayd(8) is listening.

enable-event NAME[,NAME2,...] [-k|-u] [OPTIONS]
    Enable tracing event

    A tracing event is always assigned to a channel. If -c, --channel is
    omitted, a default channel named 'channel0' is created and the event is
    added to it. For the user-space tracer, using -a, --all is the same as
    using the wildcard "*".

    If -s, --session is omitted, the session name is taken from the .lttngrc
    file.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options
    -s, --session NAME
            Apply on session name
    -c, --channel NAME
            Apply on channel name
    -a, --all
            Enable all tracepoints and syscalls. This actually enable a single
            wildcard event "*".
    -k, --kernel
            Apply for the kernel tracer
    -u, --userspace
            Apply for the user-space tracer

    --tracepoint
            Tracepoint event (default)
            - userspace tracer supports wildcards at end of string. Don't forget to
            quote to deal with bash expansion.
            e.g.:
            "*"
            "app_component:na*"
    --loglevel NAME
            Tracepoint loglevel range from 0 to loglevel. Listed in the help (-h).
    --loglevel-only NAME
            Tracepoint loglevel (only this loglevel).

            The loglevel or loglevel-only options should be combined with a
            tracepoint name or tracepoint wildcard.
    --probe [addr | symbol | symbol+offset]
            Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
            or hexadecimal (0xNNN...)
    --function [addr | symbol | symbol+offset]
            Dynamic function entry/return probe. Addr and offset can be octal
            (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
    --syscall
            System call event. Enabling syscalls tracing (kernel tracer), you will
            not be able to disable them with disable-event. This is a known
            limitation. You can disable the entire channel to do the trick.

    --filter 'expression'
            Set a filter on a newly enabled event. Filter expression on event
            fields, event recording depends on evaluation. Only specify on first
            activation of a given event within a session. Filter only allowed when
            enabling events within a session before tracing is started. If the
            filter fails to link with the event within the traced domain, the event
            will be discarded. Currently, filter is only implemented for the
            user-space tracer.

            Expression examples:

            'intfield > 500 && intfield < 503'
            '(stringfield == "test" || intfield != 10) && intfield > 33'
            'doublefield > 1.1 && intfield < 5.3'

            Wildcards are allowed at the end of strings:
            'seqfield1 == "te*"'
            In string literals, the escape character is a '\'. Use '\*' for
            the '*' character, and '\\' for the '\' character.

disable-channel NAME[,NAME2,...] [-k|-u] [OPTIONS]
    Disable tracing channel

    Disabling a channel makes all event(s) in that channel to stop tracing. You can
    enable it back by calling lttng enable-channel NAME again.

    If -s, --session is omitted, the session name is taken from the .lttngrc
    file.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options
    -s, --session NAME
            Apply on session name
    -k, --kernel
            Apply for the kernel tracer
    -u, --userspace
            Apply for the user-space tracer

disable-consumer [-k|-u] [OPTIONS]
    Disable the consumer of a tracing session.

    This call MUST be done BEFORE tracing has started.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options
    -s, --session NAME
            Apply on session name
    -k, --kernel
            Apply for the kernel tracer
    -u, --userspace
            Apply for the user-space tracer

disable-event NAME[,NAME2,...] [-k|-u] [OPTIONS]
    Disable tracing event

    The event, once disabled, can be re-enabled by calling lttng enable-event
    NAME again.

    If -s, --session is omitted, the session name is taken from the .lttngrc
    file.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options
    -s, --session NAME
            Apply on session name
    -a, --all-events
            Disable all events. This does NOT disable "*" but rather
            every known events of the session.
    -k, --kernel
            Apply for the kernel tracer
    -u, --userspace
            Apply for the user-space tracer

list [-k|-u] [SESSION [SESSION_OPTIONS]]
    List tracing session information.

    With no arguments, it will list available tracing session(s).

    With the session name, it will display the details of the session including
    the trace file path, the associated channels and their state (activated
    and deactivated), the activated events and more.

    With -k alone, it will list all available kernel events (except the system
    calls events).
    With -u alone, it will list all available user-space events from registered
    applications. Here is an example of 'lttng list -u':

    PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
          ust_tests_hello:tptest_sighandler (type: tracepoint)
          ust_tests_hello:tptest (type: tracepoint)

    You can now enable any event listed by using the name :
    ust_tests_hello:tptest.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options
    -k, --kernel
            Select kernel domain
    -u, --userspace
            Select user-space domain.

    SESSION OPTIONS:

    -c, --channel NAME
            List details of a channel
    -d, --domain
            List available domain(s)

set-session NAME
    Set current session name

    Will change the session name in the .lttngrc file.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options

start [NAME] [OPTIONS]
    Start tracing

    It will start tracing for all tracers for a specific tracing session.

    If NAME is omitted, the session name is taken from the .lttngrc file.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options

stop [NAME] [OPTIONS]
    Stop tracing

    It will stop tracing for all tracers for a specific tracing session. Before
    returning, the command checks for data availability meaning that it will wait
    until the trace is readable for the session. Use --no-wait to avoid this
    behavior.

    If NAME is omitted, the session name is taken from the .lttngrc file.

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options
    --no-wait
            Don't wait for data availability.

version
    Show version information

    OPTIONS:

    -h, --help
            Show summary of possible options and commands.
    --list-options
            Simple listing of options

view [SESSION_NAME] [OPTIONS]
    View traces of a tracing session

    By default, the babeltrace viewer will be used for text viewing.

    If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.

    OPTIONS:

    -h, --help
            Show this help
    --list-options
            Simple listing of options
    -t, --trace-path PATH
            Trace directory path for the viewer
    -e, --viewer CMD
            Specify viewer and/or options to use
            This will completely override the default viewers so
            please make sure to specify the full command. The trace
            directory path of the session will be appended at the end
            to the arguments

EXIT VALUES
On success 0 is returned and a positive value on error. Value of 1 means a command error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that something went wrong during the command.

Any other value above 10, please refer to <lttng/lttng-error.h> for a detailed list or use lttng_strerror() to get a human readable string of the error code.

ENVIRONMENT VARIABLES

Note that all command line options override environment variables.

LTTNG_SESSIOND_PATH
    Allows one to specify the full session daemon binary path to lttng command line tool. You can also use --sessiond-path option having the same effect.

SEE ALSO
babeltrace(1), lttng-ust(3), lttng-sessiond(8), lttng-relayd(8), lttng-health-check(3)

BUGS

If you encounter any issues or usability problem, please report it on our mailing list <lttng-dev@lists.lttng.org> to help improve this project or at https://bugs.lttng.org which is a bugtracker.

CREDITS

lttng is distributed under the GNU General Public License version 2. See the file COPYING for details.

A Web site is available at http://lttng.org for more information on the LTTng project.

You can also find our git tree at http://git.lttng.org.

Mailing lists for support and development: <lttng-dev@lists.lttng.org>.

You can find us on IRC server irc.oftc.net (OFTC) in #lttng.

THANKS

Thanks to Yannick Brosseau without whom this project would never have been so lean and mean! Also thanks to the Ericsson teams working on tracing which helped us greatly with detailed bug reports and unusual test cases.

Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA maintainer) and Jon Bernard for our Debian packages.

Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de Montreal for the LTTng journey.

AUTHORS

lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and David Goulet. More people have since contributed to it. It is currently maintained by David Goulet <dgoulet@efficios.com>.