LTTng 2.15 "Péché Mortel" Release Candidate

We're excited to announce the availability of the first release candidate of LTTng 2.15, codenamed "Péché Mortel"!

Only a few months after LTTng 2.14, this release delivers an impressive number of features considering the short time frame, in particular:

• CTF 2 (Common Trace Format) as the default trace format
• Reduced memory footprint for user space tracing
• Improved resiliency of shared LTTng-UST buffers
• Redesigned lttng list command with memory usage reporting
• Extended automation capabilities for the lttng add-trigger command

What's New in LTTng 2.15?

Trace Format Selection

Choose the trace format for each recording session with the new --trace-format option of the lttng create command:

lttng create my-session --trace-format=ctf-1.8

Select between CTF 1.8 and CTF 2.

CTF 2 now is the default trace format for all newly created recording sessions. Babeltrace 2 supports CTF 2 since version 2.1 (January 2025). To keep the default set to CTF 1.8 for your application, use the new --default-trace-format option of lttng-sessiond.

The LTTng project adopts CTF 2 because it has many benefits over CTF 1.8:

This per-recording session configuration replaces the mechanism introduced in LTTng 2.14 where setting the LTTNG_EXPERIMENTAL_FORCE_CTF_2 environment variable to 1 at lttng-sessiond startup forced all recording sessions to produce CTF 2 traces.

Channel Memory Reclaim

A new channel memory reclaim operation, exclusively available for user space channels, can free memory used by eligible sub-buffers by deallocating their backing memory when no longer needed.

The two ways to reclaim memory are:

Automatic Reclaim

Configure a per-channel, automatic reclaim policy which periodically tries to reclaim memory for sub-buffers older than a given age, or as soon as they're consumed, with the new --auto-reclaim-memory option of the lttng enable-channel command.

Note: This option is only available in discard mode (without the --overwrite option).

Immediate Reclaim

The new lttng reclaim-memory command reclaims memory immediately for eligible sub-buffers of one or more channels.

Only reclaim the sub-buffers older than a given age with the --older-than option:

lttng reclaim-memory --older-than=2m

Both memory reclaim modes are possible because channel ring buffers are now backed by sparse files when the file system supports them.

Buffer Preallocation Policy

Choose the buffer preallocation policy of a user space channel with the new --buffer-preallocation option of the lttng enable-channel command:

lttng enable-channel my-channel --userspace --buffer-preallocation=on-demand

LTTng can either preallocate all ring buffer memory up front (pre-2.15 behaviour; still the default) or allocate as needed. Choose preallocation to ensure predictable memory usage and avoid run-time allocation latency; choose on demand to minimize the initial footprint for low traffic, for short-lived recording sessions, or in conjunction with memory reclaims.

This feature is also enabled by sparse file-backed ring buffers.

Watchdog Timer

Before this release, an application terminated while writing to a user space ring-buffer could leave a sub-buffer stuck in a partially-written state, making it unusable.

A new health check mechanism, the watchdog timer, now periodically monitors user space ring-buffers and makes stalled sub-buffers consumable again to keep trace recording running smoothly.

This mechanism is enabled by default and can be controlled with the new --watchdog-timer option of the lttng enable-channel command. As of LTTng 2.15, it only applies to user space channels with a per-user buffer ownership model.

Redesigned lttng list Command

We completely revamped the human-readable output of the lttng list command.

The command now has a clean tree structure. Colors help improve the readability of objects and properties when the connected terminal supports it: the LTTNG_TERM_COLOR and NO_COLOR (no-color.org) environment variables provide further terminal color control.

✔ Channel `channel0` ❬42 event rules❭
  🞂 Auto. memory reclaim policy: None
  🞂 Loss mode:                   Discard newest event record
  🞂 Preallocation policy:        On demand
  🞂 Ring buffer configuration:   4 sub-buffers of 512.00 KiB per CPU,
                                 per Unix user

  🞂 Timer periods:
    🞂 Monitor timer:  1.00 s
    🞂 Read timer:     Inactive
    🞂 Switch timer:   Inactive
    🞂 Watchdog timer: 2.00 s

  🞂 Statistics:
    🞂 Discarded event records: 23,182

By default, the command:

• Adds empty lines between blocks of related information. Remove those empty lines with the new compact setting of the new --style option.
• Truncates lines, adding an ellipsis, to fit the current terminal width. Avoid truncation with the new --no-truncate option.

If you still prefer or need the previous ("legacy") version, set the LTTNG_LIST_LEGACY environment variable to 1 before you run lttng list. Please note, however, that the legacy output will never show anything related to features introduced after LTTng 2.14.

Memory Usage Reporting

The lttng list command now shows the total memory usage of each channel:

🞂 Memory usage: ❲●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳   19.07 MiB /   40.12 MiB

The new --mem-usage option controls the display mode of memory usage: compact shows the memory usage for each Unix user or process while full shows the memory usage for each CPU (if available):

🞂 Memory usage: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳   13.30 MiB /   40.12 MiB
  ┆ For UID 1000 (64-bit): ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄❳   13.30 MiB /   40.12 MiB
  ┆   CPU   0: ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  540.00 KiB /    2.51 MiB
  ┆   CPU   1: ❲●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  360.00 KiB /    2.51 MiB
  ┆   CPU   2: ❲●●●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳    1.30 MiB /    2.51 MiB
  ┆   CPU   3: ❲●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  256.00 KiB /    2.51 MiB
  ┆   CPU   4: ❲●●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳    1.20 MiB /    2.51 MiB
  ┆   CPU   5: ❲●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳    1.07 MiB /    2.51 MiB
  ┆   CPU   6: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  768.00 KiB /    2.51 MiB
  ┆   CPU   7: ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  524.00 KiB /    2.51 MiB
  ┆   CPU   8: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  792.00 KiB /    2.51 MiB

lttng status Enhancements

The lttng status command now accepts the same options as lttng list:

lttng status --mem-usage=compact --channel=my-channel

lttng status forwards its options as is to lttng list for the current recording session.

UTF-8 Control

The lttng commands won't emit multi-byte UTF-8 sequences when the LTTNG_NO_UTF_8 environment variable is set to 1, making a best effort to find alternatives.

New Trigger Conditions

New trigger conditions are now available for the lttng add-trigger command:

channel-buffer-usage-ge and channel-buffer-usage-le

Satisfied when the buffer usage of a given channel becomes greater/less than or equal to some threshold (size in bytes or ratio of the total channel buffer size):

lttng add-trigger --condition=channel-buffer-usage-ge \
                  --session=my-session --channel=my-channel \
                  --domain=kernel --threshold-ratio=0.75 \
                  --action=notify

session-consumed-size-ge

Satisfied when the total consumed size of the tracing data of all the channels of a recording session becomes greater than or equal to a threshold:

lttng add-trigger --condition=session-consumed-size-ge \
                  --session=my-session --threshold-size=100M \
                  --action=snapshot-session other-session

session-rotation-starts and session-rotation-finishes

Satisfied when the rotation operation of a recording session starts or finishes:

lttng add-trigger --condition=session-rotation-finishes \
                  --session=my-session --action=notify

liblttng-ctl C API Updates

The liblttng-ctl C API is updated to support the new features above:

• Trace format: Set the trace format of a recording session descriptor with lttng_session_descriptor_set_trace_format() before creating the recording session.

• Memory reclaim:

  • Immediately reclaim user space channel memory for eligible sub-buffers with lttng_reclaim_channel_memory()
  • Get the results with lttng_reclaim_handle_get_reclaimed_subbuffer_count() and lttng_reclaim_handle_get_pending_subbuffer_count()
  • Optionally wait for pending sub-buffers to be reclaimed with lttng_reclaim_handle_wait_for_completion()
  • When done, destroy the memory reclaim operation handle with lttng_reclaim_handle_destroy()

• Automatic memory reclamation policy: Set and get with lttng_channel_set_automatic_memory_reclamation_policy() and lttng_channel_get_automatic_memory_reclamation_policy()

• Buffer preallocation policy: Set and get with lttng_channel_set_preallocation_policy() and lttng_channel_get_preallocation_policy()

• Watchdog timer: Set and get the watchdog timer period with lttng_channel_set_watchdog_timer_interval() and lttng_channel_get_watchdog_timer_interval()

• Data stream info API: Enumerate and query the CPU ID and memory usage of the data streams of a channel:

  • lttng_channel_get_data_stream_info_sets()
  • lttng_data_stream_info_sets_get_count()
  • lttng_data_stream_info_sets_get_at_index()
  • lttng_data_stream_info_set_get_count()
  • lttng_data_stream_info_set_get_uid()
  • lttng_data_stream_info_set_get_pid()
  • lttng_data_stream_info_set_get_app_bitness()
  • lttng_data_stream_info_set_get_at_index()
  • lttng_data_stream_info_get_cpu_id()
  • lttng_data_stream_info_get_memory_usage()
  • lttng_data_stream_info_sets_destroy()

systemd Integration

The session daemon and relay daemon now support systemd's sd_notify() protocol, enabling proper synchronization with dependent systemd units by signaling when the daemons are ready to receive commands and when they're stopping.

Other Changes

• The internal v2.15 communication protocol between LTTng-tools and LTTng-UST is incompatible with the LTTng 2.14 one: you cannot use LTTng-tools 2.15 with LTTng-UST 2.14 and vice versa.

• The lttng Zsh completion function now fully completes the add-trigger command. From now on, the Debian/Ubuntu packages of LTTng-tools will contain the Zsh completion functions.

Version Name

This release is named after Péché Mortel, the iconic Imperial Stout from Montréal's Dieu du Ciel! and a long-time team favorite.

Deep black and unapologetically intense, Péché Mortel pours with a dense, mocha-tinted head and an aroma that immediately announces freshly roasted coffee. Decadent and dangerously drinkable, it is the ideal companion for long, cold winter evenings.

Important Links

Downloads

• lttng-tools-2.15.0-rc1.tar.bz2
• lttng-ust-2.15.0-rc1.tar.bz2
• lttng-modules-2.15.0-rc1.tar.bz2

Resources

Continuous Integration

• LTTng-tools CI
• LTTng-ust CI
• LTTng-modules CI

Code Review

• lttng-tools
• lttng-ust
• lttng-modules

References