tstack/lnav v0.15.0-beta1

on GitHub

lnav v0.15.0

Features:

• Added the :filter-context command to show lines surrounding
  filter matches, similar to grep's -C option. The command
  accepts one or two arguments for the number of messages to
  show before and after each match. In the LOG view, context
  is counted in whole messages (including continuation lines).
  The z/Z keys can also be used to increase/decrease the
  context by one in the LOG, TEXT, and TIMELINE views. Context
  lines are styled using the new context-line theme style.
• Added a built-in metrics_log format that recognizes CSV
  files whose first column header is Time/Timestamp/ts/
  Date... and whose subsequent rows begin with a parseable
  timestamp. Each row is rendered with a timestamp and
  a <column>=<value> pair for each numeric column. Rows
  from multiple files that have the same timestamp are
  merged into a single line. Values are right-aligned and
  decorated with reverse-video bars proportional to each
  column's observed min/max range. When the row is
  focused, an overlay below it labels each column with the
  source file stem. Exports from Excel, PowerShell, and
  Grafana should be handled as-is.
• Added the all_metrics SQL virtual table, a long-format
  view across every open metrics file. The metric column
  contains the name of the metric and value contains its
  value.
• The TIMELINE view now supports overlaying metric
  sparklines at the top of the view. The following commands
  can be used to manage this:
  • :timeline-metric [<source>.]<metric> picks a column
    from any loaded metrics file
  • :timeline-metric-sql <label> <query> takes an
    arbitrary SELECT log_time, value FROM ... that can
    target any table, including search-table columns.
  • :clear-timeline-metric <label> removes metrics.
    Up to four metrics can be added.
• Added support for "tabular" formats (e.g. CSV, TSV).
  The format definition for this type of file sets
  file-type to tabular and then defines the known
  columns. When opening a file of this type, the
  separator will be automatically detected and the header
  compared against the columns defined in the tabular
  formats. If a good match is found, it will be used as
  the format for the file. Quoted cells that span
  multiple physical lines (an embedded LF inside "...")
  are stitched back into a single log message, so an
  Excel-exported CSV with a multi-line free-text column
  shows one logline per row and SQL/search operates on
  the merged cell value.
• Added a log format for the fsck_apfs and fsck_hfs tools on
  macOS, covering both the started/completed lifecycle lines
  and legacy run entries. This replaces the previous
  fsck_hfs_log format, which only matched the start lines.
  The new format exposes device, tool, and action fields,
  groups messages by device in the TIMELINE view, and highlights
  error: lines and FILESYSTEM CLEAN status messages.
• Added a log format for the Asterisk PBX framework. The
  call ID (e.g. C-00000001) is used as the operation ID so
  the messages for a call are grouped in the TIMELINE view.
• Log format value definitions now accept a unit object
  with suffix and divisor properties. suffix specifies
  how numeric fields are humanized. divisor normalizes
  the raw value to the base unit implied by suffix —
  e.g. a field storing milliseconds with "suffix": "s"
  declares "divisor": 1000.
• The details overlay now shows per-column statistics for
  the focused message. Numeric columns get a min..max of N range summary on the value line, plus a p50/p90/p99
  percentile sub-line for columns with enough samples to
  characterize the distribution shape. Identifier and
  metrics-text columns get an estimated distinct-value
  count (~K distinct of N). Distinct counts are computed
  from a HyperLogLog sketch (~4 KB per text column,
  ~1.6% standard error). Stats render in the column's
  declared unit when one is set.
• The FILES panel now reports per-file indexing cost: an
  Index Time row shows cumulative wall-clock vs.
  thread-CPU spent indexing the file. In addition,
  Index Memory and Line Buffer rows show allocations
  for indexing and I/O-cache. The same data is queryable
  as a JSON object through the new stats column on the
  lnav_file SQL vtab, with keys polls, reads,
  index/wall-us, index/cpu-us, index/memory-bytes,
  and line-buffer-memory-bytes.
• The humanize_duration() SQL function now preserves
  nanosecond inputs: a value like 0.0000001 (100 ns)
  renders as "100ns" instead of being floored to
  "0s".
• Added support for "named" searches, which stay active
  and highlighted while other searches are run. This
  allows several patterns to be tracked at the same time.
  The commands related to named searches are:
  • :create-named-search <name> [pattern] for creating
    one. If no pattern is given, the currently active
    search is adopted and then cleared. So, a search
    can be promoted once it turns out to be worth keeping.
  • :delete-named-search <name> removes a named search.
  • :disable-named-search <name> stops a search from
    highlighting and marking without deleting it. Its
    hits are kept up-to-date, so
    :enable-named-search <name> brings it back without
    another pass over the view.
    The matching text is given a background-color derived
    from the name, so each search reads as its own block,
    with the foreground adjusted to stay readable against
    it. The n / N and < / > keys move through the
    hits of named searches as well as the current search.
    One search can be focused with the . and , keys,
    or by name with :focus-search <name>, so that those
    keys move through its hits alone. The cycle runs from
    nothing focused, through the current search, then the
    named searches, and back, so there is always a way
    back to moving through all of them. The status bar
    says which hits the keys are moving through: the name
    of the focused search, its pattern for the current
    search, or all searches with a count that covers
    every enabled search when none is focused.
    Named searches are also surfaced in:
  • The Text Filters panel, where they are listed
    below the view's filters with their hit counts.
    Pressing s creates a new one, . focuses the
    selected one, and the same keys used for filters
    work on these as well.
  • The TIMELINE view where each search is a row that
    shows the span of hits. The current search gets a
    row as well, labelled with its pattern in quotes
    since it has no name. The :hide-in-timeline search command hides all of these rows.
  • The log_named_searches column on the log tables.
    It contains a JSON list of the searches that
    matched a message.
  • The lnav_view_searches table. Rows can be
    INSERTed and DELETEd to create and remove them
    from SQL and the enabled column can be UPDATEd
    to turn them on and off.
    In the LOG view, a named search also creates a search
    table of the same name that contains the messages it
    matched, with a column for each capture in the
    pattern, so the hits can be queried without writing
    the pattern a second time. The table is dropped when
    the search is deleted and kept when it is only
    disabled. Since the name is used for the table, it
    must be a valid SQL identifier.
    Named searches are saved in the session and included
    in the output of :export-session-to.
• Added the :show-only-in-timeline command to show
  only the given row type(s) in the timeline view and
  hide all the others. If no arguments are given,
  only the type of the focused row is shown.
• Installing files with the -i option will now
  validate log format, configuration files, and SQL
  files before installation.
• Added the file split <path> management command to
  split a large log file into smaller files that lnav
  can fully index. Files are only split between log
  messages, so multi-line messages are kept together.
  The size of each piece can be limited with --lines,
  --size, or --time (e.g. --time 1h puts each hour
  of messages in its own file). If no limits are given,
  a size is picked based on the size of the file and
  lnav's indexing limits. Header lines that are needed
  to recognize formats like CSV are copied into each
  piece. The pieces are written to the current
  directory or the one given with -o and the command
  checks that the directory has enough free space,
  keeping the amount set by
  /tuning/archive-manager/min-free-space free.
  The --since and --until options can be used to
  only write the messages in a time range. They
  accept the same kinds of times as the -S and -U
  options.

Interface Changes:

• Moving horizontally now defaults to moving to the
  next "column" in the content instead of half the
  width of the view. This should make it easier to
  fit the full content of a column/captured-field
  into the view. This behavior is supported in the
  following views:
  • In the DB view, pressing the arrow keys will
    move to the adjacent column.
  • In the LOG view, pressing the arrow keys will
    move to the adjacent field as captured by the
    log format.
• Pressing G will now cycle through putting the
  end of the content at the top and bottom of the
  screen in cursor mode.

Performance:

• Indexing performance has been improved in a few
  ways:
  • Multiple files are now indexed at the same time.
  • Date-time scanning has been sped up a bit.
  • The k-way merge of log messages has been optimized.

Bug Fixes:

• Bookmarks in the TEXT view should be more stable.
  The marks now include a reference to the nearest
  anchor (e.g. header in Markdown) and use it as a
  starting point of a search for the matching line.
• Opening a file with more lines than lnav can index
  (about 134 million) would crash. Indexing now stops
  at the limit and a warning is shown for the file.
  The limit can be lowered with the
  /tuning/logfile/max-lines configuration setting.