tstack/lnav v0.14.0-beta2

on GitHub

lnav v0.14.0

Features:

• The Filter configuration panel in the TUI now supports
  editing the minimum log level and minimum/maximum
  times for the current view. Previously, only the
  :set-min-log-level, :hide-lines-before, and
  :hide-lines-after commands could be used to set the
  values. Pressing l will create/set the log level.
  Pressing m will create/set the minimum
  time and pressing Shift + M will create/set the
  maximum time. Live preview has also been added to
  show which lines will be filtered out when the min/max
  is applied.

• The -S/--since and -U/--until flags have been
  added to limit how much of a log file is indexed.
  Supported values are relative (e.g. "yesterday",
  "30 min ago") or absolute local times ("2020-01-01").
  The values on the main command-line set the defaults
  for all values that are opened. The :open command
  supports the same options to change the values from
  the default for a particular file. Files with
  content that lie completely outside of the cutoff
  will be closed to reduce resource usage.

• Broadened support for viewing files on remote hosts.
  The "tailer" program that is transferred to the
  remote host to monitor files and perform other tasks
  has been translated to Python3. If the remote host
  does not have Python, lnav will fall back to the APE
  binary.

• The "Files" panel now shows a progress bar for each
  file as it is being indexed and finishes with a
  check-mark if indexing was successful, a warning
  sign if the file has some notes, or an error mark
  if something else happened.

• Introducing "Log-Oriented Debugging", a collection of
  features to streamline mapping log messages back to
  the source code that generated them. For example,
  given the log message "Hello, Steve!" and the source
  directory containing the log statement. lnav can
  find the line of code that generated the message,
  such as logging.info("Hello, %s!", name), and
  determine the value of the substituted variables
  (name => Steve). This functionality is
  implemented using the
  log2src project.
  The following features have been added in support of
  this functionality:

  • The :add-source-path command was added to specify
    the source directories to be scanned for log
    statements.
  • Log formats can now specify source file/line and
    thread ID with the src-file-field, src-line-field,
    and thread-id-field properties. These fields can
    then be accessed in the SQL vtables as log_src_file,
    log_src_line, and log_thread_id.
  • The :breakpoint, :toggle-breakpoint, and
    :clear-breakpoints commands have been added to
    support setting/clearing breakpoints for log messages.
    The CTRL-B shortcut toggles a breakpoint on the
    focused line in the LOG view. Also, if the log
    format specifies source file/line fields, the first
    character of the source file will be underlined and
    can be clicked to toggle a breakpoint. Once
    breakpoints have been added, you can press F7/F8
    to move to the previous/next log message that
    matches a breakpoint.

• The all_opids and all_thread_ids virtual tables
  have been added to make it simple to discover all of
  the operations and threads across all log files. The
  all_opids table also supports setting a description
  for an operation using through an UPDATE.

• The :xopen command will now open text files in an
  external editor. To open the file at a particular
  line/column, add a URL fragment of the form
  L<line>C<column>.

• When opening the contents of the prompt in an external
  editor (CTRL+O), the cursor position will be preserved,
  if possible.

• The external-editor configuration has been expanded
  with extra properties to help lnav choose the right one
  to use:

  • The config-dir property specifies the name of the
    directory that stores the editor's configuration in a
    source tree. If the directory is found in an ancestor
    of the path to be opened, and it has the most recent
    modified time, the associated editor will be used.
  • The prefers property is a regular expression that
    will be tested against the full path to be opened.
    If matched, that editor will be chosen.

• The :external-access command has been added to open a
  localhost HTTP port that can be used to remotely control
  lnav. Requests can be sent to execute commands and poll
  for changes in the view state. When the external port
  is open, a globe icon (🌐) is displayed in the top-right
  corner. Clicking that icon will open a URL in a browser
  and log you into the server. The :external-access-login
  command can also be used to login.

• Custom "Apps" can be added to the "external access"
  server to provide custom browser-based user interfaces
  to lnav. See the "External Access" documentation online
  for more details.

• The ;.save SQL command has been added that can save
  tables you have created to a SQLite database file. The
  tables that lnav creates have been moved to a separate
  in-memory DB, so the main DB should only contain your
  own tables/views/etc...

• The json_object_count_of() SQL aggregate function has
  been added to make it easy to create a JSON object
  where the values are the number of times a value has
  been seen.

• The :write-json-cols-to command has been added to
  write JSON output in a column-oriented fashion.

• The encode() and decode() SQL functions now accept
  html as an algorithm.

• The opid-field can now be set to a JSON array/object
  for JSON-lines logs. For example, the spans array in
  a Rust tracing log message. The OPID for the message
  will be computed by hashing the contents of the array
  or object and the description will be the container
  itself.

• An OPID can now be constructed from multiple fields by
  leaving the opid-field blank and creating a single
  opid/description definition with a format. The
  content of the format fields will then be hashed to
  create the OPID. The builtin log formats have been
  updated to use this when appropriate. For example,
  access_log now uses c_ip and cs_user_agent as the
  OPID.

  [!NOTE]
  If you want a description, but don't want it used as
  the OPID. You can set the opid/source field to
  "from-whole-msg" and the OPID will be computed from
  the contents of the log message.

• The duration-field log format property has been added
  to specify the field that contains a duration in the
  log message. If a duration is available, it will be
  used to calculate time spans in the TIMELINE view.

• The timestamp-point-of-reference log format property
  has been added to specify the related of the timestamp
  to the operation that the message refers to, either:
  start or send. This is used in conjunction with the
  message duration to determine the time span.

• The OPID for log messages is now shown in the parser
  details overlay (revealed by pressing p) in the
  LOG view.

• Added rust_tracing_log format from @richard-hajek.

• Added macosuni_log format that understands the
  output of the macOS log stream --style=ndjson
  command.

• Added the idea_log format from @segevfiner.

• The strace_log format has been improved to handle
  more output formats and the syscalls will now show
  up in the TIMELINE view.

• The strace:// URL-handler has been added to make it
  easier to run strace on an existing process. A
  host must be given and the path should be the PID,
  such as strace://localhost/1234.

• Added the nestable flag to the log format and
  theme highlight configurations to control whether a
  highlight can be applied to text that is covered by
  another highlight.

• Search tables are now included in the output of
  the :export-session-to command.

Breaking changes:

• All of lnav's SQLite tables have been moved to a
  separate in-memory database that is attached as
  lnav_db. You may need to update some of your SQL
  statements to qualify table names with lnav_db..
  This change was made so that the main DB only contains
  user data that can be easily backed up to a new DB
  file.
• Timestamp columns and results from lnav time functions
  now have microsecond precision instead of millisecond.
• The "module format" functionality has been removed.
  This functionality tried to match log messages wrapped
  in another format (usually syslog), but it never
  really worked well and was impeding progress in other
  areas. Also, there have been many features added
  since the beginning that can serve the same use cases.
• The sudo_log format has been removed since it was
  a module-only format. Instead, a sudo_log
  search-table was added to the syslog_log format.

Interface changes:

• Mouse mode is now enabled by default.
• The CTRL+f hotkey has been remapped to the
  :toggle-filtering command.
• Aborting the prompt now requires two successive
  presses of Esc (a message will pop up on the right
  that mentions this). Since Esc is also used to
  close the completion popup, it was too easy to
  cancel the prompt. Pressing CTRL+] will still
  close the prompt immediately.
• The TIMELINE view has a few updates:
  • The header has been redesigned to be one line that
    shows the time increments at the current scale.
    This approach should more clearly convey the spans
    of time shown in the main part of the view. The
    previous design tried to show the overall time and
    the current time frame. But, the multi-line header
    was hard to interpret and didn't make it clear how
    large the time increments were.
  • Log files and threads are now shown in the view
    in addition to operations.
• The HIST view now shows the year and inserts a spacer
  row in-between gaps in time. The spacer row shows
  bullet points on a log scale to represent the amount
  of time in the gap.
• The SPECTRO view now shows the year in timestamps
  and uses additional colors to show the value range.
• The breadcrumb bar in the LOG view now includes the
  current thread, if defined.
• If there are background tasks, like the processing done
  by :add-source-path, a panel with progress bars for
  each operation will be shown just above the bottom
  status bar.
• The first line of a multi-line log message will now be
  shown in the header of the LOG view if the message has
  scrolled off the screen. When scrolled to the
  beginning of the content, the top line will indicate
  if any log messages were filtered out before the first
  message. Left-clicking on the header will scroll the
  view to the displayed line.
• In the Filters configuration panel, you can now
  create/edit the SQL expression filter by pressing the
  e key.
• The { hotkey and :prev-section command will now
  move to the first line of a multi-line log message if
  the focused line is in the middle of the message.
• The } hotkey and :next-section command will now
  move to the next log message if the focused line is
  in the middle of a multi-line message.
• The parser-details overlay now mentions any search
  tables that match the focused line. Now, you don't
  have to remember the names of the tables.
• If a text file contains invalid UTF-8 content, the
  invalid bytes will be shown as the replacement
  character (�) and, when the line is focused, an
  overlay will show a hex dump of the line.

Bug Fixes:

• If a file path contains a hash (#), check if the path
  exists before splitting around the hash and treating it
  as an internal file location.
• The initialization sequence has been cleaned up to
  hopefully make it more consistent/reliable. This
  concerns the sequence of loading files and executing
  commands from the command-line.
• In the SPECTRO view, you can now move to the next/
  previous row with bookmarks. Note that this view
  synchronizes bookmarks with the LOG view.
• Session state should no longer override commands
  passed on the command-line.

Internal:

• Added operation IDs (OpIDs) to lnav's internal logging
  to make it easier to identify the high-level operations
  that are being performed using the TIMELINE view.
  While this is mainly useful to the authors of lnav, it
  can serve as a good example of the benefits of adding
  OpIDs and the TIMELINE view.