zeek/zeek v5.1.0-rc1

v5.1.0

on GitHub

Breaking Changes

• The Packet::{l2,l3}_checksummed variables were reworked to correctly match
  the network layers that they apply to. A new Packet::l4_checksummed
  variable was added to cover the transport layer. See this GitHub issue for
  more detail: #2183.

• The STREAM mode of the ASCII reader now behaves like tail -F: when file is
  removed/replaced, it will start tracking the new file. See
  #2097 for more detail

• The Dictionary and PDict classes are now C++ templates. This may cause
  plugin/package builds to fail due to needing to modify uses of them to match.

• By default, files.log does not have the fields tx_hosts, rx_hosts
  and conn_uids anymore. These have been replaced with the more commonly
  used uid and id fields. They can be re-instantiated by loading the
  following policy script through local.zeek:

  @load frameworks/files/deprecated-txhosts-rxhosts-connuids

  Note, however, that this script will be removed with Zeek 6.1. Consumers
  of files.log should convert to using the singular uid and id
  fields instead.

• The files.log is now unrolled consistently. That is, when Zeek associates
  multiple connections with a single file, each of these connections will result
  in individual files.log entries with unique connection uids, all sharing
  the same file uid.

  This unrolling behavior always existed in a Zeek cluster when the network
  connections involved in a file transfer are load-balanced to different
  workers. Due to this affecting only a marginal ratio of files on real-world
  networks, unrolling the log was chosen as the more efficient approach over
  making the current logic cluster aware.

  The seen_bytes and missing_bytes fields of a File::Info record
  continue to represent the total number across all connections seen by the
  current instance of Zeek.

• The barnyard2 policy scripts have been removed. The integration with the
  Barnyard2 project used the pre-Broker Broccoli library, which got removed in
  Zeek 3.0.

• The unified2 analyzer and accompanying scripts have been removed without
  deprecation.

• The return value of packet_analysis::IP::ParsePacket has changed to return
  enum values. This makes it easier to look at the result and immediately know
  what it means. Unfortunately, because we can't overload a method on the return
  value alone, we aren't able to deprecate the original version of the method.
  This may cause build of packages to fail if they were using this method.

• Conditional directives (@if, @ifdef, @ifndef, @else and
  @endif) can not be placed directly following if, for or while
  statements anymore. This was interpreted non-intuitively and could lead to
  subtle bugs. The statement following the directive was placed outside of its
  intended block. Placing braces after if, for or while should
  result in the intended behavior.

• The bro symlink to the zeek binary has finally been removed.

New Functionality

• Added support for the /s regular expression modifier. Using this modifier in
  patterns in Zeek scripts will cause the '.' character to also match newline
  characters.

• Added a new telemetry framework for providing high-level access to Zeek's
  metric subsystem. This framework allows script writers to use different
  metric types (counters, gauges and histograms) for tracking metrics without
  using lower-level BiFs from telemetry.bif. Additionally, metrics can
  now be accessed from script land using Telemetry::collect_metrics() and
  Telemetry::collect_histogram_metrics().

  The framework is located in base/frameworks/telemetry.

  In addition to the Prometheus endpoint for metrics export that has existed
  since Zeek 4.1, two new log streams, telemetry.log and telemetry_histogram.log,
  can be enabled by loading policy/frameworks/telemetry/log. This policy
  script is included in local.zeek by default.

  For further details on the framework and examples, please refer to the
  Zeek documentation.

• Allow redef'ing the &log attribute of record fields:

  redef Notice::Info$email_dest -= { &log };

  While the syntax allows for any attribute, only &log is supported. The
  semantics for other record field attributes are not easy to grasp and there
  were no obvious use-cases identified.

• Introduced a global disabling_analyzer() hook to allow vetoing calls
  to disable_analyzer().

  The contract is simple: Any script can veto a disable_analyzer() call by
  breaking from this hook. The decision is local to the script taking into
  account any state attached to the connection or state stored elsewhere.
  A script breaking from the hook takes over responsibility to call
  disable_analyzer() at a later point when it finds the condition due
  to which it vetoed fulfilled (which may be never).

• Add support for iterating over indices and values of a vector using the
  same syntax as used for iterating over key-value pairs of tables, where
  value will be set to vec[idx].

  local vec = vector("zero", "one", "two");
  for ( idx, value in vec )
  print idx, value;

• The Supervisor framework now allows better control over where to place
  additional scripts in the load sequence of new nodes. It previously always
  loaded such scripts after any other user scripts, which could create pitfalls
  when users expected their scripts to run last. Scripts placed in
  NodeConfig's new addl_base_scripts and addl_user_scripts fields
  will be loaded after the base scripts (and thus before any user scripts) and
  after any user scripts, respectively. The old ``NodeConfig$scripts` field
  still adds to the very end and is deprecated.

• Added a new script-level option max_changes_per_connection to limit the
  number of tunnel_changed events that can be sent for a connection. This
  helps prevent log spam from connections that regularly swap. The option
  defaults to 5, and can be set to zero do disable the limiting.

• Added a new BIF bytestring_to_float for converting 4-byte bytestrings to
  float values.

• Added a new BIF pow.

• Added new bit-shift operators << and >> for use in scripts.

• Added a new BIF table_keys which returns a set of keys from a table.

• Added a new BIF table_values which returns a vector of keys from a
  table.

• Added new fields to the Modbus log for the Modbus PDU type, the transaction
  ID, and the unit ID. See #2281 for more
  information.

• Added support for parsing TCP option 27, and fixed validation of lengths for
  TCP options 28, 29, and 34.

• Added new packet-analzyer to handle the DLT_LINUX_SLL2 PCAP link type.

Changed Functionality

• The SSL analyzer now determines the direction of the SSL/TLS session by examining
  the packets, and no longer assumes that the connection originator is the client.
  Due to this, the is_orig field in all SSL/TLS events was renamed to is_client.

  Furthermore, the ssl_history now can indicate that the connection was flipped
  (meaning that it is not in the normal order of the originator is the client) using
  the ^ character. A new ssl_connection_flipped is raised when the connection
  is flipped. Furthermore, a SSL_unclear_connection_direction weird is raised when
  we cannot determine the connection direction, because both sides of the connection
  send packets that are associated with being a client/server.

• The default logging directory is now set globally across all log
  writers through Log::default_logdir.

• Calling Option::set() when Zeek is terminating is now a noop and returns F.
  This prevents callbacks into script-land through change handlers when parts
  of the environment have already been torn down.

• When running in cluster mode, the manager by default now imports metrics from
  all other cluster nodes and opens port 9911/tcp for Prometheus metrics exposition.

• The smb2_file_delete event will now be raised for SMB2 CREATE requests
  marked with the FILE_DELETE_ON_CLOSE option.

• Fixed bytestring_to_count to handle 3-, 5-, 6-, and 7-byte strings.

Removed Functionality

• The barnyard2 policy scripts have been removed.

• The unified2 analyzer and accompanying scripts have been removed.

Deprecated Functionality

• The PDict class is now an alias to Dictionary and has been
  deprecated. Use Dictionary directly, passing a pointer type to the
  template.

• LogAscii::logdir and per-writer log directories have been deprecated in
  favor of the new Log::default_logdir.

• The HOOK_BRO_OBJ_DTOR hook and associated methods have been
  deprecated. They are replaced by the HOOK_OBJ_DTOR hook and methods.

• The bro_int_t and bro_uint_t types have been deprecated and replaced
  by zeek_int_t and zeek_uint_t.

• The bro_inet_ntop.h and bro_inet_ntop.c files have been deprecated and
  replaced by zeek_* files.

• The BRO_PLUGIN_API_VERSION has been deprecated and replaced by
  zeek::PLUGIN_API_VERSION.

• The misc/scan.zeek script has been marked for removal in Zeek 6.1. Use
  github.com/ncsa/bro-simple-scan instead.

• The Supervisor framework's NodeConfig$scripts field has been deprecated and
  marked for removal in Zeek 6.1. Use NodeConfig$addl_user_scripts instead.