zeek/zeek v4.1.0

on GitHub

New Functionality

• Lambda functions can now use capture-list to help specify exactly which local
  variables from outer scopes need to made available while evaluating the lambda
  and also the method by which they're made available: deep vs. shallow copy.

  For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function

• Support for VN-Tag protocol headers: the new VN-Tag packet analyzer simply
  skips past the VN-Tag header, allowing for further analysis of subsequent
  packet contents.

• Support for decapsulating Geneve packets to process the inner
  payload, similar in operation to the existing VXLAN support.

• Support for Zeek script "Reaching Definitions" (RD) analysis: tracking the
  extent to which a given variable definition (assignment) can be visible
  elsewhere in the Zeek script. The analysis works on local variables in
  function/event/hook bodies, but not across them.

  The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a
  given point in a function body) and "maximal" (what possibly could reach).
  Upcoming script optimization will use the latter, but the former currently
  allows identification of places for which a value is used where it
  does not appear that it will necessarily be defined. Specifying the
  zeek -u option will generate warnings for instances where this holds for
  local variables. Specifying zeek -uu turns on additional (expensive)
  analysis to report instances where record fields might be used without
  having previously been set.

  The zeek -u option can also identify assigned-to variables that aren't
  subsequently used (i.e. "dead code") and issues a warning. A new is_used
  attribute can be used situationally to suppress such warnings.

  The base scripts have some places where the static analysis lacks sufficient
  power to tell that values are being used safely (guaranteed to have been
  identified). In order to enable users to employ zeek -u on their own
  scripts without being distracted by these instances, this change also
  includes a new attribute, &is_assigned, which can be associated with a
  variable or a record field to inform Zeek's analysis that the script writer
  asserts the value will be set, suppressing the associated warnings.

• A Telemetry API was added to assist in gathering arbitrary runtime
  metrics and allows export to Prometheus. This is still
  work-in-progress, preliminary documentation for current, low-level
  API lives at https://github.com/zeek/zeek/wiki/Telemetry for now.

• Experimental support for translating Zeek scripts to equivalent C++.
  The generated C++ can then be compiled directly into the zeek binary,
  replacing use of the interpreter and producing better runtime performance.
  See src/script_opt/CPP/README.md for a guide on how to use this feature.

• Support for more generic session management. The NetSessions class has been
  renamed to SessionMgr (with the old name marked deprecated). The new
  class allows plugins to take advantage of session management similar to how
  Connection objects were handled previously, but without the need to be based
  on IP-based protocols.

• The logging framework now provides a global policy hook, Log::log_stream_policy.
  Like the existing filter-level hooks, handlers for the new hook can provide
  additional processing and veto the log write. The new hook runs once per
  write, prior to any filter-level policy hooks. Even when it vetoes,
  filter-level policy hooks still run, but cannot "un-veto" the write.

• The ASCII writer gained a new option LogAscii::logdir, which can be used to
  change the logging output directory.

• Added a --include-plugins argument to configure. This argument
  takes a semicolon separated list of paths containing plugins that will be
  statically built into Zeek.

• Added a --plugindir argument to configure to set the
  installation path for plugins.

• The X509 analyzer now can check if a specific hostname is valid for a
  certificate. Two new BIFs were added for this, x509_check_hostname and
  x509_check_cert_hostname. A new field sni_matches_cert that tracks
  this information was added to ssl.log.

• Added new functions to dynamically enable/disable file analyzers:

  • global enable_analyzer: function(tag: Files::Tag): bool;
  • global disable_analyzer: function(tag: Files::Tag): bool;
  • global analyzer_enabled: function(tag: Files::Tag): bool;

• Zeek now includes its own BTest tooling in the distribution, enabling other
  tests (e.g. in Zeek packages) to use it. The $PREFIX/share/btest folder,
  reported via zeek-config --btest_tools_dir, includes:

  • scripts/ for btest-diff canonifiers
  • data/ for data files, including random.seed
  • data/pcaps for the test pcaps

  Configuring with --disable-btest-pcaps suppresses installation of the
  test pcaps.

• The Supervisor now defaults to starting with a minimal set of Zeek
  scripts controlled by a new init file, base/init-supervisor.zeek.
  One may still run it with a larger configuration by loading additional
  scripts, including init-default.zeek, as always. (Bare mode continues
  to work as usual, reducing the configuration to a minimum.)

  The NodeConfig record has two new members, providing additional
  control over launched nodes. The env member allows setting environment
  variables in the launched process. The bare_mode member, an optional
  boolean, provides control over the bare-mode state of the new node.
  When not provided, the node inherits the bare-mode status of the
  Supervisor, and setting the variable enables/disables bare mode.

• Zeek now includes an incomplete, preliminary version of the future
  cluster controller framework. Loading policy/frameworks/cluster/agent
  and/or policy/frameworks/cluster/agent in a Zeek running with the
  Supervisor will launch the corresponding cluster management node(s).
  An experimental management client, zeek-client, connects to the
  controller and lets you issue commands. It requires configuration with
  --enable-zeek-client. This does not yet provide a functional
  substitute for zeekctl, which users should continue to use for now.

Changed Functionality

• The default IP-based transport protocols (UDP, TCP, and ICMP) have been
  moved to the packet analysis framework. This change allows us to move other
  analyzers in the future that better align with the packet analysis framework
  than they do with session analysis.

• The input framework's handling of unset fields (commonly expressed as "-")
  in ingested data is now safer and more consistent. When reading data into
  records, Zeek now accepts unset fields in the input data only when the
  corresponding record field is &optional. Unset fields for non-optional
  fields cause the input line to be skipped. Reading data into tables with complex
  index types (such as table[count, count] of string) now also skips lines
  with unset fields, since such indexes require fields to be present.

  Note that this may change the behavior of existing scripts if you have unset
  fields in your input data.

• The version field in ssh.log is now optional and will not be set if we cannot
  determine the version that was negotiated by the client and server.

• Add a new field email_dest to NOTICEs, which defines where to
  send email to. The email-related NOTICE actions fill this now, and
  then emails will be sent to all recorded addresses at the end of
  NOTICE processing. This makes email generation more consistent and
  extensible.

• Add page and email administrator to mails processed by hostnames extension.

• SSL and X509 handling was significantly overhauled with the goal to make
  the data that is logged by Zeek more helpful and compact.

  This change means that there are significant changes to the default log files,
  as well as changes to functionality:

  • x509.log is now indexed by the sha256 of the certificate, with deduplication
    being automatically performed. By default, the same certificate is only logged
    once per day.

    This also means that the file ID is no longer present in X509 log. Similarly,
    ssl.log now contains hashes for X509 certificates.

    The hash function that is used for indexing the certificates is changeable by
    changing the X509::hash_function option.

    The time period after which a certificate is logged again can be configured by
    changing X509::relog_known_certificates_after.

    By default deduplication of certificates is done across the entire cluster using
    broker. If this is not desired due to the higher communication overhead, this
    behavior can be disabled using X509::known_log_certs_use_broker.

  • X509 certificates are, by default, no longer logged into files.log. This
    behavior is configurable and the previous default can be restored by changing
    the X509::log_x509_in_files_log option.

  • x509.log now tracks if a certificate was encountered as a end-host certificate
    or as a client certificate.

  • OCSP logging is now enabled by default.

  • ssl.log now no longer includes information about the certificate issuer and
    subject. This information is still available in X509.log. If you need this
    information in ssl.log, the old behavior can be restored by changing the
    SSL::log_include_server_certificate_subject_issuer and
    SSL::log_include_client_certificate_subject_issuer configuration options.

  • ssl.log now contains a ssl_history field, which tracks which protocol
    messages were seen in an SSL/TLS connection.

  • We added a policy script ssl-log-ext.zeek which greatly extends the amount
    of protocol information logged to SSL.log. The script is not loaded by default.

  • We added a disable-certificate-events-known-certs.zeek policy script. This script
    will completely disable X509 events for known certificates over SSL/TLS connections.

    For Zeek installations in settings where you encounter a lot of certificates,
    this could improve the performance of your installation. Before enabling this
    script, make sure that you do not use any third-party scripts that depend on the
    X509 events. The script is not loaded by default.

  • The ICSI SSL Notary script was deprecated. This functionality is superseeded by newer
    approaches, like SCT validation (which is supported by Zeek).

  • extract-certs-pem.zeek was deprecated - it never really worked in cluster modes.
    A new policy script, log-certs-base64.zeek that can be used to log raw certificates
    was added instead.

• The CT logs listed in ct-list.zeek are now derived from the list of CT
  logs that are accepted by Google Chrome. In the past, we allowed the list
  of all known CT logs. This no longer makes sense since nowadays logs exist that
  contain, e.g., only outdated or invalid certificates. If the old behavior is
  desired, you can re-add Logs to SSL::ct_logs.

• The Mozilla CA list was updated to the state of NSS 3.67.

• SQLite was updated to 3.36.0.

• The list of subnets that are considered private addresses was updated to mostly
  match IANA's list unroutable addresses. This brings it in line with Chrome, the
  W3C spec, and Python's ipaddress module.

Removed Functionality

• Support for the RocksDB Broker data store was previously broken and unusable,
  so all code/options related to it are now removed.

• Support for the ENABLE_MOBILE_IPV6 compiler variable has been removed. Mobile
  IPv6 is now enabled by default. The --enable-mobile-ipv6 option for
  configure now returns a warning that it will be removed in v5.1 and no
  longer has any effect.

Deprecated Functionality

• Lambda/closure support: automatic capturing of references to variables
  outside a lambda's scope is now deprecated. An explicit capture
  list which also specifies the desired copy-semantics is now required when
  writing lambda functions that refer to local variables of an outer scope.

  For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function

• The IterCookie version of iteration over Dictionary and PDict
  objects was marked as deprecated. It was replaced by standard-library
  compatible iterators. This enables the use of standard constructs such
  as ranged-for loops to iterate over those objects.

• The zeek::util::zeekenv() function is deprecated since use of all
  environment variables prefixed by BRO_ is now removed and calling
  getenv() directly with ZEEK_ environment variables can be done.

• supervisor_rotation_format_func is renamed to archiver_rotation_format_func

• The MemoryAllocation() function implemented by a number of interfaces
  is now deprecated. In testing we found that the values returned were mostly
  incorrect and weren't useful. The val_size and global_sizes BIF
  methods have also both been marked deprecated.