zeek/zeek v7.1.0-rc1

on GitHub

Breaking Changes

• The OpaqueVal::DoSerialize and OpaqueVal::DoUnserialize methods were
  marked as deprecated in v7.0 and have now been removed as per the Zeek
  deprecation policy. Plugins that were overriding these methods and were not
  updated will fail to compile. Those plugins should be updated to override the
  new OpaqueVal::DoSerializeData and OpaqueVal::DoUnserializeData
  methods.

• Certain internal methods on the broker and logging classes have been changed to
  accept std::vectorthreading::Value parameters instead of threading::Value**
  to leverage automatic memory management, reduce the number of allocations
  and use move semantics to express ownership.

  The DoWrite() and HookLogWrite() methods which can be provided by plugins
  are not affected by this change, so we keep backwards compatibility with
  existing log writers.

• Func::Name() was deprecated, use Func::GetName() instead.

New Functionality

• The following dependencies have had updates:

  • The bundled version of Spicy was updated to 1.12.0. See
    https://github.com/zeek/spicy/releases/tag/v1.12.0 for notes on what's new
    with Spicy.

  • The bundled version of c-ares has been updated to v1.34.2, which required
    some updates to Zeek's internal DNS resolver due to changes in the c-ares
    API. At least version v1.28.0 is now required to build Zeek.

  • Python 3.9 is now required for Zeek and all of it's associated subprojects.

• IP-based connections that were previously not logged due to using an unknown
  IP protocol (e.g. not TCP, UDP, or ICMP) now appear in conn.log. All conn.log
  entries have a new ip_proto column that indicates the numeric IP protocol
  identifier used by the connection. A new policy script at
  policy/protocols/conn/ip-proto-name-logging.zeek can be loaded to also add
  an ip_proto_name column with a string version of the ip_proto value.
  This entire feature can be disabled by loading the new
  policy/protocols/conn/disable-unknown-ip-proto-support.zeek policy script.

• New Cluster::publish(), Cluster::subscribe() and Cluster::unsubscribe()
  functions have been added. In contrast to their Broker counterparts, these
  will operator on whichever cluster backend is enabled. Going forward, in-tree
  Broker::publish() usages will be replaced with Cluster::publish() and
  script writers should opt to prefer these over the Broker-specific functions.

• Zeek now includes a PostgreSQL protocol analyzer. This analyzer is enabled
  by default. The analyzer's events and its postgresql.log should be
  considered preliminary and experimental until the arrival of Zeek's next
  long-term-stable release (8.0).

  If you observe unusually high CPU consumption or other issues due to this
  analyzer being enabled by default, the easiest way to disable it is via the
  Analyzer::disabled_analyzers const as follows:

  redef Analyzer::disabled_analyzers += {
  Analyzer::ANALYZER_POSTGRESQL,
  };

  If you observe PostgreSQL traffic in your environment, please provide feedback
  about the analyzer and structure of the new log.

• Broker's message I/O buffering now operates on per-peering granularity at the
  sender (it was previously global) and provides configurable overflow handling
  when a fast sender overwhelms a slow receiver, via the following new tunables
  in the Broker module:

  const peer_buffer_size = 2048 &redef;
  const peer_overflow_policy = "disconnect" &redef;
  const web_socket_buffer_size = 512 &redef;
  const web_socket_overflow_policy = "disconnect" &redef;

  When a send buffer overflows (i.e., it is full when a node tries to transmit
  another message), the sender may drop the message and unpeer the slow receiver
  (policy disconnect, the default), drop the newest message in the buffer
  (drop_newest), or drop the oldest (drop_oldest). Buffer sizes are
  measured in number of messages, not bytes. Note that "sender" and "receiver"
  are independent of the direction in which Zeek established the peering. After
  disconnects Zeek automatically tries to re-establish peering with the slow
  node, in case it recovers.

  Zeek notifies you in two ways of such disconnects:

  • A cluster.log entry for the sending node indicates that a slow peered node
    has been removed. Here node worker01 has removed a peered ``proxy01`:

  1733468802.626622 worker01 removed due to backpressure overflow: 127.0.0.1:42204/tcp (proxy01)

  • The labeled counter metric zeek_broker_backpressure_disconnects_total
    in the telemetry framework tracks the number of times such disconnects
    happen between respective nodes. The following scraped telemetry indicates
    the same disconnect as above:

  zeek_broker_backpressure_disconnects_total{endpoint="worker01",peer="proxy01"} 1

  To implement custom handling of a backpressure-induced disconnect, add a
  Broker::peer_removed event handler, as follows:

  event Broker::peer_removed(endpoint: Broker::EndpointInfo, msg: string)
  {
  if ( "caf::sec::backpressure_overflow" !in msg )
  return;

    # The local node has disconnected the given endpoint,
    # add your logic here.
            }
  

  These new policies fix a problem in which misbehaving nodes could trigger
  cascading "lockups" of nodes, each ceasing to transmit any messages.

• The LDAP analyzer now supports handling of non-sealed GSS-API WRAP tokens.

• StartTLS support was added to the LDAP analyzer. The SSL analyzer is enabled
  for connections where client and server negotiate to TLS through the extended
  request/response mechanism.

• The unknown_protocols() event now includes the name of all packet
  analyzer used for processing the packet when the event is raised. The
  unknown_protocol.log file was extended to include this information.

• The MySQL analyzer now generates a mysql_user_change() event when the user
  changes mid-session via the COM_USER_CHANGE command.

• The DNS analyzer was extended to support TKEY RRs (RFC 2390). A corresponding
  dns_TKEY event was added.

• The signature_match() and custom signature events now receive the end of
  match offset within the data parameter as an optional parameter named
  end_of_match.

  event signature_match(state: signature_state, msg: string, data: string, end_of_match: count);

• A new plugin hook InitPreExecution() has been added to allow introspection
  of Zeek's AST after ZAM optimizations ran. This hook executes right before
  the zeek_init() event is enqueued.

• The SQLite logger now supports setting the value of the SQLite synchronous mode,
  as well as of the journal mode. For example, WAL mode can be enabled by setting:

  redef LogSQLite::journal_mode=LogSQLite::SQLITE_JOURNAL_MODE_WAL;

• A pseudo protocol analyzer StreamEvent has been added. Attaching this analyzer
  to TCP connections allows processing the connection's stream data in the
  scripting layer. One example use-case is interactive terminal sessions over
  HTTP connections upgraded to TCP.

  redef HTTP::upgrade_analyzers += {
  ["tcp"] = Analyzer::ANALYZER_STREAM_EVENT,
  };

  event stream_deliver(c: connection, is_orig: bool, data: string);

  This comes with performance caveats: For use-cases with high-data rates
  a native protocol analyzer with dedicated events will be far more efficient.

• Experimental support for pluggable cluster backends has been added. New plugin
  components have been introduced to support switching Zeek's Broker-based
  publish-subscribe and remote logging functionality to alternative implementations.

  redef Cluster::backend = Cluster::CLUSTER_BACKEND_ZEROMQ;

  Besides the backend, the serialization format used for events and log-writes
  has become pluggable as well.

• The Zeek distribution now includes an experimental ZeroMQ based cluster backend.
  To experiment with it, load the following script on each cluster node.

  @load frameworks/cluster/backend/zeromq/connect

  Note that Broker-dependent scripts or integrations will become non-functional
  when doing so as Zeek nodes will not listen on Broker ports anymore, nor will
  they establish a peering to other nodes.

• Zeek now ships with an experimental Spicy-based SSL analyzer, which is
  disabled by default. This analyzer can be enabled using the
  --enable-spicy-ssl conifgure-time option. The Spicy-based analyzer has
  full support for SSL and TLS, just like the current binpac analyzer. It does,
  however, not support any version of DTLS. Enabling it will disable DTLS
  parsing in Zeek.

  The analyzer is currently mostly interesting if you want to experiment with
  SSL; we do not yet recommend to enable it in normal Zeek deployments.

• The majority of the metrics reported via stats.log are also now reported via
  the Telemetry framework, and are visible in the output passed to Prometheus.

• A new weird DNS_unknown_opcode was added to the DNS analyzer to report
  when it receives opcodes that it cannot process.

Changed Functionality

• Heuristics for parsing SASL encrypted and signed LDAP traffic have been
  made more strict and predictable. Please provide input if this results in
  less visibility in your environment.

• The MySQL analyzer has been improved to better support plugin authentication
  mechanisms, like caching_sha2_password, as well as recognizing MySQL query
  attributes.

• The mysql.log for user change commands will contain just the username
  instead of the remaining parts of the command, including auth plugin data.

• The POP3 parser has been hardened to avoid unbounded state growth in the
  face of one-sided traffic capture or when enabled for non-POP3 traffic.
  Concretely, the Redis protocol's AUTH mechanism enables the POP3 analyzer
  for such connections through DPD.

• Batching and flushing for local log writers can now be controlled via the
  options Log::flush_interval and Log::write_buffer_size. Previously
  the Threading::heartbeat_interval was used for flushing and the buffer
  size fixed at 1000.

• Logging of the FTP PASS command in ftp.log now honors FTP::default_capture_password
  and the password is blanked with "". Previously, the argument for the PASS
  command would be logged in clear.

• The ASCII input reader now suppresses warnings for consecutive invalid lines,
  producing a summary of total suppressions once a valid line is encountered.

• The Telemetry::sync() hook is now invoked on demand. Either when the metrics
  of a node are scraped via the Prometheus HTTP endpoint, or one of the collect
  methods is invoked from Zeek script.

• The community-id-logging.zeek policy script was used to set c$conn$community_id
  during new_connection() rather than connection_state_remove(), allowing
  other scripts to reuse its value early.

• Calling Broker::publish() now uses the event time of the currently
  executing event as network time metadata attached to the remote event.
  Previously, network_time() was used. This matters if Broker::publish()
  is called within scheduled events or called within remote events.

• The SSL analyzer now reports the correct version when an SSLv2 client hello is
  used. Zeek previously always reported these as v2, even when the v2 client
  hello indicated support for a later version of SSL.

Deprecated Functionality

• The Broker::auto_publish() function has been deprecated and should
  be replaced with explicit Broker::publish() invocations that are
  potentially guarded with appropriate @if or @ifdef directives.

• The misspelled complte_flag in the dns_binds_rr record has been deprecated.
  The new complete_flag uses type count instead of string.