zeek/zeek v3.1.0

on GitHub

This release primarily adds new features: configuration options, scripting language functionality, and a new "supervisor" deployment mode (alternative to ZeekControl). Performance is improved, especially the handling of SYN-scans and also JSON logging. The main I/O loop of Zeek is rewritten with better idle behavior and reduced CPU load. It also includes many smaller bug fixes and improvements.

Reminder: Zeek 3.0.x is the current Long-Term Support release, receiving bug fixes until at least October 2020.

New Functionality

• Add a new supervisor framework that enables Zeek to operate clusters
  of processes itself without any external help.

  The Supervisor framework provides an entirely new deployment mode
  for Zeek, one that supervises a set of Zeek processes that are meant
  to be persistent. A Supervisor automatically revives any process
  that dies or exits prematurely and also arranges for an ordered
  shutdown of the entire process tree upon its own termination. This
  Supervisor mode for Zeek provides the basic foundation for process
  configuration/management that could be used to deploy a Zeek cluster
  similar to what ZeekControl does, but is also simpler to integrate
  as a standard system service.

  This mode is still experimental and will evolve over time. The
  command-line argument of -j toggles Zeek to run in "Supervisor
  mode" to allow for creation and management of child processes. If
  you're going to test this, please note that you will need some
  custom script code to configure the processes you want Zeek to run.
  See the documentation for more information:
  https://docs.zeek.org/en/current/frameworks/supervisor.html

• Add a new option, dpd_late_match_stop, which can be used in conjuction
  with the option dpd_match_only_beginning and the new event
  protocol_late_match to help annotate the conn.log with a field
  to speculate on the protocol/service in cases where the DPD buffer
  was already exhausted and can't analyze the full connection anymore,
  but where there was still a late signature match. A new script,
  policy/protocols/conn/speculative-service.zeek, was added as an
  example of how to perform this tuning and add a "speculative_service"
  field to conn.log, but it's not loaded by default.

• There is now a new tcp_options event that is raised for each TCP header
  that contains options.

• Added a new option, Log::print_to_log that can be set to automatically
  redirect the output from "print" statements to a real log stream (e.g.
  instead of writing to stdout).

• There is now a new &on_change attribute that can be used to be notified
  of changes to tables and sets.

Changed Functionality

• A C++17-capable compiler and CMake 3.0+ are now required to compile Zeek

• The backwards-compability wrappers & work-arounds introduced in 3.0
  for the "Bro to Zeek rename" have either changed their operation, or in some
  cases been removed. Generally, anything that reported a
  naming-related warning in 3.0 now aborts with a corresponding error
  message. In cases where 3.0 silently continued to accept old names,
  3.1 now reports warnings. Most importantly, that's loading of
  scripts with ".bro" endings, which are now flagged and should be
  renamed.

• Broker has switched versions for the underlying CAF communication
  library from 0.16 to 0.17. CAF has changed its wireformat between
  those versions, which means that previous Broker and Zeek versions
  won't be able to connect to the new ones. In other words, all Zeek
  instances, as well as other Broker clients, need to upgrade at the
  same time. In case of version mismatches, Broker now reports better
  error messages to point out the problem.

• The key type of Known::service_store has changed to
  Known::AddrPortServTriplet and Known::services is now a table
  instead of just a set.

• The DNS class name for Hesiod in the DNS::classes table is now spelled
  correctly as "C_HESIOD" instead of "C_HESOD". For reference, this
  class name string may appear in the dns.log file or in any custom
  script logic that inspects the qclass field of DNS::Info
  after a dns_request event.

• The configuration input reader now ignores trailing spaces at the end of
  configuration lines.

• The tcp_option event is now correctly raised.

• The base scripts shipped with Zeek now use the new
  successful_connection_remove event instead of
  connection_state_remove where possible (when the logic doesn't
  pertain to unestablished TCP connections). There's a performance
  benefit to this switch, but it may potentially break custom scripts
  that depended on accessing or modifying state via specific
  &priority ordering of connection_state_remove event
  handlers. However, since most of Zeek's base scripts previously just
  used that event with the intention of writing a finalized log as the
  last thing executed for a given connection, and the new
  successful_connection_remove event handlers all run after
  connection_state_remove, it's not likely this change to the base
  scripts causes any incompatibility with user scripts.

  There's also a new event called connection_successful and a new
  connection record field named "successful" to help indicate this
  new property of connections.

• The JSON output formatters now use the RapidJSON library. This
  improves their performance considerably over the library that was
  previously used. Output from the formatters remains nearly
  identical.

• The decompose_uri function no longer raises an error when parsing
  URIs with an empty port number (e.g. http://example.org:/). Instead,
  the portnum component of the returned URI value is left
  uninitialized.

• Replace old select-based IO loop with a new architecture that doesn't
  spin checking for active IO sources. The new architecture now waits for the
  the sources to actively notify it when activity occurs and only processes
  data once it's ready. This helps heavily reduce the CPU usage on idle
  network connections. This includes a couple of breaking changes:

  • Only a single packet source is allowed to be specified from the
    command-line now. If you pass combinations of multiple -r and/or
    -i flags, Zeek will return an error at startup.
  • The IOSource API changed fairly wildly. The GetFds() and
    NextTimestamp methods no longer exist. If you had previously
    implemented a custom IOSource, you will need to look at the new API
    and make changes to your code to accomodate it. This does not include
    packet sources, which should remain functional with little to no
    changes, since the entirety of the changes should be in PktSrc.

• Remove a large number of headers from being included by various files across
  the entire code base, which leads to a sizeable build time improvement. This
  set of changes has the potential to cause plugins to not build anymore. The
  fixes for this potential breakage should just be a matter of including the
  necessary headers in the plugin code.

Removed Functionality

• Removed the current_conns_extern field from the ConnStats record
  type. Zeek only maintains a single timer manager now, and without the
  manager tags that came with multiple tiemr managers, we don't track
  whether a connection is external anymore.

Deprecated Functionality

• The C++ API typedefs for int{8,16,32,64} and uint{8,16,32,64} are deprecated
  in favor of the real types they alias. E.g. use int8_t instead of
  int8.

• The C++ API functions "safe_snprintf" and "safe_vsnprintf" are deprecated.
  Use "snprintf" and "vsnprintf" instead.