zeek/zeek v3.2.0

on GitHub

Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least November 2020 while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately November 2020 when the next 4.0.x LTS release series is expected to begin.

New Functionality

• X509 Certificate caching:

  Zeek now caches certificates if they have (by default) been encountered
  more than 10 times in 62 seconds. Information for cached certificates is
  retained; if the certificate is encountered again it does not have to
  be re-parsed and already existing information is used to raise the events.

  This should especially help with performance in environments where the
  same certificates are seen very often.

  Certificate caching is very configureable; it is possible to disable the
  feature, change the time intervals or even suppress X509 events.
  For details see scripts/base/files/x509/main.zeek.

• Add parsing support for Remote Desktop Protocol UDP Transport Extension
  (RDPEUDP versions 1 and 2). This primarily only adds "rdpeudp" to
  connection record service fields when an RDPEUDP session handhake is
  detected, but also provides a few other events related to the RDPEUDP
  connection establishment.

• Add the udp_content_ports configuration option. Any port added to
  this set will cause the udp_contents event to be raised.

• Add the udp_content_delivery_ports_use_resp option which can be used
  to specify how the destination port for the udp_content_delivery_ports_orig
  and udp_content_delivery_ports_orig options is determined. The current value
  keeps behavior as it was in previous versions of Zeek.

• Add a file signature to identify ISO9660 disk images (application/x-iso9660-image)

• Add file signature to identify Python bytecode (application/x-python-bytecode)

• Events and hooks are now allowed to have multiple, alternate prototype
  declarations. This allows for extending event/hook parameters in a way that
  won't break an existing user's handlers and also allows users to define their
  own custom event/hook prototypes that consume a subset of the parameters
  (convenience of typing/memory/etc). This feature is documented in detail
  here: https://docs.zeek.org/en/current/script-reference/types.html#type-event

• Add flags parameters to rdp_connect_request,
  rdp_negotiation_response, and rdp_negotiation_failure events.

• Reporter::conn_weird now correctly handles weirds for expired connections,
  for which no connection state information is available in the core anymore. These
  cases will raise the new expired_conn_weird event.

• Broker Store table synchronization (experimental).

  Zeek now supports synchronizing tables/sets across clusters using a backing Broker
  store. The same feature also allows persistent storage of data in tables/sets
  over Zeek restarts. This feature is implemented using the new &backend attribute.

  To synchronize a table over a cluster, you can, e.g., use:

  global t: table[string] of count &backend=Broker::MEMORY;

  This feature is documented in detail here:

  https://docs.zeek.org/en/current/frameworks/broker.html#broker-store-backed-zeek-tables-for-data-synchronization-and-persistence

  Note: this feature is experimental and the syntax/specifics can change in the future.

Changed Functionality

• Several C++ functions have been changed to pass smart pointers
  (class IntrusivePtr<>) instead of raw pointers. This makes the
  code more robust. External plugins may need to be updated to this
  API change.

• BIFs that use @ARG@, @ARGS@, or @ARGC@ may break since their type has
  changed: BIF arguments are now passed as a std::vector<IntrusivePtr<Val>>
  rather than a val_list (i.e. List<Val*>).

• RocksDB support for Broker data stores is now opt-in instead of automatically
  detected and used at configuration-time. Use the --enable-rocksdb and
  --with-rocksdb= flags to opt-in.

• At the C++ level, a large number of functions had their return
  values and/or arguments changed to use bool types instead of
  int. This includes some virtual methods, which may cause build
  failures in plugins that were overriding those methods. Those
  plugins will need to be updated to match these API changes. To make
  sure to catch changes you need to make to your plugin, double-check
  that all virtual method implementations use "override".

• Many C++ classes were marked "final" which also has some performance benefits
  due to devirtualization optimizations.

• Data members of many C++ classes/structs were reordered to achieve better
  packing and smaller memory footprint.

• "using namespace std" was removed from the Zeek header files; Zeek now always
  explicitly specifies std when using STL functionality in headers. This may
  necessitate small changes in external plugins, if they relied on the using
  statement in Zeek headers.

• The connection_external event was removed. This functionality that could
  raise this event (injecting connections via broccoli) was removed a while ago;
  the event handler served no purpose anymore.

• Reorganize the file signatures to break them out into more groups. This may
  break scripts that had been explicitly loading any signature files that moved.

• The DCE/RPC operation string of "NetrLogonSamLogonWithFlags" has been
  corrected from "NetrLogonSameLogonWithFlags".

• AsRecord() and AsNonConstRecord() have changed to return
  std::vector<IntrusivePtr<Val>>*.

• AsVector() has changed to return std::vector<IntrusivePtr<Val>>*.

• Moved a large number of classes from the global namespace into either the
  zeek or zeek::detail namespace. See #266
  for the rationale behind these changes. Most types that were moved and functions
  that used them have been marked as deprecated and will generate compiler
  warnings if used (a few exceptions will not generate compiler warnings,
  but the Deprecated Functionality section below will mention those
  ones specifically).

  This includes a number of renames to classes, removing 'Bro' from their
  names. Deprecation warnings should notify plugin developers of these
  name changes.

Removed Functionality

• The fmt() function which takes a va_list argument is replaced, use
  the new vfmt() function for equivalent functionality. The former is
  deprecated because overloading it with the variadic fmt() function
  can cause the unintended overload to be chosen depending on how the
  platform implements va_list.

Deprecated Functionality

• The plugin::Plugin::HookCallFunction() method is deprecated. Note
  that compilers will not emit a deprecation warning, but the replacement
  method to now use is called HookFunctionCall and uses IntrusivePtr
  arguments and return value.

• The Func::Call(val_list*, ...) method is now deprecated. Use Invoke()
  instead which takes a zeek::Args (std::vector<IntrusivePtr<Val>>).
  There's also a variadic template for Invoke() that forwards all arguments
  into a zeek::Args for you.

• The EventMgr::QueueEvent() and EventMgr::QueueEventFast() methods
  are now deprecated, use EventMgr::Enqueue() instead.

• The Connection::ConnectionEvent(), Connection::Event(), and
  Connection::ConnectionEventFast() methods are now deprecated, use
  Connection::EnqueueEvent() instead.

• The file_analysis::File::FileEvent() methods taking val_list
  arguments are now deprecated, use the overload that takes a zeek::Args
  instead.

• The analyzer::Analyzer::ConnectionEvent(), analyzer::Analyzer::Event,
  and analyzer::Analyzer::ConectionEventFast() methods are deprecated, use
  analyzer::Analyzer::EnqueueConnEvent() instead.

• All val_mgr methods starting with "Get" are deprecated, use the new
  val_mgr methods that return IntrusivePtr.

• Connection::BuildConnVal() is deprecated, use Connection::ConnVal().

• Analyzer::BuildConnVal() is deprecated, use Analyzer::ConnVal().

• BifEvent::generate_ functions are deprecated, use zeek::BifEvent::enqueue_.

• binpac::bytestring_to_val() is deprecated, use binpac::to_stringval().

• binpac::string_to_val() is deprecated, use StringVal constructor.

• Returning Val* from BIFs is deprecated, return IntrusivePtr instead.

• Various methods of converting protocol structures, like IP or packet headers,
  to associated Val type are now deprecated, the deprecation warning
  message will advise what new method to use instead.

• Various methods of Tag classes are deprecated with the warning
  message advising what new method to use instead.

• The utf16_bytestring_to_utf8_val() function is deprecated, use
  utf16_to_utf8_val() instead.

• RecordType::FieldType() is deprecated, use RecordType::GetFieldType()

• BroType::HasField() and BroType::FieldType() are deprecated, use
  the methods of RecordType directly.

• BroType::YieldType() is deprecated, use BroType::Yield().

• ID::AsType() is deprecated, use ID::IsType() and ID::GetType().

• ID::Type() is deprecated, use ID::GetType().

• ID::ID_Val() is deprecated, use ID::GetVal().

• internal_type() is deprecated, use zeek::id::find_type().

• internal_val() and internal_const_val() are deprecated, use
  zeek::id::find_val() or zeek::id::find_const().

• internal_func() is deprecated, use zeek::id::find_func().

• opt_internal_val() is deprecated, use lookup_ID() or
  zeek::id::find_val().

• Val::Type() is deprecated, use Val::GetType.

• Most global type/value pointers in NetVar.h are deprecated, but one can
  still always perform the lookup themselves.

• Scope::Lookup() is deprecated, use Scope::Find().

• All generated BroType* names in the BifType:: namespaces are
  deprecated, but there's an equivalent name in zeek::BifType:: of
  IntrusivePtr type to use instead.

• All generated BifConst:: names are deprecated, but there's an
  equivalent name now in zeek::BifCont::, and changed to IntrusivePtr
  if the old name was some Val* type.

• Constructors for Val types that take a BroType* are all generally
  deprecated, with alternatives that instead take an IntrusivePtr argument.

• FuncType::Args() is deprecated, use FuncType::Params().

• FuncType::ArgTypes() is deprecated, use FuncType::ParamList().

• RecordVal::Assign(int, Val*) is deprecated, use the overload taking
  IntrusivePtr.

• RecordVal::Lookup(int) is deprecated, use RecordVal::GetField(int).

• RecordVal::LookupWithDefault(int) is deprecated, use
  RecordVal::GetFieldOrDefault(int).

• RecordVal::Lookup(const char*, bool) is deprecated, use either
  RecordVal::GetField() or RecordVal::GetFieldOrDefault().

• TableVal::Assign methods taking raw Val* are deprecated, use the
  overloads taking IntrusivePtr.

• TableVal::Lookup() is deprecated, use TableVal::Find() or
  TableVal::FindOrDefault().

• VectorVal::Assign and Insert methods taking raw Val* are
  deprecated, use the methods that take IntrusivePtr.

• VectorVal::Lookup() is deprecated, use VectorVal::At().

• The file analysis/analyzer API has deprecated methods taking raw
  RecordVal* for analyzer arguments and replaced those with methods
  taking IntrusivePtr.

• The Val(double, TypeTag) constructor is deprecated, use either
  IntervalVal(), TimeVal() or DoubleVal() constructors.

• The "BroString.h" file is deprecated, use "ZeekString.h"

• The str_split() BIF is deprecated, use str_split_indices(). Note
  that the former returns a vector with indices starting at 1 while the
  later returns a vector with indices starting at 0.

• The icmp_conn parameter of ICMP events is deprecated, there's an
  alternate version with an icmp_info parameter to use instead.
  The icmp_conn record passed to ICMP events has always been re-used
  amongst all events within an ICMP "connection", so the
  itype, icode, len, and hlim fields as inspected in
  handlers never appears to change even if the underlying packet data
  has different values for those fields. However, it's not known if
  anyone relied on that behavior, so the new icmp_info record is
  introduced with the more-expected behavior of being created and
  populated for each new event. It also removes the orig_h/resp_h
  fields since those are redundant with what's already available in
  the connection parameter.

• External plugins should include Zeek header files like
  #include <zeek/Foo.h> instead of #include <Foo.h>. The later
  style is considered deprecated. Reliance on zeek-config --include_dir
  to contain $prefix/include/zeek is also deprecated: its replacement
  output is expected to be just $prefix/include, with it currently
  outputting both paths, delimited by a colon, during the deprecation period.