Malcolm v4.0.0 consists of a major restructuring of the underlying data schema used to represent Zeek logs (and, going forward, logs from other data sources) in the Elasticsearch data store. As the Malcolm project uses semantic versioning when choosing version numbers, this backwards-compatibility breaking change is the reason for bumping the major version number from 3 to 4 despite no significant new functionality being introduced.
The details of the drivers behind this change can be found at idaholab#64 and idaholab#16. This change, though somewhat painful, will make it easier to integrate more data sources into Malcolm in the future and potentially makes Malcolm's network session data more compatible with other tools that use the Elastic Common Schema.
BREAKING CHANGES:
- as many field names have changed, custom saved dashboards and/or bookmarks to Kibana or Arkime visualizations may need to be adjusted accordingly
- old network session data (stored in the
sessions2-*indices in Elasticsearch) will not be visible (as the indices are now namedarkime-sessions3-*)
A fresh install of Malcolm is recommended with this release. Upgrading from previous versions of Malcolm to v4.0.0+ is not suggested.
Changes:
- added GitHub workflow files which contain instructions for GitHub to build the docker images and sensor and Malcolm installer ISOs.
- moved many fields that were named zeek-specific to generic ECS-specified (or at least "ECS-inspired") field names, updating related parsing code and dashboard definitions
- changed Zeek-specific field naming schema (e.g.,
zeek_foo.barbecomeszeek.foo.bar) - added Corelight's Microsoft Excel privilege escalation detection (CVE-2021-42292) plugin
- integrated updates to the LDAP parser which improve the detail given from observed LDAP searches
- improved and genericized the code for mapping MAC addresses to vendor OUIs, replacing the use of logstash-filter-ieee_oui
- updated some Dockerfiles to use Debian 11 "bullseye" instead of Debian 10 "buster"