3.3.0 (2024-03-01)
Configuration
- Add new optional configuration parameters for
intelmq.bots.collectors.stomp.collector
andintelmq.bots.outputs.stomp.output
(PR#2408 by Jan Kaliszewski):auth_by_ssl_client_certificate
(Boolean, default: true; if false then
ssl_client_certificate
andssl_client_certificate_key
will be ignored);username
(STOMP authentication login, default: "guest"; to be used only
ifauth_by_ssl_client_certificate
is false);password
(STOMP authentication passcode, default: "guest"; to be used only
ifauth_by_ssl_client_certificate
is false).
- Add the possibility to set the
ssl_ca_certificate
configuration parameter for
intelmq.bots.collectors.stomp.collector
and/orintelmq.bots.outputs.stomp.output
to an empty string - which means that the SSL machinery used for STOMP communication
will attempt to load the system’s default CA certificates (PR#2414 by Jan Kaliszewski).
Core
intelmq.lib.message
: For invalid message keys, add a hint on the failure to the exception: not allowed by configuration or not matching regular expression (PR#2398 by Sebastian Wagner).intelmq.lib.exceptions.InvalidKey
: Add optional parameteradditional_text
(PR#2398 by Sebastian Wagner).- Change the way we discover bots to allow easy extending based on the entry point name. (PR#2413 by Kamil Mankowski)
intelmq.lib.mixins
: Add a new class,StompMixin
(defined in a new submodule:stomp
),
which provides certain common STOMP-bot-specific operations, factored out from
intelmq.bots.collectors.stomp.collector
andintelmq.bots.outputs.stomp.output
(PR#2408 and PR#2414 by Jan Kaliszewski).intelmq.lib.upgrades
: Replace deprecated instances ofurl2fqdn
experts by the newurl
expert in runtime configuration (PR#2432 by Sebastian Wagner).intelmq.lib.bot
: Ensure closing log files on reloading (PR#2435 by Kamil Mankowski).- AMQP Pipeline: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
- Only load the config once when starting intelmqctl (which makes IntelMQ API calls take less time) (PR#2444 by DigitalTrustCenter).
Development
- Makefile: Add codespell and test commands (PR#2425 by Sebastian Wagner).
Data Format
Bots
Collectors
intelmq.bots.collectors.stomp.collector
(PR#2408 and PR#2414 by Jan Kaliszewski):- Drop support for versions of
stomp.py
older than4.1.12
. - Update the code to support new versions of
stomp.py
, including the latest (8.1.0
);
fixes #2342. - Add support for authentication based on STOMP login and passcode, introducing three
new configuration parameters (see above: Configuration). - Add support for loading the system’s default CA certificates, as an alternative to
specifying the CA certificate(s) file path explicitly (see above: Configuration). - Fix (by carefully targeted monkey patching) certain security problems caused by
SSL-related weaknesses that some versions ofstomp.py
suffer from. - Fix the reconnection behavior: do not attempt to reconnect after
shutdown
. Also,
never attempt to reconnect if the version ofstomp.py
is older than4.1.21
(it
did not work properly anyway). - Add coercion of the
port
config parameter toint
. - Add implementation of the
check
hook (verifying, in particular, accessibility
of necessary file(s)). - Remove undocumented and unused attributes of
StompCollectorBot
instances:
ssl_ca_cert
,ssl_cl_cert
,ssl_cl_cert_key
.- Minor fixes/improvements and some refactoring (see also above: Core...).
- Drop support for versions of
intelmq.bots.collectors.amqp
: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).intelmq.bots.collectors.shadowserver.collector_reports_api
:- The 'json' option is no longer supported as the 'csv' option provides better performance (PR#2372 by elsif2).
intelmq.bots.collectors.alienvault_otx.collector
(PR#2449 by qux-bbb):- Fix modified_pulses_only is always False.
Parsers
intelmq.bots.parsers.netlab_360.parser
: Removed as the feed is discontinued. (#2442 by Filip Pokorný)intelmq.bots.parsers.webinspektor.parser
: Removed as the feed is discontinued. (#2442 by Filip Pokorný)intelmq.bots.parsers.sucuri.parser
: Removed as the feed is discontinued. (#2442 by Filip Pokorný)intelmq.bots.parsers.shadowserver._config
:- Switch to dynamic configuration to decouple report schema changes from IntelMQ releases by regularly downloading them from the Shadowserver server (PR#2372 by elsif2).
intelmq.bots.parsers.cymru
: Save current line. (PR by Kamil Mankowski)
Experts
intelmq.bots.experts.jinja
(PR#2417 by Mikk Margus Möll):- Add optional
socket_perms
andsocket_group
parameters to change
file permissions on socket file, if it is in use.
- Add optional
intelmq.bots.experts.ripe
(PR#2461 by Mikk Margus Möll):- Handle "No abuse contact found for" messages for non-ASN resources
Outputs
intelmq.bots.outputs.stomp.output
(PR#2408 and PR#2414 by Jan Kaliszewski):- Drop support for versions of
stomp.py
older than4.1.12
. - Update the code to support new versions of
stomp.py
, including the latest (8.1.0
). - Add support for authentication based on STOMP login and passcode, introducing three
new configuration parameters (see above: Configuration). - Add support for loading the system’s default CA certificates, as an alternative to
specifying the CA certificate(s) file path explicitly (see above: Configuration). - Fix (by carefully targeted monkey patching) certain security problems caused by
SSL-related weaknesses that some versions ofstomp.py
suffer from. - Fix
AttributeError
caused by attempts to get unset attributes ofStompOutputBot
(ssl_ca_cert
et consortes). - Add coercion of the
port
config parameter toint
. - Add implementation of the
check
hook (verifying, in particular, accessibility
of necessary file(s)). - Add
stomp.py
version check (raiseMissingDependencyError
if not>=4.1.12
). - Minor fixes/improvements and some refactoring (see also above: Core...).
- Drop support for versions of
intelmq.bots.outputs.stomp.output
(PR#2423 by Kamil Mankowski):- Try to reconnect on
NotConnectedException
.
- Try to reconnect on
intelmq.bots.outputs.smtp_batch.output
(PR #2439 by Edvard Rejthar):- Fix ability to send with the default
bcc
- Fix ability to send with the default
intelmq.bots.outputs.amqp
: fix SSL context to pointing to create a client-side connection that verifies the server (PR by Kamil Mankowski).
Documentation
- Add a readthedocs configuration file to fix the build fail (PR#2403 by Sebastian Wagner).
- Add a guide of developing extensions packages (PR#2413 by Kamil Mankowski)
- Update/fix/improve the stuff related to the STOMP bots and integration with the n6's
Stream API (PR#2408 and PR#2414 by Jan Kaliszewski). - Complete documentation overhaul. Change to markdown format. Uses the mkdocs-material (PR#2419 by Filip Pokorný).
- Adds warning banner if not browsing the latest version of the docs (PR#2445 by Filip Pokorný).
- Fix logo path in index.md when building the docs (PR#2445 by Filip Pokorný).
Packaging
- Add
pendulum
to suggested packages, as it is required for the sieve bot (PR#2424 by Sebastian Wagner). debian/control
: inSuggests
field, replacepython3-stomp.py (>= 4.1.9)
with
python3-stomp (>= 4.1.12)
, i.e., fix the package name by removing the.py
suffix and bump the minimum version to4.1.12
(PR#2414 by Jan Kaliszewski).
Tests
Tools
intelmq_psql_initdb
:- got support for providing custom harmonization file, generating view for storing
raw
fields separately, and addingIF NOT EXISTS
/OR REPLACE
clauses (PR#2404 by Kamil Mankowski). - got support for generating JSONB fields for PostgreSQL schema (PR#2436 by Kamil Mankowski).
- got support for providing custom harmonization file, generating view for storing