Documentation: https://docs.intelmq.org/latest/
NEWS
Most important changes potentially requiring administration attention
Requirements
Python 3.8 or newer is required.
Bots
CIF 3 API Output deprecation
The CIF 3 API Output bot is not compatible with Python version greater or equal to 3.12 and will be removed in the future due to lack of maintenance.
See https://lists.cert.at/pipermail/intelmq-users/2024-December/000474.html for more information.
Twitter Collector removal
As the bot does not work anymore and uses an unmaintained library, it is removed from IntelMQ.
Please remove if from your setup.
intelmqctl check
and intelmqctl upgrade-config
command warns if you have the bot in use.
Twitter Parser renaming
The Twitter parser is renamed to IoC Extractor Parser (intelmq.bots.parsers.ioc_extractor
).
intelmqctl upgrade-config
will automatically adapt the configuration.
The previous module name is left as a stub to load the IoC Extractor parser for backwards-compatibility.
Packaging
Packages are now also available for Ubuntu 24.04.
To upgrade an Ubuntu 22.04 installation to 24.04 please refer to the Ubuntu documentation: https://documentation.ubuntu.com/server/how-to/software/upgrade-your-release/index.html
Full changelog
Configuration
Core
- AMQP: Fix maintaining pipeline connection when during interrupted connections (PR#2533 by Kamil Mankowski).
- Python 3.8 or newer is required (PR#2541 by Sebastian Wagner).
intelmq.lib.utils.list_all_bots
/intelmqctl check
: Fix check for bot executable in $PATH by using the bot name instead of the import path (fixes #2559, PR#2564 by Sebastian Wagner).
Bots
Collectors
intelmq.bots.collectors.shadowserver.collector_reports_api.py
:- Fixed behaviour if parameter
types
value is empty string, behave the same way as not set, not like no type.
- Fixed behaviour if parameter
intelmq.bots.collectors.misp
: UsePyMISP
class instead of deprecatedExpandedPyMISP
(PR#2532 by Radek Vyhnal)intelmq.bots.collectors.http.collector_http
: Log the downloaded size in bytes to ease troubleshooting (PR#2554 by Sebastian Wagner).intelmq.bots.collectors.mail.collector_mail_url
:- Log the downloaded size in bytes to ease troubleshooting (PR#2554 by Sebastian Wagner).
- Fix import for Timeout exception preventing another exception (fixes #2555, PR#2556 by Sebastian Wagner).
- Remove
intelmq.bots.collectors.twitter
as it uses an unmaintained library and does not work any more (fixes #2346, #2441, PR#2568 by Sebastian Wagner).
Parsers
intelmq.bots.parsers.shadowserver._config
:- fix error message formatting if schema file is absent (PR#2528 by Sebastian Wagner).
intelmq.bots.parsers.shadowserver.parser
:- Fix to avoid schema download if not configured #2530.
intelmq.bots.parsers.misp.parser
: Replace deprecated datetime functionutcfromtimestamp
for Ubuntu 24.04 compatibility (PR#2577 by Sebastian Wagner, fixes #2576, #2571).intelmq.bots.parsers.cleanmx.parser
: Replace deprecated datetime functionutcfromtimestamp
for Ubuntu 24.04 compatibility (PR#2577 by Sebastian Wagner, fixes #2576, #2571).- Renamed
intelmq.bots.parsers.twitter
tointelmq.bots.parser.ioc_extractor
(PR#2568 by Sebastian Wagner).- Added
intelmq.bots.parsers.twitter
as a stub to load the IoC Extractor parser.
- Added
Experts
intelmq.bots.experts.securitytxt
:- Added new bot (PR#2538 by Frank Westers and Sebastian Wagner).
intelmq.bots.experts.misp
: UsePyMISP
class instead of deprecatedExpandedPyMISP
(PR#2532 by Radek Vyhnal).intelmq.bots.experts.fake.expert
: New expert to fake data (PR#2567 by Sebastian Wagner).
Outputs
intelmq.bots.outputs.cif3.output
:- The requirement can only be installed on Python version < 3.12.
- Add a check on the Python version and exit if incompatible.
- Add a deprecation warning (PR#2544 by Sebastian Wagner).
intelmq.bots.outputs.sql.output
:- Treat an empty string
fields
parameter as unset parameter, fixing a crash in default configuration (PR#2548 by Sebastian Wagner, fixes #2548).
- Treat an empty string
Documentation
docs/admin/installation/linux-packages
: Add[signed-by=]
options, add wget command as alternative to curl (PR#2547 by Sebastian Wagner).- Add documentation on the Redis pipeline (databases, configuration), fix generic pipeline documentation and add missing information on parameters, add unlinked intelmqctl docs to the index and TOC (PR#2560 by Sebastian Wagner).
- Remove empty page tutorials/intelmq-manager (PR#2562 by Sebastian Wagner).
Packaging
- Packages for Ubuntu 24.04 (by Sebastian Wagner, fixes #2571).
Tests
- Install build dependencies for
pymssql
on Python 3.8 as there are no wheels available for this Python version (PR#2542 by Sebastian Wagner). - Install
psql
explicitly for workflow support on other platforms such as act (PR#2542 by Sebastian Wagner). - Create intelmq user & group if running privileged to allow dropping privileges (PR#2542 by Sebastian Wagner).
intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge
: Also skip on Python 3.11 and 3.12 besides on 3.8 when running on CI (PR#2542 by Sebastian Wagner).- Full pytest workflow: Version-independent install of postgres client, for Ubuntu 24.04 (default on GitHub now) test environment compatibility (PR#2557 by Sebastian Wagner).
- Debian package build workflow: Use artifact upload v4 instead of v3 (PR#2565 by Sebastian Wagner).
Known issues
This is short list of the most important known issues. The full list can be retrieved from GitHub.
- intelmqctl: interactive run ignores custom log level (#2563).
intelmq.parsers.html_table
may not process invalid URLs in patched Python version due to changes inurllib
(#2382).- Breaking changes in 'rt' 3.0 library (#2367).
- Type error with SQL output bot's
prepare_values
returning list instead of tuple (#2255). intelmq_psql_initdb
does not work for SQLite (#2202).- intelmqsetup: should install a default state file (#2175).
- Misp Expert - Crash if misp event already exist (#2170).
- Spamhaus CERT parser uses wrong field (#2165).
- Custom headers ignored in HTTPCollectorBot (#2150).
- intelmqctl log: parsing syslog does not work (#2097).
- Bash completion scripts depend on old JSON-based configuration files (#2094).
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
- Corrupt dump files when interrupted during writing (#870).