certtools/intelmq 3.2.0

3.2.0 Feature release: Running IntelMQ bots as Python Library

on GitHub

IEP007: Running IntelMQ bots as Python Library is implemented.

Installation: https://intelmq.readthedocs.io/en/develop/user/installation.html
Upgrade: https://intelmq.readthedocs.io/en/develop/user/upgrade.html

The accompanying 3.2.0 release of intelmq-api switches it's backend from the library hug to fastapi.
Deb-packages of intelmq-api 3.2.0 are delayed for some distributions because of necessary changes in packaging.

Core

• intelmq.lib.utils:
  • resolve_dns: Deprecate dnspython versions pre-2.0.0 and disable search domains (PR#2352)
• Fixed not resetting destination path statistics in the stats cache after restarting bot (Fixes #2331)
• Force flushing statistics if bot will sleep longer than flushing delay (Fixes #2336)
• intelmq.lib.upgrages: Fix a bug in the upgrade function for version 3.1.0 which caused an exception if a generic csv parser instance had no parameter type (PR#2319 by Filip Pokorný).
• intelmq.lib.datatypes: Adds TimeFormat class to be used for the time_format bot parameter (PR#2329 by Filip Pokorný).
• intelmq.lib.exceptions: Fixes a bug in InvalidArgument exception (PR#2329 by Filip Pokorný).
• intelmq.lib.harmonization:
  • Changes signature and names of DateTime conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorný).
  • Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes #2377)
• intelmq.lib.bot.Bot: Allow setting the parameters via parameter on bot initialization.

Development

• CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).

Bots

Collectors

• intelmq.bots.collector.rt:
  • restrict python-rt to be below version 3.0 due to introduced breaking changes,
  • added support for Subject NOT LIKE queries,
  • added support for multiple values in ticket subject queries.
• intelmq.bots.collectors.rsync: Support for optional private key, relative time parsing for the source path, extra rsync parameters and strict host key checking (PR#2241 by Mateo Durante).

Parsers

• intelmq.bots.parsers.shadowserver._config:
  • Reset detected feedname at shutdown to re-detect the feedname on reloads (PR#2361 by @elsif2, fixes #2360).
• intelmq.bots.parsers.shadowserver._config:
  • Added 'IPv6-Vulnerable-Exchange' alias and 'Accessible-WS-Discovery-Service' report. (PR#2338)
  • Removed unused p0f_genre and p0f_detail from the 'DNS-Open-Resolvers' report. (PR#2338)
  • Added 'Accessible-SIP' report. (PR#2348)
  • Added 'IPv6-Open-HTTP-Proxy' and 'IPv6-Accessible-HTTP-Proxy' aliases. (PR#2348)
  • Removed duplicate mappings from the 'Spam-URL' report. (PR#2348)
• intelmq.bots.parsers.generic.parser_csv: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
• intelmq.bots.parsers.html_table.parser: Changes time_format parameter to use new TimeFormat class (PR#2329 by Filip Pokorný).
• intelmq.bots.parsers.turris.parser.py Updated to the latest data format (issue #2167). (PR#2373 by Filip Pokorný).

Experts

• intelmq.bots.experts.sieve:
  • Allow empty lists in sieve rule files (PR#2341 by Mikk Margus Möll).
• intelmq.bots.experts.cymru_whois:
  • Ignore AS names with unexpected unicode characters (PR#2352, fixes #2132)
  • Avoid extraneous search domain-based queries on NXDOMAIN result (PR#2352)
• intelmq.bots.experts.sieve:
  • Added :before and :after keywords (PR#2374)

Outputs

• intelmq.bots.outputs.cif3.output: Added (PR#2244 by Michael Davis).
• intelmq.bots.outputs.sql.output: New parameter fail_on_errors (PR#2362 by Sebastian Wagner).
• intelmq.bots.outputs.smtp_batch.output: Added a bot to gathering the events and sending them by e-mails at a stroke as CSV files (PR#2253 by Edvard Rejthar)

Documentation

• API: update API installation to be aligned with the rewritten API, and clarify some missing steps.

Tests

• New decorator skip_installation and environment variable INTELMQ_TEST_INSTALLATION to skip tests requiring an IntelMQ installation on the test host by default (PR#2370 by Sebastian Wagner, fixes #2369)

Tools

• intelmqsetup:
  • SECURITY: fixed a low-risk bug causing the tool to change owner of / if run with the INTELMQ_PATHS_NO_OPT environment variable set. This affects only the PIP package as the DEB/RPM packages don't contain this tool. (PR#2355 by Kamil Mańkowski, fixes #2354)
• contrib.eventdb.separate-raws-table.sql: Added the missing commas to complete the sql syntax. (PR#2386, fixes #2125 by Sebastian Kufner)
• intelmq_psql_initdb:
  • Added parameter -o to set the output file destination. (by Sebastian Kufner)
• intelmqctl:
  • Increased the performance through removing unnecessary reads. (by Sebastian Kufner)

Known Issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (#2382).
• Breaking changes in 'rt' library (#2367).
• Stomp collector failed (#2342).
• Type error with SQL output bot's prepare_values returning list instead of tuple (#2255).
• intelmq_psql_initdb does not work for SQLite (#2202).
• intelmqsetup: should install a default state file (#2175).
• Misp Expert - Crash if misp event already exist (#2170).
• Turris greylist has been updated (#2167).
• Spamhaus CERT parser uses wrong field (#2165).
• Custom headers ignored in HTTPCollectorBot (#2150).
• intelmqctl log: parsing syslog does not work (#2097).
• Bash completion scripts depend on old JSON-based configuration files (#2094).
• Bot configuration examples use JSON instead of YAML (#2066).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
• Corrupt dump files when interrupted during writing (#870).