certtools/intelmq 2.0.2

on GitHub

Install documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.2/docs/UPGRADING.md

As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.

Core

• intelmq.lib.bot.CollectorBot: Support the deprecated parameter feed until version 2.2 as the documentation was not properly updated (#1445).
• intelmq.lib.bot.Bot:
  • _dump_message: Wait for up to 60 seconds instead of 50 if the dump file is locked (the log message was said 60, but the code was for only 50).
• intelmq.lib.upgrades.v202_fixes
  • Migration of deprecated parameter feed for Collectors.
  • Ripe expert parameter query_ripe_stat_ip was not correctly configured in v110_deprecations, now use query_ripe_stat_asn as default if it does not exist.
• intelmq.lib.upgrades.v110_deprecations: Fix upgrade of ripe expert configuration.
• intelmq.lib.bot_debugger:
  • Fix handling of empty messages generated by parser when user wanted to show the result by "--show-sent" flag.
  • Fix handling of sent messages for bots using the path_permissive paramter (#1453).
• intelmq.lib.pipeline.Amqp:
  • use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.
  • Reconnect once on sending messages if disconnect detected.

Bots

Collectors

• intelmq.bots.collectors.api.collector_api:
  • Handle non-existing IO loop in shutdown.
  • Close socket on shutdown, fixes reloading.
  • Marked as non-threadable.
• intelmq.bots.collectors.rt.collector_rt: Check for matching URLs if no attachment_regex is given.
• intelmq.bots.collectors.stomp.collector_stomp: Handle disconnects by actively reconnecting.

Parsers

• intelmq.bots.cymru.parser_cap_program: Fix parsing of the new $certname_$date.txt report format (#1443):
  • Support protocol ICMP.
  • Fix error message for unsupported protocols.
  • Support fields destination_port_numbers, port.
  • Support for all proxy types without ports.
  • Use Country Code of AS as source.geolocation.cc.
  • Support for 'scanner' and 'spam' categories.
  • Handle bogus lines with missing separator.
  • Fix bug preventing use of old format after using the new format.
  • Handle postfix (total_count:..) for destination port numbers.

Experts

• intelmq.bots.experts.cymru_whois.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
• intelmq.bots.experts.modify.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).
• intelmq.bots.experts.reverse_dns.expert: Add optional parameter overwrite, current behavior was True, default if not given is True now, will change to False in 3.0.0 (#1452, #1455).

Outputs

• intelmq.bots.outputs.amqptopic.output: use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.

Packaging

• Rules:
  • Exclude intelmqsetup tool in packages
  • Include update-rfiprisk-data in packages

Tests

• Tests for intelmq.lib.upgrades.v202_fixes.
• Tests for intelmq.lib.upgrades.v110_deprecations.
• Extended tests for intelmq.bots.parser.cymru.parser_cap_program.

Tools

• intelmqctl:
  • More and more precise logging messages for botnet starting and restarting, enable and disable.
  • No error message for disabled bots on botnet reload.
  • Fix upgrade-conf is state file is empty or not existing.
  • Use arpgarse's store_true action for flags instead of store_const.
  • If the loading of the defaults configuration failed, a variable definition was missing and causing an exception (#1456).

Contrib

• Check MK Statistics Cronjob:
  • Use statistics_* parameters.
  • Make file executable
  • Handle None values in *.temporary.* keys and treat them as 0.
• systemd:
  • Add PIDFile parameter to service file.

Known issues

• MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
• ctl: shell colorizations are logged (#1436)
• http stream collector: retry on regular connection problems? (#1435)
• tests: capture logging with context manager (#1342)
• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
• n6 parser: mapping is modified within each run (#905)
• reverse DNS: Only first record is used (#877)
• Corrupt dump files when interrupted during writing (#870)