certtools/intelmq 2.1.3

Bugfix release

on GitHub

Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md

Requirements

• The python library requests is (again) listed as dependency of the core (#1519).

Core

• intelmq.lib.upgrades:
  • Harmonization upgrade: Also check and update regular expressions.
  • Add function to migrate the deprecated parameter attach_unzip to extract_files for the mail attachment collector.
  • Add function to migrate changed Taichung URL feed.
  • Check for discontinued Abuse.CH Zeus Tracker feed.
• intelmq.lib.bot:
  • ParserBot.recover_line: Parameter line needs to be optional, fix usage of fallback value self.current_line.
  • start: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).
  • start: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
  • _dump_message: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
• intelmq.lib.test:
  • BotTestCase.run_bot: Add parameters allowed_error_count and allowed_warning_count to allow set the number per run, not per test class.
  • Set source_pipeline_broker and destination_pipeline_broker to pythonlist instead of the old broker, fixes intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising.
  • Fix test for (allowed) errors and warnings.
• intelmq.lib.exceptions:
  • InvalidKey: Add KeyError as parent class.
  • DecodingError: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
• intelmq.lib.pipeline:
  • Decode messages in Pipeline.receive not in the implementation's _receive so that the internal counter is correct in case of decoding errors (#1494).
• intelmq.lib.utils:
  • decode: Raise new DecodingError if decoding fails.

Harmonization

• protocol.transport: Adapt regular expression to allow the value nvp-ii (protocol 11).

Bots

Collectors

• intelmq.bots.collectors.mail.collector_mail_attach:
  • Fix handling of deprecated parameter name attach_unzip.
  • Fix handling of attachments without filenames (#1538).
• intelmq.bots.collectors.stomp.collector: Fix compatibility with stomp.py versions > 4.1.20 and catch errors on shutdown.
• intelmq.bots.collectors.microsoft:
  • Update REQUIREMENTS.txt temporarily fixing deprecated Azure library (#1530, PR#1532).
  • intelmq.bots.collectors.microsoft.collector_interflow: Add method for printing the file list.

Parsers

• intelmq.bots.parsers.cymru.parser_cap_program: Support for protocol 11 (nvp-ii) and conficker type.
• intelmq.bots.parsers.taichung.parser: Support more types/classifications:
  • Application Compromise: Apache vulnerability & SQL injections
  • Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
  • C2 Sever: Attack controller
  • DDoS
  • DoS: DNS, DoS, Excess connection
  • IDS Alert / known vulnerability exploitation: backdoor
  • Malware: Malware Proxy
  • Warn on new unknown types.
• intelmq.bots.parsers.bitcash.parser: Removed as feed is discontinued.
• intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target: Removed as feed is discontinued.
• intelmq.bots.parsers.malwaredomains.parser: Correctly classify C&C and phishing events.
• intelmq.bots.parsers.shadowserver.parser: More verbose error message for missing report specification (#1507).
• intelmq.bots.parsers.n6.parser_n6stomp: Always add n6 field name as malware.name independent of category.
• intelmq.bots.parsers.anubisnetworks: Update parser with new data format.
• intelmq.bots.parsers.bambenek: Add new feed URLs with Host faf.bambenekconsulting.com (#1525, PR#1526).
• intelmq.bots.parsers.abusech.parser_ransomware: Removed, as the feed is discontinued (#1537).
• intelmq.bots.parsers.nothink.parser: Removed, as the feed is discontinued (#1537).
• intelmq.bots.parsers.n6.parser: Remove not allowed characters in the name field for malware.name and write original value to event_description.text instead.

Experts

• intelmq.bots.experts.cymru_whois.lib: Fix parsing of AS names with Unicode characters.

Outputs

• intelmq.bots.outputs.mongodb:
  • Set default port 27017.
  • Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).

Documentation

• Feeds:
  • Remove unavailable feed Abuse.CH Zeus Tracker.
  • Remove the field status, offline feeds should be removed.
  • Add a new field public to differentiate between private and public feeds.
  • Adding documentation URLs to nearly all feeds.
  • Remove unavailable Bitcash.cz feed.
  • Remove unavailable Fraunhofer DDos Attack feeds.
  • Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
  • Update information on Bambenek Feeds, many require a license now (#1525).
  • Remove discontinued Nothink Honeypot Feeds (#1537).
• Developers Guide: Fix the instructions for /opt/intelmq file permissions.

Packaging

• Patches: fix-logrotate-path.patch: also include path to rotated file in patch.
• Fix paths from /opt to LSB for setup.py and contrib/logrotate/intelmq in build process (#1500).
• Add runtime dependency debianutils for the program which, which is required for intelmqctl.

Tests

• Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
• intelmq.tests.bots.experts.cymru_whois:
  • Drop missing ASN test, does not work anymore.
  • IPv6 to IPv4 test: Test for two possible results.
• intelmq.lib.test: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).
• intelmq.bots.collectors.tcp.test_collector: Removing custom mocking and bot starting, not necessary anymore.
• Added tests for intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline.
• Fix and split tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json.
• Added tests for invalid encodings in input messages in intelmq.tests.lib.test_bot and intelmq.tests.lib.test_pipeline (#1494).
• Travis: Explicitly enable RabbitMQ management plugin.
• intelmq.tests.lib.test_message: Fix usage of the parameter blacklist for Message hash tests (#1539).

Tools

• intelmqsetup: Copy missing BOTS file to IntelMQ's root directory (#1498).
• intelmq_gen_docs: Feed documentation generation: Handle missing/empty parameters.
• intelmqctl:
  • IntelMQProcessManager: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492).
  • IntelMQController: Fix exit codes of check command for JSON output (now 0 on success and 1 on error, was swapped, #1520).
• intelmqdump:
  • Handle base64-type messages for show, editor and recovery actions.

Contrib

• intelmq/bots/experts/asn_lookup/update-asn-data: Use pyasn_util_download.py to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, hadiasghari/pyasn#62).

Known issues

• HTTP stream collector: retry on regular connection problems? (#1435).
• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
• Reverse DNS: Only first record is used (#877).
• Corrupt dump files when interrupted during writing (#870).