certtools/intelmq 2.3.0

2.3.0 Feature release

on GitHub

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.

Configuration

Core

• intelmq.lib.bot:
  • ParserBot.recover_line_json_stream: Make line parameter optional, as it is not needed for this method (by Sebastian Wagner).
  • Bot.argparser: Added class method _create_argparser (returns argparse.ArgumentParser) for easy command line arguments parsing (PR#1586 by Filip Pokorný).
  • Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorný).
  • Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
  • Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner).
• intelmq.lib.upgrades:
  • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559, by Sebastian Wagner).
• intelmq.lib.exceptions:
  • PipelineError: Remove unused code to format exceptions (by Sebastian Wagner).
• intelmq.lib.utils:
  • create_request_session_from_bot:
    • Changed bot argument to optional, uses defaults.conf as fallback, renamed to create_request_session. Name create_request_session_from_bot will be removed in version 3.0.0 (PR#1524 by Filip Pokorný).
    • Fixed setting of http_verify_cert from defaults configuration (PR#1758 by Birger Schacht).
  • log: Use RotatingFileHandler for allow log file rotation without external tools (PR#1637 by Vasek Bruzek).
• intelmq.lib.harmonization:
  • The IPAddress type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner).
  • DateTime.parse_utc_isoformat: Add parameter return_datetime to return datetime object instead of string ISO format (by Sebastian Wagner).
  • DateTime.convert: Fix utc_isoformat format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner).
  • DateTime.from_timestamp: Ensure that time zone information (+00:00) is always present (by Sebastian Wagner).
  • DateTime.__parse now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner).
• intelmq.lib.upgrades:
  • Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner).
  • Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht).

Development

• intelmq.bin.intelmq_gen_docs:
  • Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht).
  • Merged into docs/autogen.py (PR#1622 by Birger Schacht).

Bots

Collectors

• intelmq.bots.collectors.eset.collector: Added (PR#1554 by Mikk Margus Möll).
• intelmq.bots.collectors.http.collector_http:
  • Added PGP signature check functionality (PR#1602 by sinus-x).
  • If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner).
• intelmq.bots.collectors.kafka.collector: Added (PR#1654 by Birger Schacht, closes #1634).
• intelmq.bots.collectors.xmpp.collector: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
• intelmq.bots.collectors.shadowserver.collector_api:
  • Added (#1683, PR#1700 by Birger Schacht).
  • Change file names in the report to .json instead of the original and wrong .csv (PR#1769 by Sebastian Wagner).
• intelmq.bots.collectors.mail: Add content of the email's Date header as extra.email_date to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner).
• intelmq.bots.collectors.http.collector_http_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
• intelmq.bots.collectors.shodan.collector_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
• intelmq.bots.collectors.twitter.collector_twitter:
  • Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner).
  • Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum of one (PR#1754 by Sebastian Wagner).

Parsers

• intelmq.bots.parsers.eset.parser: Added (PR#1554 by Mikk Margus Möll).
  • Ignore invalid "NXDOMAIN" IP addresses (PR#1573 by Mikk Margus Möll).
• intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559, by Sebastian Wagner).
• intelmq.bots.parsers.cznic.parser_haas: Added (PR#1560 by Filip Pokorný and Edvard Rejthar).
• intelmq.bots.parsers.cznic.parser_proki: Added (PR#1599 by sinus-x).
• intelmq.bots.parsers.key_value.parser: Added (PR#1607 by Karl-Johan Karlsson).
• intelmq.bots.parsers.generic.parser_csv: Added new parameter compose_fields (by Sebastian Wagner).
• intelmq.bots.parsers.shadowserver.parser_json: Added (PR#1700 by Birger Schacht).
• intelmq.bots.parsers.shadowserver.config:
  • Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer).
  • Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer).
  • Ignore value 0 for source.asn and destination.asn in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner).
• intelmq.bots.parsers.abusech.parser_ip: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus).
• intelmq.bots.parsers.malwaredomainlist: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).

Experts

• intelmq.bots.experts.rfc1918.expert:
  • Add support for ASNs (PR#1557 by Mladen Markovic).
  • Speed improvements.
  • More output in debug logging mode (by Sebastian Wagner).
  • Checks parameter length on initialization and in check method (by Sebastian Wagner).
• intelmq.bots.experts.gethostbyname.expert:
  • Added parameter fallback_to_url and set to True (PR#1586 by Edvard Rejthar).
  • Added parameter gaierrors_to_ignore to optionally ignore other gethostbyname errors (#1553).
  • Added parameter overwrite to optionally overwrite existing IP addresses (by Sebastian Wagner).
• intelmq.bots.experts.asn_lookup.expert:
  • Added --update-database option (PR#1524 by Filip Pokorný).
  • The script update-asn-data is now deprecated and will be removed in version 3.0.
• intelmq.bots.experts.maxmind_geoip.expert:
  • Added --update-database option (PR#1524 by Filip Pokorný).
  • Added license_key parameter (PR#1524 by Filip Pokorný).
  • The script update-geoip-data is now deprecated and will be removed in version 3.0.
• intelmq.bots.experts.tor_nodes.expert:
  • Added --update-database option (PR#1524 by Filip Pokorný).
  • The script update-tor-nodes is now deprecated and will be removed in version 3.0.
• intelmq.bots.experts.recordedfuture_iprisk.expert:
  • Added --update-database option (PR#1524 by Filip Pokorný).
  • Added api_token parameter (PR#1524 by Filip Pokorný).
  • The script update-rfiprisk-data is now deprecated and will be removed in version 3.0.
• Added intelmq.bots.experts.threshold (PR#1608 by Karl-Johan Karlsson).
• Added intelmq.bots.experts.splunk_saved_search.expert (PR#1666 by Karl-Johan Karlsson).
• intelmq.bots.experts.sieve.expert:
  • Added possibility to give multiple queue names for the path directive (#1462, by Sebastian Wagner).
  • Added possibility to run actions without filtering expression (#1706, PR#1708 by Sebastian Waldbauer).
  • Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer).
• intelmq.bots.experts.maxmind_geoip.expert:
  • Fixed handing over of overwrite parameter to event.add (PR#1743 by Birger Schacht).

Outputs

• intelmq.bots.outputs.rt: Added Request Tracker output bot (PR#1589 by Marius Urkis).
• intelmq.bots.outputs.xmpp.output: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
• intelmq.bots.outputs.smtp.output: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner).

Documentation

• Feeds:
  • Add ESET URL and Domain feeds (by Sebastian Wagner).
  • Remove unavailable HPHosts Hosts file feed (#1559 by Sebastian Wagner).
  • Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorný and Edvard Rejthar).
  • Added CZ.NIC Proki feed (PR#1599 by sinus-x).
  • Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorný).
  • Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner).
  • Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner).
  • Fixed parsing of the public field in the generated feeds documentation (PR#1641 by Birger Schacht).
  • Change the rate_limit parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds).
  • Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorný and Sebastian Wagner).
  • Added Shadowserver Reports API (by Sebastian Wagner).
  • Change the rate_limit parameter for many feeds from 2 days to the default one day (by Sebastian Wagner).
  • Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
• Bots:
  • Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic and Sebastian Wagner).
  • Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar).
  • Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new --update-database option (PR#1524 by Filip Pokorný).
  • Added documentation for Shadowserver API collector and parser (PR#1700 by Birger Schacht and Sebastian Wagner).
• Add n6 integration documentation (by Sebastian Wagner).
• Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner).
• Generate documentation using Sphinx (PR#1622 by Birger Schacht).
  • The documentation is now available at https://intelmq.readthedocs.io/en/latest/
  • Refactor documentation and fix broken syntax (#1639, PRs #1638 #1640 #1642 by Birger Schacht).
• Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht).

Packaging

• Fix paths in the packaged logcheck rules (by Sebastian Wagner).
• Build the sphinx documentation on package build (PR#1701 by Birger Schacht).
• Ignore non-zero exit-codes for the intelmqctl check call in postinst (#1748, by Sebastian Wagner).

Tests

• Added tests for intelmq.lib.exceptions.PipelineError (by Sebastian Wagner).
• intelmq.tests.bots.collectors.http_collector.test_collector: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
• intelmq.tests.bots.outputs.restapi.test_output:
  • Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
  • Add a test for checking the response status code (by Sebastian Wagner).
• intelmq.tests.bots.collectors.mail.test_collector_url: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
• intelmq.tests.bots.experts.ripe.test_expert: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
• The test flag (environment variable) INTELMQ_TEST_LOCAL_WEB is no longer used (by Sebastian Wagner).
• Added tests for intelmq.harmonization.DateTime.parse_utc_isoformat and convert_fuzzy (by Sebastian Wagner).
• Move from Travis to GitHub Actions (PR#1707 by Birger Schacht).
• intelmq.lib.test:
  • test_static_bot_check_method checks the bot's static check(parameters) method for any exceptions, and a valid formatted return value (#1505, by Sebastian Wagner).
  • setUpClass: Skip tests if cache was requests with use_cache member, but Redis is deactivated with the environment variable INTELMQ_SKIP_REDIS (by Sebastian Wagner).
• intelmq.tests.bots.experts.cymru_whois.test_expert:
  • Switch from example.com to ns2.univie.ac.at for hopefully more stable responses (#1730, PR#1731 by Sebastian Waldbauer).
  • Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner).
• intelmq.tests.bots.parsers.abusech: Remove tests cases of discontinued feeds (PR#1741 by Thomas Bellus).
• Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner).

Tools

• intelmqdump:
  • Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic).
• intelmqctl:
  • intelmq list queues: --sum, --count, -s flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic).
  • intelmq check: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner).
  • Allow setting the pipeline host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).

Contrib

• EventDB:
  • Add SQL script for keeping track of the oldest inserted/update "time.source" information (by Sebastian Wagner).
• Cron Jobs: The script intelmq-update-data has been renamed to intelmq-update-database (by Filip Pokorný).
• Dropped utterly outdated contrib modules (by Sebastian Wagner):
  • ansible
  • vagrant
  • vagrant-ansible
• logrotate:
  • Do not use the deprecated "copytruncate" option as intelmq re-opens the log anyways (by Sebastian Wagner).
  • Set file permissions to 0644 (by Sebastian Wagner).

Known issues

• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
• Corrupt dump files when interrupted during writing (#870).
• CSV line recovery forces Windows line endings (#1597).
• intelmqdump: Honor logging_path variable (#1605).
• Timeout error in mail URL fetcher (#1621).
• AMQP pipeline: get_queues needs to check vhost of response (#1746).