certtools/intelmq 2.0.1

on GitHub

Install documentation:
https://github.com/certtools/intelmq/blob/2.0.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.0.1/docs/UPGRADING.md

As always: read the NEWS file, upgrade according to the documentation
and have fun! If you get any errors, please report them here or in the
bug tracker.

Core

• intelmq.lib.harmonization:
  • IPAddress: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
  • All types: Handle None for validation and sanitation gracefully.
• intelmq.lib.bot:
  • fix parameters of ParserBot and CollectorBot constructors, allowing intelmqctl run with these bots again (#1414).
  • Also run rate_limit after retry counter reset (#1431).
• __version_info__:
  • is now available in the top level module.
  • uses integer values now instead of strings for numerical version parts
• Also provide (empty) ROOT_DIR for non-pip installations.
• intelmq.lib.upgrades: New library file upgrades with upgrade functions.
• intelmq.lib.utils:
  • New function setup_list_logging for intelmqctl check an possibly others.
    • Fix return values (#1423).
  • New function version_smaller for version comparisons.
  • New function lazy_int for version conversions.
  • parse_logline: Handle thread IDs.
  • log takes a new argument logging_level_stream for the logging level of the console handler.
  • New constant LOG_FORMAT_SIMPLE, used by intelmqctl.
  • New function write_configuration to write dicts to files in the correct json formatting.
  • New function create_request_session_from_bot.
• intelmq.lib.pipeline:
  • AMQP:
    • Actually use source/destination_pipeline_amqp_virtual_host parameter.
    • Support for SSL with source/destination_pipeline_ssl parameter.
  • pipeline base class: add missing dummy methods.
  • Add missing return types.
  • Redis: Evaluate return parameter of queue/key deletion.
• Variable STATE_FILE_PATH added.

Development

• intelmq.bin.intelmq_gen_docs: For yaml use safe_load instead of unsafe load.

Harmonization

• IPAddress type: Remove Scope/Zone IDs for IPv6 addresses in sanitation.
• TLP: Sanitation handles now more cases: case-insensitive prefixes and arbitrary whitespace between the prefix and the value (#1420).

Bots

Collectors

• intelmq.bots.collectors.http.collector_http: Use utils.create_request_session_from_bot.
• intelmq.bots.collectors.http.collector_http_stream: Use utils.create_request_session_from_bot and thus fix some retries on connection timeouts.
• intelmq.bots.collectors.mail.collector_mail_url: Use utils.create_request_session_from_bot.
• intelmq.bots.collectors.microsoft.collector_interflow: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
• intelmq.bots.collectors.rt.collector_rt: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
• intelmq.bots.collectors.twitter.collector_twitter: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts for non-twitter connections.

Parsers

• intelmq.bots.parsers.n6.parser_n6stomp: use malware-generic instead of generic-n6-drone for unknown infected system events.
• intelmq.bots.parsers.abusech.parser_ip: Support LastOnline column in feodo feed (#1400) and use it for time.source if available.
  • Use lower case malware names as default, should not make a difference in practice.
  • Fix handling of CSV header for feodotracker (#1417, #1418).
• intelmq.bots.parsers.netlab_360.parser: Detect feeds with https:// too.

Experts

• intelmq.bots.experts.generic_db_lookup: Recommend psycopg2-binary package.
• intelmq.bots.experts.modify.expert:
  • Compile regular expressions (all string rules) at initialization, improves the speed.
  • Warn about old configuration style deprecation.
• intelmq.bots.experts.do_portal.expert:
  • Use utils.create_request_session_from_bot and thus fix retries on connection timeouts (#1432).
  • Treat "502 Bad Gateway" as timeout which can be retried.
• intelmq.bots.experts.ripe.expert: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.
• intelmq.bots.experts.url2fqdn.expert: Support for IP addresses in hostnames (#1416).
• intelmq.bots.experts.national_cert_contact_certat.expert: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.

Outputs

• intelmq.bots.outputs.postgresql: Recommend psycopg2-binary package.
• intelmq.bots.outputs.amqptopic:
  • Shutdown: Close connection only if connection exists.
  • Add support for pika > 1. Pika changed the way it indicates (Non-)Acknowledgments of sent messages.
  • Gracefully handle unroutable messages and give advice.
  • Support for connections without authentication.
  • Replace deprecated parameter type with exchange_type for exchange_declare, supporting pika >= 0.11 (#1425).
  • New parameters message_hierarchical_output, message_with_type, message_jsondict_as_string.
  • New parameter use_ssl for SSL connections.
  • New parameter single_key for sending single fields instead of the full event.
• intelmq.bots.outputs.mongodb.output: Support for pymongo >= 3.0.0 (#1063, PR#1421).
• intelmq.bots.outputs.file: time.* field serialization: support for microseconds.
• intelmq.bots.outputs.mongodb.output: Support for authentication in pymongo >= 3.5 (#1062).
• intelmq.bots.outputs.restapi.output: Use utils.create_request_session_from_bot and thus fix retries on connection timeouts.

Documentation

• Add certbund-contact to the ecosystem document.
• Rename the IDEA expert to "IDEA Converter".
• Add the new configuration upgrade function to the docs.
• User Guide:
  • Clarify on Uninstallation

Packaging

• Do not execute the tcp collector tests during debian and ubuntu builds as they fail there.

Tests

• intelmq.lib.test: Disable statistics for test runs of bots.
• contrib.malware_name_mapping: Added tests.
• Travis: Also run tests of contrib.

Tools

• intelmqsetup: Only change directory ownerships if necessary.
• intelmqctl:
  • Provide new command upgrade-conf to uprade configuration to a newer version.
    • Makes backups of configurations files on its own.
    • Also checks for previously skipped or new functions of older versions and catches up.
  • Provides logging level on class layer.
  • Fix -q flag for intelmqctl list queues by renaming its alternative name to --non-zero to avoid a name collision with the global --quiet parameter.
  • For console output the string intelmqctl: at the beginning of each line is no longer present.
  • check: Support for the state file added. Checks if it exists and all upgrade functions have been executed successfully.
  • Waits for up to 2 seconds when stopping a bot (#1434).
  • Exits early on restart when stopping a bot did not work (#1434).
  • intelmqctl run process -m debugging: Mock acknowledge method if incoming message is mocked too, otherwise a different message is acknowledged.
  • Queue listing for AMQP: Support non-default monitoring URLs, see User-Guide.

Contrib

• logcheck rules: Adapt ignore rule to cover the instance IDs of bot names.
• malware name mapping:
  • Ignore lines in mapping starting with '#'.
  • Optionally include malpedia data.
  • Fix command line parsing for not arguments (#1427).

• bash-completion: Support for intelmqctl upgrade-config added.

Known issues

• http stream collector: retry on regular connection problems? (#1435)
• tests: capture logging with context manager (#1342)
• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
• n6 parser: mapping is modified within each run (#905)
• reverse DNS: Only first record is used (#877)
• Corrupt dump files when interrupted during writing (#870)