certtools/intelmq 2.0.0

on GitHub

Installation instructions:
https://github.com/certtools/intelmq/blob/2.0.0/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/2.0.0/docs/UPGRADING.md

There are some features considered as beta and marked as such in the documentation, do not use them in production yet.

See also the changelog for 2.0.0.beta1 below.

Configurations

• Defaults: New parameters statistics_host, statistics_port, statistics_databasae, statistics_password for statistics redis database (#1402).

Core

• Add more and fix some existing type annotations.
• intelmq.lib.bot:
  • Use statistics_* parameters for bot's statistics (#1402).
  • Introduce collector_empty_process for collectors with an empty process() method, hardcoded 1s minimum sleep time, preventing endless loops, causing high load (#1364).
  • Allow to disable multithreading by initialization parameter, used by intelmqctl / the bot debugger (#1403).
• intelmq.lib.pipeline: redis: OOM can also be low memory, add this to log message (#1405).
• intelmq.lib.harmonization: ClassificationType: Update RSIT mapping (#1380):
  • replace botnet drone with infected-system
  • replace infected system with infected-system
  • replace ids alert with ids-alert
  • replace c&c with c2server
  • replace malware configuration with malware-configuration
  • sanitize replaces these values on the fly
• Allow using non-opt/ (LSB) paths with environment variable INTELMQ_PATHS_NO_OPT.
• Disable/disallow threading for all collectors and some other bots.

Development

• Applied isort to all core files and core-related test files, sorting the imports there (every thing except bots and bots' tests).

Harmonization

• See the Core section for the changes in the allowed values for classification.type.

Bots

• Use the new RSIT types in several types, see above

Parsers

• intelmq.bots.parsers.spamhaus.parser_cert: Added support for extortion events.

Experts

• added intelmq.bots.experts.do_portal.expert.

Outputs

• intelmq.bots.outputs.elasticsearch.output: Support for TLS added (#1406).
• intelmq.bots.outputs.tcp.output: Support non-intelmq counterparts again. New parameter counterpart_is_intelmq, see NEWS.md for more information (#1385).

Packaging

• Update IntelMQ path fix patch after INTELMQ_PATHS_NO_OPT introduction, provide INTELMQ_PATHS_OPT environment variable for packaged instances.

Tests

• test_conf: For yaml use safe_load instead of unsafe load.
• Travis: Switch distribution from trusty to xenial, adapt scripts.
  • Add Python 3.7 to tests.
• Don't use Cerberus 1.3 because of pyeve/cerberus#489

Tools

• intelmqdump: Fix creation of pipeline object by providing a logger.
• intelmqctl: Disable multithreading for interactive runs / the bot debugger (#1403).

Known issues

• tests: capture logging with context manager (#1342)
• pymongo 3.0 deprecates used insert method (#1063)
• pymongo >= 3.5: authentication changes (#1062)
• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
• n6 parser: mapping is modified within each run (#905)
• reverse DNS: Only first record is used (#877)
• Corrupt dump files when interrupted during writing (#870)