certtools/intelmq 2.0.0.beta1

2.0.0 Beta 1

on GitHub

Installation instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md

There are some features considered as beta and marked as such in the documentation, do not use them in production yet.

• upgraded all files to python3-only syntax, e.g. use super() instead of super(..., ...) in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer.

Removals of deprecated code:

• Removed compatibility shim intelmq.bots.collectors.n6.collector_stomp, use intelmq.bots.collectors.stomp.collector instead (see #1124).
• Removed compatibility shim intelmq.bots.parsers.cymru_full_bogons.parser, use intelmq.bots.parsers.cymru.parser_full_bogons instead.
• Removed compatibility shim handing deprecated parameter feed for collectors. Use name instead.
• Removed deprecated and unused method intelmq.lib.pipeline.Pipeline.sleep.
• Removed support for deprecated parameter query_ripe_stat in intelmq.bots.experts.ripe.expert, use query_ripe_stat_asn and query_ripe_stat_ip instead (#1291).
• Removed deprecated and unused function intelmq.lib.utils.extract_tar.

Core

• lib/pipeline:
  • Allow setting the broker of source and destination independently.
  • Support for a new AMQP broker. See User Guide for configuration. (#1179)
• lib/bot:
  • Dump messages locks the dump file using unix file locks (#574).
  • Print idle/rate limit time also in human readable format (#1332).
  • set_request_parameters: Use {} as default proxy value instead of None. Allows updating of existing proxy dictionaries.
  • Bots drop privileges if they run as root.
  • Save statistics on successfully and failed processed messages in the redis database 3.
• lib/utils
  • Function unzip to extract files from gzipped and/or tar-archives.
  • New class ListHandler: new handler for logging purpose which saves the messages in a list.
  • Add function seconds_to_human.
  • Add function drop_privileges.
  • parse_relative: Strip string before parsing.
  • parse_logline: Do not convert the timestamps to UTC, leave them as is.
• lib/cache:
  • Allow ttl to be None explicitly.
  • Overwrite existing cache keys in the database instead of discarding the new data.
• lib/bot:
  • Basic, but easy-to-configure multi-threading using python's threading library. See the User-Guide for more information (#111, #186).
• bin/intelmqctl:
  • Support for Supervisor as process manager (#693, #1360).

Harmonization

Bots

Collectors

• added intelmq.bots.parsers.opendxl.collector (#1265).
• added intelmq.bots.collectors.api: collecting data using an HTTP API (#123, #1187).
• added intelmq.bots.collectors.rsync (#1286).
• intelmq.bots.collectors.http.collector_http:
  • Add support for uncompressing of gzipped-files (#1270).
  • Add time-delta support for time formatted URLs (#1366).
• intelmq.collectors.blueliv.collector_crimeserver: Allow setting the API URL by parameter (#1336).
• intelmq.collectors.mail:
  • Use internal lib for functionality.
  • Add intelmq.bots.collectors.mail.collector_mail_body.
  • Support for ssl_ca_certificate parameter (#1362).

Parsers

• added intelmq.bots.parsers.mcafee.parser_atd (#1265).
• intelmq.bots.parsers.generic.parser_csv:
  • New parameter columns_required to optionally ignore parse errors for columns.
• added intelmq.bots.parsers.cert_eu.parser_csv (#1287).
  • Do not overwrite the local time.observation with the data from the feed. The feed's field 'observation time' is now saved in the field extra.cert_eu_time_observation.
  • Fix parsing of asn (renamed to source asn, source.asn internally) and handle existing feed.accuracy for parsing confidence.
  • Update columns and mapping to current (2019-04-02) data.
• added intelmq.bots.parsers.surbl.surbl
• added intelmq.bots.parsers.html_table (#1381).
• intelmq.bot.parsers.netlab_360.parser: Handle empty lines containing blank characters (#1393).
• intelmq.bots.parsers.n6.parser_n6stomp: Handle events without IP addresses.
• intelmq.bots.parsers.cymru.parser_cap_program: Handle new feed format.
• intelmq.bots.parsers.shadowserver:
  • Add support for the Accessible-FTP feed (#1391).
• intelmq.bots.parsers.dataplane.parser:
  • Fix parse errors and log more context (#1396).
• added intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py (#1373).

Experts

• added intelmq.bots.experts.recordedfuture_iprisk (#1267).
• added intelmq.bots.experts.mcafee.expert_mar (1265).
• renamed intelmq.bots.experts.ripencc_abuse_contact.expert to intelmq.bots.experts.ripe.expert, compatibility shim will be removed in version 3.0.
  • Added support for geolocation information in ripe expert with a new parameter query_ripe_stat_geolocation (#1317).
  • Restructurize the expert and de-duplicataion (#1384).
  • Handle '?' in geolocation country data (#1384).
• intelmq.bots.experts.ripe.expert:
  • Use a requests session (#1363).
  • Set the requests parameters once per session.
• intelmq.bots.experts.maxmind_geoip.expert: New parameter use_registered to use the registered country (#1344).
• intelmq.bots.experts.filter.expert: Support for paths (#1208).

Outputs

• added intelmq.bots.experts.mcafee.output_esm (1265).
• added intelmq.bots.outputs.blackhole (#1279).
• intelmq.bots.outputs.restapi.expert:
  • Set the requests parameters once per session.
• intelmq.bots.outputs.redis:
  • New parameter hierarchichal_output (#1388).
  • New parameter with_type.
• intelmq.bots.outputs.amqptopic.output: Compatibility with pika 1.0.0 (#1084, #1394).

Documentation

• added documentation for feeds
  • CyberCrime Tracker
  • Feodo Tracker Latest
• Feeds: Document abuse.ch URLhaus feed (#1379).
• Install and Upgrading: Use intelmqsetup tool.

Packaging

Tests

• Add tests of AMQP broker.
• Travis: Change the ownership of /opt/intelmq to the current user.

Tools

• intelmqctl check: Now uses the new ListHandler from utils to handle the logging in JSON output mode.
• intelmqctl run: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour.
• intelmqdump: Inspecting dumps locks the dump file using unix file locks (#574).
• intelmqctl:
  • After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown.
• intelmqsetup: New tool for initialize an IntelMQ environment.

Contrib

• malware_name_mapping:
  • Added the script apply_mapping_eventdb.py to apply the mapping to an eventdb.
  • Possibility to add local rules using the download tool.
• check_mk:
  • Added scripts for monitoring queues and statistics.

Known issues

• Multi-threaded bots require multiple SIGTERMs (#1403)
• Stats can't be saved with AMQP if redis is password-protected (#1402)
• Update taxonomies to current RSIT and vice-versa (#1380)
• stomp collector bot constantly uses 100% of CPU (#1364)
• tests: capture logging with context manager (#1342)
• Consistent message counter log messages for all kind of bots (#1278)
• pymongo 3.0 deprecates used insert method (#1063)
• pymongo >= 3.5: authentication changes (#1062)
• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
• n6 parser: mapping is modified within each run (#905)
• reverse DNS: Only first record is used (#877)
• Corrupt dump files when interrupted during writing (#870)