certtools/intelmq 1.1.1

on GitHub

Installation instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md
Upgrade instructions:
https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md

Core

• lib/harmonization.py: Change parse_utc_isoformat of DateTime class from private to public (related to #1322).
• lib/utils.py: Add new function object_pair_hook_bots.
• lib.bot.py:
  • ParserBot's method recover_line_csv now also handles given tempdata.
  • Bot.acknowledge_message() deletes __current_message to free the memory, saves memory in idling parsers with big reports.
  • start(): Warn once per run if error_dump_message is set to false.
  • Bot.start(), ParserBot.process(): If errors happen on bots without destination pipeline, the on_error path has been queried and lead to an exception being raised.
  • start(): If error_procedure is pass and on pipeline errors, the bot retries forever (#1333).
• lib/message.py:
  • Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (#1335).
  • Do not ignore empty or ignored (as defined in _IGNORED_VALUES) values of extra.* fields for backwards compatibility (#1335).
• lib/pipeline.py (Redis.receive): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334).

Default configuration

• Set error_dump_message to true by default in defaults.conf.
• Fixed typo in defaults.conf: proccess_manager -> process_manager

Development

• bin/rewrite_config_files.py: Fix ordering of BOTS file (#1327).

Harmonization

Update to 2018-09-26 version. New values are per taxonomy:

• Taxonomy 'intrusions':
  • "application-compromise"
  • "burglary"
  • "privileged-account-compromise"
  • "unprivileged-account-compromise"
• Taxonomy 'fraud':
  • "copyright"
  • "masquerade"
  • "unauthorized-use-of-resources"
• Taxonomy 'information content security':
  • "data-loss"
• Taxonomy 'vulnerable':
  • "ddos-amplifier"
  • "information-disclosure"
  • "potentially-unwanted-accessible"
  • "vulnerable-system"
  • "weak-crypto"
• Taxonomy 'availability':
  • "dos"
  • "outage"
  • "sabotage"
• Taxonomy 'abusive-content':
  • "harmful-speech"
  • "violence"
• Taxonomy 'malicious code':
  • "malware-distribution"
• Taxonomy 'information-gathering':
  • "social-engineering"
  • "sniffing"
• Taxonomy 'information content security':
  • "Unauthorised-information-access"
  • "Unauthorised-information-modification"

Bots

Collectors

• intelmq.bots.collectors.http.collector_http:
  • Fix parameter name extract_files in BOTS (#1331).
  • Fix handling of extract_files parameter if the value is an empty string.
  • Handle not installed dependency library requests gracefully.
  • Explain extract_files parameter in docs and use a sane default in BOTS file.
• intelmq.bots.collectors.mail.collector_mail_url:
  • Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling.
  • Handle HTTP errors (bad status code and timeouts) with error_procedure == 'pass' but marking the mail as read and logging the error.
  • Handle not installed dependency library requests gracefully.
• intelmq.bots.collectors.http.collector_http_stream:
  • Handle not installed dependency library requests gracefully.
• intelmq.bots.collectors.microsoft.collector_interflow:
  • Handle not installed dependency library requests gracefully.
• intelmq.bots.collectors.rt.collector_rt:
  • Handle not installed dependency library requests gracefully.
• added intelmq.bots.collectors.shodan.collector_stream for collecting shodan stream data (#1096).
  • Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers.
• intelmq.bots.collectors.microsoft.collector_interflow:
  • Add check if Cache's TTL is big enough compared to not_older_than and throw an error otherwise.

Parsers

• intelmq.bots.parsers.misp: Fix Object attribute (#1318).
• intelmq.bots.parsers.cymru.parser_cap_program:
  • Add support for new format (extra data about botnet of 'bots').
  • Handle AS number 0.
• intelmq.bots.parsers.shadowserver:
  • Spam URL reports: remove src_naics, src_sic columns.
  • fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271).
  • Add support in parser to ignore some columns in config file by using False as intelmq key.
  • Add support for the Outdated-DNSSEC-Key and Outdated-DNSSEC-Key-IPv6 feeds.
  • Add support for the Accessible-Rsync feed.
  • Document support for the Open-LDAP-TCP feed.
  • Add support for Accessible-HTTP and Open-DB2-Discovery-Service (#1349).
  • Add support for Accessible-AFP (#1351).
  • Add support for Darknet (#1353).
• intelmq.bots.parsers.generic.parser_csv: If the skip_header parameter was set to True, the header was not part of the raw field as returned by the recover_line method. The header is now saved and handled correctly by the fixed recovery method.
• intelmq.bots.parsers.cleanmx.parser: Use field first instead of firsttime for time.source (#1329, #1348).
• intelmq.bots.parsers.twitter.parser: Support for url-normalize >= 1.4.1 and recommend it. Added new optional parameter default_scheme, passed to url-normalize (#1356).

Experts

• intelmq.bots.experts.national_cert_contact_certat.expert:
  • Handle not installed dependency library requests gracefully.
• intelmq.bots.experts.ripencc_abuse_contact.expert:
  • Handle not installed dependency library requests gracefully.
• intelmq.bots.experts.sieve.expert:
  • check method: Add missing of the harmonization for the check, caused an error for every check.
  • Add text and more context to error messages.
  • README: Fix 'modify' to 'update' (#1340).
  • Handle empty rules file (#1343).
• intelmq.bots.experts.idea.expert: Add mappings for new harmonization classification.type values, see above.

Outputs

• intelmq.bots.outputs.redis:
  • Fix sending password to redis server.
  • Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354).
  • Use Redis class instead of deprecated StrictRedis for redis-py >= 3.0.0 (#1355).
• intelmq.bots.outputs.mongodb:
  • New parameter replacement_char (default: '_') for non-hierarchical output as dots in key names are not allowed (#1324, #1322).
  • Save value of fields time.observation and time.source as native datetime object, not as string (#1322).
• intelmq.bots.outputs.restapi.output:
  • Handle not installed dependency library requests gracefully.

Documentation

• FAQ
  • Explanation and solution on orphaned queues.
  • Section on how and why to remove raw data.
• Add or fix the tables of contents for all documentation files.
• Feeds:
  • Fix Autoshun Feed URL (#1325).
  • Add parameters name and provider to intelmq/etc/feeds.yaml, docs/Feeds.md and intelmq/bots/BOTS (#1321).
• Add SECURITY.md file.

Packaging

• Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).

Tests

• intelmq.tests.lib.test_bot: Skip test_logging_level_other on python 3.7 because of unclear behavior related to copies of loggers (#1269).
• intelmq.tests.bots.collectors.rt.test_collector: Remove test because the REST interface of the instance has been closed (see also python-rt/python-rt#28).

Tools

• intelmqctl check: Shows more detailed information on orphaned queues.
• intelmqctl:
  • Correctly determine the status of bots started with intelmqctl run.
  • Fix output of errors during bot status determination, making it compatible to IntelMQ Manager.
  • check subcommand: Show bot ID for messages also in JSON output.
  • run [bot-id] process -m [message] works also with bots without a configured source pipeline (#1307).

Contrib

• elasticsearch/elasticmapper: Add tlp field (#1308).
• feeds-config-generator/intelmq_gen_feeds_conf:
  • Add parameters to write resulting configuration directly to files (#1321).
  • Handle collector's feed.name and feed.provider (#1314).

Known issues

• Bots started with IntelMQ-Manager stop when the webserver is restarted (#952).
• Tests: capture logging with context manager (#1342).
• stomp collector bot constantly uses 100% of CPU (#1364).