certtools/intelmq 1.0.0.dev6 on GitHub

Changes between 0.9 and 1.0.0.dev6

Dropped support for Python 2, Python >= 3.3 is needed
Dropped startup.conf and system.conf. Sections in BOTS can be copied directly to runtime.conf now.
Support two run modes: 'stream' which is the current implementation and a new one 'scheduled' which allows scheduling via cron or systemd.
Helper classes for parser bots
moved intelmq/conf to intelmq/etc
cleanup in code and repository
All bots capable of reloading on SIGHUP
packages
pip wheel format instead of eggs
unittests for library and bots
bots/BOTS now contains only generic and specific collectors. For a list of feeds, see docs/Feeds.md

DEV: intelmq_gen_harm_docs: added to generate Harmonization documentation
intelmq_psql_initdb: creates a table for a postgresql database using the harmonization fields
intelmqctl: reworked argument parsing, many bugfixes
intelmqdump: added to inspect dumped messages and reinsert them into the queues
DEV: rewrite_config_files: added to rewrite configuration files with consistent style

added alienvault, alienvault otx, bitsight, blueiv, file, ftp, misp, n6, rtir, xmpp collector
removed hpfeeds collector
removed microsoft DCU collector
renamed and reworked URL collector to HTTP
reworked Mail collectors

source specific parsers added: abusech, alienvault, alienvault otx, anubisnetworks, autoshun, bambenek, bitcash, bitsight, blocklistde, blueliv, ci army, cleanmx, cymru_full_bogons, danger_rulez, dataplane, dshield (asn, block and domain), dyn, fraunhofer_dga, hphosts, malc0de, malwaredomains, misp, n6, netlab_360, nothink, openphish, proxyspy, spamhaus cert, taichung, turris, urlvir
generic parsers added: csv, json
specific parsers dropped: abusehelper (broken), arbor (source unavailable), bruteforceblocker, certeu, dragonresearchgroup parser (discontinued), hpfeeds, microsoft_dcu (broken), taichungcitynetflow, torexitnode parser
renamed intelmq.bots.parsers.spamhaus.parser to intelmq.bots.parsers.spamhaus.parser_drop
renamed intelmq.bots.parsers.malwarepatrol.parser-dansguardian to intelmq.bots.parsers.malwarepatrol.parser_dansguardian
renamed intelmq.bots.parsers.taichungcitynetflow.parser to intelmq.bots.parsers.taichung.parser
major rework of shadowserver parsers
enhanced all parsers

Added experts: asnlookup, cert.at contact lookup, filter, generic db lookup, gethostbyname, modify, reverse dns, rfc1918, tor_nodes, url2fqdn
removed experts: contactdb, countrycodefilter (obsolete), sanitizer (obsolete)
renamed intelmq.bots.expers.abusix.abusix to bots.expers.abusix.expert
intelmq.bots.experts.asnlookup.asnlookup to intelmq.bots.experts.asn_lookup.expert
intelmq.bots.experts.cymru.expert to intelmq.bots.experts.cymru_whois.expert
intelmq.bots.experts.deduplicator.deduplicator to intelmq.bots.experts.deduplicator.expert
intelmq.bots.experts.geoip.geopip to intelmq.bots.experts.maxmind_geoip.expert
intelmq.bots.experts.ripencc.ripencc to intelmq.bots.experts.ripencc_abuse_contact.expert
intelmq.bots.experts.taxonomy.taxonomy to intelmq.bots.experts.taxonomy.expert
enhanced all experts
changed configuration syntax for bots.experts.modify to a more simple variant

TST: testing framework for core and tests. Newly introduced components should always come with proper unit tests.
ENH: intelmqctl has shortcut parameters and can clear queues
STY: code obeys PEP8, new code should always be properly formatted
DOC: Updated user and dev guide
Removed Message.contains, Message.update methods Message.add ignore parameter

ENH: New parameter and field named accuracy to represent the accuracy of each feed
Consistent naming "overwrite" to switch overwriting capabilities of bots (as opposed to override)
Renamed http_ssl_proxy to https_proxy
parameter hierarchical_output for many output bots
deduplicator bot has a new required parameter to configure deduplication mode filter_type
deduplicator bot key ignore_keys was renamed to filter_keys
The tor_nodes expert has a new parameter overwrite, which is by default false.

ENH: Additional data types: integer, float and Boolean
ENH: Added descriptions and matching types to all fields
DOC: harmonization documentation has same fields as configuration, docs are generated from configuration
BUG: FQDNs are only allowed in IDN representation
ENH: Removed UUID Type (duplicate of String)
ENH: New type LowercaseString and UppercaseString, doing automatic conversion
ENH: Removed UUID Type (duplicate of String)
ENH: FQDNs are converted to lowercase
ENH: regex, iregex and length checks when data is added to messages

(source|destination).bgp_prefix is now (source|destination).network
(source|destination).cc is now (source|destination).geolocation.cc
(source|destination).reverse_domain_name is (source|destination).reverse_dns
(source|destination).abuse_contact is lower case only
misp_id changed to misp.event_uuid
protocol.transport added, a fixed list of values is allowed
protocol.application is lower case only
webshot_url is now screenshot_url
additional_information renamed to extra, must be JSON
os.name, os.version, user_agent removed in favor of extra
all hashes are lower case only
added malware.hash.(md5|sha1|sha256), removed malware.hash
New parameter and field named feed.accuracy to represent the accuracy of each feed
New parameter and field named feed.provider to document the name of the source of each feed
New field classification.identifier
-classification.taxonomy is now lower case only

Harmonization: hashes are not normalized and classified, see also issue #394 and pull #634