nDPI 4.4 (July 2022)

New Features

• Add risk information that describes why a specific risk was triggered also providing metadata
• Added API call ndpi_check_flow_risk_exceptions() for handling risk exceptions
• Split protocols in: network (e.g. TLS) and application protocols (e.g. Google)
• Extended confidence level with two new values (NDPI_CONFIDENCE_DPI_PARTIAL and NDPI_CONFIDENCE_DPI_PARTIAL_CACHE)
• Added ndpi_get_flow_error_code() API call

New Supported Protocols and Services

• Add protocol detection for:
  • UltraSurf
  • i3D
  • RiotGames
  • TSAN
  • TunnelBear VPN
  • collectd
  • PIM (Protocol Indipendent Multicast)
  • Pragmatic General Multicast (PGM)
  • RSH
  • GoTo products (mainly GoToMeeting)
  • Dazn
  • MPEG-DASH
  • Agora Software Defined Real-time Network (SD-RTN)
  • Toca Boca
  • VXLAN
  • MDNS/LLMNR

Improvements

• Improve protocol detection for:
  • SMTP/SMTPS now supports STARTTLS
  • OCSP
  • TargusDataspeed
  • Usenet
  • DTLS (added support for old versions)
  • TFTP
  • SOAP via HTTP
  • GenshinImpact
  • IPSec/ISAKMP
  • DNS
  • syslog
  • DHCP (various bug fixes and improvements)
  • NATS
  • Viber
  • Xiaomi
  • Raknet
  • gnutella
  • Kerberos
  • QUIC (Added support for v2drft 01)
  • SSDP
  • SNMP
• Improved DGA detection
• Improved AES-NI check
• Add flow risk:
  • NDPI_PUNYCODE_IDN
  • NDPI_ERROR_CODE_DETECTED
  • NDPI_HTTP_CRAWLER_BOT
  • NDPI_ANONYMOUS_SUBSCRIBER
• NDPI_UNIDIRECTIONAL_TRAFFIC

Changes

• Added support for 64 bit bins
• Added Cloudflare WARP detection patterns
• Renamed Z39.50 -> Z3950
• Replaced nDPI's internal hashmap with uthash
• Reimplemented 1kxun application protoco
• Renamed SkypeCall to Skype_TeamsCall
• Updated Python Bindings
• Unless --with-libgcrypt is used, nDPI now uses its internal gcrypt implementation

Fixes

• Fixes for some protocol classification families
• Fixed default protocol ports for email protocols
• Various memory and overflow fixes
• Disabled various risks for specific protocols (e.g. disable missing ALPN for CiscoVPN)
• Fix TZSP decapsulation

Misc

• Update ASN/IPs lists
• Improved code profiling
• Use Doxygen to generate the API documentation
• Added Edgecast and Cachefly CDNs.

Raw Changelog

• Label SMTP w/ STARTTLS as SMTPS and dissect TLS clho. (#1639)
• Compilation fix
• Fix handling of NDPI_UNIDIRECTIONAL_TRAFFIC risk (#1636)
• SMTP with STARTTLS is now identified as SMTPS
• Detect SMTPs w/ STARTTLS as TLS and dissect client/server hello. Fixes #1630. (#1637)
• Run regression tests from different locations at the same time w/o side effects on the results. (#1638)
• Exported username in flow information
• Updated ndpi_check_flow_risk_exceptions() signature
• Cleaned-up issuer DN check code adding u_int8_t ndpi_check_issuerdn_risk_exception(struct ndpi_detection_module_struct *ndpi_str, char *issuerDN);
• Set CiscoVPN as a network protocol
• Updated JA3/SSL fingerprints.
• Replaced malicious JA3-md5/SSL-cert-sha1 ac automata with hashmaps.
• Added UltraSurf protocol dissector. (#1618)
• Add two new confidence values: confidence by partial DPI (#1632)
• Update host content list match (#1633)
• Sync Psiphon unit test. (#1634)
• Added Psiphon detection patterns. See #566 and #1099. (#1631)
• OCSP: improve detection (#1629)
• Added i3D and RiotGames protocol dissectors. (#1609)
• TargusDataspeed: avoid false positives (#1628)
• Update ASN/IPs lists (#1627)
• bins: add support for 64bit bins (#1626)
• Skinny: rework and improve classification (#1625)
• Skype_Teams, Mining, SnapchatCall: fix flow category (#1624)
• Minor changes in how classification results are set (#1623)
• Usenet: improve dissection (#1622)
• Fix category for mail sessions (#1621)
• TLS: add support for old DTLS versions and for detection of mid-sessions (#1619)
• Fix a compilation warning (#1620)
• Generate profiling results as PNG.
• gprof test/CI integration
• Improved TFTP. Dissect Read/Write Request filenames. (#1617)
• Added TSAN support. (#1613)
• Fix byte-order issue during ndpiReader tcp/udp src/dst port serialization. Fixes #1608. (#1614)
• Added Cloudflare WARP detection patterns. (#1615) (#1616)
• Fixed SMTP default port 587
• Added TunnelBear VPN detection patterns. (#1615)
• Updated (C)
• Removed space from "Genshin Impact"
• sync unit tests (#1612)
• Fix after the protocol name update
• Renamed Z39.50 -> Z3950 as the '.' breaks the naming convention QUIC is a network protocol
• Enhanced TLS risk info reported to users
• Added default port for syslog TCP
• Fix compilation and sync unit tests results (#1606)
• Added unidirectional traffic flow risk
• Improved SOAP via HTTP. (#1605)
• Improved GenshinImpact protocol dissector. (#1604)
• Added collectd dissector (again). (#1601)
• Replaced nDPI's internal hashmap with uthash. (#1602)
• Improved IPSec/ISAKMP detection. (#1600)
• Added new test pcaps
• Add some statistics to ndpiReader (#1587)
• Add support for PIM (Protocol Indipendent Multicast) protocol (#1599)
• Improved WhatsApp detection. (#1595)
• Fix invalid memory access (#1596)
• DNS: fix TTL check and sync unit test results (#1594)
• Updated DNS alert triggered only with TTL == 0
• Restored ndpi_set_proto_defaults() prototype Updated test results
• Added check for DGA names that resolve to a valid record
• Improved DNS traffic analysis Added ability to identify application and network protocols
• Added DNS record TTL check
• Added gprof CPU/HEAP profiling support. (#1592)
• Removed Makefile references to legacy code. (#1589)
• Added Pragmatic General Multicast (PGM) protocol detection
• Dissect host line if SSDP contains such. (#1586)
• Reimplemented 1kxun application protocol. (#1585)
• Prevent compilation failure if, for whatever reason, NDPI_API_VERSION is empty. (#1584)
• Fixed syslog false negatives. (#1582)
• Fix some debug messages (#1583)
• Updated test results
• Fixed invalid DHCP dissection
• Fixed DHCP dissection bug
• Added RSH dissector. Fixes #202. (#1581)
• Add support for GoTo products (mainly GoToMeeting) (#1580)
• Fix syslog heap overflow introduced in 09fbe0a. (#1579)
• Fixed syslog false positives. (#1577)
• Fix heap buffer overflow mentioned in #1574. (#1576)
• TLS: fix use-of-uninitialized-value error (#1573)
• Removed README.nDPI as it does not provide any new information not covered by README.md (#1572)
• Removed LGTM ql query for packet payload integer arithmetic. (#1570)
• Force roaring bitmap to use ndpi memory wrappers. (#1569)
• TLS: fix stack-buffer-overflow error (#1567)
• Updated risk results
• Improved message for known proto on non std port
• Added check
• Updated README.md (#1562)
• TLS: fix use-of-uninitialized-value error (#1566)
• Redefined type name to avoid conflicts
• Added ability to return risk info in JSON format in ndpi_get_flow_risk_info()
• Support word diff for tests/do.sh for better readability. (#1565)
• Prohibit MPEG-DASH to set HTTP as application protocol. (#1560)
• HTTP: fix heap-buffer-overflow error (#1564)
• Certificate timestamps should be printed in UTC (#1563)
• Fixed dispay bug for risk_info
• Updated tests results Code cleanup
• Added RiskInfo string
• Fix dissection of IPv4 header (#1561)
• Dazn: add support for Dazn streaming service (#1559)
• Compilation fixes for old ggc's
• Comment
• Added detection for WordPress exploits Fixed ndpi_iph_is_valid_and_not_fragmented() that was bugged with non UDP traffic
• Use Doxygen to generate the API documentation. (#1558)
• Added MPEG-DASH dissector. Fixes #1223. (#1555)
• Fixed HTTP lower/upper protocol mess for Aimini/IPP. (#1557)
• Compilation fixes for old gcc compiler
• Compilation fixes
• Version cut fix
• Fixes compilation issues on RedHat systems
• Sync unit test results (#1554)
• Updated SkypeCall -> Skype_TeamsCall
• Fixed false positives with NATS
• Added script to compare and verify the output of `make dist'. (#1551)
• Replaced obsolete autoconf macros. (#1553)
• Fixed windows-latest build error. (#1552)
• Improved invalid host detection
• Added invalid SNI check in QUIC
• Improved detection of invalid SNI and hostnames in TLS, HTTP
• Added room for storing information used by custom third-party dissectors
• Moved RTSP http patterns to the protocol source file.
• Yet another approach to fix #1499 (basically a copy&pasta from @socketpair).
• Removed MacOS XCode integration.
• Moved mgcp.pcapng to tests/pcap/ instead of tests/
• DNS-over-QUIC: update default port (#1548)
• Improved Viber (TCP) detection. (#1547)
• Improved Xiaomi HTTP detection. (#1546)
• Removed TLS patterns in the CiscoVPN aka Anyconnect dissector as mentioned in PR #1534. (#1543)
• Added Softether(-VPN) DDNS service detection. (#1544)
• Improved TLS alert detection. (#1542)
• Improved TLS application data detection. (#1541)
• Added Edgecast and Cachefly CDNs. (#1540)
• Replaced ndpiReader's libjson-c support with libnDPI's internal serialization interface. (#1535)
• Fix compilation (if --enable-debug-messages is used) (#1539)
• Added extra check to make sure that the guessed protocol is the one we expect and not another one
• Fixes bug that prevents triggering alerts for traffic on non-standard ports that have been defined in the custom protocols file
• Fixes outdated description
• Modified risk labels
• Added some Pluralsight Hostnames/SNIs. May fix #1501. (#1538)
• Updated RRD dependencies
• Improved suspicious http user agent detection. (#1537)
• Added ndpi_get_flow_error_code() API call Fixed typo
• Improved AES-NI check. (#1536)
• Improved AES-NI check on Linux to avoid crashes on CPUs that do not support it (e.g. Intel Celeron N2930)
• Sync unit tests results (#1533)
• Improved TLS application data detection. (#1532)
• Added BPF filtering for discarding non-IP packets
• String messages have been shrinked
• Added ability to store custom category file in patricia tree
• Add ndpi_json_string_escape to the API
• Raknet: fix heap-buffer-overflow (#1531)
• Added generic user agent setter. (#1530)
• XIAOMI: add detection of Xiaomi traffic (#1529)
• Added RakNet protocol dissector. (#1527)
• Code cleanup (removed redundancy)
• Tiny gnutella improvement if gtk-gnutella used. (#1525)
• Updated `utils/whatsapp_ip_addresses_download.sh' to scrape the required IP addresses/ranges. (#1524)
• Add some scripts to easily update some IPs lists (#1522)
• Reduce ndpiReader's -h' spam. -H' does this job now. (#1523)
• Added proprietary Agora Software Defined Real-time Network (SD-RTN) protocol dissector. (#1520)
• Added Toca Boca protocol dissector. (#1517)
• Removed superfluous ifdef'd includes. (#1519)
• Improved sflow protocol detection false-positives. (#1518)
• Kerberos: fix Undefined-shift error (#1516)
• DGA improvements
• Minor fix.
• Merge pull request #1491 from utoni/fix/windows-msys2
• Fixed msys2 build warnings and re-activated CI Mingw64 build.
• Kerberos: fix some memory access errors (#1514)
• Extended list of cybersecurity domains
• fix(ndpi_main):Fix memory leak about ndpi_str; (#1513)
• TINC: fix invalid memory read (#1512)
• Improved ASN.1 parsing for Keberos. Fixes #1492. (#1497)
• QUIC: handle retransmissions and overlapping fragments in reassembler (#1195) (#1498)
• Fix JSON-C.
• Python bindings fix.
• Added ndpi_find_outliers() API call using Z-Score
• Added -z flag
• ndpiReader: fix compilation (#1510)
• Removed un-necessary guess in mining
• update
• Fixed incompatibilities due to #1509
• DGA improvements
• Waring fixes
• ndpireader: add json output back. (#1509)
• Improvements for CUSTOM_NDPI_PROTOCOLS
• Moved geneated file to a separate folder
• Improved twitter detection
• Removed SRV record from suspicious DNS traffic
• Improved DGA detection
• [autoconf] Fixed .git submodule detection test. (#1507)
• Added code for identifiying anomalies with metrics stored in InfluxDB
• reader_util: add support for userAgent in SSDP (#1502)
• Add support for Pluralsight site (#1503)
• Fix CI tests results (#1504)
• Reducing the size of the ndpi_detection_module_struct structure. (#1490)
• [SSDP] Extract HTTP user-agent when available. (#1500)
• Improved DGA detection skipping names containign at least 3 consecutive digits in the first word
• QUIC: add support for version 2 draft 01 (#1493)
• Mining: cleanup registration (#1496)
• Trying to improve QUIC reassembler (#1195) (#1489)
• Update Python bindings guide.
• Fix typo.
• Add HOWTO Python.
• Fix python bindings CI.
• Complete rework of nDPI Python bindings (cffi API, automatic generation, packaging and CI integration)
• Extended the list of cybersecurity protocol
• Bug fixing. (#1487)
• QUIC: convert logs to standard mechanism (#1485)
• QUIC: fix dissection of draft-34 (#1484)
• Extend tests coverage (#1476)
• Improved ASN/IP update scripts and CI integration. (#1474)
• Implement CI on Windows. (#1483)
• Some small fixes (#1481)
• Extracting the Azure Origin url from the download link (#1480)
• Errors fixed (#1482)
• EthernetIP: fix integer conversion on big-endian archs (#1477)
• Fixed a bug for BE architectures (#1478)
• configure: fix usage of libgpg-error with --with-local-libgcrypt (#1472)
• Added autoconf option `--enable-tls-sigs'. (#1471)
• Drop support for non-gcrypt builds. (#1469)
• Internal crypto: increase size of authentication buffer (#1468)
• reader_util: fix parsing of MPLS packets (#1467)
• Add ICMP checksum check and set risk if mismatch detected. (#1464)
• Typo
• Added configureable ndpi packet processing limit. (#1466)
• Fix libgcrypt(-light/-internal) compile error. (#1465)
• Add a new flow risk NDPI_ANONYMOUS_SUBSCRIBER (#1462)
• Unless --with-libgcrypt is used, nDPI now uses its internal gcrypt implementation
• Removed some unused fields (#1461)
• Bug fixing. (#1459)
• Added `--enable-code-coverage' build using lcov for coverage generation. (#1430)
• reader_util: fix TZSP decapsulation (#1460)
• Add some scripts to easily update some IPs lists (#1449)
• Provide some API functions for convenience. (#1456)
• Win fixes
• Replaced strdup with ndpi_strup
• Directly drop malformed packets (#1455)
• reader_util: fix parsing of IPv6 extension headers (#1453)
• reader_util: fix infinite loop in packet dissection (#1454)
• fuzz: purge old sessions (#1451)
• DTLS: fix access to certificate cache (#1450)
• EthernetIP: add missing initialization (#1448)
• Add support for Google Cloud (#1447)
• fuzz: make fuzz_ndpi_reader faster (#1446)
• Added lightweight implementation of libgcrypt. (#1444)
• Fix compilation and sync unit tests results (#1445)
• Added newflow risk NDPI_HTTP_CRAWLER_BOT
• Silenced NDPI_SUSPICIOUS_DGA_DOMAIN, NDPI_BINARY_APPLICATION_TRANSFER, NDPI_HTTP_NUMERIC_IP_HOST, NDPI_MALICIOUS_JA3,
• Extended cybersecurity protocol dissection
• Added SNMP error code check
• Exteended cybersecurity list
• Invalid prototupe fix
• HSRP: fix dissection over IPv6 (#1443)
• Added cybersecurity category mapping to string
• Added cybersecurity protocol and category that groups traffic towards leading cybersecurity companies and CDNs, useful to make destinations that should be marked as trusted in firewalls and security gateways
• HSRP: add support for IPv6 (#1440)
• Added VXLAN dissector (#1439)
• Fix memory access in ndpi_strnstr() (#1438)
• Add few scripts to easily update some IPs lists (#1436)
• Increment current/total number of active flows on successful flow insertion (#1434)
• Added ndpi_serialize_string_string_len() APi call Fixed CSV string serialization
• Added HSRP protocol detection Removed attic directory now obsolete
• Added check to ignore multicast packets marking the as Skype
• Improved MDNS/LLMNR detection. (#1437)
• TLS: fix parsing of certificate elements (#1435)
• Sync utests (#1433)
• Add comment
• Updated test results
• Added NDPI_ERROR_CODE_DETECTED risk
• Renamed DCERPC to more generic RPC protocol so we can use also for other types of RPCs (not limited to DCE) Extended HTTP plugin to support RPC Improved HTTP crear text detection to limit it to Basic and Digest
• Typo
• Improved risks description
• Updated risk documentation
• Added new IDN/Punycode risk for spotting internationalized domain names
• Added missing __sync_fetch_and_add() definition in Windows