ntop/nDPI 4.8

4.8 Stable

on GitHub

Major Changes

• Reworked lists implementation that decreased memory usage of orders of magnitude
• Improved code robustness via extensive code fuzzing
• Various improvements to overall library performance
• Extended IPv6 support

New Supported Protocols and Services

• Add "Heroes of the Storm" video game signature detection. (#1949)
• Add Apache Thrift protocol dissector. (#2007)
• Add Remote Management Control Protocol (RMCP).
• Add Service Location Protocol dissector. (#2036)
• Add VK detection (#1880)
• Add Yandex services detection (#1882)
• Add a new protocol id for generic Adult Content traffic (#1906)
• Add a new protocol id for generic advertisement/analytics/tracking stuff (#1904)
• Add bitcoing protocol dissector. (#1992)
• Add detection of Roblox games (#2054)
• Add support for (un-encrypted) HTTP/2 (#2087)
• Add support for Epic Games and GeForceNow/Nvidia (#1990)
• Add support for SRTP (#1977)
• Added BACnet dissector. (#1940)
• Added HAProxy protocol. (#2088)
• Added OICQ dissector. (#1950)
• Added OperaVPN detection
• ProtonVPN: add basic detection (#2006)
• Added detection of Facebook Reels and Stories
• Add an heuristic to detect fully encrypted flows (#2058)
• Added NDPI_MALWARE_HOST_CONTACTED flow risk
• Added NDPI_TLS_ALPN_SNI_MISMATCH flow risk

Improvements

• Improve protocol detection for:
• FreeBSD compilation fix (C) update
• Gnutella: improve detection (#2019)
• H323: fix false positives (#1916)
• HTTP: fix another memory access error (#2049)
• HTTP: fix extraction of filename (#2046)
• HTTP: fix heap-buffer-overflow (#2044)
• HTTP: improve extraction of metadata and of flow risks (#1959)
• HTTP: remove useless code about XBOX (#1958)
• HTTP: rework state machine (#1966)
• Hangout: detect Hangout/Duo/GoogleMeet/... in the STUN code (#2025)
• Enhance DNS risk for long hostnames (> 32)
• Enhanced MS teams STUN/Azure detection
• Enhanced custom port definition and improved error reporting in case of duplications
• Improve detection of Alibaba flows (#1991)
• Improve detection of crawler/bot traffic (#1956)
• Improve detection of crawlers/bots (#1968)
• Improved MGCP detection by allowing '\r' as line feed.
• Improved MS Teams detection with heuristic
• Improved Steam detection by adding steamdiscover pattern. (#2105)
• Improved Wireguard detection
• Improved checks for duplicated entries in protocols file
• Improved classification further reducing memory used
• Improved detection of invalid chars in DNS names
• Improved domain search tet unit
• Improved helper scripts. (#1986)
• MS Teams enhancement
• MySql: improve detection (#1928)
• zabbix: improve detection (#2055)

Tools

• ndpiReader: allow to configure LRU caches TTL and size (#2004)
• ndpiReader: fix VXLAN de-tunneling (#1913)
• ndpiReader: fix export of DNS/BitTorrent attributes (#1985)
• ndpiReader: fix export of HTTP attributes (#1982)
• ndpiReader: fix flow stats (#1943)
• ndpiReader: fix print of flow payload (#1960)
• ndpiReader: improve printing of payload statistics (#1989)
• ndpiReader: print how many packets (per flow) were needed to perform full DPI (#1891)
• ndpireader: fix detection of DoH traffic based on packet distributions (#2045)

Misc

• ARM compilation fix
• Add ndpi_domain_classify_finalize() function (#2084)
• Add a configuration knob to enable/disable loading of gambling list (#2047)
• Add a new flow risk about literal IP addresses used as SNI (#1892)
• Add an heuristic to detect/ignore some anomalous TCP ACK packets (#1948)
• Add another example of custom rules (#1923)
• Add support for multiline json
• Add support for roaring_bitmap_xor_inplace (#1983)
• Add support for vxlan decapsulation (#1441) (#1900)
• Added Source Engine dissector. (#1937)
• Added lists/gambling.list to extra dist.
• Added slackb.com SNI. (#2067)
• Added ability to define an unlimited number of custom rules IP:port for the same IP (it used tobe limited to 2)
• Added check to avoid skype heuristic false positives
• Added comment
• Added coverage targets to Makefile.am for convenience. (#2039)
• Added fix for better handling exceptions rollback in case of later match
• Added hyperlink
• Added ndpi_binary_bitmap data structure
• Added ndpi_bitmap64 support
• Added ndpi_bitmap_andnot API call
• Added ndpi_bitmap_copy() API call
• Added ndpi_bitmap_is_empty() and ndpi_bitmap_optimize() API calls
• Added ndpi_domain_classify_XXX(0 API
• Added ndpi_filter_add_multi() API call
• Added ndpi_murmur_hash to the nDPI API
• Added new API calls for implementing Bloom-filter like data structures
• Added printf/fprintf replacement for some internal modules. (#1974)
• Added scripts to auto generate hostname/SNI *.inc files. (#1984)
• Added sub-domain classification fix
• Added the ability to define custom protocols with arbitrary Ids in proto.txt
• Added vlan_id in ndpi_flow2json() prototype
• Adds new pcap for testing "funny" HTTP servers
• All protocols should be excluded sooner or later (#1969)
• Allow init of app protocols w/o any hostnames set. (#2057)
• Avoid calling ndpi_reconcile_protocols() twice in ndpi_detection_giveup() (#1996)
• Boundary check
• CI: fix Performance job (#1936)
• Centos7 fixes
• Changed logging callback function sig. (#2000)
• Changes for supporting more efficient sub-string matching
• Classification fixes
• DNS: extract geolocation information, if available (#2065)
• Debian 12 fixes
• Disabled query string validation in MDNS in order to avoid zapping chars that in DNS (instead) are not permitted
• DisneyPlus/Hulu ip lists should be auto-generated (#1905)
• Extend content list of Microsoft protocols (#1930)
• Extend content-match list (#1967)
• Fix LRU/Patricia/Automa stats in ndpiReader with multiple threads (#1934)
• Fix MS Teams detection with heuristic (#1972)
• Fix access to packet/flow information (#2013)
• Fix an heap-buffer-overflow (#1994)
• Fix classification-by-ip in ndpi_detection_giveup (#1981)
• Fix compilation (#2011)
• Fix compilation in CI jobs (#2048)
• Fix compilation on Windows (#2072)
• Fix compilation with GCC-7 and latest RoaringBitmap code (#1886)
• Fix detection of packet direction and NDPI_UNIDIRECTIONAL_TRAFFIC risk (#1883)
• Fix export/serialization of flow->risk (#1885)
• Fix for buffer overflow in serialization
• Fix insert of ip addresses into patricia tree(s) (#1895)
• Fix missing u_char, u_short and u_int typedefs for some platforms e.g.: (#2009)
• Fix packet counters (#1884)
• Fix some errors found by fuzzers (#2078)
• Fix some memory errors triggered by allocation failures (#1995)
• Fix some prototypes (#2085)
• Fix string truncation. (#2056)
• Fixed OpenWRT arm related build issues. (#2104)
• Fixed heap-buffer-overflow issue
• Fixed heap-overflow if compiled with --enable-tls-sigs. (#2038)
• Fixed invalid use of ndpi_free(). Sorry, my fault. (#1988)
• Fixed missing AS_HELP_STRING in configure.ac. (#1893)
• Fixed two OpenWRT arm related build issues. (#2103)
• Fixes matches with domain name strings that start with a dot
• Fixes risk mask exception handling while improving the overall performance
• Implemented Count-Min Sketch [count how many times a value has been observed]
• Implemented Zoom/Teams stream type detection
• Implemented ndpi_XXX_reset() API calls whre XXX is ses, des, hw
• Implemented ndpi_predict_linear() for predicting a timeseries value overtime
• Improved debug output. (#1951)
• Improved invalid logging via printf().
• Improved line protocol dissection with heuristic
• Improved missing usage of nDPIs malloc wrapper. Fixes #1978. (#1979)
• Improved protocol detection exploiting IP-based guess Reworked ndpi_reconcile_protocols() that is now called only in front of a match (less overhead)
• Improvement for reducing false positives
• Included Gambling website data from the Polish hazard.mf.gov.pl list (#2041)
• Keep master protocol in ndpi_reconcile_protocols
• Leak fix
• Language fix
• Line: fix heap-buffer-overflow error (#2015)
• Made VK protocol detection more strict
• Make Bittorrent LRU cache IPv6 aware. (#1909)
• Merged new and old version of ndpi_domain_classify.c code
• Mullvad VPN service added (based on entry node IP addresses) (#2062)
• Numeric truncation at ndpi_analyze.c at lines 101, 104, 107, 110 (#1999)
• Numeric truncation at tls.c:1010 (#2005)
• Ookla: rework detection (#1922)
• Optimizes and fixes possible out0of0boundary write in ndpi_fill_prefix_v4()
• ProtonVPN: split the ip list (#2060)
• QUIC: add support for QUIC version 2
• QUIC: export QUIC version as metadata
• QUIC: fix a memory access error
• QUIC: fix dissection of packets forcing VN
• RDP: improve detection over UDP (#2043)
• RTP: remove dead-code (#1953)
• RTP: rework code (#2021)
• Refreshed ASN lists Enhanced the Line IP list with https://ipinfo.io/AS23576/125.209.252.0/24 used by line
• Remove some useless checks (#1993)
• Remove special handling of some TCP flows without SYN (#1965)
• Removed overlapping port
• Renamed HTTP/2 to HTTP2 as the '/' can have side effects with applications sitting on top of nDPI
• Replaces free() with ndpi_free()
• Rework CI jobs to try reducing CI duration (#1903)
• Reworked domain classification based on binary filters
• Reworked initialization
• Reworked ndpi_filter_xxx implementation using compressed bitmaps
• Reworked teams handling
• RiotGames: add detection of flows (#1935)
• STUN: add dissection of DTLS handshake (#2018)
• STUN: avoid FacebookVoip false positives (#2029)
• STUN: fix Skype/MsTeams detection and monitoring logic (#2028)
• STUN: fix detection of Google Voip apps (#2031)
• STUN: fix detection over TCP
• STUN: improve WhatsappCall detection
• STUN: keep monitoring/processing STUN flows (#2012)
• STUN: tell RTP from RTCP while in monitoring state (#2027)
• Serialization fix
• Set _DEFAULT_SOURCE and _GNU_SOURCE globally. (#2010)
• Simplify ndpi_internal_guess_undetected_protocol() (#1941)
• Simplify the report of streaming multimedia info (#2026)
• SoftEther: fix invalid memory access
• Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic approach for handling Internet domain names.
• Sync unit tests results
• Sync unit tests results
• Sync unit tests results (#1962)
• Sync utests results (#1887)
• TLS: add basic, basic, detection of Encrypted ClientHello (#2053)
• TLS: fix another interger overflow in certificate processing (#1915)
• TLS: fix parsing of certificate elements (#1910)
• Test files for riit games
• Test multiple ndpiReader configurations (#1931)
• Thrift: fix heap-buffer-overflow (#2024)
• Update GitHub runners versions (#1889)
• Update every ip lists (#2079)
• Update libinjection code (#1918)
• Update protocols documentation (#2081)
• Update roaring bitmap code
• Updated line test result
• Updated pcap detection results after Facebook Reel/Stories support
• Updated results
• Updated results after the latest changes
• Win include change
• Windows code rework
• Windows compilation fixes
• Windows warning checks
• add 2 ns from fdn.fr to DoH section (#1964)
• add support for gre decapsulation (#1442) (#1921)
• added bimap and/or with allocation
• added feature to extract filename from http attachment (#2037)
• added new domain names (#2002)
• configure: add an option to enable debug build, i.e -g (#1929)
• fix Stack overflow caused by invalid write in ndpi_automa_match_strin… (#2035)
• fixed numeric truncation error
• fixed numeric truncation error in diameter.c (#2034)
• fixed numeric truncation error in kerberos.c (#2032)
• fixed numeric truncation error in ndpi_main.c:6837 (#1998)
• fixed numeric truncation error in rtcp.c (#2033)
• fuzz: add a new fuzzer to test TLS certificates (#1901)
• fuzz: add a new fuzzer triggering the payload analyzer function(s) (#1926)
• fuzz: add fuzzer for DGA detection code (#2042)
• fuzz: add fuzzer to test internal gcrypt code (#1920)
• fuzz: add fuzzers to test bitmap64 and domain_classify data structures (#2082)
• fuzz: add fuzzers to test reader_util code (#2080)
• fuzz: extend coverage (#2073)
• fuzz: extend fuzz coverage (#1888)
• fuzz: extend fuzzers coverage (#1952)
• fuzz: extend fuzzing coverage (#2040)
• fuzz: extend fuzzing coverage (#2052)
• fuzz: extend fuzzing coverage (#2083)
• fuzz: simplify fuzzers dependencies in CIFuzz (#1896)
• fuzz: some improvements and add two new fuzzers (#1881)
• fuzzing: extend fuzzing coverage
• in case of failure, failing result files are not listed
• minor fixes (#2023)
• oss-fuzz: sync build script with upstream
• remove redefinition to vxlanhdr struct in vxlan dissector (#1911)
• removed useless call of ndpi_set_risk func (#2022)
• tests: add an option to force the overwrite of the unit tests results (#2001)
• tests: restore some old paths as symbolic links (#2050)
• tftp: check for Option Acknowledgements
• tftp: check incrementation for DATA and ACK packets
• tftp: rework request checking to account for options
• tftp: update pcap results
• version of dirent.c that is liked by both VC++ and MinGW