dmachard/DNS-collector v3.0.0-rc3 on GitHub

What's Changed

This release introduces major performance, scalability, and security improvements:

~40% lower memory footprint across the DNStap collector, wire-DNS decoders, and JSON serialization.
Optional multi-worker pool (num-workers) for improved high-throughput scalability.
New fast direct Wire DNSTap decoder
Distroless runtime image for a smaller Docker image and reduced attack surface.
New DNS message batching across all workers to increase throughput
Support Batch insertion for Clickhouse logger
New Top-N logger for periodic DNS traffic summaries without external infrastructure.
New frequency-filtering transformer for real-time heavy-hitter detection, sampling, and filtering.


        ================================================================================
                  PERFORMANCE COMPARISON: v2.5.0 vs v3.0.0
        ================================================================================
        Metric                      v2.5.0               v3.0.0              Delta
        --------------------------------------------------------------------------------
        Total Messages Processed    1000000              1000000              -
        Execution Time              1.298s               686ms                -47.13%
        Total CPU Time (User+Sys)   2.147s               542ms                -74.73%
        Peak Memory (Max RSS)       105680 KB (103.20 MB) 63428 KB (61.94 MB)  -39.98%
        Throughput                  770451.10            1457310.66           +89.15%
        ================================================================================

⚠️ Deprecations

Worker Queue: The legacy per-worker chan-buffer-size setting is deprecated in favor of global.worker.buffer-size (default: 512) and global.worker.batch-size (default: 64). A non-blocking deprecation warning is logged if legacy options are detected.

💡 Enhancements

feat(transformers): add frequency-filtering transformer with Counting Cuckoo Filter by @dmachard in #1334
feat(kafka): use higher-level Writer abstraction from segmentio/kafka-go by @dmachard in #1322
feat(fileingestor): add Linux SLL cooked-mode PCAP support by @dmachard in #1323
feat(transformers): add Unique Response Tracking (UDR) support with Cuckoo support by @dmachard in #1325 #1330
feat(powerdns): support AuthRequest, Ede, EdeText, and OpenTelemetryTraceID by @dmachard in #1314
feat(clickhouse): add JSONEachRow batching, DNS flags support and connection pooling (#877, #921) by @dmachard in #1318
feat(syslog): support dynamic hostname based on DNStap identity, peer-name or query-ip by @dmachard in #1320
feat(prometheus): add Top N countries and ASNs GeoIP metrics by @dmachard in #1321
feat: add UPX compression for linux, darwin and freebsd release binaries by @dmachard in #1300
feat(docker): switch runtime image to distroless/static-debian13 with UPX compression by @dmachard in #1301
refactor(core): code cleanup to introduce decentralized pluggable component registry and eliminate panic calls by @dmachard in #1329 #1331 #1333 #1332

🚀 Performance

perf(dnsutils): add zero-allocation IPv6 formatting by @dmachard in #1340
perf(userprivacy): optimize IPv6 anonymization with zero-alloc stack formatting by @dmachard in #1342
perf(dnsutils): eliminate heap allocations in PCAP and DNSTap IP parsing using netip.ParseAddr by @dmachard in #1343
perf(dnsutils): add compiled matcher engine with zero allocations by @dmachard in #1337
perf(transformers): optimize reordering with zero-alloc and batched flush by @dmachard in #1309 #1315
perf(transformers): zero-allocation in-place IP anonymization for userprivacy by @dmachard in #1310
perf(extract,dnsutils): zero-allocation base64 JSON encoding and fast hex extraction by @dmachard in #1312
perf(transformers): zero-allocation key generation and repetitive traffic lookup for reducer by @dmachard in #1311
perf(transform reducer): use 32-way map sharding with FNV-1a hashing by @dmachard in #1313
perf(dnsutils): implement zero-copy PCAP packet serialization by @dmachard in #1316
perf(workers): optimize prometheus logger with lazy LRU init by @dmachard in #1307
perf: channel DNS message batching for high throughput and memory reduction by @dmachard in #1306
feat(dnstap): zero-alloc Protobuf wire decoder for DNSTap collector by @dmachard in #1302
perf(dnsutils,workers): binary IP representation with zero-alloc lazy formatting by @dmachard in #1303
perf(dnsutils): fast direct JSON encoder & modular transformer encoding by @dmachard in #1296
perf(dnsutils,workers): precompile text formatter with opcode dispatch by @dmachard in #1297
perf(dnsutils): add zero-alloc fast-path for uncompressed DNS question decoding by @dmachard in #1298
perf(collectors,workers): add lazy timestamp formatting by @dmachard in #1299
perf(workers): optimize flat-json encoding across loggers and add benchmarks by @dmachard in #1294
perf(workers): update worker telemetry channels with atomic counters by @dmachard in #1295
perf(prom-collector): optimize prometheus metrics using atomic counters by @dmachard in #1293
perf(dnsutils): optimize memory and CPU by using DNSMessage pointers by @dmachard in #1291
perf(dnstap collector): add configurable worker pool for high-throughput scalability by @dmachard in #1289
perf(dnsutils): optimize JSON serialization and DNS decoder by @dmachard in #1290
perf(dnstap collector): add configurable socket read buffer size by @dmachard in #1288
perf(dnstap collector): optimize processor hot path allocations and enum lookups by @dmachard in #1287
refactor(tcp logger): move transport buffer flush outside of the frame processing loop for batch efficiency by @dmachard in #1338
refactor: improve lifecycle management, thread safety, and HTTP client connection handling across transformers and workers. by @dmachard in #1341

🔧 Bugfixes

fix(pcap file): prevent duplicate DNS messages on partial read & ignore temp files in file ingestor
fix(statd logger): prevent memory leaks by purging unbounded statsd stream maps during flush cycles by @dmachard in #1339
fix(filtering): prevent nil IPSet panic when both drop-queryip and keep-queryip files are configured by @dmachard in #1326
fix(restapi): prevent unlimited memory usage with bounded LRU caches and TTL by @dmachard in #1319

📖 Documentation

docs(k8s): add Kubernetes deployment manifest and documentation guide by @dmachard in #1317
docs: fix two typos by @vaibhav8a in #1308
chore: fix docs by @dmachard in #1336

Bug reporters

New Contributors

@vaibhav8a made their first contribution in #1308

Full Changelog: v2.5.0...v3.0.0-rc3