ail-project/ail-framework v6.5

AIL v6.5 – I2P Crawler, Image Descriptions, and Enhanced Search

on GitHub

AIL v6.5 introduces several major improvements to strengthen dark web monitoring and analysis workflows:

• I2P Crawling Support
  The crawler now supports I2P, extending coverage beyond Tor and traditional web sources.

• Enhanced Search with Description Indexing
  Search capabilities have been improved with description indexing, making it easier to discover and correlate relevant content across large datasets.

• Improved Image Analysis Workflows
  Image analysis has been optimized to provide more efficient processing, categorization, and contextual enrichment of visual material.

🚀 New Features

• Crawler
  • Added full I2P crawler support, including auto-discovery.
• Search Engine
  • Now supports searching descriptions of images, screenshots, and domains.
  • Added new description indexes for domains, images, and screenshots.
• Image Engine
  • Introduced a function to automatically describe all images for better indexing and search.
• System
  • Added support for kvrocks installed via deb package.
  • Flask will now skip SSL context if certificates or keys are missing, improving deployment flexibility.

Changes & Improvements

• Correlation Engine
  • Added functions to search for IDs in one-depth correlations, with support for returning intermediate objects.
• I2P Crawler
  • Improved domain filtering (e.g., unreachable, unknown, invalid destinations).
  • Enhanced crawler stats reporting.
• psl_faup
  • Added support for additional TLDs (like .b32.i2p).
  • Performance improvements for parsing.
• Settings
  • Dashboard now shows number of active organizations, logged users, and active users.
• Search Engine
  • Enhanced image description handling and UI display.
• Build System
  • Moved YARA installation from update_thirdparty.sh to installing_deps.sh for consistency.

🛠 Fixes

• Crawler / I2P
  • Fixed localhost redirection issues.
  • Corrected filtering of errored and invalid I2P pages.
  • Fixed crawler stats accuracy.
• Onion Module
  • Fixed domain recrawling and general module issues.
• Search Engine
  • Fixed indexing naming issues.
  • Corrected blueprint redirects and pagination.
  • Fixed broken links in image/object descriptions.
• UI & Dashboard
  • Fixed domain type selector in search by date range.
  • Improved crawler dashboard layout and fixed HTML tag issues.
• Tests & Docs
  • Fixed API/UI tests.
  • Corrected documentation for pystemon config directory.

Contributors

• Raphaël Vinot
• Thirion Aurélien

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on the misp-lea.org website.