v3.3.0

This release introduces major enhancements to deployment infrastructure, data leak monitoring capabilities, and SSL certificate tracking. The update focuses on improving operational flexibility with a modern modular deployment system, adding regex pattern matching for keywords, migrating to the actively maintained SearxNG project, implementing comprehensive SSL certificate expiration monitoring, and introducing a local CertStream server for improved certificate transparency monitoring.

⚠️ Important - Breaking Changes

This is a major infrastructure update that requires manual configuration changes:

✅ New CertStream service must be added to docker-compose.yml
✅ SearxNG migration: Service renamed from searx to searxng
✅ Environment variables: Three variables must be updated in .env
✅ Configuration file: New certstream-config.yaml required
✅ Database migrations: New fields for regex keywords and SSL tracking

Update Procedure

For existing deployments:

Important: This release includes significant infrastructure changes. Please follow these steps carefully.

Step 1: Backup your data

Before proceeding, ensure you have a complete backup of your database and configuration files.

Step 2: Pull the latest version

docker compose pull

Step 3: Stop running containers

docker compose down

Step 4: Update docker-compose.yml

This release introduces a new CertStream service for local certificate transparency monitoring. Update your docker-compose.yml file with the following changes:

Add the CertStream service (add this section before the searxng service):

certstream:
  container_name: certstream
  image: 0rickyy0/certstream-server-go:latest
  restart: always
  networks:
    default:
      ipv4_address: 10.10.10.7
  volumes:
    - ./certstream-config.yaml:/app/config.yaml:ro
  ports:
    - "8080:8080"
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8080/"]
    interval: 10s
    timeout: 5s
    retries: 5
    start_period: 10s
  environment:
    - TZ=${TZ}

Rename the Searx service from searx to searxng:
- Change container_name: searx to container_name: searxng
- Change hostname: searx to hostname: searxng
- Update image from searx/searx:1.1.0-69-75b859d2 to searxng/searxng:latest
- Update volume mount from ./Searx/searx:/etc/searx:rw to ./Searx/searx:/etc/searxng:rw
- Remove the command: ${SEARX_COMMAND:-} line (no longer needed)

Update the Watcher service dependencies:

depends_on:
  db_watcher:
    condition: service_healthy
  searxng:
    condition: service_started
  certstream:
    condition: service_healthy

Add health check to db_watcher (if not already present):

healthcheck:
  test: mysqladmin --user=${DB_USER} --password=${DB_PASSWORD} status
  interval: 2s
  timeout: 1s
  retries: 10
  start_period: 30s

Update the Watcher service environment to bypass proxy for internal services:

environment:
  no_proxy: "10.10.10.3,10.10.10.5,10.10.10.6,10.10.10.7,certstream"

Step 5: Update your `.env` file

The following environment variables must be updated:

SearxNG URL (Required):

DATA_LEAK_SEARX_URL=http://searxng:8080/

CertStream URL (Required - New variable):
```
CERT_STREAM_URL=ws://certstream:8080
```

NO_PROXY configuration (Required for proper internal routing):

NO_PROXY=certstream,10.10.10.7,localhost,127.0.0.1,10.10.10.3,10.10.10.5,10.10.10.6

Optional updates:
- Review and update SEARX_HOSTNAME if needed (default: searxng:8080)
- You can remove SEARX_COMMAND as it's no longer used

Step 6: Add CertStream configuration file

Create or update the certstream-config.yaml file in your project root directory:

# Configuration for certstream-server-go
# This file is mounted read-only in the Docker container

webserver:
  # Listen on all interfaces for Docker networking
  listen_addr: "0.0.0.0"
  listen_port: 8080
  
  # WebSocket endpoints
  lite_url: "/"                    # Recommended endpoint for Watcher
  full_url: "/full-stream"         # Full stream with all certificate data
  domains_only_url: "/domains-only" # Only domain names
  
  # Enable real IP detection for reverse proxy scenarios
  real_ip: true

# Prometheus metrics for monitoring
prometheus:
  enabled: true
  listen_addr: "0.0.0.0"
  listen_port: 9090
  metrics_url: "/metrics"

This configuration file is already included in the repository. If you've customized it, ensure your changes are preserved.

Step 7: Apply database migrations

docker compose run watcher bash
python manage.py migrate

Step 8: Rebuild and restart containers

docker compose up -d

Step 9: Verify the deployment

Check that all services are running correctly:

docker compose ps

For new modular deployment (recommended for production):

The new deployment system provides enhanced reliability and maintainability. To migrate:

Navigate to the new deployment directory:
```
cd deployment
```
Initialize the environment:
```
make init
```
Configure your environment:
- Copy .env.example to .env and customize
- Update domain and network settings
- Configure database credentials
Deploy the stack:
```
make deploy
```

For detailed migration instructions, see the deployment README.

What's Changed

Deployment Infrastructure Overhaul

Modern Modular Docker Deployment System

A complete redesign of the deployment architecture provides improved maintainability, reliability, and operational flexibility:

Modular Compose Architecture
- compose_apps.yaml: Application services (Watcher, SearxNG, CertStream)
- compose_databases.yaml: Database services with optimized configurations
- compose_reverse_proxy.yaml: Traefik reverse proxy with TLS termination
- Main docker-compose.yml orchestrates all services with health checks
Automated Deployment Scripts (deployment/scripts/)
- init.sh: Complete environment initialization and validation
- deploy.sh: One-command full stack deployment
- migrate.sh: Database migration automation
- backup-db.sh: Automated database backup utility
- populate-db.sh: Sample data population
- create-superuser.sh: Interactive superuser creation
- check-network.sh: Docker network validation and creation
- replace-tls.sh: Automatic TLS configuration updates
- openssl-certificates-generator.sh: TLS certificate generation
Makefile Integration
- Simple commands: make up, make down, make deploy, make migrate
- Automatic environment validation before operations
- Integrated backup and initialization workflows
Enhanced Configuration Management
- Centralized .env file with comprehensive examples
- Template-based configuration with validation
- Automatic TLS certificate generation
- Network isolation with custom subnet management
Traefik Reverse Proxy
- Automatic TLS termination with custom certificates
- Dynamic routing configuration
- Access logging and request tracing
- Dashboard for monitoring (optional)

Data Leak Module Enhancements

Regex Pattern Matching for Keywords

Advanced keyword matching capabilities significantly improve data leak detection:

Regex Support
- New is_regex boolean field in Keyword model
- Backend validation of regex patterns before saving
- Compilation error handling with user-friendly messages
- Performance optimization with compiled pattern caching
Frontend UI Improvements
- Checkbox to enable/disable regex mode per keyword
- Visual badge indicators:
  - "RegEx" badge for regex patterns (blue)
  - "Exact" badge for literal matches (green)
Use Cases
- Email pattern matching: [a-zA-Z0-9._%+-]+@company\.com
- Domain variations: (company|brand|product)\.(com|net|org)
- Complex string patterns: API[_-]?KEY[:\s]+[A-Za-z0-9]+

Search Engine Infrastructure

Migration to SearxNG

Transition from deprecated Searx to actively maintained SearxNG ensures long-term reliability:

Service Update
- Container name: searx to searxng
- Official SearxNG image with latest security patches
- Optimized configuration for data leak detection
Configuration Improvements
- Simplified environment variables
- JSON format enabled for API compatibility
- Optimized for containerized environments
- Enhanced engine configuration (GitHub, GitLab, Bitbucket, NPM, StackOverflow)
- Disabled rate limiting for internal use
Settings Synchronization
- Unified configuration between settings.py and .env
- Consistent URL handling across all modules

SSL Certificate Monitoring

Comprehensive SSL Expiration Tracking

New infrastructure for monitoring SSL certificate expiration across all domains:

New SSL Fields
- LegitimateDomain.ssl_expiry: Track certificate expiration dates
- Site.ssl_expiry: Monitor site certificate validity
- Site.domain_created_at: Track domain registration dates
- Alert.new_ssl_expiry / Alert.old_ssl_expiry: Track certificate changes
SSLCertificateChecker Utility
- Automatic certificate fetching and parsing
- Expiration date extraction and validation
- Support for various certificate formats
- Domain name cleaning and normalization
- Issuer information extraction
- Subject Alternative Names (SAN) parsing
Automated Monitoring
- Periodic certificate checks
- Expiration alerts and notifications
- Integration with existing alert system
- Historical tracking of certificate changes

CertStream Infrastructure

Enhanced Certificate Transparency Monitoring

This release introduces a local CertStream server deployment, replacing the reliance on external CertStream services. This architectural change provides better reliability, control, and performance for certificate transparency monitoring.

Local CertStream Server (New)
- Self-hosted CertStream server using 0rickyy0/certstream-server-go:latest
- Local WebSocket endpoint: ws://certstream:8080
- Eliminates dependency on external services (e.g., wss://certstream.calidog.io)
- Improved reliability and reduced latency
- Configurable via certstream-config.yaml
- Health checks for service reliability
- Runs on dedicated network IP: 10.10.10.7
New CertStreamClient Class (dns_finder/certstream_client.py)
- WebSocket client with automatic reconnection
- Enterprise proxy support with authentication
- Automatic proxy detection and bypass for internal URLs
- Periodic ping mechanism to maintain connections
- Smart handling of connection failures with exponential backoff
Proxy Intelligence
- Automatic detection of internal networks
- Smart proxy bypass for Docker networking (NO_PROXY configuration)
- Support for HTTP_PROXY, HTTPS_PROXY environment variables
- Seamless integration with corporate proxy environments
Configuration File
- New certstream-config.yaml for service configuration
- WebSocket endpoint customization
- Prometheus metrics integration
- Flexible endpoint configuration (lite, full, domains-only)
Docker Integration
- Updated to use 0rickyy0/certstream-server-go:latest
- Health checks for service reliability
- Dedicated network configuration with static IP
- Volume mounting for configuration management
- Environment variable support for timezone and proxy settings

New Contributors

@TheoBhang made their first contribution in #266)

Full Changelog: v3.2.2...v3.3.0

What's Changed

Release v3.3.0 – Modular deployment overhaul, local CertStream server, SSL monitoring, and SearxNG migration by @ygalnezri in #267

Full Changelog: v3.2.2...v3.3.0

thalesgroup-cert/Watcher v3.3.0 on GitHub

v3.3.0

⚠️ Important - Breaking Changes

Update Procedure

For existing deployments:

Step 1: Backup your data

Step 2: Pull the latest version

Step 3: Stop running containers

Step 4: Update docker-compose.yml

Step 5: Update your .env file

Step 6: Add CertStream configuration file

Step 7: Apply database migrations

Step 8: Rebuild and restart containers

Step 9: Verify the deployment

For new modular deployment (recommended for production):

What's Changed

Deployment Infrastructure Overhaul

Data Leak Module Enhancements

Search Engine Infrastructure

SSL Certificate Monitoring

CertStream Infrastructure

New Contributors

What's Changed

thalesgroup-cert/Watcher v3.3.0
on GitHub

Step 5: Update your `.env` file