natlas/natlas v0.5.0

Initial release

on GitHub

This release is to provide a consistent, relatively bug free experience for deploying the natlas server and natlas agents. It does not yet contain all of the desired features initially laid out for the project, but it provides a functional set of software that can be used to begin distributed scanning and centralized storage of data to an Elasticsearch back-end.

Development was done on Ubuntu 18.10
Testing was done on Ubuntu 18.10 desktop and 18.04.1 LTS server (headless).

Requirements

The agent can run on relatively low cost hardware, and can scale based on the number of threads you define.

If running Elasticsearch on the same server that your natlas server is running on, you'll likely want at least 3GB of memory. The public test instance is running one server and one agent on 4GB and is hovering around 70% memory usage. Elastic tries to allocate a large chunk of memory at startup by default.

Server Features

• Automatic setup via setup scripts
  • setup-elastic script to download and install Elasticsearch 6.6.0
  • setup-server script to add a natlas user, install all necessary prereqs, initialize the metadata database
  • Python dependencies are contained within a virtual environment
• Server-side scripts for the following:
  • Take or restore elasticsearch snapshots (elastic-snapshot)
  • Add administrative users with a generated random password (add-admin)
  • Promote existing users to administrators (add-admin)
  • Add scope and blacklist items from files (add-scope)
• Server-defined list of services for agents to scan for
  • Ships with augmented nmap default including popular services such as memcached and mongodb.
• Server defined options for agents to use while scanning
  • Enable/disable OS Fingerprinting
  • Enable/disable Version Detection
  • Enable/disable Default Scripts
  • Enable/disable Open ports only
  • Scan timeout in seconds before killing an nmap scan
  • Enable/disable web screenshots
  • Enable/disable VNC screenshots
• Basic user access control to the platform
  • Differentiation between admin user and regular user
  • Setting to require login in order to view results
  • Setting to allow open registration to the site
  • Admin users can:
    • Control settings to modify agent behavior
    • Upload new versions of the servicedb, add individual services, export existing services
    • Add, remove, import, export, view, and toggle scope and blacklist networks.
    • Invite new users, delete users, and promote/demote users to/from admin status
    • Optional login required (default no login required)
    • Optional registration allowed (default invite only)
    • Redirect the natlas server to another Elasticsearch cluster
    • Modify the mail server settings that are used to send invitation and password reset emails
  • Normal users can:
    • change their password
    • change the number of results to display per page
    • change the length for nmap summaries in search results.
• Web Service Front end
  • Browse recent results
  • Search for specific results using elasticsearch queries such as specific ports, scans from a specific time period, and full-text search of any other data that can be read from nmap output
  • Browse scan history for any host, each scan is individually addressable
  • View associated web and vnc screenshots for any host
• API for agent access
  • Unauthenticated (for now)
  • /api/getwork
    • Pseudorandom cyclical target selection allows for a seemingly random distribution of scans while providing some assurance that, if the server runs long enough, all items in scope will have been scanned. Edge cases apply.
      • When the server restarts, the cycle restarts as well.
      • When an agent restarts, it loses whatever targets are in it's work queue as well as whatever targets it's currently working on. This can be defined as 2*agent.config.max_threads for each agent deployed.
    • Provides a randomly generated scan identifier that gets used by agents for filenames
    • Provides the current configuration options to use for each scan
    • Provides a hash of the current servicedb, so the agent can determine if it needs a new one
  • /api/natlas-services
    • Returns the current version of the servicedb and ensures that the hash is correct
  • /api/submit
    • Ensures that an nmap scan report is found in a submission
    • Ensures the target of the scan is both in scope and not blacklisted
    • Retrieves the hostname from the scan report
    • Retrieves a list of open ports from the scan report
    • Timestamps the scan results at the time of submission
    • Rejects suspicious scan results where no open ports are found or too many (500+) open ports are found
  • Provided nginx config specifies 10Mb limit on scan result submissions
• Example systemd units provided that should just work after running the setup scripts.
• Example nginx config provided for a TLS protected front end, reverse proxying to gunicorn

Agent Features

• Nmap to do port scans and store nmap, xml, and gnmap data
• Headless screenshots of web servers using aquatone
• Headless screenshots of VNC servers using vncsnapshot
• Configurable environment variables via a .env file
  • Server address from which to get work and submit results to
  • Number of threads to use for scanning (how many simultaneous hosts will you scan)
  • Switch to allow or deny scanning private IP ranges
    • Off by default, enable if using natlas to enumerate the inside of your network
  • Exponential backoff when attempting to contact the server for new work or submit a completed scan
    • Initial timeout value for scans (default 15 seconds)
    • Maximum backoff period (default 5 minutes)
    • Base value to begin the backoff (default 1 second)
    • A jitter of some number of milliseconds below 1 second is automatically determined and applied to each request
• Standalone mode
  • Scan a single target with --target x.x.x.x
  • Scan a single CIDR range with --target x.x.x.x/xx
  • Scan a list of line-separated targets and CIDR ranges with --target-file myfile.txt
  • NOTE: The server will still reject scans for hosts that are not in scope or are explicitly blacklisted.