• added the ability to specify an example page for filtering pages that are similar to the given example
  • --filter-similar-to requests the page passed to it via CLI, after which it hashes
    the response body using the SSDeep algorithm. All subsequent
    pages are hashed and compared to the original request's hash. If the comparison of the two hashes meets a certain
    percentage of similarity (currently 95%), then that request will be filtered out.
  • SSDeep was selected as it does a good job of identifying near-duplicate pages once content-length reaches a certain
    size, while remaining performant. Other algorithms were tested but resulted in huge performance hits (orders of
    magnitude slower on requests/second).