ibnaleem/gosearch v2.0.2

v2.0.2 - Faster searches, module refactor and 6 new sites

on GitHub

Summary

Runs now take about 15s instead of about 48s. @AtahanPoyraz rebuilt the modules around one interface, @ramnbt turned on post-quantum TLS, and there are 6 new sites.

 main
-  for each module: Search, print           # one after another
+  runModules
+    start every module at once, each writing to its own buffer
+    print each buffer in module order as soon as it's done
   searchSite
-    probe each site (30s client timeout)
+    probe each site, give up after 15s, report it as "could not be checked"

Speed and sites (#139, b3f5b4d)

• All modules run at the same time, and their output still prints in the usual order.
• No site can hold the search for more than 15s. A site that stalls partway through its response is reported, not dropped.
• Added Hugging Face, crates.io, Hex, NuGet, Pixelfed and Exercism. Removed PHUCKS, whose host no longer answers. hoodri.ch and frail.lol were also added.

Architecture overhaul by @AtahanPoyraz (#128)

• Every module implements modules.Module with its own client (3ca3842)
• GitHub finds mutual followers and emails from commits (e60966c)
• More TLDs, and one output file per run (002c77e)
• Site checks follow redirects and only count a 200 from the same domain (953d4e5)
• Failed site checks are counted and reported instead of being treated as not found (71011d3)
• One shared HTTP client and JSON fetch helper (1016f77, 462391d)
• Keybase's INPUT_ERROR means "no profile" instead of an error (518e8f4)
• Tests run in CI, and data.json is validated, which caught a duplicate Lobsters entry (d4326fc, 99d7c10)

Security and fixes

• Hybrid post-quantum TLS key exchange (X25519MLKEM768), by @ramnbt (#137, 158a938)
• Usernames are escaped in site URLs (#132) and sanitised in output paths (#133) (b50bf1b, 3c76016)
• The data.json download has a timeout (#130) and falls back to the local copy, and gosearch no longer deletes your local data.json (#129) (45695af, e5ff16e, 81c5c6d)
• GitHub API errors are reported instead of showing as "not mutual" (323531b)
• Keybase lookup no longer needs PGP (#123, 9ab5b4e)
• CI checks every commit with gofmt, vet, go mod tidy, race tests and data.json validation (#135), and a workflow got explicit permissions (#136)
• Dependabot bumps: brotli 1.2.6, tablewriter 1.1.5, golang.org/x/crypto 0.52.0 (#120, #122, #124, #125, #126, #127)

Evidence

• Before (v2.0.1): gosearch -u ibnaleem finished in 47.8s. One dead host held the site search for the full 30s on every run.
  After: the same command finished in 15.3s and found the same 37 sites. That's one paired run, so treat it as roughly 3x.
• Before: a test with three modules taking 400, 300 and 200ms ran them one after another in 901ms.
  After: the same test finishes in under 700ms, with output still in module order.
• Before: a site that stalled while sending its response was dropped with no warning.
  After: the test passes and reports 1 site(s) could not be checked.

Merge Danger

Door: two-way

Anyone can stay on v2.0.1. The site list is the exception: gosearch downloads data.json from main, so v2.0.1 installs already see the new sites.

Blast Radius: output

Module output is printed in the same order but arrives in bursts as each module finishes. The output file layout comes from #128 (one file per run), so check any script that reads it.

Thanks to @AtahanPoyraz and @ramnbt.

Full Changelog: v2.0.1...v2.0.2