tonikelope/megabasterd v8.53

MegaBasterd 8.53

on GitHub

TL;DR

• Fixes #773 -- memory footprint of long-running MegaBasterd is now substantially smaller and stops creeping under heavy bursts. Two concrete changes attack the real cost drivers (native thread-stack RSS + GC pressure from per-chunk allocations), and the JVM-RAM status label gained a click-to-release action introduced in the unreleased 8.52 development cut.
• Single biggest contributor was the unbounded global thread pool, which under realistic bursts (transferences provisioning + smart proxy + per-slot managers + streamer) routinely peaked at 100-200 live JVM threads -- each one carrying a ~512 KB-1 MiB native stack that no GC can ever reclaim.

New in 8.53

#773 -- Cap global THREAD_POOL at 64 workers

• What changed. MainPanel.THREAD_POOL used to be Executors.newCachedThreadPool(...), i.e. unbounded. Now it is a ThreadPoolExecutor(0, 64, 60s, LinkedBlockingQueue, daemonFactory). Daemon-thread factory is unchanged.
• Why this is the high-leverage change. Each JVM thread carries a native stack outside the Java heap (~512 KB-1 MiB depending on platform), so a 150-thread burst was adding ~75-150 MB of resident memory that no System.gc() can reclaim. Capping at 64 erases that contribution entirely and reflects directly in the Task Manager RSS that motivated #773.
• Why nothing regresses. The queue is unbounded, so no task is ever rejected -- past the cap they queue for microseconds until a worker frees up. Per-transfer chunk workers live in each Download/Upload's own thread pool (Download.java:219, 265), not in this one, so the global cap does not throttle download parallelism. 64 leaves ample headroom for the long-lived managers + supervisors + handful of short-lived setup tasks that actually use it.

#773 -- Reuse the 1 MiB in-memory buffer in ChunkDownloaderMono

• What changed. The mono (single-slot) chunk loop used to do new byte[(int) chunk_size] inside the chunk loop, throwing away ~1 MiB of byte[] per chunk. The buffer is now hoisted to a worker-level local and grown on demand; after the warmup ramp (128 KiB, 256 KiB, ..., 896 KiB, then 1 MiB steady state) the array stops growing and every subsequent chunk reuses it.
• Why it matters. A stable mono download processes ~60 chunks/min, so the old code translated to ~60 MB/min of pure allocation pressure per active mono download. G1 recycles young-gen fast enough that the live set stayed small, but the constant churn kept the heap commit elevated and made auto-shrink slow. The new path is exactly one allocation per worker for the entire transfer.
• Atomic-commit semantics preserved. The reason mono buffers in memory at all is the comment in ChunkDownloaderMono.java:256-266: writing directly to the sequential output stream while reading would let a mid-chunk network failure shift every subsequent chunk and corrupt the file. The cipher stream still fills the buffer fully before output_stream.write() runs, and the MAX_CHUNK_BUFFER_BYTES guard above the allocation still applies. Only the allocation strategy changed; the safety invariant did not.

Rolled in from the unreleased 8.52 development cut

• #773 -- click-to-release on the JVM-RAM status label + auto-GC hint at 70 %. The FORCE_GARBAGE_COLLECTION_MAX_MEMORY_PERCENT = 0.7 constant had been declared in MainPanel for a long time but never read. The 2 s memory-monitor loop now requests a GC when live usage crosses 70 % of -Xmx, throttled to once per 60 s so it cannot pin a core during heavy chunk decryption. Left-clicking the JVM-RAM used: ... label fires the same hint on demand; hover for a tooltip explaining the behaviour and suggesting -Xmx for a hard cap. The G1 collector that ships as default since Java 9 honours System.gc() and, after a couple of cycles, returns free regions to the OS -- which is the user-visible reduction in Task Manager that #773 was about.

Recommended launch flag for users on systems with lots of RAM

Java's default -Xmx is roughly 1/4 of physical RAM on Java 9+. On a 16 GB box the JVM happily commits up to ~4 GB even if the live working set is small. If you want a hard ceiling, launch with:

java -Xmx512m -jar MegaBasterd_8.53.jar

512 MB is plenty for everyday use; bump to -Xmx1g only if you see OutOfMemoryError in the DEBUG LOG tab under heavy parallel downloads.

Other

• Version bumped to 8.53 in pom.xml and MainPanel.VERSION.
• No transfer-path logic changed beyond the buffer reuse described above. Multi-slot downloads, uploads, streaming server, and SmartProxy paths are untouched.