github tonikelope/megabasterd v8.55
MegaBasterd 8.55

one hour ago

TL;DR

  • Fixes #776 — "Checking previous downloads/uploads, please wait..." stayed forever on the status bar. A regression introduced in 8.53 (the THREAD_POOL cap added for #773) effectively froze every background task submitted to the global pool after the first one, including the startup resume of previous downloads and uploads, the SmartProxy refresh, the single-instance watchdog, and the MEGAcrypter reverse server. The status label that says "Checking previous..." is set on the EDT but cleared from the resume worker — and the resume worker never got to run, so the label stayed forever. Visible on a clean install with an empty queue too. No previous downloads or accounts were actually being checked — the runnable that does that work was just stuck in the queue.

  • Cause: classic ThreadPoolExecutor(0, N, LinkedBlockingQueue) antipattern. ThreadPoolExecutor.execute() only spawns a recovery worker after queuing when workerCountOf == 0. The first task on the pool (the while (!_exit) memory monitor added in 8.52 for #773) grabbed the one worker forever and every later submit enqueued behind it without ever waking another worker. Documented in the JDK source for years; same trap that bites everyone who reaches for ThreadPoolExecutor instead of newCachedThreadPool.

  • Fix is a 5-line, byte-exact replacement of the pool config. corePoolSize == maxPoolSize == 64, allowCoreThreadTimeOut(true), same unbounded LinkedBlockingQueue. The 64-thread RSS cap from #773 is preserved, idle workers still die after the 60 s keepAlive, and nothing else changes.

  • Withdrawn binaries. 8.53 and 8.54 are defective and have been removed from the releases page. Their original changelogs are preserved at the bottom of these notes (minus the broken THREAD_POOL section of 8.53) so the #773 mono-buffer work and the #774 post-finish hooks are not lost from the changelog.

What was wrong

8.53 wired the global pool as:

new ThreadPoolExecutor(0, 64, 60s, new LinkedBlockingQueue<>(), factory)

ThreadPoolExecutor.execute() does this:

  1. If workerCountOf(c) < corePoolSize, spawn a new core worker. With corePoolSize=0 this branch never fires.
  2. Otherwise, offer the task to the queue. An unbounded LinkedBlockingQueue always accepts, so this always succeeds.
  3. After enqueuing, if workerCountOf(recheck) == 0, spawn a non-core recovery worker. This is the only way a worker ever gets created in this configuration — and it requires workerCount to already be zero.

The startup sequence runs three long-lived submits very early:

  • The memory monitor (while (!_exit) Thread.sleep(2000)).
  • resumeDownloads().
  • resumeUploads().

First submit: queue is empty, no workers exist, the recheck-recovery branch sees workerCount == 0, creates worker #1, worker #1 picks up the memory monitor runnable and loops forever inside while (!_exit). Worker #1 never returns to the pool.

Second submit: workerCount == 1, queue accepts the task, recheck-recovery doesn't fire because workerCount != 0. Task sits in the queue indefinitely.

Third submit: same. Plus every later submit for the entire lifetime of the process (SmartProxy refresh, watchdog accept loop, MEGAcrypter reverse, etc.).

Result for #776: the "Checking previous downloads, please wait..." label is set on the EDT inside resumeDownloads() and is cleared inside the worker's THREAD_POOL.execute(() -> { ... }) body. Since the body never runs, the label stays. Same story for "Checking previous uploads..." set inside resumeUploads().

What changes

MainPanel.THREAD_POOL is now built as:

ThreadPoolExecutor tpe = new ThreadPoolExecutor(
        GLOBAL_THREAD_POOL_MAX, GLOBAL_THREAD_POOL_MAX,
        60L, TimeUnit.SECONDS,
        new LinkedBlockingQueue<>(),
        _megabasterdDaemonThreadFactory());
tpe.allowCoreThreadTimeOut(true);

That is the canonical "grow up to N on demand, shrink back when idle" pool: new submits create workers up to the cap, idle workers die after the 60 s keepAlive, the unbounded queue still buffers anything past 64 concurrent tasks so nothing is ever rejected. Identical RSS bound as 8.53 once idle because allowCoreThreadTimeOut(true) lets all 64 slots release back to zero workers when the pool is quiet — so the #773 memory-mitigation goal is preserved end-to-end.

Audit of every other commit between 8.52 and 8.54

Done before shipping this fix to make sure nothing else regressed in the same window:

  • 4651ad4 (#773 mono chunk buffer reuse). Pure local change inside ChunkDownloaderMono: lifts the per-chunk new byte[chunk_size] to a worker-level scratch buffer that grows on demand. No behavioural change outside that method. Clean.
  • e396b14 / 6a9c0ee / 1f876f5 (#774 post-finish hooks + SettingsDialog rows + i18n keys). New onAllTransferencesFinished() hook in TransferenceManager, plus two new config keys in SettingsDialog. The hook only fires when finish > 0 && pre+prov+wait+run == 0 && _isOKFinishedInQueue(), which is impossible to satisfy at startup with an empty queue. Cannot have produced the #776 symptom. Clean.

So the entire #776 symptom traces back to one commit (92a9dbe) and is fixed by reverting its pool-config pattern to one that actually creates workers.

Other

  • Version bumped to 8.55 in pom.xml and MainPanel.VERSION.
  • No changes to transfer-path logic, the resume queue persistence, the account-login flow, the MEGA error popup machinery, the SmartProxy machinery, the #774 post-finish hooks, or any of the i18n bundles.
  • For users who were on 8.53 or 8.54 and noticed pending downloads/uploads not resuming on startup, they will resume normally on the first launch of 8.55. The queue persistence on disk (the downloads_queue / uploads_queue SQLite rows) was always being read correctly; the problem was that the runnable doing the reading never got scheduled.

Closes #776.


Preserved changelog from withdrawn v8.53 and v8.54

The 8.53 and 8.54 binaries have been removed from the releases page because the broken THREAD_POOL cap shipped in both. Everything below is the original changelog text from those releases, minus the broken Cap global THREAD_POOL at 64 workers section of 8.53 (which is exactly what this 8.55 release withdraws). The mono-buffer optimisation, the JVM-RAM click-to-release UI, and the #774 post-finish hooks all shipped intact in 8.55 too, so this section exists purely so the changelog history of those features is not lost when their original release entries are deleted.

Originally shipped in v8.54 (withdrawn)

#774 — Independent post-finish hooks for downloads and uploads

  • What changed. MainPanel grows two new pairs of fields modelled on the existing 509 command: _run_command_dl_finish + _run_command_dl_finish_path (with LAST_DL_FINISH_COMMAND_TIMESTAMP) and the upload-side equivalents. Two new public methods, MainPanel.run_dl_finish_command() and MainPanel.run_ul_finish_command(), each short-circuit when their feature flag is off and otherwise spawn a non-blocking ProcessBuilder.inheritIO().start() exactly the same way the legacy 509 path has been doing for years. The actual argv-resolution body (single-arg if the configured value is a real file, whitespace-token split otherwise) lives in a single shared _spawnExternalProcess(...) helper used by all three call sites so the three commands stay byte-for-byte identical in execution semantics.

  • Where the trigger lives. TransferenceManager gains a virtual hook onAllTransferencesFinished() (default no-op) and a new latch _finish_command_fired separate from the existing _all_finished / _tray_icon_finish latches. _genStatus() fires the hook exactly once per "non-empty → empty with at least one OK finish" transition, regardless of main-window visibility (the existing tray-notification block still requires !isVisible() and is left untouched). The latch resets together with _all_finished whenever a new transference enters the pipeline, so each fresh batch retriggers the command on completion. DownloadManager and UploadManager override the hook to call their respective MainPanel.run_*_finish_command().

  • Why a separate latch and timestamp. Sharing LAST_EXTERNAL_COMMAND_TIMESTAMP (or even a single new finish timestamp) with both queues would mean a download finish landing inside the cooldown window of an upload finish silently skips its command, which is the opposite of what #774 asks for. Each command gets its own cooldown so all three (509, downloads-finish, uploads-finish) can fire on the same tick if the timing works out. The separate _finish_command_fired latch (one per TransferenceManager subclass, since each side has its own instance) keeps the natural "queue must transition from non-empty to empty" gate intact without coupling to the visibility-aware tray notification.

  • Why gate on _isOKFinishedInQueue(). A queue that emptied because the user cancelled everything, or because every transfer failed with an error, almost certainly does not have files for the post-processing script to act on. Silently no-firing on all-error / all-cancelled is the safer default than running the script against an empty result set. If you want the script to fire regardless, leaving at least one successful transfer in the finished queue before clearing it satisfies the gate.

Settings dialog (Advanced tab)

  • Two new checkbox-and-textbox rows appear directly below the existing "Execute this command when MEGA download limit is reached:" row:
    • Execute this command when ALL downloads finish:
    • Execute this command when ALL uploads finish:
  • Each row has its own Test button that spawns the configured command immediately, identical to the 509 row's Test. The textbox starts disabled and is enabled by toggling the checkbox above.
  • Persistence: four new SQLite settings rows (run_command_dl_finish, run_command_dl_finish_path, run_command_ul_finish, run_command_ul_finish_path) loaded in the dialog's constructor and saved by the Save handler alongside the existing run_command / run_command_path keys. No migration is needed; missing rows default to "feature disabled".

i18n

  • New keys execute_this_command_when_all_downloads_finish and execute_this_command_when_all_uploads_finish added to all 8 bundles. EN carries the canonical English literal, ES has the Spanish translation, and IT/DE/HU/TR/VI/ZH carry the English placeholder pending translator pickup via the #397 contributor thread. LabelTranslatorSingleton's inverse-index lookup resolves the literal in SettingsDialog to the active locale's key at component-walk time, so the new checkboxes render localised wherever the bundle has a real translation and fall through to English otherwise.

Originally shipped in v8.53 (withdrawn)

#773 — Reuse the 1 MiB in-memory buffer in ChunkDownloaderMono

  • What changed. The mono (single-slot) chunk loop used to do new byte[(int) chunk_size] inside the chunk loop, throwing away ~1 MiB of byte[] per chunk. The buffer is now hoisted to a worker-level local and grown on demand; after the warmup ramp (128 KiB, 256 KiB, ..., 896 KiB, then 1 MiB steady state) the array stops growing and every subsequent chunk reuses it.
  • Why it matters. A stable mono download processes ~60 chunks/min, so the old code translated to ~60 MB/min of pure allocation pressure per active mono download. G1 recycles young-gen fast enough that the live set stayed small, but the constant churn kept the heap commit elevated and made auto-shrink slow. The new path is exactly one allocation per worker for the entire transfer.
  • Atomic-commit semantics preserved. The reason mono buffers in memory at all is the comment in ChunkDownloaderMono.java:256-266: writing directly to the sequential output stream while reading would let a mid-chunk network failure shift every subsequent chunk and corrupt the file. The cipher stream still fills the buffer fully before output_stream.write() runs, and the MAX_CHUNK_BUFFER_BYTES guard above the allocation still applies. Only the allocation strategy changed; the safety invariant did not.

Rolled in from the unreleased 8.52 development cut

  • #773 — click-to-release on the JVM-RAM status label + auto-GC hint at 70 %. The FORCE_GARBAGE_COLLECTION_MAX_MEMORY_PERCENT = 0.7 constant had been declared in MainPanel for a long time but never read. The 2 s memory-monitor loop now requests a GC when live usage crosses 70 % of -Xmx, throttled to once per 60 s so it cannot pin a core during heavy chunk decryption. Left-clicking the JVM-RAM used: ... label fires the same hint on demand; hover for a tooltip explaining the behaviour and suggesting -Xmx for a hard cap. The G1 collector that ships as default since Java 9 honours System.gc() and, after a couple of cycles, returns free regions to the OS — which is the user-visible reduction in Task Manager that #773 was about.

Recommended launch flag for users on systems with lots of RAM

Java's default -Xmx is roughly 1/4 of physical RAM on Java 9+. On a 16 GB box the JVM happily commits up to ~4 GB even if the live working set is small. If you want a hard ceiling, launch with:

java -Xmx512m -jar MegaBasterd_8.55.jar

512 MB is plenty for everyday use; bump to -Xmx1g only if you see OutOfMemoryError in the DEBUG LOG tab under heavy parallel downloads.

Don't miss a new megabasterd release

NewReleases is sending notifications on new releases.