Nukesor/pueue v4.0.0 on GitHub

[4.0.0] - 2025-03-09

This release aims to further improve Pueue and to rectify some old design decisions.
Large portions of both the library and the executables' code have been refactored and the protocol has been reworked, which completely breaks backwards compatibility.

Read the Change section for more details.

Removing internal channel communication

TLDR: Commands that start/stop/pause tasks now only return when the task is actually started/stopped/paused.
killing commands still takes a short while, as the process needs to be cleaned up properly.

Until recently, Pueue managed subprocess (task) states in a dedicated thread.
Client commands affecting subprocesses, such as pueue start --immediate, were relayed to this special thread via an mpsc channel for processing.

This setup caused short delays before the instructions were executed.
For instance, tasks would begin a few hundred milliseconds after the client received an Ok from the daemon, despite using the --immediate flag.
This behavior was unintuitive and often led to commands like pueue add --immediate install_something && pueue send 0 'y\n' failing, as the task had not started by the time pueue send was called.

The new state design resolves this issue by allowing Pueue to manipulate subprocess states directly within the client message handlers, eliminating any delays.

New editing

Task editing was a bit tedious until recently.
One could only edit a single task at a time and you had to specify which properties you wanted to add.
Each property was then opened in a new $EDITOR session, which meant that users had to open and close editors up to four times to edit a single task.

After a lot of consideration, tasks editing has been redesigned, provoding two ways that allow simple and convenient editing of multiple tasks at once.

File based approach

For this, a temporary directory is created for every task to edit and a new file for every property, resulting in the following structure:

   📁 0/
   │ * command
   │ * label
   │ * path
   └ * priority
   📁 1/
   │ * command
   │ * label
   │ * path
   └ * priority

You can then just navigate the resulting file structure and edit the properties you want in the editor of your choice.
This solution is obviously flawed if your editor doesn't provide an in-editor file tree like helix.

This mode is enabled by setting client.edit_mode to files.

Toml based approach

In this mode, all tasks to be edited are serialized into a single toml file that's then opened in the editor of your choice.

[1]
id = 1
command = "ls"
path = "/tmp"
priority = 0

This mode is very convenient for editing simple tasks, but has the drawback that users need to know the TOML format and have to make sure to not mess up character escaping.
For example, multi-line commands or commands that contain special/reserved characters in TOML will need special care.

Based on some feedback, this mode is apparently the most asked for, so this is enabled by default.
It can also be explicitly set via client.edit_mode: "toml".

Runtime invariants

TLDR: A new task state representation has been introduced, which is significantly cleaner and fixes some bugs.
However, it breaks compatibility with old states, so ensure there are no important tasks in your queue before updating. You'll also need to recreate groups.

Previously, various task-state related invariants were manually enforced during runtime. For example, a Queued task should not have a start or enqueued_at time set.
Turns out, doing this manually is highly error-prone, as it is difficult to account for every state transition and ensure everything is set or cleaned up correctly.

Fortunately, this issue can be addressed in a more elegant way by using Rust's struct enums. This method enforces invariants via the type system at compile time.
Although the affected code became slightly more verbose (about 25% larger), it eliminated an entire class of bugs.
During this refactoring, I discovered at least two instances where I had forgotten to clear a variable, leading to inconsistent state.

Since the new structure differs significantly from the old one, it completely breaks backward compatibility.
Upon updating Pueue and restarting the daemon, the previous state will be wiped, resulting in a clean slate.

Change

Breaking: Refactor internal task state. Some task variables have been moved into the TaskStatus enum, which now enforces various invariants during compile time via the type system.
Due to this, several subtle time related inconsistencies (task start/stop/enqueue times) have been fixed. #556
Important: This completely breaks pre v4.0 states.
Important: The Pueue daemon needs to be restarted and the state will be wiped clean.
Breaking: Streamlined pueue log parameters to behave the same way as start, pause or kill. #509
Breaking: Remove the --children commandline flags, that have been deprecated and no longer serve any function since v3.0.0.
Send log output to stderr instead of stdout #562.
Change default log level from error to warning #562.
Bumped MSRV to 1.85 and Rust edition to 2024.
Breaking: Redesigned task editing process #553.
Pueue now allows editing all properties a task in one editor session. There're two modes to do so: toml and files, which is configurable via the client.edit_mode settings field.
Revisited, fixed and cleaned up CLI help texts.
Print most of Pueue's info/log messages to stderr. Only keep useful stuff like json and task log output on stdout.
Breaking: Allow --immediate flag instead of --start--immediately on pueue restart for consistency with pueue add.
Breaking: Ported from anyhow to color_eyre for prettier log output.
Breaking: Switch cbor handling library, breaking backwards-compatible communication on a data format level.
Breaking: Switch protocol message representation, completely breaking backwards compatibility.

Add

Add support for NetBSD.
Add --all and --group to pueue log. #509
Add --all and --group to pueue enqueue. #558
Add --all and --group to pueue stash. #558
Add pueue reset --groups [group_names] to allow resetting individual groups. #482
This also refactors the way resets are done internally, resulting in a cleaner code architecture.
Ability to set the Unix socket permissions through the new unix_socket_permissions configuration option. #544
Add command filter to pueue status. #524 #560
Allow pueue status to order tasks by enqueue_at. #554
Added Windows service on Windows to allow a true daemon experience. #344 #567
Add queued_count and stashed_count to callback template variables. This allows users to fire callbacks when whole groups are finished. #578
Add new subcommand to set or unset environment variables for tasks. #503
Add add --follow flag that may be called in combination with --immediate #592
Add option to save the state in compressed form. This can be toggled with the daemon.compress_state_file config file.
Preliminary testing shows significant compression ratios (up to x15), which helps with large states in embedded and I/O bound environments.
On my local machine with a state of 400 tasks, state file size shrinks from ~2MB to ~120KB and save time increases from ~8ms to ~20ms.
Due to the very repetitive nature of the state's data (mostly environment variables), the gzip compression algorithm with the flate2 implementation has been chosen.
It shows similar compression rations to zstd on level 7, which is more than enough and the dependency is significantly lighter than zstd.
snappy, which is already a dependency, has also been considered, but it has much worse compression ratios (~2MB -> ~300KB).

Fixed

Fixed delay after sending process related commands from client. #548
Callback templating arguments were html escaped by accident. #564
Print incompatible version warning info as a log message instead of plain stdout input, which broke json outputs #562.
Fixed -d daemon mode on Windows. #344
Fixed a pueued crash when malformed secret exchange messages are sent by a connecting client #619.

Remove

status-format subcommand. This was a temporary workaround until the status query feature was finished, which is now the case.