mutagen-io/mutagen v0.10.0 on GitHub

Overview

This is large and exciting release with a number of new features, including network forwarding and session orchestration! The goal of this release is to enable developers to create remote containerized development environments that work with their local tools. For more information, check out the recently renovated project site at https://mutagen.io.

Please bear in mind that network forwarding and session orchestration are still a little experimental and subject to change, so please report any feedback or issues you encounter on the issue tracker or Spectrum chat.

Changes

This release includes the following changes from the v0.9.x series:

Added support for port forwarding between endpoints, including TCP and Unix domain socket forwarding (see below)
Added support for project-based session orchestration (see below)
Switched to YAML for configuration (see below)
- Legacy TOML synchronization configurations are still supported, but deprecated
Reorganized Mutagen command structure to account for expanded functionality
- Synchronization session management commands are still available as hidden commands directly under the root command in order to avoid breaking existing scripts and tools
Sessions can now be created in a paused state using the -p/--paused flag
Sessions can now have user-friendly names assigned using the -n/--name flag
Session specifications must now be either a session name or session identifier (fuzzy matching is no longer supported)
Added the ability to customize the Mutagen data directory location using the MUTAGEN_DATA_DIRECTORY environment variable, allowing for isolated daemon instances (or just simple relocation of the directory)
Modified session management commands to automatically start the daemon if not running
- This behavior can be disabled by setting the environment variable MUTAGEN_DISABLE_AUTOSTART=1
Added logging infrastructure
Switched the default scan mode to accelerated

Forwarding

With v0.10.0, Mutagen adds support for network forwarding sessions. These sessions function very similarly to Mutagen's synchronization sessions, with a few necessary differences.

Forwarding sessions are created using the mutagen forward create command:

mutagen forward create <source> <destination>

Unlike with synchronization, where endpoints are referred to as "alpha" and "beta", forwarding endpoints are referred to as "source" and "destination" since network forwarding is, by its nature, unidirectional.

Forwarding endpoint URLs are similar to synchronization endpoint URLs, except that they encode a network address instead of a filesystem path.

Type	Format
Local	`<network-endpoint>`
Docker	`docker://[<user>@]<container>:<network-endpoint>`
SSH	`[<user>@]<host>[:<port>]:<network-endpoint>`

Each <network-endpoint> has the form <protocol>:<address>. The <protocol> and <address> components allow the same values as the network and address arguments (respectively) of Go's net.Dial and net.Listen functions, except that <protocol> is restricted to tcp, tcp4, tcp6, and unix.

As with synchronization, each endpoint URL can be either local or remote, and they can be combined in arbitrary ways. Basic network forwarding might look like:

mutagen forward create tcp:localhost:8080 docker://appcontainer:tcp:localhost:8080

whereas reverse forwarding would look like:

mutagen forward create docker://appcontainer:tcp:localhost:8080 tcp:localhost:8080

You can even do remote-to-remote forwarding, e..g.:

mutagen forward create user@example.org:tcp:[::1]:8080 : docker://appcontainer:tcp:localhost:8080

With Unix domain socket forwarding, you can expose remote resources:

mutagen forward create unix:~/Projects/docker.sock user@host.example.org:/var/run/docker.sock

You can even mix-and-match protocols:

mutagen forward create unix:~/mysocket.sock user@example.org:tcp:localhost:9090

Forwarding sessions can then be managed in a very similar fashion to synchronization sessions with the following commands:

mutagen forward list
mutagen forward monitor
mutagen forward pause
mutagen forward resume
mutagen forward terminate

Mutagen's network forwarding is still new and experimental, so please provide feedback about your experience via the Spectrum chat or issue tracker.

YAML-based configuration

To help support Mutagen's growing functionality, configuration has been switched to use YAML instead. The legacy TOML-based configuration (such as ~/.mutagen.toml and files passed to mutagen sync create with the -c/--configuration-file flag) will still be supported, but support for new configuration options, including network forwarding options, will not be added to these files.

The Mutagen YAML format looks like:

sync:
  defaults:
    mode: ...
    maxEntryCount: ...
    maxStagingFileSize: ...
    probeMode: ...
    scanMode: ...
    stageMode: ...
    symlink:
      mode: ...
    watch:
      mode: ...
      pollingInterval: ...
    ignore:
      paths:
        - ...
        - ...
      vcs: ...
    permissions:
      defaultFileMode: ...
      defaultDirectoryMode: ...
      defaultOwner: ...
      defaultGroup: ...

forward:
  defaults:
    socket:
      overwriteMode: ...
      owner: ...
      group: ...
      permissionMode: ...

All fields optional and the syntax for field values is otherwise unchanged (except where YAML requires certain values to be quoted).

Mutagen now looks for its global configuration file in ~/.mutagen.yml (falling back to ~/.mutagen.toml, if it exists, for synchronization sessions) and the -c/--configuration-file flags have been updated to support this YAML-based format.

Orchestration

Building on its new YAML-based configuration, Mutagen is adding experimental support for synchronization and forwarding session orchestration via the mutagen project command. This lets you create project configurations defining the synchronization and forwarding sessions for your project and have Mutagen automatically manage those sessions. This is particularly powerful when combined with a tool like Docker Compose to set up sessions. This functionality is still in its infancy, so any and all feedback is welcome, though either the Spectrum chat or issue tracker.

In order to use this functionality, you need to create a mutagen.yml file that contains your session configuration(s). The format for this file is based off of the YAML layout defined above. In addition to a defaults key, you can specify named sessions with pre-defined endpoint URLs and configuration options. An example of this file can be found in the Mutagen repository, along with a corresponding Docker Compose configuration and underlying Dockerfile.

Using the orchestration infrastructure, your workflow might look something like:

# Start Docker Compose services
docker-compose up --build --force-recreate --detach
# Start Mutagen project sessions
mutagen project start

You can then work interactively in containers by starting a shell with a command like:

docker-compose exec <service-name> /bin/sh

Finally, to shut down the orchestration infrastructure, you can do:

# Terminate Mutagen project sessions
mutagen project terminate
# Terminate Docker Compose services
docker-compose down --rmi=all

Once you have that workflow set up, you can even move the entire development infrastructure off of your laptop using the DOCKER_HOST environment variable when invoking docker-compose up, mutagen project start, and docker-compose down (Mutagen will lock in DOCKER_HOST and other Docker environment variables and doesn't need them re-specified for each command).

The mutagen project command also provides other subcommands for managing projects, including:

mutagen project list (to list project sessions and their status)
mutagen project flush (to flush project synchronization sessions)
mutagen project pause (to pause project sessions)
mutagen project resume (to resume paused sessions or sessions that require authentication)
mutagen project terminate (to terminate project sessions)