Release 2.1.0
What's New
- Connect-V2: Sessionless SDK Dial - SDKs can dial services without a per-service controller session; the router authorizes the dial locally against the Router Data Model
- ZAC Bootstrapping CLI - CLI commands to download, configure, and serve the Ziti Admin Console without hand-editing YAML
- Verify Traffic with ext-jwt-signers -
ziti ops verify trafficcan authenticate with an ext-jwt-signer (OIDC) to test the certless ephemeral-cert path - Cluster Quorum Recovery - A mechanism for recovering clusters that have irrevocably lost the ability to form a quorum
- Quickstart Cluster -
ziti run quickstart clusterbrings up a multi-node HA cluster in a single command for testing and development and learning - Fully Connected Controller Mesh - Controllers now proactively keep the cluster mesh fully connected
- Config Type Target Field - Config types now have a target field indicating whether they apply to services, routers or other entities
- Wildcard OIDC Issuers - Controllers with a wildcard server-certificate SAN can serve OIDC for explicitly allow-listed hostnames
- Router Configs - Allow routers to have a list of associated configs
- Multiple LAN Interfaces for tproxy -
lanIfnow accepts a single interface or a list of interfaces - Multiple Resolver Addresses for tproxy -
resolvernow accepts a single address or a list of addresses - DNS Upstream Query Modes - choose how multiple DNS upstreams are queried: parallel fan-out (default) or serial fail-through
- Logging Now Uses slog with an Async Handler - Logging moves to Go's
log/slogbehind an asynchronous sink; output is unchanged by default, with new flags to tune buffering - Security Advisories - Includes the two control-plane certificate validation fixes first released in 2.0.2
Security Advisories
This release includes the two control-plane certificate and identity validation fixes first released in
2.0.2 and 1.6.18. Anyone upgrading from 2.0.1 or earlier is picking them up here for the first time. See
the linked GitHub Security Advisories for full details, impact, and affected versions.
- GHSA-mrpr-756c-xm47 (Critical) - Improper peer certificate validation on the controller
cluster mesh, router links, and metrics endpoint. TLS peer checks accepted a connection when any presented
certificate chained to the trusted CA while taking the peer identity from the leaf certificate, allowing a
peer to be admitted under a forged identity without possessing a trusted key. On HA/clustered controllers
this allows joining the controller cluster as an arbitrary controller. - GHSA-cc5m-7mhm-xh9f (Medium) - Control-channel connections carrying a channel-type header
bypassed router certificate and identity verification, allowing an attacker that can reach the controller
control port to be admitted as an arbitrary router identity and manipulate that router's fabric terminators,
faults, and circuit routing. Impact is limited to router data model metadata (service and identity names) and
control-plane manipulation; it does not by itself grant access to the services the network protects.
Connect-V2: Sessionless SDK Dial
Ziti SDKs built on sdk-golang v2 can now dial a service without first obtaining a
per-service session token from the controller. With Connect-V2 the router authorizes the
dial locally against its copy of the Router Data Model, the same policy and posture data
the controller already distributes. Dropping the per-dial session round-trip to the
controller lowers dial latency and controller load, most noticeably for identities that
dial many services.
There is nothing to configure. Connect-V2 is negotiated automatically: a router advertises
the capability, an SDK that supports it uses the new path, and older SDKs transparently
continue to use the session-based dial. Policy and posture checks are still enforced on
every dial; they simply run against the Router Data Model on the router instead of during
session creation on the controller.
Connect-V2 requires both a router and an SDK from this release (or newer). Mixed networks
are fine: a dial uses Connect-V2 only when both ends support it, and otherwise falls back
to the session-based path.
ZAC Bootstrapping CLI
Three new commands make it easy to get the Ziti Admin Console running without manual file editing.
Download ZAC:
ziti ops console download [version] --location <dir>
Downloads a ZAC release from GitHub and extracts it into the given directory. version defaults to
latest. The target directory must be empty or not yet exist; the command never removes or overwrites
existing files.
Configure a controller to serve it:
ziti ops console configure <controller-config.yml> --all --location <dir>
Edits the controller config YAML in place, adding or updating the spa web-listener binding so the
controller serves ZAC from the given directory. Selects listeners with --all or --name; prompts
interactively when neither is given. The file's comments and structure are preserved.
Serve it locally:
If you prefer to serve the console locally rather than have a controller host it, you can serve ZAC
straight from the ziti CLI:
ziti run console --location <dir>
ziti run console --version latest
Serves the ZAC SPA over HTTPS on 127.0.0.1:8443. Generates a self-signed cert automatically if none
is supplied. The browser points ZAC at whatever controller you choose inside the console itself.
To serve with your own certificate on a specific address and port:
ziti run console --location <dir> --bind-address 0.0.0.0 --port 9443 \
--tls-cert ./server.pem --tls-key ./server.key
Bootstrap it with the quickstart:
ziti run quickstart --zac
Adding --zac to the quickstart downloads ZAC and configures the controller to serve it in one step,
so the console is available at https://<ctrl-address>:<ctrl-port>/zac as soon as the network is up.
The assets install under <home>/console by default (override with --zac-location), and
--zac-version selects the release (defaults to latest). Re-running against an existing --home
reconciles the binding, so adding --zac on a later run enables the console on a network that was first
brought up without it. ziti run quickstart cluster --zac installs the assets once and serves the
console from every node.
Two related quickstart re-run fixes ship alongside this:
- Re-running against an already-initialized
--homenow adopts the controller and router ports/addresses
the environment was first created with, instead of falling back to the flag defaults. Previously a
re-run that omitted--ctrl-portwould wait on the default port while the controller listened on the
original one. - Re-running against an existing
--homewith a flag that only takes effect when the environment is
first created (for example--ctrl-port) now reports that the flag was ignored, rather than silently
dropping it.
Verify Traffic with ext-jwt-signers
ziti ops verify traffic can now authenticate with an ext-jwt-signer (OIDC), so the certless
ephemeral-certificate path can be traffic-tested.
--ext-jwt-signer <name>selects the signer to authenticate with and generate a cert from.--ext-jwt-redirect-urlsets the OIDC redirect URL (defaulthttp://localhost:20314/auth/callback).
Cluster Quorum Recovery
A new offline CLI command, ziti ops cluster recover <controller-config>, lets
operators rebuild a stuck HA controller cluster after losing quorum. Use it when
enough controllers are permanently gone that ziti ops cluster add and
ziti ops cluster remove fail with "no leader" — for example, a 2-node cluster
where one node is unrecoverable, or a 3-node cluster that lost two nodes at once.
The command must be run while the surviving controller process is stopped. It
reads the same controller config the controller would, opens the raft data
directory, calls raft.RecoverCluster to force the configuration down to a
single local node, and aligns the FSM-tracked member list and snapshot data so
no stale peers leak through on restart. After it succeeds, restart the
controller and add new peers normally with ziti ops cluster add.
End-to-End Encryption (e2ee) Improvements
- Add support for negotiating e2ee scheme during Dial/Accept handshake
- Allow hosting-side crypto material to be generated on per connection basis (instead of per terminator)
Quickstart Cluster
ziti run quickstart cluster stands up a multi-node HA controller cluster with a single command, for
testing, learning, and local development. It launches one quickstart child process per node (default 3,
minimum 3, configurable with --size), initializes the first
node, joins the rest, and prints a banner once the whole cluster is online listing each node's controller
address, router address, process id, and per-node log file.
Each node runs as its own operating-system process, so you can stop, restart, or attach a debugger to any
single node to explore HA behavior without disturbing the others. The banner prints the exact
ziti run quickstart command needed to start each node by hand.
Lifecycle mirrors the single-node quickstart. Pass --home for a persistent cluster you can stop and start
again: restarting against an existing --home rejoins the existing cluster rather than re-initializing it,
and the nodes start together to re-form a quorum. Omit --home to run from a temporary directory that is
removed on a clean shutdown. Pressing Ctrl-C stops every node.
Fully Connected Controller Mesh
In an HA cluster, controllers form a mesh of channel connections that raft uses to
communicate. Previously we followed the raft library's lead: connections were made as
needed to allow elections, and after that only the leader maintained connections to its
followers. That is enough for raft itself, but it means most nodes have no direct link to
most other nodes.
Now controllers keep the mesh fully connected. This gives better visibility into system
state from any node, not just the leader, and it provides a baseline for building
additional non-raft coordination features on top of the mesh.
Each controller runs a PeerDialer that proactively dials every known cluster member and
works to keep the mesh fully connected. Failed dials are retried with exponential backoff,
and when two controllers dial each other at the same time a deterministic tie-break (based
on their SPIFFE IDs) keeps a single connection rather than a redundant pair. The dialer's
state can be inspected with ziti fabric inspect ctrl-peer-dialer.
The dialer is tunable via a new optional cluster.dialer config section. All values have
defaults, so no configuration change is required:
cluster:
dialer:
minRetryInterval: 1s # minimum time between dial retries
maxRetryInterval: 1m # maximum time between dial retries
retryBackoffFactor: 2.0 # multiplier applied to the retry interval after each failure
fastFailureWindow: 30s # if a connection is lost within this window, apply backoff instead of resetting the retry delay
dialTimeout: 10s # maximum time a single dial attempt may run
scanInterval: 30s # period of the full scan that reconciles dial state against membership
queueCheckInterval: 5s # how often expired entries are popped from the retry queueA related cluster.nonMemberGrace setting (default 1m) controls how long a leader will
let a TLS-valid but non-member controller stay connected to the mesh before dropping it.
This gives a controller that is being added to the cluster time to be accepted as a member
before its connection is reaped.
Config Type Target Field
Config types now have an optional target field that indicates what kind of entity the config type
is intended for. Valid values are "service", "router", and "other". The field is set on creation
and is immutable afterward.
This is the first step toward controller-managed router configuration. The target field lets us
distinguish between config types meant for services, config types meant for routers, and config types
meant for other purposes, which keeps UIs, APIs, and validation clean. See
doc/design/ctrl-managed-router-config.md for the full design.
A database migration sets target = "service" on all existing config types. Services and identity
service config overrides now require that referenced configs have a config type with
target = "service".
The CLI has been updated to support the new field:
ziti edge create config-typenow accepts a--targetflagziti edge list config-typesnow shows aTargetcolumn
Wildcard OIDC Issuers
Controllers can serve OIDC for hostnames covered by a wildcard server-certificate SAN. Prior to 2.1,
wildcard SANs were excluded from the set of valid OIDC issuers, so /oidc/* requests to a
wildcard-covered hostname returned 404.
Wildcard SANs cannot be used as a literal OIDC issuer (https://*.example.com/oidc is not a usable URL).
Instead, the edge-oidc API binding now accepts an allowedHostnames option listing the exact hostnames
(covered by the wildcard) that may be served as issuers. If omitted, the wildcard contributes no issuers
and the controller logs a warning at startup; a malformed entry (a non-string value, or one containing a
wildcard character) is a startup error:
web:
- name: client-management
apis:
- binding: edge-oidc
options:
allowedHostnames:
- ctrl.example.comEach entry must be an exact hostname (no patterns) that an active server-certificate SAN actually covers;
entries are matched against wildcard SANs using standard X.509 hostname rules. The resulting OIDC issuers
are therefore concrete, fixed hostnames, so the set of valid iss values stays closed. Concrete
(non-wildcard) SANs continue to be served as issuers automatically and do not need to be listed.
Router Configs
Routers (edge, transit, and fabric) now have a configs field that holds a list of config IDs the
router should use. This is the second step toward controller-managed router configuration: routers
can now be associated with configs in the same way services already can.
Validation rules:
- Every config referenced by a router must use a config type with
target = "router". Configs
withtarget = "service"(or anything else) are rejected. - A router may reference at most one config per config type. Attempting to attach two configs of
the same type is rejected with a duplicate-config error naming both configs. - Deleting a config automatically removes it from the
configslist of any router that referenced
it, so dangling references are not possible.
The configs field is exposed on router create, update, patch, and detail responses across the
edge, transit, and fabric router REST APIs.
The CLI has been updated to support the new field:
ziti edge create edge-routeraccepts--config <id>(repeatable)ziti edge create transit-routeraccepts--config <id>(repeatable)ziti edge update edge-routeraccepts--config <id>to replace the router's config listziti fabric create routeraccepts--config <id>(repeatable)ziti fabric update routeraccepts--config <id>to replace the router's config list
Status: in progress. The target field, the router configs field, and the built-in
router.link.v1 config type together establish the data model and distribution path:
config types can be targeted at routers, configs can be attached to routers, and the
controller distributes router-targeted configs to the affected routers through the Router
Data Model, where a router-side registry receives them. Routers do not yet consume these
configs to drive their runtime behavior, so controller-managed router configuration is not
ready for production use in this release. See doc/design/ctrl-managed-router-config.md
for the overall design.
Multiple LAN Interfaces for tproxy
The lanIf option in xgress_edge_tunnel tproxy configs now accepts either a
single string (as before) or a YAML list of interface names. For each intercepted
service address the router inserts one iptables ACCEPT rule per configured
interface. Existing single-interface configs are unchanged.
# single interface (unchanged)
- binding: tunnel
options:
mode: tproxy
lanIf: enp0s5
# multiple interfaces
- binding: tunnel
options:
mode: tproxy
lanIf:
- enp0s5
- enp0s6The CLI --lanIf flag also accepts a comma-separated list or repeated flags:
ziti tunnel tproxy --lanIf enp0s5,enp0s6
# or
ziti tunnel tproxy --lanIf enp0s5 --lanIf enp0s6Multiple Resolver Addresses for tproxy
The resolver option in xgress_edge_tunnel tproxy configs now accepts either a
single string (as before) or a YAML list of udp:// addresses. A single shared
resolver instance handles all listeners so hostname→IP mappings remain consistent
across interfaces. Existing single-address configs are unchanged.
# single address (unchanged)
- binding: tunnel
options:
mode: tproxy
resolver: udp://172.18.102.70:53
# multiple addresses
- binding: tunnel
options:
mode: tproxy
resolver:
- udp://172.18.102.70:53
- udp://192.168.10.1:53DNS Upstream Query Modes
When more than one DNS upstream is configured for tproxy host-side resolution, the
resolver previously always fanned out: every upstream was queried in parallel and
the first NOERROR response won. For a high-volume router with several upstreams this
multiplies outbound DNS traffic, since every request hits every upstream.
A new dnsUpstreamMode option (router config) / --dnsUpstreamMode flag
(ziti tunnel) controls how multiple upstreams are queried:
parallel(default) - query all upstreams at once; first NOERROR wins. Unchanged
behavior.serial- query upstreams one at a time in order, starting at the first. Fail
through to the next only on timeout, transport error, or a non-NOERROR response.failover- likeserial, but the upstream that last answered is reused as the
starting point, so a healthy upstream isn't re-probed from the top on every query.random- likeserial, but each query starts at a randomly chosen upstream.
In all serial modes a healthy upstream costs exactly one outbound query regardless of
how many upstreams are configured. The single-upstream case is unaffected.
In a router's xgress_edge_tunnel tproxy config:
- binding: tunnel
options:
mode: tproxy
dnsUpstreamMode: serial
dnsUpstream:
- udp://10.96.0.10:53
- tcp://8.8.8.8:53The standalone ziti tunnel gains a matching --dnsUpstreamMode flag (default
parallel):
ziti tunnel run --dnsUpstreamMode serial \
--dnsUpstream udp://10.96.0.10:53 \
--dnsUpstream tcp://8.8.8.8:53
Logging Now Uses slog with an Async Handler
The controller, router, and ziti tunnel now log through Go's standard
log/slog library behind a single asynchronous sink. By default the output is
unchanged - the same human-readable format as before - so no configuration
change is required. Existing pfxlog/logrus log statements continue to work;
they are bridged into the new sink rather than rewritten, so the migration to
slog is gradual and nothing is lost in the meantime.
The motivation is performance under load. Previously every log call contended on
a single process-wide formatter-plus-writer mutex, which could block many
goroutines at once. Writing is now handed to a background goroutine, off the hot
path of the code doing the logging.
Asynchronous writes and dropped records
Because writes are buffered through a bounded queue, behavior under saturation
is the one change worth knowing about:
- Records at or above a configurable block threshold (default
warn) block the
caller until there is room, so warnings, errors, and fatal messages are never
silently dropped. - Lower-priority records (
info,debug,trace) are dropped when the queue is
full instead of blocking. Whenever drops occur, a summary line is emitted
periodically reporting how many records were dropped per level, so the loss is
always visible in the logs.
Under normal load nothing is dropped; this only engages when log volume outruns
the writer.
New CLI flags
ziti controller run, ziti router run, and ziti tunnel gain three optional
flags to tune the async sink. All have sensible defaults, so leaving them unset
preserves current behavior:
--log-queue-size(default4096) - capacity of the async log queue.--log-block-threshold(defaultwarn) - lowest level that blocks rather than
dropping under saturation (panic|fatal|error|warn|info|debug|trace).--log-summary-interval(default5s) - how often the dropped-record summary
line is emitted.
The existing --log-formatter flag is unchanged: pfxlog (default), json
(unchanged JSON shape), and text (logrus-style key=value).
Per-channel log levels are slog-only
ziti agent set-channel-log-level <name> <level> now adjusts only code that has
been migrated to the new slog loggers. Call sites still using pfxlog.Logger()
or pfxlog.ChannelLogger(...) continue to follow the global level, so because
most call sites are not yet migrated, per-channel overrides have limited reach
today and expand as packages are converted. The global ziti agent set-log-level <level> still affects everything.
Component Updates and Bug Fixes
-
github.com/openziti/channel/v5: v4.3.11 -> v5.0.15
- Issue #258 - Add a hook to inject hello headers derived from the peer's certificate
- Issue #267 - NewSingleChannelWithUnderlay panics on underlays with nil headers (e.g. websocket)
- Issue #269 - Make channel logging pluggable via injectable slog.Logger
- Issue #264 - Multi-underlay group reconnect can reject-storm under load
- Issue #265 - Support reconfiguring heartbeat intervals on a running channel
- Issue #261 - Add ContentTypeReceiver, a self-describing receive handler
- Issue #250 - BackoffDialPolicy cannot (re)establish a grouped channel: never sets IsFirstGroupConnection
- Issue #252 - BackoffDialPolicy misclassifies multi-underlay constraint fill as short-lived/flapping
- Issue #247 - Config.Binder exposes unexported
*channelImplin its public signature - Issue #255 - Flaky Test_MultiUnderlayChannels: CloseRandom can close the last required underlay
- Issue #253 - Multi-underlay channel delays below-Min closure when a dial/backoff is in progress
- Issue #246 - classic_dialer leaks underlay FD when hello handshake fails
- Issue #241 - Allow calling LoadOptions on an Options instance
- Issue #238 - Channelv5
-
github.com/openziti/edge-api: v0.31.0 -> v0.35.2
- Issue #198 - Advertise edge router capabilities in the service edge-router list
-
github.com/openziti/foundation/v2: v2.0.91 -> v2.0.99
- Issue #489 - Add graceful shutdown and idle-wait support to goroutines.Pool
- Issue #488 - Add package-level Fatal and SyncEmit helpers to logging
- Issue #484 - Add slog logging core (foundation/v2/logging) for upstream libraries
-
github.com/openziti/identity: v1.0.129 -> v1.0.137
-
github.com/openziti/runzmd: v1.0.90 -> v1.0.91
-
github.com/openziti/sdk-golang/v2: v1.7.0 -> v2.0.0-pre3
- Issue #958 - ConnectV2 xgress client conn not marked closed on router-initiated teardown
- Issue #967 - Move RouterCapabilityConnectV2 constant into edge_client.proto
- Issue #951 - Add an SDK acceptance-test framework
- Issue #952 - xgress client half-close not delivered to legacy edge hosts
- Issue #936 - Implement Connect-V2: sessionless SDK dial
- Issue #945 - Migrate to channel/v5
- Issue #948 - AddControllerUrlsUpdateListener remover unsubscribes the wrong event
- Issue #941 - Prep for channel v5: explicit receive handler registration, drop send priorities
- Issue #924 - Make controller http timeout configurable, with a default of 30s
- Issue #925 - Switch controllers on a broader set of errors
- Issue #926 - Refresh OIDC token using a window to avoid race conditions and herding
- Issue #927 - Apply exponential backoff to auth retry attempts
- Issue #932 - API Session Certificate chain is not preserved
-
github.com/openziti/secretstream: v0.1.49 -> v0.1.51
-
github.com/openziti/transport/v2: v2.0.215 -> v2.0.216
-
github.com/openziti/xweb/v3: v3.0.4 -> v3.0.5
-
github.com/openziti/ziti/v2: v2.0.0 -> v2.1.0
- Issue #4149 - upgrading a running 1.x controller/router to 2.x fails to create the service user
- Issue #3990 - Expose service change subscriptions to external SDKs over protobuf
- Issue #4108 - Controller retains bbolt-managed memory past transaction (create-circuit response SIGSEGV)
- Issue #4067 - JWKS peer signer kid is undecodable raw bytes (should be hex) -- %s vs %x on sha1.Sum in oidc_auth/storage.go KeySet()
- Issue #4071 - Unify router capabilities into a single shared bitmask across control and edge channels
- Issue #3998 - Add router-side managed configuration registry
- Issue #4069 - Controller can cache empty apiAddresses forever due to startup race between raft mesh and xweb config load
- Issue #4066 - bootstrap zac for controller with quickstart
- Issue #4060 - support ext-jwt-signer in ziti verify traffic
- Issue #4036 - Own the metrics wire format (MetricsMessage) in ziti
- Issue #4137 - ziti tunnel ignores --dnsSvcIpRange
- Issue #4052 - ziti cli cached creds don't use refresh token
- Issue #3881 - Add Capability for DNSUPSTREAMS to be used in serial
- Issue #4035 - Controller /metrics endpoint produces duplicate TYPE declarations causing Prometheus to drop samples
- Issue #4045 - Reduce controller link/router management lock contention under high link churn
- Issue #3884 - Implement Connect-V2: sessionless SDK dial
- Issue #3929 - [Backport-2.0] Router does not enforce api-session or identity revocations on live connections
- Issue #3841 - Controller Cluster - new controllers must be able to be dialed by the leader in order to join successfully
- Issue #3933 - edge enrollment: add the list of controllers to successful enrollment response
- Issue #3992 - Overlay edge-oidc listener panics when its redirect_uri is not in the redirectURIs allow-list
- Issue #4039 - cryptic x509 error when cached CA is stale
- Issue #4010 - Non-admin identity with
enrollmententity permission can create enrollments for any identity including admins, achieving full privilege escalation to admin - Issue #4011 - Router deletes terminators ~12m after creation when host SDK replies with wrong inspect content type
- Issue #4002 - make quickstart easier - ZAC
- Issue #4007 - add cluster option to ziti run quickstart
- Issue #3983 - Migrate to channel/v5
- Issue #3916 - Convert router/forwarder to slog
- Issue #3976 - Distribute routers and router-target configs through the RDM
- Issue #3974 - Add router.link.v1 config type
- Issue #3994 - support additional addresses in deployments
- Issue #3934 - Consolidate the fabric and edge services data stores
- Issue #3910 - Install slog and route agent log-level callbacks through common/logging
- Issue #3927 - Router does not enforce api-session or identity revocations on live connections; revoked OIDC sessions keep dialing/hosting until access-token expiry
- Issue #3906 - Add named-logger registry, logrus bridge, and pfxlog-shape JSON
- Issue #3904 - Add slog AsyncHandler in preparation for moving to slog for logging
- Issue #3902 - Add agent IPC capability discovery and channel-based log-level commands
- Issue #3893 - Import openziti/agent library into common/agent
- Issue #3894 - Consolidate duplicated agent channel-upgrade code into common/agent
- Issue #3952 - externalIdClaim on CA returns HTTP 500 with empty body for most matcher/parser combinations
- Issue #3908 - Router posture-data updates don't revoke SDK-hosted xgress circuits or hosted terminators
- Issue #3780 - Add configs field to routers
- Issue #3961 - Router panics evaluating an AnyOf process posture check when client reports no process/OS data
- Issue #1593 - Expanded attribute query support in management API; add policy attribute support and usage count
- Issue #3949 - DeleteById swallows errors when firing change events
- Issue #3867 - Tunneler skips iptables rules for services sharing an intercept hostname
- Issue #3891 - oidc auth fails with wildcard server-cert SANs
- Issue #3914 - ziti login fails with oidc + wildcard certs
- Issue #3938 - Carry the link id in a link header instead of only in the channel identity token
- Issue #3945 - Increase certificate serial number namespace to 159 bits
- Issue #3942 - Prep for channel v5: bind handler invocation, send priorities
- Issue #3920 - ziti pki create csr ignores --key-name flag
- Issue #3744 - Add a target field to config type
- Issue #3684 - Keep controller mesh fully connected, as much as possible
- Issue #3864 - e2ee: allow hosting SDK to return e2ee public key in the dial response
- Issue #3849 - Add a recover mechanism for when a controller cluster can't form a quorum