Release 2.0.9
What's New
- Bug fixes
- Controller Advertise Address Updates - Changing an HA controller's
advertiseAddressand restarting now updates the address the cluster and routers use - Edge Router Tunneler Policy Enforcement - Removing a tunneler's dial access now closes its live circuits, and the controller verifies the dial policy for tunneler circuits. Read the posture check warning.
- Removal of the edge router API proxy (see Removed Features below)
- Router Data Model Subscriber Metrics - New router metrics for diagnosing router data model performance issues
- Security Advisories - Two security advisories
Security Advisories
This release addresses two security advisories. See the linked GitHub Security Advisories for full details,
impact, and affected versions.
- GHSA-5m4x-r6jg-v3wq (High) - Router enrollment and
certificate renewal copied the URI SANs a router's CSRs requested into the signed certificates, SPIFFE IDs
included. A router could obtain a certificate naming a controller, which an HA cluster whose edge signer
shares the controllers' root CA accepts as a cluster peer, or naming another identity's API session, which
satisfies the certificate binding of that session's access tokens. Router CSRs that request a SPIFFE ID are
now rejected withROUTER_CSR_SPIFFE_ID_NOT_ALLOWED. A router whose configuration lists a SPIFFE URI under
csr.sans.urican't enroll or renew until that entry is removed. - GHSA-ccq6-vc5v-4ghh (Low) - The controller and the router
wrote complete service session tokens into structured log fields during edge control operations, several of
them at Error or Info, so the values appeared in logs at default verbosity during ordinary operation. A
service session token does not authenticate a client on its own, because the controller validates it only
against a matching api session token, but logging it placed one half of a credential pair into logging
infrastructure, which carries its own access controls, retention, backups and exports. These log events now
carry the session id, which is not a credential.
Router Data Model Subscriber Metrics
Routers now report metrics for processing router data model subscriber updates, to help diagnose
router data model performance issues.
rdm.subscriber.event- a timer of processing one queued subscriber eventrdm.subscriber.event.queue_time- a timer of how long a subscriber event waited in the queuerdm.subscriber.event.queue_size- a gauge of the number of subscriber events waitingrdm.subscriber.scan- a timer of a scan of identities marked for a check while the event queue was fullrdm.subscriber.scan.queue_time- a timer of how long a scan request waitedrdm.subscriber.scan.identities- a histogram of the number of identities each scan processedrdm.subscriber.full_sync- a timer of a check of every subscription, as done after the router data model is replaced
Controller Advertise Address Updates
In an HA cluster, a controller's ctrl.options.advertiseAddress was only read when the cluster was
initialized or the node joined. After that the address lived in the raft configuration, so changing it in
the config file did nothing: other controllers kept dialing the old address, and routers and new enrollment
JWTs kept getting it. The usual way into this is putting a proxy in front of the ctrl listener, on a
different port, after the cluster already exists. A single-node cluster had no way out short of
re-bootstrapping.
The config file is now the source of truth. On startup, if a controller's configured advertise address
differs from the one stored in the cluster, it asks the leader to update it, retrying until the change
lands. The member keeps its voter or non-voter status, and connected routers get the new address through
the usual cluster membership update.
The leader refuses the change if another member already has the address, or if a different controller
answers there. If the leader can't reach the new address at all, it makes the change anyway and logs a
warning, since some controllers only dial out.
Running ziti agent cluster add or ziti fabric cluster add for an existing member at a new address also
updates its address in place, instead of removing and re-adding it.
The leader must be running this version for the update to happen. An older leader ignores the request and
the stored address stays as it is. The requesting controller logs a warning and keeps retrying, so the
change goes through once the leader is upgraded.
Edge Router Tunneler Policy Enforcement
When an edge router tunneler (ER/T) intercepts a service and the router's identity later loses dial
access to that service, through a service policy change, an identity being disabled, or the identity
being deleted, the router now closes the circuits it established for that service. Previously only new
intercepts were stopped, and existing connections kept flowing until the application or the router
ended them. The same applies to circuits created through the router-embedded SDK. See
#4330.
Three related gaps are closed at the same time:
- The router's pre-dial policy check for tunneler circuits was miswritten and never denied a dial.
- The controller did not verify the dial service policy when a router requested a tunneler circuit;
it verified only edge router and service edge router policy. It now rejects the request when no dial
policy grants the router's identity access to the service. - Adding a posture check to a service policy did not re-check existing circuits or hosted terminators
when the identity already had that check through another policy. This affected SDK clients using
OIDC sessions, whose posture the router enforces directly. The router now re-checks them whenever
a policy granting the service changes.
Warning: posture checks and edge router tunnelers
Edge routers do not submit posture data. Because the tunneler's dial check previously never denied,
a service policy that carried posture checks and granted access to a router identity, whether
directly or through #all, let the router dial the service as though the checks had passed.
That is no longer the case. The router now evaluates the dial policy the same way it does for any
other identity, and with no posture data every posture check fails. After upgrading, a router
granted dial access only through a policy with posture checks will be denied, and connections that
were flowing through such a policy before the upgrade will not re-establish. Before upgrading, find
any service policies that both carry posture checks and include router identities, and split the
routers out into their own policy without posture checks. ziti edge list service-policies shows the
Identity Roles and Posture Check Roles of each policy; #all in Identity Roles includes every router
identity.
Removed Features
- Edge router API proxy - The
edge.apiProxyrouter option, which served the
controller's Edge API from a router listener, has been removed. Certificate
authentication through the proxy did not work, and the proxy doesn't fit the
planned end-to-end encryption and certificate proof of possession work.
Routers log a warning and ignore the section if it is still present in config.
Clients that used the proxy should connect to the controller directly, or
through a load balancer that passes TLS through.
Component Updates and Bug Fixes
-
github.com/openziti/channel/v4: v4.3.13 -> v4.3.15
- Issue #317 - [Backport-4.x] Hellos with a value other than 4 bytes under the ReplyFor header key are rejected
- Issue #313 - [Backport-v4] A queued message can be discarded unwritten after Send reports success
-
github.com/openziti/foundation/v2: v2.0.104 -> v2.0.105
-
github.com/openziti/identity: v1.0.143 -> v1.0.144
-
github.com/openziti/runzmd: v1.0.96 -> v1.0.97
-
github.com/openziti/sdk-golang: v1.8.2 -> v1.8.3
- Issue #1049 - [Backport-1.8] Write-failed payload is written to the peer as empty data, failing encrypted reads with invalid input
- Issue #1037 - [Backport-1.8] Closing an SDK-hosted xgress conn before its circuit starts blocks for the circuit start timeout
- Issue #1043 - [Backport-1.8] Cost and precedence updates on a hosted service are lost when its listener rebinds
- Issue #1032 - [Backport-1.8] Service refresh retry panics in rand.Int63n when RefreshInterval is 10s or less
- Issue #1056 - [Backport-1.8] Xgress conn closed locally ignores the router's StateClosed and retransmits forever
-
github.com/openziti/transport/v2: v2.0.225 -> v2.0.228
-
github.com/openziti/xweb/v3: v3.0.5 -> v3.0.6
-
github.com/openziti/ziti/v2: v2.0.8 -> v2.0.9
- Issue #4375 - router on release-v2.0.x never seeds MFA posture from the api session token
- Issue #4694 - [Backport-2.0] Router does not relay an SDK xgress terminator's half-close to tunneler, proxy, or embedded SDK initiators
- Issue #4707 - [Backport-2.0] Version built-in config types independently of the database version
- Issue #4768 - [Backport-2.0] Controller RouterDataModel rebuild (BuildAll) drops all existing revocations
- Issue #4715 - [Backport-2.0] Identity PUT re-enables a disabled identity
- Issue #4717 - [Backport-2.0] Terminator PUT clears hostId, peerData and sourceCtrl
- Issue #4719 - [Backport-2.0] Fabric router PUT crashes the controller
- Issue #4657 - [Backport-2.0] Router doesn't tell SDKs to stop sending on circuits it no longer has
- Issue #4755 - [Backport-2.0] Routers drop controllers on a stale member list from a leader restarted after a snapshot install
- Issue #4729 - [Backport-2.0] API request timeouts surface as TLS 'bad record MAC' errors instead of a 503
- Issue #4479 - [Backport-2.0] Router never reconnects to a standalone controller after the heartbeat check closes the control channel
- Issue #4574 - [Backport-2.0] Router can drop controllers on a follower's stale controller list and never re-add them
- Issue #4735 - [Backport-2.0] JWKS peer signer kid is undecodable raw bytes (should be hex) -- %s vs %x on sha1.Sum in oidc_auth/storage.go KeySet()
- Issue #4389 - [Backport-2.0] Router applies posture responses out of order and without bound
- Issue #4392 - Router posture access checks read posture data without synchronization
- Issue #4390 - [Backport-2.0] Circuit endpoint faults are sent inline from arbitrary goroutines and lost on failure
- Issue #4585 - [Backport-2.0] Terminator cost, precedence and peer data don't converge at the controller after a takeover or update
- Issue #4599 - [Backport-2.0] ER/T terminator re-create reverts health check cost and precedence
- Issue #4594 - [Backport-2.0] CLI id-or-name resolution is duplicated, and the fabric version misreads names starting with 'id'
- Issue #4556 - [Backport-2.0] transport_udp and proxy_udp listeners can deadlock their own event loop
- Issue #4597 - [Backport-2.0] Compilation fails on Illumos
- Issue #4605 - [Backport-2.0] ziti tunnel host mode: contextProvider.accept dials the upstream and completes the accept inline, so one slow connection blocks every later dial to that service
- Issue #4563 - [Backport-2.0] Identity admin protection is decided from the serving controller's view before dispatch
- Issue #4564 - [Backport-2.0] Granular permissions are granted to a session before MFA completes
- Issue #4565 - [Backport-2.0] The default admin's isAdmin flag can be cleared
- Issue #4487 - [Backport-2.0] Sync a controller's raft advertise address with its config on startup
- Issue #4533 - [Backport-2.0] Validate commands on the raft leader before they enter the log, and halt on an entry a member cannot decode
- Issue #4540 - [Backport-2.0] Forwarded API errors lose their structured cause, and duplicate-name or referenced-delete errors return 500 from a follower
- Issue #4550 - [Backport-2.0] Identity creation with authenticators and entity renames swallow their errors
- Issue #4664 - [Backport-2.0] Add metrics for the router data model subscriber goroutine
- Issue #4490 - [Backport-2.0] Router panics sending a link state update to a controller it has unregistered
- Issue #4505 - [Backport-2.0] Router pins a CPU core reporting metrics while no controller is registered
- Issue #4525 - [Backport-2.0] Allow case-insensitive terminator addressing via host.v1
- Issue #4647 - [Backport-2.0] Remove the edge router API proxy
- Issue #4407 - [Backport-2.0] Router data model root CA public keys carry different intermediates on each controller
- Issue #4612 - [Backport-2.0] Controller deadlocks when posture data changes for an identity with legacy API sessions
- Issue #4416 - Set clientAuth and serverAuth extended key usages on identity client certificates
- Issue #4332 - [Backport-2.0] ER/T does not close existing circuits when the router loses dial access to a service
- Issue #4698 - [Backport-2.0] Router xgress Close can block indefinitely on a congested circuit