Percona Monitoring and Management 3.9.1

Release date: 19 August, 2026

Percona Monitoring and Management (PMM) is an open source database monitoring, management, and observability solution for MySQL, PostgreSQL, MongoDB, Valkey and Redis. PMM empowers you to:

• monitor the health and performance of your database systems
• identify patterns and trends in database behavior
• diagnose and resolve issues faster with actionable insights
• manage databases across on-premises, cloud, and hybrid environments

Release summary

PMM 3.9.1 is a security-focused release that fixes a high-severity vulnerability in the Grafana ClickHouse data source, which connected using a privileged account. The fix replaces it with a dedicated, read-only user. We strongly recommend upgrading as soon as possible.

This release also fixes an encryption key rotation bug that corrupted stored credentials, and an issue where PMM Client stayed disconnected after a network interruption.

🔒 Security updates

High-severity vulnerability fixed: upgrade now

A high-severity vulnerability has been identified in the Grafana ClickHouse data source that enables arbitrary SQL execution and potential exposure of internal databases and cloud credentials.

Immediate actions required

Your system may be exposed to unauthorized database access and credential theft. Take the following steps immediately to secure your infrastructure:

1. UPGRADE IMMEDIATELY to PMM 3.9.1 (strongly recommended).
2. CREATE THE READ-ONLY USER FIRST if you use an external ClickHouse instance: set PMM_CLICKHOUSE_DATASOURCE_USER and PMM_CLICKHOUSE_DATASOURCE_PASSWORD before upgrading, or the data source fails to authenticate. See Restrict the ClickHouse data source to a read-only user.
3. AUDIT ACCESS LOGS for potential unauthorized access if your PMM instance is publicly accessible, and consider rotating credentials for connected services.

Vulnerability details

This vulnerability stems from the Grafana ClickHouse data source connecting with a privileged account that enables:

• arbitrary SQL execution by any signed-in user, including those with the lowest-privilege Viewer role
• access to internal databases beyond Query Analytics data
• outbound HTTP requests from ClickHouse, exposing AWS instance metadata and cloud credentials on cloud deployments
• potential takeover of the PMM administrator account

Affected installations

All PMM deployments running version 3.9 and earlier are affected, regardless of deployment method (Docker, Podman, Helm, or AMI).

Mitigation options

PREFERRED: Upgrade to PMM 3.9.1

This release replaces the privileged ClickHouse data source account with a dedicated, read-only user.

To secure your system:

1. If you use an external ClickHouse instance, [create the read-only user](https://docs.percona.com/percona-monitoring-and-management/3/reference/third-party/clickhouse.html#restrict-the-clickhouse-data-source-to-a-read-only-user) and set `PMM_CLICKHOUSE_DATASOURCE_USER` and `PMM_CLICKHOUSE_DATASOURCE_PASSWORD`.
2. [Upgrade to PMM 3.9.1](https://docs.percona.com/percona-monitoring-and-management/3/pmm-upgrade/index.html).
3. Verify that Query Analytics dashboards load correctly after the upgrade.

TEMPORARY: If you cannot upgrade immediately

If an upgrade is not immediately possible, run the mitigation script to close the exploitation path. It creates a least-privilege ClickHouse user for Grafana and points the ClickHouse data source at it, replacing the default superuser.

Before running the script, back up the `pmm-data` volume.

1. On the host running PMM Server, save the script below as `pmm-ch-user.sh`. The script requires `docker`, `jq`, and `openssl`:

    ```sh
    #!/bin/bash
    # Create a least-privilege ClickHouse identity for Grafana and point the
    # ClickHouse datasource at it, replacing the default superuser.


    set -euo pipefail

    CONTAINER=${CONTAINER:-pmm-server}
    PMM_HOST=${PMM_HOST:-localhost}
    PMM_PORT=${PMM_PORT:-443}
    GRAFANA_URL="https://${PMM_HOST}:${PMM_PORT}"
    ADMIN_PASS=${ADMIN_PASS:-$(cat /root/pmm-admin-password)}
    # Drop-ins are loaded from users.d (users_config defaults to users.xml ->
    # users.d), NOT default-users.d.
    BOOTSTRAP_XML=/etc/clickhouse-server/users.d/zz-provision-bootstrap.xml

    CH_PASS=$(openssl rand -hex 24)
    CH_HASH=$(printf '%s' "$CH_PASS" | sha256sum | awk '{print $1}')
    BOOT_PASS=$(openssl rand -hex 24)
    BOOT_HASH=$(printf '%s' "$BOOT_PASS" | sha256sum | awk '{print $1}')

    ch_wait () {
      local user=$1 pass=$2 i
      for i in $(seq 1 45); do
        if docker exec -i "$CONTAINER" clickhouse-client --host 127.0.0.1 \
             --user "$user" --password "$pass" -q "SELECT 1" >/dev/null 2>&1; then
          return 0
        fi
        sleep 2
      done
      echo "ERROR: clickhouse did not accept $user within 90s" >&2
      return 1
    }

    # PMM's ClickHouse default superuser has access_management disabled, so it
    # cannot run CREATE USER / GRANT even with its known password. Install a
    # short-lived admin to run the DDL instead. Drop-ins must live in users.d;
    # and a plaintext <password> is rejected outright at startup because PMM
    # ships allow_plaintext_password=0.
    docker exec -u root "$CONTAINER" mkdir -p /etc/clickhouse-server/users.d
    docker exec -u root -i "$CONTAINER" bash -c "cat > $BOOTSTRAP_XML" <<XMLEOF
    <clickhouse>
        <users>
            <provision_admin>
                <password_sha256_hex>$BOOT_HASH</password_sha256_hex>
                <networks><ip>127.0.0.1</ip><ip>::1</ip></networks>
                <profile>default</profile>
                <quota>default</quota>
                <access_management>1</access_management>
            </provision_admin>
        </users>
    </clickhouse>
    XMLEOF
    docker exec -u root "$CONTAINER" chown pmm:root "$BOOTSTRAP_XML"
    docker exec -u root "$CONTAINER" supervisorctl restart clickhouse
    ch_wait provision_admin "$BOOT_PASS"

    # grafana_ro holds SELECT and nothing else. Without the SOURCES family it
    # cannot call url(), s3(), mongodb(), remote() or file(); readonly=1
    # additionally prevents it overriding server settings such as
    # max_http_get_redirects. ALTER runs unconditionally so that re-running
    # this script rotates the password rather than failing.
    docker exec -i "$CONTAINER" clickhouse-client --host 127.0.0.1 \
      --user provision_admin --password "$BOOT_PASS" --multiquery <<SQLEOF
    CREATE SETTINGS PROFILE IF NOT EXISTS grafana_ro_profile SETTINGS
        readonly = 1, allow_ddl = 0, max_execution_time = 60;

    CREATE USER IF NOT EXISTS grafana_ro IDENTIFIED WITH sha256_hash BY '$CH_HASH';
    ALTER USER grafana_ro IDENTIFIED WITH sha256_hash BY '$CH_HASH'
        SETTINGS PROFILE grafana_ro_profile;

    REVOKE ALL ON *.* FROM grafana_ro;
    GRANT SELECT ON pmm.*            TO grafana_ro;
    GRANT SELECT ON default.*        TO grafana_ro;
    GRANT SELECT ON system.tables    TO grafana_ro;
    GRANT SELECT ON system.columns   TO grafana_ro;
    GRANT SELECT ON system.databases TO grafana_ro;
    GRANT SELECT ON system.one       TO grafana_ro;
    GRANT SELECT ON system.numbers   TO grafana_ro;
    SQLEOF

    docker exec -u root "$CONTAINER" rm -f "$BOOTSTRAP_XML"
    docker exec -u root "$CONTAINER" supervisorctl restart clickhouse
    ch_wait grafana_ro "$CH_PASS"

    # Repoint the datasource. The UID is assigned by PMM, so look it up.
    DS_UID=$(curl -sk -u "admin:$ADMIN_PASS" "$GRAFANA_URL/graph/api/datasources" \
      | jq -r '.[] | select(.type == "grafana-clickhouse-datasource") | .uid' | head -1)
    if [ -z "$DS_UID" ]; then
      echo "ERROR: no grafana-clickhouse-datasource found" >&2
      exit 1
    fi

    # Transient files hold the CH password; keep them in a private dir and
    # always remove them, even if a curl below fails.
    umask 077
    TMPD=$(mktemp -d)
    trap 'rm -rf "$TMPD"' EXIT
    curl -sk -u "admin:$ADMIN_PASS" \
      "$GRAFANA_URL/graph/api/datasources/uid/$DS_UID" > "$TMPD/ds-ch.json"
    jq --arg p "$CH_PASS" \
      '.jsonData.username = "grafana_ro" | .secureJsonData.password = $p' \
      "$TMPD/ds-ch.json" > "$TMPD/ds-ch.new.json"
    curl -sk -u "admin:$ADMIN_PASS" -X PUT -H 'Content-Type: application/json' \
      -d @"$TMPD/ds-ch.new.json" \
      "$GRAFANA_URL/graph/api/datasources/uid/$DS_UID" >/dev/null

    # Fail the build rather than come up believing this worked.
    if docker exec -i "$CONTAINER" clickhouse-client --host 127.0.0.1 \
         --user grafana_ro --password "$CH_PASS" \
         -q "SELECT count() FROM url('http://169.254.169.254/latest/user-data','LineAsString','line String')" \
         >/dev/null 2>&1; then
      echo "FATAL: grafana_ro can still reach url()" >&2
      exit 1
    fi
    if docker exec -i "$CONTAINER" clickhouse-client --host 127.0.0.1 \
         --user grafana_ro --password "$CH_PASS" \
         -q "CREATE TABLE default.zz_provision_check (x String) ENGINE=Memory" \
         >/dev/null 2>&1; then
      echo "FATAL: grafana_ro can still run DDL" >&2
      exit 1
    fi

    unset CH_PASS BOOT_PASS ADMIN_PASS
    echo "ClickHouse datasource now authenticates as grafana_ro."
    ```

2. Run the script, replacing `XXXXX` with your PMM admin password. Adjust `CONTAINER`, `PMM_HOST`, and `PMM_PORT` if your setup differs from the defaults:

    ```sh
    CONTAINER=pmm-server PMM_HOST=localhost PMM_PORT=443 ADMIN_PASS=XXXXX bash ./pmm-ch-user.sh
    ```

3. Confirm the script finishes with `ClickHouse datasource now authenticates as grafana_ro.` If it exits with an `ERROR` or `FATAL` message instead, the mitigation is not in place. Do not ignore the failure: resolve the reported problem and re-run the script. Re-running is safe and rotates the `grafana_ro` password.

4. Verify that Query Analytics dashboards still load.

Support & additional resources

If you require further clarification or assistance, we are available 24/7:

• Technical support portal for customers
• Technical support for community

Removed percona-telemetry-agent from PMM Server image

We removed the percona-telemetry-agent binary from the PMM Server image. It was built on Go 1.26.5 and carried eight high-severity Go standard library vulnerabilities, so removing it eliminates those CVEs entirely.

Multiple component upgrades

PMM 3.9.1 upgrades several internal components to address known vulnerabilities.

Zero vulnerabilities in PMM's own components

All PMM-owned binaries (exporters, VictoriaMetrics, Nomad, pmm-managed, pmm-agent, pmm-admin, vmalert, vmproxy, qan-api2, and pmm-dump) report zero known vulnerabilities in this release.

Any remaining risks are in third-party dependencies where upstream fixes are not yet available, and none are exploitable in a typical PMM deployment.

Residual security risk

As part of modern software development, complex systems commonly rely on third-party components, and some level of residual risk may remain at release time despite ongoing vulnerability management.

This is particularly the case when vulnerabilities in upstream dependencies are disclosed close to a planned release.

The CVEs below remain present in third-party dependencies included with PMM. After assessment, the residual risk to PMM deployments is considered low due to limited exploitability, existing architectural boundaries, and available mitigating controls.

We will continue to monitor upstream disclosures, reassess their impact on PMM, and remediate in future releases.

Go standard library vulnerabilities in Grafana binaries

Tracked as CVE-2026-33818, CVE-2026-39821, CVE-2026-46600, CVE-2026-56853, CVE-2026-56858, CVE-2026-56859, CVE-2026-56860, and CVE-2026-56862.

Affected component

Grafana binaries (grafana, grafana-cli, grafana-server). PMM is built on Go 1.26.5, while the fixes for these CVEs require Go 1.26.6 or later. The CVEs were published on August 13, 2026 as part of the Go 1.26.6 security release.

PMM-specific context

These are Go standard library issues affecting encoding/asn1, encoding/xml, html/template, net/http, net/url, crypto/tls, and golang.org/x/net. All are denial of service or require pre-authenticated access. None allow remote code execution or data access.

• CVE-2026-33818 (encoding/asn1 DoS): PMM does not parse untrusted ASN.1 input.
• CVE-2026-39821 (Punycode privilege escalation): PMM does not perform internationalized domain name lookups.
• CVE-2026-46600 (DNS parsing DoS): Only internal DNS is used.
• CVE-2026-56853 (unencrypted HTTP/2 DoS): Grafana is behind nginx; direct access to the Grafana port is not required.
• CVE-2026-56858 (html/template XSS): Requires authenticated access and template injection, neither of which is possible through normal PMM usage.
• CVE-2026-56859 (XML decoding DoS): PMM does not parse untrusted XML.
• CVE-2026-56860 (URL path resolution DoS): Requires crafted URLs to reach Grafana endpoints.
• CVE-2026-56862 (TLS KeyUpdate DoS): Grafana is behind nginx TLS termination.

Mitigating factors

• PMM authentication is required before any Grafana endpoint is reachable.
• Grafana is behind nginx, which terminates TLS and proxies HTTP.
• None of these CVEs allow data access or remote code execution.

Remediation plan

These will be addressed in a future release by rebuilding Grafana on Go 1.26.6 or later.

kin-openapi authentication bypass (GHSA-r277-6w6q-xmqw)

Affected component

Grafana binary (third-party dependency, github.com/getkin/kin-openapi v0.133.0, fixed in v0.144.0).

PMM-specific context

This vulnerability causes ValidationHandler.Load() to fail open, defaulting to NoopAuthenticationFunc when loading OpenAPI specs, potentially bypassing authentication.

In PMM, Grafana's OpenAPI validation is not exposed as a standalone endpoint, and the vulnerable code path is not reachable without prior PMM authentication.

Mitigating factors

• PMM authentication is required to access Grafana.
• The vulnerable ValidationHandler.Load() path is not invoked by PMM's Grafana configuration.

Remediation plan

This will be addressed in a future dependency update once upstream ships the fix.

Denial of service and information disclosure (CVE-2026-21728 and CVE-2026-28377)

Affected component

Grafana binary. Tempo is compiled into Grafana as an optional data source plugin.

PMM-specific context

Tempo is a distributed tracing backend. PMM does not use Tempo, does not configure a Tempo datasource, and does not accept tracing data.

The denial of service via large queries and the S3 encryption key disclosure via the status endpoint cannot be triggered in a PMM deployment.

Mitigating factors

• PMM does not configure or enable the Tempo datasource.
• No PMM component sends or receives tracing data through Tempo.
• The vulnerable endpoints are not exposed in PMM's Grafana configuration.

Remediation plan

This will be addressed in a future dependency update once upstream ships the fix.

Prometheus library information disclosure (CVE-2026-42151)

Affected component

The Prometheus library embedded in Grafana for PromQL evaluation.

PMM-specific context

PMM uses VictoriaMetrics as its metrics backend, not Prometheus. This CVE discloses Azure OAuth client secrets via the Prometheus config API. PMM does not use Azure OAuth for Prometheus, and the config API is not exposed externally.

Mitigating factors

• PMM authentication is required to access Grafana and execute queries.
• PMM does not use Azure OAuth or Prometheus remote read.

Remediation plan

This will be addressed in a future dependency update once upstream ships the fix.

golang.org/x/text denial of service (CVE-2026-56852)

Affected component

Grafana binary (golang.org/x/text v0.37.0, fixed in v0.39.0).

PMM-specific context

This is a denial of service via invalid UTF-8 input in text processing. PMM does not pass untrusted text data through this code path.

Mitigating factors

• PMM authentication is required to access Grafana.
• The impact is limited to denial of service, not data access or code execution.

Remediation plan

This will be addressed in a future dependency update once upstream ships the fix.

gRPC-Go xDS RBAC and HTTP/2 vulnerabilities (GHSA-hrxh-6v49-42gf)

Affected components

Grafana and Grafana ClickHouse Datasource plugin (third-party dependencies using google.golang.org/grpc < 1.82.1).

PMM-specific context

PMM does not use xDS service discovery or RBAC policies in any component, so the two xDS issues cannot be triggered. The HTTP/2 DoS requires direct access to a gRPC server port; in PMM, all affected components communicate internally within the PMM Server container and are not directly exposed to external networks.

Mitigating factors

• PMM does not configure xDS or xDS-based RBAC policies in any component.
• All affected components are bound to internal interfaces within the PMM Server container.

Remediation plan

This will be addressed in a future dependency update once upstream ships the fix.

Go standard library vulnerabilities in ClickHouse datasource plugin

Tracked as CVE-2026-33818, CVE-2026-39821, CVE-2026-39822, CVE-2026-46600, CVE-2026-56853, CVE-2026-56858, CVE-2026-56859, CVE-2026-56860, and CVE-2026-56862.

Affected component

Grafana ClickHouse Datasource plugin (third-party, not Percona-maintained). Built on Go 1.26.4; fixes require Go 1.26.6 or later.

PMM-specific context

The plugin only connects to PMM's internal ClickHouse instance over localhost. No external or user-controlled input reaches the ClickHouse datasource query path without prior PMM authentication.

Mitigating factors

• The plugin only connects to ClickHouse within the PMM Server container.
• PMM Server runs on Linux; Windows-specific CVEs do not apply.
• No untrusted external input reaches the ClickHouse datasource query path without prior PMM authentication.

Remediation plan

The fix requires an upstream rebuild of the plugin with Go 1.26.6 or later. This will be addressed once it is available.

How to reduce your exposure now

While upstream fixes are pending, you can lower your risk by:

• restricting network access to PMM Server to trusted networks and users
• keeping the number of PMM admins small and enforcing strong authentication
• applying resource limits to PMM Server containers where possible

📈 Improvements

• PMM-15309: The Grafana ClickHouse data source now connects as a dedicated, read-only user instead of a privileged account.

✅ Fixed issues

• PMM-15188: Fixed a bug in pmm-encryption-rotation that corrupted stored TLS certificates and keys, as well as AWS and Azure credentials, for monitored services. Each rotation re-encrypted these values instead of decrypting them first, adding extra encryption layers that eventually made them unreadable.

  Even a single pre-upgrade rotation makes the affected credentials unrecoverable. If you rotated the encryption key before upgrading, remove and re-add the affected services to recreate their credentials. See Corrupted credentials after encryption key rotation.

• PMM-15198: When you monitored MongoDB router (mongos) nodes in a sharded cluster, the exporter kept index statistics from only one shard and dropped the rest, and flooded the system logs with duplicate mongodb_indexstats_accesses_ops metric errors. Collection statistics could also be attributed to the wrong shard. Index and collection statistics are now labeled with the shard they come from, so you get complete and correctly attributed data from every shard, with no duplicate-metric errors. Non-sharded deployments are unaffected.

• PMM-15310: Fixed an issue where PMM Client could get stuck showing as Disconnected after a network interruption, with metrics and Query Analytics data no longer flowing. The connection now recovers automatically within about a minute.

🚀 Ready to upgrade to PMM 3.9.1?

• New installation
• Upgrading from PMM 3
• Upgrading from PMM 2