permitio/opal 0.10.0-rc.3 on GitHub

Release candidate. Docker :latest and a plain pip install stay on 0.9.9 until 0.10.0 ships, so nothing moves unless you opt in. 0.10.0-rc.3 is 0.10.0-rc.2 plus the scopes-mode clone repack below. See the 0.10.0-rc.2 notes for the security fixes and the 0.10.0-rc.1 notes for the move to pydantic v2 and Starlette 1.x.

Scope clones stop growing without bound (#961)

This affects opal-server in scopes mode (OPAL_SCOPES=1) only.

The problem. The scopes git fetcher (pygit2 on libgit2) writes one pack file per fetch and never merges them. On repositories with many branches, a fetch can also re-download history the clone already has. A scope clone's disk usage therefore grows until the volume holding the clones fills up.

The fix. Once a successful fetch leaves a clone with at least OPAL_SCOPES_GIT_REPACK_PACK_LIMIT pack files (default 50), opal-server runs git repack -a -d -k on it:

It merges the packs into one and drops the duplicate objects.
It runs at the end of the sync pass, after the pass's notifications.
A single-scope refresh repacks its own clone right after its sync.
It never starts while another sync pass is running in the same process.

⚠️ Before upgrading opal-server in scopes mode

This is a behaviour change: repacking is on by default. To turn it off, set OPAL_SCOPES_GIT_REPACK_PACK_LIMIT=0.

Check these first:

git: the git binary must be on PATH. The official images ship it.
Memory: a repack takes about 0.5 KB per object of the repository, inside the opal-server container. Leave that room under the container's memory limit.
- On Linux, git runs with oom_score_adj 1000, so the kernel picks the repack first.
- On Kubernetes with cgroup v2, an OOM kill takes the whole container. The clone is then left alone for an hour, so a repack that keeps running out of memory restarts the container at most hourly instead of in a loop.
Disk: git writes the merged pack before it deletes the old ones, so the volume needs room for one more copy of the largest clone's packs. On a Kubernetes emptyDir with a sizeLimit, the kubelet evicts the pod when the volume passes the limit, so size the limit with that headroom.
Filesystem: repacks are serialized per pod with an flock on the clones directory. It must be a local filesystem, or NFS mounted with local_lock=flock.

New settings and metrics

Settings:

OPAL_SCOPES_GIT_REPACK_PACK_LIMIT (default 50): the pack count that triggers a repack. 0 or negative turns repacking off, and 1 counts as 2.
OPAL_SCOPES_GIT_REPACK_TIMEOUT (default 120 seconds): a hard limit on one repack.

Metrics:

opal_server.scopes.git_repack{outcome}
the opal_server.scopes.git_repack_seconds distribution
opal_server.scopes.git_op_skipped, with reasons repack_running and pass_in_flight

A clone whose repack failed is left alone for an hour by every process of the pod. An INFO line names it.

How to opt in

Docker: every image is published under the 0.10.0-rc.3 tag, plus -alpine:

permitio/opal-server:0.10.0-rc.3
permitio/opal-client:0.10.0-rc.3
permitio/opal-client-standalone:0.10.0-rc.3
permitio/opal-client-cedar:0.10.0-rc.3
permitio/opal-client-eopa:0.10.0-rc.3

pip: pre-releases are skipped unless you ask for them:

pip install --pre opal-client        # or: opal-client==0.10.0rc3
pip install --pre opal-server