DirektorBani/DataSafeS3 v1.5.0 on GitHub

[1.5.0] - 2026-10-12

Minor release. Durable event delivery, optional Kafka, scripted HA promote, and per-rule gateway sync. VERSION is 1.5.0. This commit does not push the git tag. Images are published when release.yml runs on v1.5.0.

Upgrade

Postgres migrations 020 (event_outbox) and 021 (gateway sync schedule) run on startup. Bolt opens the outbox buckets when the database opens. A v1.4.x binary leaves them in place and ignores them. The down migrations drop the table and its events, or the schedule columns.
The no-op Kafka and RabbitMQ sinks are removed. Kafka delivers only when STORAGE_KAFKA_BROKERS is set. An empty value does not dial. Webhook and NATS settings are unchanged. These sinks are an external event feed, not the cluster interconnect.
Lab ports moved. Erasure defaults are console 8084, S3 9004, Postgres 5444. Site A defaults are console 8086, S3 9006, Postgres 5446. Delete an old .env.erasure or .env.site-a and copy the example again. A copied file keeps the previous ports.
Automatic storage-leader takeover waits out the current lease. A 10s lease takes about 9s. The default 30s lease takes about 26s. A process that already held the lease and then lost it does not acquire it again.

Guides: events EN · RU · HA EN · RU · upgrade EN · RU.

Added

Gateway sync schedule (opt-in). A replication rule with an empty sync_schedule still follows STORAGE_GATEWAY_FULL_SYNC_INTERVAL (default 1h; STORAGE_GATEWAY_CONTINUOUS_SYNC=true shortens that global scan to 5 minutes). That scan queues objects and does not write a sync job. off stops the scheduled full scan for that rule; object events still replicate. A duration from 1m to 168h copies only that rule, writes a sync job, and stores the last run (last_sync_at, last_sync_status, last_sync_error, last_sync_objects). The console rule card shows the schedule, the last run, and the error. PATCH /api/v1/gateway/replication/{id} changes the schedule. Postgres migration 021. This is not continuous multi-destination sync.
HA lab port split. Erasure defaults are console 8084, S3 9004, Postgres 5444. Site A defaults are console 8086, S3 9006, Postgres 5446. Both overlays override the storage port, so they no longer publish :9000. Site A’s cluster endpoint defaults to its own S3 port. Vault stays 9001 / 5434 / 8200. HA stays 8082 / 9002 / 5442, with standby HTTP 19011 and 19013. The HA NATS profile publishes 14222 and 18222. scripts/ha/*.ps1 read the stand .env when it exists. An older copied .env.erasure or .env.site-a keeps its previous ports until it is recreated from the example.
Promote drill without host Python. bash scripts/ha/run-promote-drill.sh runs its helpers inside the webhook container. A static Docker client is downloaded once into the drill state directory and mounted there. Git Bash on Windows does not need Python installed. Linux state stays /tmp/datasafe-promote-drill. The SSH lab gate still uses host python3.
Scripted promote and a DR evidence pack. scripts/ha/promote.sh and scripts/ha/promote.ps1 fence the old Postgres primary, promote the standby, and on a fresh promote clear ha_leader_lock. A second run is safe and leaves a live lease in place. scripts/cluster/promote.sh and .ps1 are the Patroni switchover with the same exit codes. Dry-run prints the plan and changes nothing. scripts/collect-dr-evidence.sh and .ps1 write a folder, summary.md, evidence.json, and a zip (timing, lag, healthz, object write/read, outbox, dispatcher on the new leader, loss, at-least-once duplicates, versions). bash scripts/ha/run-promote-drill.sh is the one-command lab; CI job ha-promote-drill runs it. Host ports default away from :9000 and :9001. This is scripted promote, not automatic multi-AZ failover. See EN · RU.
Event delivery observability. The overview dashboard has an Event delivery row: outbox pending, delivered/failed/dead rates by destination, dropped rate by reason, delivery lag p50/p95/p99, the degraded gauge, and Kafka bridge restarts (datasafe_events_bridge_restarts_total, datasafe_events_sink_degraded). Example alerts are deploy/docker/prometheus/alerts.yml (pending above 1000 or growing for 10 minutes, any destination degraded for 5 minutes, dead deliveries increasing, pending-cap drops increasing). Compose loads that file. CI job event-observability runs promtool check rules and provisions Grafana 13.2.3 to read dashboard datasafe-overview. Activity writes events_sink_degraded on the first real failure of a destination and events_sink_recovered when it connects again; retries do not add rows. The console Activity filter lists those actions and events_outbox_dropped. Before the first successful probe, delivery status is degraded with last_error waiting for first successful connection. See EN · RU.
Kafka sink (opt-in). STORAGE_KAFKA_BROKERS registers a franz-go producer (pure Go, no CGO) with acks=all, the default idempotent producer, record key tenant/bucket/key, and topic STORAGE_KAFKA_TOPIC (default datasafe.events). TLS CA/cert/key files, SASL PLAIN (refused unless TLS is on), and SCRAM-SHA-256/512. Username and password accept *_FILE. An empty broker list does not create a client, a goroutine, or a connection. The body is S3-notification JSON (Records[], eventName, s3.bucket.name, s3.object.key/size/eTag/versionId) plus datasafe (tenant_id, event_id, schema_version: 1). Webhook and NATS keep their envelopes. Put, copy, and multipart store etag, size, and the object version id from the committed object on the outbox row at insert. The dispatcher holds per-object order for every sink: a later row for the same tenant/bucket/key is not published to a sink while an earlier row is still pending on that sink. One stuck key blocks only that key. When the earlier attempt goes dead, later rows proceed and the dead row is a gap. Settings → Event delivery shows connected/degraded/disabled, last error, and queue depth; Send test event is administrator-only (GET /api/v1/events/delivery, POST /api/v1/events/test). A broker-list change between starts is Activity kafka_brokers_changed. Lab overlay deploy/compose/docker-compose.kafka.yml (apache/kafka:3.9.2) and installer profile kafka are not in the default profiles. See EN · RU.
Single-node regression gate. CI job single-node-gate (pull requests into main, and pushes to main) builds the storage-server image and runs only that container the way installer profile core does: Bolt, STORAGE_DEV=false, no NATS or Kafka, Postgres hostname set but no Postgres process. Smoke covers /healthz, admin login and first-run setup, S3 put/get/list with the AWS CLI, Activity, inventory create/list/download, and an evidence-pack zip. It fails on broker dials in logs or outbound TCP, and when the outbox pending metric is not zero with no sinks. Idle RSS and startup-to-healthy are medians of five interleaved runs against ghcr.io/direktorbani/datasafe-storage-server:v1.4.1 on the same runner; the job fails above baseline +10% RSS or baseline +1s startup. ./install.sh --yes --profiles core runs the same smoke. .\install.ps1 -Yes -Profiles core is the Windows command; CI parses that script and does not boot Docker on Windows. See EN · RU.
Durable event outbox. Admin notification events (object, bucket, user, share, multipart, trash) are stored in the metadata backend before delivery: bolt buckets event_outbox and event_outbox_due, or Postgres table event_outbox (migration 020). One dispatcher per process delivers to webhook and NATS with exponential backoff and jitter, per-sink attempts, and dead after STORAGE_EVENTS_MAX_ATTEMPTS. Delivered rows expire after STORAGE_EVENTS_RETENTION (default 24h); dead and cap-dropped rows after STORAGE_EVENTS_DEAD_RETENTION (default 168h). STORAGE_EVENTS_MAX_PENDING (default 100000) drops the oldest pending rows, increments datasafe_events_dropped_total, and writes Activity events_outbox_dropped. Object writes are not failed by that cap. In HA only the Postgres leader dispatches. None of the STORAGE_EVENTS_* variables are required. See EN · RU. ADR-0002 is Accepted.

Changed

Event delivery page. Each sink shows delivered and failed counts from datasafe_events_delivered_total and datasafe_events_failed_total (GET /api/v1/events/delivery). Those counters are process memory and reset to zero when storage-server restarts. The sentence that Kafka stays off until STORAGE_KAFKA_BROKERS is set is shown only while that sink is disabled. The console sidebar shows the server version from GET /api/v1/health.
GET /healthz and GET /api/v1/health include version and commit, set with -ldflags from VERSION and the git commit in deploy/docker/Dockerfile. release.yml passes both when it publishes the tag.
Webhook and NATS delivery goes through the outbox. STORAGE_NATS_URL and console webhook settings are unchanged. An empty NATS URL does not register a sink and does not dial. A webhook sink is registered only while at least one webhook is enabled. With no configured sink, admin mutations do not write outbox rows, and events from that period are not replayed later. Each console webhook keeps one delivery-history row per outbox event; retries update that row.
Removed the no-op RabbitMQ and Kafka sinks from process wiring. There is one events.EventSink interface. Kafka is a real sink only when STORAGE_KAFKA_BROKERS is set. The container image runs franz-go in /usr/lib/datasafe/datasafe-kafka-bridge and starts it only then. A go build without -tags datasafe_sinkexec links it into the binary and still does not dial when brokers are empty.
S3 SigV4 object events (put, copy, delete including delete markers, multipart complete) write outbox rows in the same metadata transaction when a sink is registered. With no sink and no matching bucket notification, those PUTs do not grow the outbox.
Bucket-notification webhooks are delivered by the outbox dispatcher. Optional prefix/suffix filters apply when the row is inserted. The POST body stays {"event","bucket","key","size","time"}.
Webhook retry classification. Transport errors, timeouts, HTTP 5xx, 408, and 429 are retried. Retry-After is honoured and capped at 5 minutes. Any other 4xx marks that sink dead immediately; last_error contains the status.
NATS JetStream. If the connected server has JetStream and a stream covers STORAGE_NATS_SUBJECT.<event>, publish waits for an ack (failure is retried). Otherwise core publish is used. There is no extra environment variable.

Fixed

Fresh install console. Default Compose copies /srv/console from DATASAFE_CONSOLE_IMAGE (ghcr.io/direktorbani/datasafe-console:v1.5.0) into the console-dist volume and Caddy serves that volume. A git archive tree has no web/console/dist, so the old bind mount was an empty directory and GET / was 404 while /healthz still returned 200. install.sh and install.ps1 now require GET / 200 and the id="root" marker on the edge (port PUBLIC_HTTP_PORT, default 8080). Host web/console/dist is the overlay deploy/compose/docker-compose.local-console.yml.
Simultaneous start on empty Postgres. Several nodes no longer exit with user already exists. Bootstrap of the admin user runs under the migration advisory lock, and an existing admin is success.
Webhook delivery after HA takeover. A follower reloads sinks from the database when it becomes leader and about every 2 seconds, so a webhook created after that process started is registered. An outbox row for a sink that is not registered yet stays pending and does not count attempts toward dead. Takeover waits out the current lease (about 9s for a 10s lease, about 26s for the default 30s lease).
Promote-drill Docker client on Git Bash. The helper looked for an executable bit. Git Bash does not set that on the Linux client, so every run downloaded the 82 MB archive again. The script now keeps a non-empty file, writes the download to a partial tarball, and resumes with curl -C - --retry 5 --retry-delay 3, including through HTTPS_PROXY.
Gateway connection create response. POST /api/v1/gateway/connections no longer returns secret_key. The list endpoint already omitted it. The stored credential is unchanged.
Concurrent PUT of one key. Writers on one node no longer share <key>.tmp or insert a second unversioned row. Temps are unique, Bolt Put and Postgres ON CONFLICT keep the last writer, and that writer's bytes, etag, and size agree. Twenty parallel PUTs of one key all return 200.
Parallel Postgres migrate. Nodes that start together on an empty database take a session advisory lock around migrations, so CREATE EXTENSION in 001_init does not hit pg_extension_name_index.
Short HA leader lease. A lease under 15 seconds used to renew on the same interval as the lease. The renew interval is now at most one third of the lease.
Follower lease takeover. A storage node that has never held the lease keeps trying and takes it only after the previous lease expires. A process that already led and then failed renew still drops the lease immediately and does not acquire it again. This is not Postgres failover and not a power fence.
Gateway rule card. The console shows the rule name when the rule has one.
Event bridge restarts during a broker outage. A produce that outlasted the parent deadline used to kill the child and increment datasafe_events_bridge_restarts_total. The child now answers within its own deadline and stays up. The counter moves only when the process exits.
Dispatcher republish after a read-only primary. A promote that fenced Postgres used to leave the old leader publishing the same outbox row on every tick until renew failed, because the webhook had already been sent and the delivered mark could not be written. A read-only or not-leader error from the store now stops the dispatcher and drops the lease. That process does not publish the row again. The new leader may deliver it once.
Single-node Prometheus scrape. The shipped cluster-nodes.json target list is empty, so job datasafe-cluster-nodes does not scrape storage-server:9000 a second time. Add extra nodes there when you have them.
Windows installer stderr. install.ps1 runs every docker / docker compose call through Invoke-NativeCommand. Windows PowerShell 5.1 with $ErrorActionPreference = 'Stop' was treating compose progress on stderr as a terminating error when the script output was redirected (*> log), so -Yes -Profiles core exited 1 while the stack was already up. The helper keeps ErrorActionPreference at Continue, sets PSNativeCommandUseErrorActionPreference to false on PowerShell 7.3+, returns stderr as text, and uses $LASTEXITCODE as the only success check. -Yes still waits for http://127.0.0.1:8080/healthz and exits 0 when that responds. CI on windows-latest runs that helper under redirected output in Windows PowerShell 5.1 and PowerShell 7. The Docker install itself stays on the Linux gate.
Console Playwright buckets.spec.ts no longer matches both the bucket-table search and the command palette. In Russian both placeholders start with «Поиск бакетов». The spec now matches only Search buckets... / Поиск бакетов.... gateway.spec.ts accepts the English, Russian, and French page titles, and the Russian empty state «Подключения шлюза не настроены».

Honesty

Promote is a script. Nothing in this tree elects a new Postgres primary on its own, and nothing fails a node over between availability zones. A storage follower that shares the database retries the leader lease and takes it only after expiry. A process that already held the lease and then lost it does not try again. The fence is default_transaction_read_only on the old primary plus an immediate drop of the storage-server leader lease when renew fails. A superuser can turn that GUC off in a session. The script does not rebuild the old primary as a standby. Object bytes stay on the node that wrote them; the drill's post-promote read is a new object on the new leader. Delivery during the drill is at-least-once: one extra post is allowed for a row that was published and then could not be marked delivered. The compose drill runs the image from deploy/docker/Dockerfile. CI job ha-promote-drill loads the docker-build artifact instead of building that image a second time. Node-a is started and becomes leader before node-b starts. The SSH lab gate runs the promote dry-run and a healthz evidence pack. The timed Patroni step is still the existing kill-leader drill.
Container images for this release are published when release.yml runs on the git tag v1.5.0. This commit does not push that tag. GET /healthz and GET /api/v1/health report version and commit from the image ldflags. The single-node gate still compares RSS and startup with the published v1.4.1 image, because v1.5.0 is not on GHCR until the tag is pushed. The installer step retags the pull-request server image and builds the console image as v1.5.0 locally so Compose uses the pull-request bits. A fresh git archive plus install.sh --yes --profiles core must return GET / 200.
datasafe_events_sink_degraded is 1 from registration until the first successful probe, so a destination that has not connected yet looks degraded. The 5-minute alert does not fire for a probe that succeeds immediately. Activity does not record the waiting sentence, only a real error and the later recovery. Dropped events have a reason label, not a destination label. Compose pins Grafana 13.2.3 and Prometheus v3.15.0. Helm Prometheus and Grafana image tags in values.yaml stay v1.2.0. The DataSafeS3 server and console tags in that file are v1.5.0. The Helm overview JSON includes the same Event delivery row, and Helm Prometheus does not mount the example alert file.
The gate's resource limits can still fail on a noisy GitHub runner when most PR samples are slow. The step summary prints min and max beside the medians. The job does not widen the +10% or +1s limits.
GHCR login in single-node-gate is optional. It is skipped for fork pull requests and when no token is present, and a failed login does not fail the job. ghcr.io/direktorbani/datasafe-storage-server:v1.4.1 is a public image.
The installer step retags the PR server image as ghcr.io/direktorbani/datasafe-storage-server:v1.5.0 and tags a locally built ghcr.io/direktorbani/datasafe-console:v1.5.0 so Compose uses the pull request bits. It does not push those tags. The resource baseline stays the published v1.4.1 image. Outbox emptiness is the Prometheus gauge, not a Bolt key count. The TCP check does not see UDP DNS.
Object bytes are written before the metadata transaction. The S3 outbox row commits with the object or delete-marker row. Soft-delete commits the event with the original-object delete, after the trash copy and trash row. Multipart completion deletes the upload record in a following transaction and does not enqueue replication. Creating a tenant member writes the user, membership, and groups first, then inserts the event. Bucket create is one transaction with the bucket row; visibility and team updates after that are separate.
Bucket notification time is the object record time at commit, not the HTTP send time. event stays put or delete (multipart posts put). bucket stays the storage key. The stored payload includes _webhook_url, which is stripped before the POST. There is no console delivery-history row for a per-bucket URL. Copy and put share the event value put, so a Copy-only filter also matches a normal PUT. Delete and delete-marker share delete. PutBucketNotificationConfiguration still keeps one URL (last topic, then last queue). s3:ObjectCreated:* now matches the posted event put; previously the generic :* check prevented that.
A bucket notification that fails is retried. The previous HTTP path discarded the error. 3xx is terminal, same as 4xx other than 408 and 429. The console webhook sink retries every matching hook together: one hook's 429 retries hooks that already returned 2xx. The sink is dead immediately only when every failure on that attempt is terminal.
NATS core publish does not confirm subscriber delivery. A JetStream ack confirms the stream stored the message, not that a consumer processed it. The mode is cached until reconnect.
Delivery is at-least-once. A second leader that appears while a publish is in flight can deliver the same row again. Followers do not dispatch. Bolt has no leader lock, so the single process always dispatches. Per-object order is per sink and per non-empty key. It does not stop a duplicate of an earlier event_id. An earlier row that goes dead is not published; later rows for that key are. Empty keys (UserCreated, bucket events) are not ordered against each other. Kafka consumers should dedupe by event_id. Idempotent produce covers retries inside one client session, not a process restart. Put, copy, and multipart copy etag, size, and versionId from the committed object into the outbox row at insert. versionId is empty when versioning is off. Webhook data and the NATS payload object include etag and size for those events; the envelopes stay {"event","timestamp","data"} and {"event","payload"}. Bucket-notification posts stay {"event","bucket","key","size","time"}. ObjectCreated is mapped to s3:ObjectCreated:Put, including copies, because the stored type does not distinguish copy. The object key in JSON is the raw key, not AWS URL-encoding. "Connected" on the delivery page is a cached status: the sink is registered, a probe or a publish has succeeded, and the last publish did not fail. GET /api/v1/events/delivery does not ping the broker. A background probe uses a 2s timeout, and publish failures update the same cache, so a down broker returns degraded with the dial or topic error in under a second. It is not a consumer-lag probe.
The storage-server image does not link franz-go into /storage-server. The client is the static child /usr/lib/datasafe/datasafe-kafka-bridge, started only when STORAGE_KAFKA_BROKERS is set. The image builds the server with -tags datasafe_sinkexec (no CGO). The child inherits the server environment; it is not a separate security boundary. go test and a normal go build link the client in-process. If the child exits, the server logs event bridge exited, marks the sink degraded, leaves the rows pending, and restarts the child with backoff. The child speaks only on stdin and stdout; it has no TCP listener, and its stderr is copied to the server log. It exits when the server exits. Restarting the child starts a new idempotent producer session, the same as restarting the server. The child produces one record at a time. Produce and the status ping are bounded so a stopped broker cannot hold a request for minutes. Built here with the same Docker builder, the image is 19,719,481 bytes of compressed content versus 15,872,560 for a rebuild of origin/main (530eec9), a delta of +3,846,921 bytes (+24.2%). docker images shows 75.6MB versus 62.1MB. The published ghcr.io/direktorbani/datasafe-storage-server:v1.4.1 image is 15,801,193 bytes (61.8MB). The child binary in the image is 9,511,074 bytes. The sink never creates topics. There is no STORAGE_KAFKA_AUTO_CREATE_TOPIC. A missing topic stays retriable and last_error is topic does not exist plus UNKNOWN_TOPIC_OR_PARTITION. The lab overlay creates datasafe.events with kafka-topics.sh and sets auto.create.topics.enable=false. The integration test creates its own topic the same way, through the Kafka protocol, and does not rely on broker auto-create.
Default single-node still opens no broker connection. With no NATS URL, no enabled webhook, and no bucket notification it writes no outbox rows, including for S3 PUTs. Events from that period are not replayed. Deleting every bucket notification and restarting leaves queued bucket posts without a sink.
Internal object writes (trash copy, inventory, lifecycle) do not emit global ObjectCreated / ObjectDeleted rows. They still enqueue a bucket notification when the bucket rule matches.

Container images

ghcr.io/direktorbani/datasafe-storage-server:v1.5.0
ghcr.io/direktorbani/datasafe-console:v1.5.0

CycloneDX SBOM files and cosign signatures are attached. See SECURITY.md for cosign verify instructions.