rhoopr/kei v0.12.0

on GitHub

Added

• Shared-library discovery notice on first sync. Users on the default --library PrimarySync who also have iCloud shared libraries on their account now see a one-time warning at sync time naming how many shared libraries exist and how to opt in ([filters] library = "all" or --library all). The notice fires at most once per account via a marker in the state DB's metadata table. Accounts without shared libraries re-probe on every sync so the notice still fires if a library is added later. Users who've set --library explicitly see nothing. The notice never auto-changes sync behavior.
• --bandwidth-limit accepts decimal values. 1.5M, .5K, and 2.5Gi all parse now. Previously kei rejected decimals with a confusing error about the unit suffix (the parser was splitting 1.5M at the first non-digit, leaving .5m as the unit). Values that round to less than 1 byte/sec (e.g. 0.0001K) are rejected with a clear "rounds to zero" message; negative values and NaN/infinity stay rejected. Existing integer forms (10M, 500K, 1Mi) are unchanged.
• --recent accepts a days-based form. --recent 100 still means "the 100 most-recent assets" (unchanged). --recent 30d now means "assets created in the last 30 days" and translates internally to --skip-created-before 30d. TOML [filters] recent accepts both 100 (integer) and "30d" (string). Passing both --recent Nd and --skip-created-before on the same invocation errors with a "pick one" message since they're equivalent controls. import-existing only accepts the count form (it scans files on disk, not iCloud creation dates) and errors on the days form.
• --notification-script fires a new sync_started event. The script is now invoked with KEI_EVENT=sync_started immediately before each sync cycle begins (after the watch-mode skip check, so skipped cycles don't fire). Useful for sending "sync running" pings to Slack or push notifications. Payload env vars match the other non-terminal events (KEI_MESSAGE, KEI_ICLOUD_USERNAME); per-cycle stats aren't available yet at start, so the extended KEI_DOWNLOADED/KEI_FAILED/etc. vars are omitted - they're set on sync_complete and sync_failed as before.
• --report-json gains a TOML key. Users can now set [report] json = "/path/to/run.json" in the config file instead of carrying KEI_REPORT_JSON separately. Resolution order is CLI > TOML > unset; kei config show round-trips the setting through the [report] section.
• sync_report.json has a new "interrupted" status. A cycle cut short by SIGINT/SIGTERM/SIGHUP now reports status = "interrupted" instead of "success" (when no failures) or "partial_failure" (when there were mid-flight recorded failures). Priority: session_expired > interrupted > partial_failure > success. Operators who alert on status != "success" should add "interrupted" to their benign-status allowlist unless they want to page on every graceful shutdown.
• --http-bind / KEI_HTTP_BIND / [server] bind. New flag to control which interface the /healthz + /metrics server listens on. Default stays 0.0.0.0 so Docker's -p 9090:9090 keeps working out of the box, but desktop users who don't want kei's stats reachable from the local network can set 127.0.0.1 to restrict to loopback. Accepts any IPv4 or IPv6 literal; invalid values fail at startup.
• kei import-existing --dry-run. Scans the tree, runs filename + size matching, and reports the match/unmatched counts without writing to the state DB and without the SHA256 compute a real import would do. Useful before committing: a user with 50k existing photos can verify that --folder-structure and --keep-unicode-in-filenames match the tree layout without locking in bad results, and the preview finishes in seconds rather than hours. The completion banner reads "Import complete (DRY RUN - no changes written to state DB)" so operators can't accidentally miss that nothing was persisted.
• kei reconcile caps its per-issue listing at 200 lines. Same treatment as kei verify and kei status — a library with thousands of missing files previously printed one MISSING: ... line per asset with no cap. The Marked failed: count in the summary still reflects every re-queued row; only the per-line listing is capped. A tail line reads ... and N more issue(s) not listed (listing capped at 200).
• kei verify caps its per-issue listing at 200 lines. Previously the command printed one MISSING: ... / CORRUPTED: ... line per affected file with no cap — a library with thousands of missing files produced a wall of output. The summary section still shows the true totals; listings beyond the cap print a tail line (... and N more issue(s) not listed (listing capped at 200)). Same cap used by kei status and sync_report.json so operators see consistent detail levels across the three surfaces. Follow-ups (--fix, --sample, --format json, --since <DATE>) tracked in a scratch design note.
• kei status --failed / --pending / --downloaded now caps the per-section listing at 200 rows. Previously the three listings printed every matching row from the state DB — a library with 100k+ downloaded assets or thousands of failures produced a wall of output. Listings beyond the cap print a tail line (... and N more (listing capped at 200)) with the omitted count. Summary counts (at the top of the output) are unchanged and always show the full totals. The cap matches the failed_assets cap used by sync_report.json so operators see consistent detail levels across the two surfaces. A --limit N override is tracked in a follow-up design note.
• --save-password now explains itself instead of silently no-op'ing. The flag saves the password to the credential store after successful auth, but only for ephemeral sources (--password / ICLOUD_PASSWORD). Previously, pairing --save-password with --password-file, --password-command, an already-saved credential, or a fresh interactive prompt would quietly do nothing - the user had no signal the flag was ineffective. kei now emits a targeted warning for each case naming the actual persistent source, pointing at kei password set for interactive bootstrap, or calling out that a rotating secret-manager password shouldn't be snapshotted into the store.

Changed

• --password-command / [auth] password_command now has a 30-second timeout per invocation. If the command hasn't exited by then, kei kills the child and fails the auth attempt with password command timed out after 30s: <cmd>. This matters most in watch mode, where a hung secret manager (network-stalled Vault, sleepy gpg-agent) would otherwise freeze the sync indefinitely. Commands that finish in milliseconds are unaffected.
• --password-command / [auth] password_command is now rejected on Windows at startup. kei runs the command via sh -c, which isn't on a stock Windows PATH, so previously the flag failed at auth time with a cryptic "No such file or directory". The new error fires at config load and points at --password-file or WSL. Unix behavior is unchanged.
• --notification-script is now a no-op with a one-time warning on Windows. kei invokes scripts via /bin/sh, which isn't available on a stock Windows PATH, so the flag previously failed silently at spawn time on every event. kei now logs a single warning at startup when the flag is set on Windows and skips script invocation for the rest of the run. Unix behavior is unchanged.
• --notify-systemd auto-detects under systemd. When neither the CLI flag, env var, nor TOML key is set, kei now enables sd_notify messages automatically if NOTIFY_SOCKET is present (which systemd publishes for Type=notify units). Users who set up a kei.service no longer have to remember the flag. An explicit --notify-systemd=false still suppresses even under systemd. Zero false positives: only systemd's Type=notify sets NOTIFY_SOCKET.
• Initial retry delay now derives from --max-retries. kei picks the initial exponential-backoff delay based on how patient the user asked to be: max 1-2 gets 2s, max 3 stays at 5s (today's default), max 4-6 gets 10s, max 7+ gets 30s. Higher max means "give the failing service time to recover", not "hammer faster" - Apple's 429 and 5xx failures both benefit from longer breathing room, and fast retry-storms make rate limiting worse. Explicit --retry-delay still wins during the deprecation window.
• --threads-num CLI flag renamed to --threads. KEI_THREADS_NUM becomes KEI_THREADS; the TOML key [download] threads_num becomes [download] threads. The old spellings still parse and log a deprecation warning; they'll be removed in v0.20.0. Passing --threads and --threads-num together (or both TOML keys) errors with "pick one". No short flag was added; --threads-num never had one.
• --skip-videos + --skip-photos with a live-photo-mode that drops everything now errors at startup. Previously a soft tracing::warn! would fire when --skip-videos, --skip-photos, and --live-photo-mode skip were all set and then the sync would silently complete with zero downloads. The validation now lives in Config::build and rejects the combination with a clear error explaining the outcome. The obscure legitimate case - --live-photo-mode video-only to download only Live Photo MOV companions - is still allowed.
• --directory CLI flag renamed to --download-dir. The short form -d keeps working, and KEI_DIRECTORY becomes KEI_DOWNLOAD_DIR. The TOML key [download] directory is unchanged. The old flag was generic sitting next to --config, --data-dir, and --folder-structure; the new name echoes the TOML section and makes Docker and systemd samples self-documenting.
• Minimum supported Rust version (MSRV) is now 1.91. src/download/paths.rs calls str::floor_char_boundary, which was stabilized in 1.91, so older toolchains never compiled cleanly anyway — they failed with a cryptic E0658. Cargo.toml now sets rust-version = "1.91", so users on < 1.91 get Cargo's package requires rustc 1.91 or newer error at the start of the build. The clippy::incompatible_msrv lint is enabled in CI to catch future MSRV regressions at PR time. Users on stable Rust ≥ 1.91 see no change. (#263)

Fixed

• kei import-existing no longer fails on a fresh install. On a brand-new machine (no prior kei sync or kei login run), the command bailed out with Failed to open database at ...: unable to open database file: Error code 14 because the ~/.config/kei/cookies/ parent directory didn't exist yet, and SQLite won't mkdir -p for you. SqliteStateDb::open now creates the parent directory before opening the DB, so import-existing works without the previously-required kei login workaround. (#264)
• iOS 17+ HEICs with Apple uri item references now embed XMP cleanly. From iOS 17, Apple-shot HEICs carry an extra infe entry with item_type = "uri " pointing at tag:apple.com,2023:photos/<id>. The mp4-atom 0.10.1 release didn't read the trailing item_uri_type cstr and rejected every such file with under decode: infe, so kei's metadata-embed pass failed on every iOS 17+ HEIC and the file landed without XMP. kei now pins mp4-atom to a post-fix git rev (PR kixelated/mp4-atom#123, awaiting upstream v0.11.0 release) and adds a regression test that round-trips a synthetic uri -bearing HEIC. The HEIF writer surfaces a typed HeifError enum (Decode { offset, total }, UnparsableTail, MissingMeta, Encode, Io) so future failures land with byte-offset context instead of opaque anyhow strings. (#274)
• First-pass HEIC metadata writes no longer fail with a spurious "Opening … for XMP update" warning. When --set-exif-datetime (or any other embed flag) was set, every HEIC asset logged a Failed to write metadata warning during its first download cycle, set a retry marker, and got rewritten correctly on the next sync. The metadata writer dispatched between the in-tree HEIF writer and Adobe XMP Toolkit by file-extension, but the embed step runs on the <base32>.kei-tmp part file before the atomic rename — so HEIC bytes routed to XMP Toolkit, which has no HEIF handler. Dispatch now sniffs the first 12 bytes for an ISO-BMFF ftyp HEIF brand, so part files dispatch correctly on the first attempt. No on-disk format change; existing pending markers from before the fix get cleared by the next sync as before. (#269)
• HEIC metadata reads no longer redundantly rewrite already-correct fields. The read-side mirror of #269: probe_exif always routed through Adobe's XMP Toolkit, which has no HEIF handler, so every HEIC silently came back as ExifProbe::default(). The "field already present, skip" gate in plan_metadata_write had nothing to compare against, so kei rewrote DateTimeOriginal / GPS / rating / description on every iPhone HEIC even when the file already carried the same value. No data corruption (the HEIF writer reads existing XMP before mutating), but every embed-flag run produced unnecessary writes. probe_exif now content-sniffs the first 12 bytes and parses HEIC files via the in-tree HEIF reader, matching the dispatch fix in #271. (#272)
• Default Docker image now starts cleanly out of the box. The v0.11.0 default CMD passed --watch 24h, but no such flag exists — running the published image with no overrides errored at startup with unexpected argument '--watch' found. Regression from the always-on-server work in #237; CI does not exercise the default CMD, so it slipped past review. The image's CMD now uses --watch-with-interval 86400 (and --download-dir per the rename below), so docker run ghcr.io/rhoopr/kei works without a custom CMD.
• --file-match-policy name-id7 no longer risks producing a path separator inside a filename. The 7-char asset-ID suffix baked into every filename was generated via standard base64, whose alphabet includes / and +. A CloudKit asset ID containing certain bytes (e.g. a ? early in the ID) would produce / as the 4th base64 character, causing kei to drop that asset into a surprise subdirectory instead of the expected filename. kei now uses URL-safe base64 (- / _) for the suffix. Files that previously got the bug land at a new path on next sync and may re-download once. Added a fuzz test that walks every single-byte input and asserts the suffix only contains [A-Za-z0-9_-].

Security

• Permissive-mode warning for --password-file. kei now logs a warning at startup when --password-file or [auth] password_file points to a file that's readable by group or other users (any of the 0o077 permission bits set). The warning names the path, reports the current mode, and suggests chmod 600 <path>. Container-secret mount points (/run/secrets/ and /var/run/secrets/) are exempted since Docker and Kubernetes publish those files as world-readable by design and enforce isolation at the mount layer. Unix only - no-op on Windows. Warns once per path per process, so watch mode doesn't spam.

Removed

• BREAKING: [auth] password TOML key. Plaintext passwords in config files are a standing security risk, and the credential store plus file/command sources already cover the use case. Config files that still set it now error on startup with a migration message. Move to kei password set (OS keyring or encrypted file), [auth] password_file, or [auth] password_command.

Deprecated

• --cookie-directory CLI flag and [auth] cookie_directory TOML key. Use --data-dir and top-level data_dir instead. The old spellings still work and log a deprecation warning; they'll be removed in v0.20.0.
• --directory CLI flag and KEI_DIRECTORY env var. Use --download-dir and KEI_DOWNLOAD_DIR instead. Both still parse and populate the same field but log a deprecation warning on use; they'll be removed in v0.20.0. Passing both forms together (CLI or env var) errors with "pick one" instead of silently picking a winner - docker-compose users upgrading from a pre-0.12 file should drop KEI_DIRECTORY before rolling out.
• --skip-live-photos CLI flag, KEI_SKIP_LIVE_PHOTOS env var, and [filters] skip_live_photos TOML key. Use --live-photo-mode skip and [photos] live_photo_mode = "skip" instead. All three still work and map to the same behavior, but now log a deprecation warning on use (previously the TOML path mapped silently); they'll be removed in v0.20.0.
• --no-incremental CLI flag. Use kei reset sync-token before the sync for the same effect. Both trigger a full enumeration and store a fresh token for subsequent incremental runs. The flag still parses and logs a deprecation warning; it'll be removed in v0.20.0. A narrow failure-path difference (a failed --no-incremental run preserved the prior token; a failed reset + sync did not) wasn't worth the flag's ongoing cost.
• --retry-delay CLI flag, KEI_RETRY_DELAY env var, and [download.retry] delay TOML key. The initial retry delay is now derived from --max-retries via a smart table; the explicit setting became redundant for almost every use case. All three still work with a deprecation warning; they'll be removed in v0.20.0. Remove the explicit setting to pick up the smart default.
• KEI_METRICS_PORT env var and [metrics] port TOML section now name v0.20.0 as the removal target. Both were deprecated in 0.11.0 in favor of KEI_HTTP_PORT / [server] port, but the deprecation messages said "a future release" without a concrete deadline. They now say v0.20.0.
• Hidden compat flags on kei sync now name v0.20.0 as the removal target. --auth-only, --list-albums / -l, --list-libraries, and --reset-sync-token have been hidden compat aliases since the subcommand refactor; each now emits a deprecation warning naming v0.20.0 and pointing at the replacement (kei login, kei list albums, kei list libraries, kei reset sync-token). --reset-sync-token previously emitted no warning at all; that's fixed. The shared deprecation_warning() helper used across the CLI was updated to always include the v0.20.0 target, so every flag-rename warning in the codebase is now consistent.

Full changelog: CHANGELOG.md