endurain-project/endurain v0.20.0-beta1 on GitHub

v0.20.0-beta1 - Event-driven activities, background jobs and scale-out

⚠️ Warning: beta release

This is a pre-release. Do not run it against data you cannot restore.
The DB has schema changes (a large set of new migrations). Backup your DB and data folder before updating; restoring that backup is the only way back to v0.19.x.
This release contains breaking API changes (see below). Third-party clients and integrations using the REST API will need updating.
Review the environment variable changes below.

Highlights

The backend foundation for this release is tracked in #785; related PRs are linked below.

Activity pipeline rebuilt on an event-driven core. Activities now publish events (activity.created, ...) and independent subscribers react to them: map thumbnails, reverse geocoding, notifications, source-file retention. Failures in one subscriber no longer affect the request or the others, and scheduled backfills repair anything missed.
Durable background jobs (#796). Optional job queue (JOBS_ENABLED=true) with a transactional outbox, retries with backoff, leases and a dead-letter state. File uploads, bulk import and provider refresh (Strava/Garmin) now run as jobs.
Ready to scale out (#771). New deployment profiles (local / distributed), shared Redis state, S3-compatible storage, PostgreSQL advisory locks for scheduled jobs, multiple web workers and a dedicated worker container (APP_ROLE=worker). The default single-container setup needs no changes. See Scaling and background jobs in the docs.
Platform package. Storage, events, jobs, geocoding and migrations now come from the jasil package.
Hardened security, see below.
Timezone-correct stats (#910). Week/month boundaries, gear component dates and health data now use the user's configured timezone and week start; activities without a GPS track use the user timezone too.

Import folders moved (action required if you import from disk)

Folder-based imports are now isolated per user, and the Strava export layout changed:

Bulk import: data/activity_files/bulk_import/ becomes data/activity_files/bulk_import/<your user id>/.
Strava export: data/activity_files/strava_import/ (with activities.csv, activities/, media/) becomes data/activity_files/strava_import/<your user id>/, or just upload the export ZIP from the Import page.
Files that fail to import are no longer left in bulk_import/import_errors; they are kept in the bulk_import_errors storage area under <user id>/<import id>/<file name>.

Files already sitting in the old locations are not picked up; move them before starting an import. Details in the Importing data docs page.

Environment Variables & Configuration

New Variables

DEPLOYMENT_PROFILE (default: local) - local, distributed or custom. Endurain refuses to start on an inconsistent topology and names the variable at fault.
WEB_WORKERS (default: 1) - number of web-server processes. Above 1 requires REDIS_URL.
REDIS_URL (default: empty) - default DSN for every Redis-backed capability (rate limits, login lockout, pending MFA, websocket tickets, event bus).
STATE_URI / EVENTS_URI / LOCK_URI / STORAGE_URI - fine-grained overrides. STORAGE_URI accepts local:// (default) or s3://bucket/prefix.
JOBS_ENABLED (default: false) plus JOBS_RUN_IN_PROCESS_WORKER, JOBS_MAX_ATTEMPTS, JOBS_BACKOFF_BASE_SECONDS, JOBS_BACKOFF_MAX_SECONDS, JOBS_LEASE_SECONDS, JOBS_BATCH_SIZE, JOBS_POLL_INTERVAL_SECONDS, JOBS_RETENTION_DAYS - durable job queue tuning.
APP_ROLE (default: api) - worker runs a dedicated job-worker container from the same image.
EVENT_LOG_ENABLED / EVENT_LOG_RETENTION_DAYS (defaults: true / 90).
Docker build arg PYTHON_EXTRAS=distributed - installs the Redis/S3 client libraries. The default image does not include them; if you use REDIS_URL or s3:// storage, build your own image with this extra.

Removed

RATE_LIMIT_STORAGE_URI and AUTH_SECURITY_STORAGE_URI - replaced by REDIS_URL / STATE_URI. Endurain refuses to start while either is still set, so replace them with REDIS_URL in your compose file before updating.
JAEGER_ENABLED, JAEGER_PROTOCOL, JAEGER_HOST, JAEGER_PORT - OpenTelemetry/Jaeger tracing has been removed, along with the Jaeger service in the example compose files.

Other

New data folders are created at start: activity_files/upload_incoming and activity_files/upload_staging.

Breaking API changes

If you only use the Endurain web UI you do not need to do anything. API consumers, note the main route changes (full list in the OpenAPI reference):

Before (v0.19.x)	Now
`POST /activities/create/upload`	`POST /activities/upload` (returns `202`, poll `GET /activities/ingestion-jobs/{job_id}`; supports `Idempotency-Key`)
`POST /activities/create/bulkimport`	`POST /activities/bulk-import` and `POST /activities/bulk-import/folder` (one durable job)
`GET /activities/refresh`	`POST /activities/refresh` (returns `202`)
`POST /strava/import/activities`	removed; Strava export imports go through `POST /activities/bulk-import` (upload the export ZIP as-is) or `POST /activities/bulk-import/folder`
`POST /activities_media/upload/activity_id/{id}`	`POST /activities/{id}/media`
`GET /activities_media/activity_id/{id}`, `DELETE /activities_media/{media_id}`	`GET /activities/{id}/media`, `DELETE /activities/{id}/media/{media_id}`
`GET /activity_media/{media}` (static)	`GET /activities/{id}/media/{media_id}/file` (signed, expiring token)
`GET /activity_thumbnails/{thumbnail}` (static)	`GET /activities/{id}/thumbnail` (signed, expiring token)
`GET /user_images/{img}` (static)	`GET /users/{user_id}/photo` (signed, expiring token)
`PUT /activities/edit`, `PUT /activities/visibility/{v}`	`PATCH /activities` (bulk) and `PATCH /activities/{id}` (supports `If-Match`)
`DELETE /activities/{id}/delete`	`DELETE /activities/{id}`
`GET /activities/number`, `/user/{id}/page_number/{n}/num_records/{m}`, `/name/contains/{name}`	`GET /activities` and `GET /activities/users/{user_id}` with pagination and filters; totals are in the response envelope
`GET /activities/user/{id}/followed/...`	`GET /activities/feed` (keyset pagination)
`GET /activities/user/{id}/thisweek/stats`, `/thismonth/stats`	`GET /activities/users/{user_id}/stats`
`GET /activities/gear/{id}`, `/gear/{id}/list`, `/gear/{id}/number`	`GET /activities/gears/{gear_id}`
`GET /activities_summaries/{view_type}`	`GET /activities/summaries`
`GET /activities_streams/activity_id/{id}/all` (also laps, sets, workout steps)	`GET /activities/{id}/streams`, `/laps`, `/sets`, `/workout-steps`
`GET /activities_streams/activity_id/{id}/stream_type/{t}`	`GET /activities/{id}/streams/{stream_type}`
`GET /activities_exercise_titles/all`	`GET /activities/exercise-titles/all`
`GET /public/activities_*/activity_id/{id}/all`	`GET /public/activities/{id}/streams`, `/laps`, `/sets`, `/workout-steps` (same pattern as above)
`POST /followers/create/targetUser/{id}`	`POST /followers/users/{user_id}/followers`
`PUT /followers/accept/targetUser/{id}`	`PATCH /followers/follow-requests/{follower_id}` to accept; `DELETE` on the same path to decline a pending request
`DELETE /followers/delete/follower/...`, `/delete/following/...`	`DELETE /followers/users/{user_id}/followers/{follower_id}`
`GET /followers/user/{id}/followers/all`, `/following/all` and the `count/...` routes	`GET /followers/users/{user_id}/followers`, `/following` (paginated, counts in the envelope)
`GET /followers/user/{id}/targetUser/{t}`	`GET /followers/users/{user_id}/relationship`

Authentication routes (login, refresh, MFA) are unchanged in this release.

General changes:

Paginated list envelopes. Activity, follower, lap, set, workout-step and stream list endpoints return a paginated envelope with totals instead of bare arrays. The activity feed uses keyset pagination. Empty results are [], never null.
Nested child resources. Activity child resources (laps, sets, streams, workout steps, media, exercise titles) live under /activities/{id}/....
202 Accepted for uploads and provider refresh. File upload and Strava/Garmin refresh now return 202 and run as durable jobs. Uploads support an Idempotency-Key header.
PATCH for bulk activity edit and activity ETags.
RFC 9457 problem documents for all error responses (replaces {"detail": ...}), documented in the OpenAPI schema.
Signed, token-gated URLs for activity media, thumbnails and user profile photos. Tokens expire; fetch fresh URLs from the API rather than caching them.
Followers API reworked (new model, request/accept notifications, privacy-aware lists; unknown fields in follow decisions are rejected).
Activity responses use Read schemas; timezone formatting fields were removed from activity and lap schemas. Datetimes are normalised to UTC.
Thumbnails are now WebP, generated through the event bus and storage provider (#774). Existing PNGs are migrated.

Security

This release fixes the following advisories:

Advisory	Severity	Summary
GHSA-xqfv-wgw6-xf2x	Medium	Activity streams, laps, sets and workout steps readable by any authenticated user
GHSA-jwrv-j6hh-fw3c	Medium	Activity `hide_*` privacy flags (HR, power, ...) not enforced on API responses
GHSA-464j-c746-pg75	Medium	Rate limit bypass with an arbitrary Bearer header
GHSA-69pf-842h-jm7g	High	Route-map thumbnails and user images served without authentication
GHSA-7mf8-q8fp-38x8	Medium	Missing gear ownership check let users attach activities to another user's gear
GHSA-5pxq-cv3v-89qg	High	Strava OAuth callback allowed integration hijack via state theft
GHSA-6pc3-ww8c-x2x9	High	Missing authentication on `PUT /strava/link`

Thank you to everyone who reported these responsibly.

Additional hardening:

Visibility masking for activity metrics and private notes based on viewer ownership; visibility policy for live Strava API activities; private-profile exports enforced; owner-scoped bulk reads.
Rate limits are now keyed per caller (JWT signature verified), with new limits on file uploads and follower actions.
SSRF protection with async DNS resolution for map-tile rendering and other outbound requests; Stadia URL validation; HTTPS enforced for tile API keys.
Safer uploads: all UploadFile handling goes through a single module, user filenames are never used for paths, bulk-import rejects symlinks and is isolated per user.
Control-character and path-encoding validation on redirects; log fingerprinting for sensitive values.
Replaced bleach with nh3; migrated httpx to httpx2.
Content-Security-Policy now includes object storage origins for blobs.

Backend

Backend restructured into modules with enforced boundaries; services own transactions and publish events.
Unified activity ingestion (ActivityCore) with content-hash and dedup_key deduplication, UTC normalisation and idempotent re-imports; activity media deduplicated by content_hash.
Activity source-file storage, a pluggable file-parser registry and orphan sweeps for files, media and thumbnails.
Reverse geocoding with location backfill and defer logic for unresolved activities.
HR zone percentages are no longer stored; they are computed from your current max heart rate when streams are read.
Thumbnail regeneration with retry scheduling.
Distributed locking for scheduled jobs; user activity locking during deletion; bulk-deletion events.
Follower invariants enforced with check constraints; activity exercise-title uniqueness.
Legacy timezone aliases canonicalised.
Notifications are created idempotently from source event IDs.
Week start respected in stats (#910).
Pruning of event_log, processing_jobs and event_outbox in bounded batches.
Dependency updates: FastAPI, SQLAlchemy, Alembic, Pydantic, psycopg.

Frontend

Pending-thumbnail placeholders while map images are generated.
Feed and search updated for paginated results.
Gear component dates use calendar dates in the local timezone.
Vite PWA precache excludes the large login background image.

Docs

New API reference (OpenAPI) page under Integrate: browse every REST endpoint, parameters and responses without installing anything. It is generated from the backend code at docs build time, so it always matches the release.
New: scaling and background jobs, reverse proxy, operations, troubleshooting, export/import, error handling, API reference for Garmin/Strava, events and subscribers, activity ingestion, supported types.