kobotoolbox/kpi 2.026.37b

on GitHub

attachments: stop storage counters from drifting and going negative (#7680)

Storage usage no longer goes below zero or drifts away from the files
actually stored.

Some accounts showed a negative storage usage, which gave them more room
than their plan allows. Others counted more or less than what they
really store. This happened when files were deleted, restored, or moved
to the trash at the same time as other changes, when a submission was
edited, when a project was emptied from the trash, or when a project was
transferred to another user. Storage usage is now always computed from
the projects themselves, and every change counts each file exactly once,
even when several happen at the same time.

👷 Description for instance maintainers

• The weekly sync-storage-counters task is removed, and a migration
  deletes it from the Celery Beat schedule. It compared the user profile
  counter with the sum of the user's projects; that profile counter is not
  updated anymore, so the task would have found every profile out of sync
  and suspended submissions while recounting them.
• Do not run update_attachment_storage_bytes (directly, or through
  soft_delete_orphan_attachments_2_023_37c_fix) until the follow-up PR,
  which rewrites it: the current version still resets and recounts
  profiles, with submissions suspended.
• Counters that are already wrong are not repaired here. A follow-up PR
  adds a long-running migration that recounts them.

openRosa: lock catch-all counters in date order (#7674)

Permanently deleting several projects of the same account at the same
time no longer fails with a database deadlock.

When several projects of the same account were permanently deleted at
the same time, for example after emptying the trash, some of those
deletions could fail and had to be retried later. They now go through
without getting in each other's way.

trashBin: throttle attachment auto-deletion and stop locking the Celery Beat schedule row (#7663)

This PR stops trash bin operations from locking Celery Beat's schedule
row, makes attachment auto-deletion run in small batches, and keeps
storage counters and trash clean-up correct under load.

👷 Description for instance maintainers

New optional settings (environment variables):

• AUTO_DELETE_ATTACHMENTS_MAX_PER_USER (default 100)
• AUTO_DELETE_ATTACHMENTS_USERS_PER_RUN (default 5)
  Each trashed attachment keeps its own Beat task until it is
  hard-deleted, so raising these values grows the number of tasks Celery
  Beat has to scan.
  Migration logger.0055 adds a partial index on logger_attachment.
  When SKIP_HEAVY_MIGRATIONS is set, create it manually with the SQL the
  migration prints.
  Trash bin tasks may now start up to 5 minutes after their due time.

Improve unit tests in parallel mode.