kobotoolbox/kpi 2.026.27

on GitHub

• api: add endpoint to return the total length of all audio (#7151)

  This PR adds a new endpoint that returns the duration of audio
  attachments, allowing the front-end to calculate total transcription
  time without downloading and processing each file.

  This PR introduces a new endpoint, POST /api/v2/assets/{asset_uid}/attachments/audio-duration/, which accepts a
  list of attachment UIDs and returns the duration of each audio file
  along with the total duration.

  A new audio_length field has been added to the Attachment model to
  cache audio durations. When a request is made:

  • If Attachment.audio_length is already populated, the stored value is
    returned directly.
  • Otherwise, the duration is extracted using ffprobe, saved to
    audio_length, and then returned in the response.

  This caching mechanism improves performance and avoids repeatedly
  processing the same files. The endpoint also returns per-attachment
  durations to support future front-end features that may need to display
  file-level duration information.

• api: remove temporary v1 usage tracking model and middleware (#6904)

  Deletes the V1UserTracker model, V1AccessLoggingMiddleware, and
  associated tests now that v1 endpoints have been decommissioned.

  As planned during the initial implementation, this PR removes the
  temporary tracking system used to identify legacy users. With the final
  removal of v1 endpoints, the V1UserTracker model, the middleware for
  intercepting deprecated paths, and the related test suite are no longer
  required.

• asset: allow owners to delete empty assets (#7090)

  Allow users to delete assets they have created if there are no
  submissions.

  Extends permission to delete an asset to a user if:

  1. They created the asset
  2. They have the manage_asset permission
  3. The asset has no submissions

• auditLogs: make log lifespan an environment setting (#7108)

  Move audit log retention configurations from Constance to an environment
  configuration.

• auditLogs: util for getting max lookback days (#7122)

• auditLogs: filter list endpoints by lookback (#7137)

  Restrict API endpoints for audit logs to only look back the number of
  days allowed by the user's subscription.

  Restricts the following endpoints:

  • audit logs
  • access logs (personal and global)
  • project history logs (project-specific and global)
    If stripe is not enabled, the number of days allowed is set based on the
    log retention settings of the server.

• auditLogs: limit export lookback (#7140)

  Limit audit logs exported according to the maximum lookback allowed to
  the user.

  Uses the same limits as the API endpoints.

• auth: add failed login attempts (#7166)

  Added application-level logging for failed login attempts and exposed
  them in the access logs UI and API

  Failed authentications are tracker similar to how successful logins are
  recorded, using audit logs. It also tracks failed logins for
  non-existent accounts to help identify automated attacks (e.g. repeated
  attempts across random usernames from a single IP).

  • For this it hooks into Django's user_login_failed signal to create
    AccessLog entries with a new AUTH_FAILED action. Safely handles
    instances without an attached HTTP request or user object
  • Updated AccessLogSerializer to expose the new action attribute. For
    non-existent accounts where the user relationship is null, it falls back
    to exposing the attempted_username stored in the metadata
  • Updated OpenAPI outputs and TypeScript models to include action. Added
    a "Status" column to the AccessLogsSection component table to clearly
    distinguish between Success and Failed logins.

• bulkProcessing: implement background execution and polling for bulk actions (#7080)

  📣 Summary
  Users can now successfully run bulk transcription and translation jobs
  in the background, with real-time progress tracking and automatic
  recovery if a network interruption occurs.

  📖 Description
  This PR implements the core background processing engine for
  SubsequenceBulkAction using Celery. It ensures that large-scale audio
  processing and translation tasks run asynchronously without blocking the
  main API threads.

  Key Architectural Highlights:

  • Job Orchestration: Added start_bulk_item_job to safely delegate
    individual submissions to the existing
    SubmissionSupplement.revise_data() flow, ensuring complete
    compatibility with existing NLP schemas.
  • Progress Polling: Implemented update_batch_status utilizing
    pessimistic database locking (select_for_update(skip_locked=True)) to
    safely calculate and update the parent job's overall progress percentage
    without triggering deadlocks.
  • Feature Auto-Provisioning: Updated the creation serializer to
    automatically provision the necessary QuestionAdvancedFeature (the
    backend column) if it does not already exist for the requested language.
  • System Resilience: Introduced a resume_stuck_bulk_actions
    watchdog task (run every 5 minutes) to automatically resume polling for
    jobs that experience worker restarts, timeouts, or pod terminations.
  • Async Guard Fix: Updated the execution guard in base.py to
    ensure long-running Google Cloud operations properly trigger the
    downstream polling mechanism without getting trapped in an infinite
    in_progress state.
  • BULK_ACTION_STATUS_POLL_INTERVAL (Default: 30 seconds): Determines
    how frequently the background task queries individual child submissions
    to compute the overall parent batch progress percentage.
  • BULK_ACTION_STUCK_THRESHOLD (Default: 300 seconds / 5 minutes):
    The expiration window after which an active bulk action job is flagged
    as "stuck" due to potential worker pod crashes or unexpected container
    restarts.

• bulkProcessing: add OpenAPI schema documentation for bulk actions APIs (#7111)

  This PR implements the OpenAPI specifications for the
  SubsequenceBulkAction ViewSet using drf-spectacular.

  This PR also adds the progress and failure_error fields to the API
  response schemas, allowing clients to track the progress of background
  bulk tasks and inspect failure details when errors occur.

• bulkProcessing: implement bulk processing history log action (#7114)

  Add ProjectHistoryLog entries for bulk transcription and translation
  jobs, including progress and final status updates.

  Bulk processing actions now appear in a ProjectHistoryLog. When a user
  starts a bulk transcription or translation job, Backend creates a single
  ProjectHistoryLog entry that tracks the job's progress and status. The
  log is updated as submissions are processed, allowing users to monitor
  bulk-processing operations directly from the project's activity history.

• bulkProcessing: hide feature based on admin setting (#7113)

  Only users with ASR/MT enabled can see or use bulk transcription and
  translation in the data table.

• bulkProcessing: handle new activity log (#7116)

  Project Activity now supports the new backend bulk-processing history
  action, with clear user-facing messages for transcription/translation
  jobs and a proper Bulk processing filter option.

• bulkProcessing: skip already-processed submissions in bulk actions (#7142)

  Previously, if any submission in a bulk action request had an existing
  result or an active conflicting job, the entire request was rejected
  with a 400. This PR changes the behavior so eligible submissions are
  processed and ineligible ones are silently skipped, with their UUIDs
  returned in skipped_uuids on the POST response. A 400 is only returned
  if every submission is ineligible.

  Previously _validate_no_existing_results and
  _validate_no_active_bulk_conflicts raised a 400 if any submission was
  ineligible. Now, 400 is only raised if all submissions are ineligible.
  The POST response includes skipped_uuids so callers know which
  submissions were excluded.

• bulkProcessing: add bulk transcription modal (#7109)

  Adds modal to allow bulk transcribing multiple audio submissions

• bulkProcessing: make bulk activity message interactive (#7153)

  Activity log now displays ongoing bulk transcription and translation
  jobs with real-time progress tracking and the ability to cancel them.

• bulkProcessing: add bulk translation modal (#7134)

  Adds modal to allow bulk translation for multiple transcripts

• bulkProcessing: in progress banner and request toast (#7156)

  Show bulk processing banner immediatelly after bulk job is created. Show
  additional activity log link for users who created jobs. Also show a
  toast on bulk job creation success.

• bulkProcessing: add toast notification when bulk translation request succeeds (#7175)

  Show toast notification when bulk translation request succeeds.

• bulkProcessing: poll for ongoing bulk action and update table (#7106)

  Data Table now keeps bulk-processing rows up to date automatically by
  polling active jobs and refreshing completed cells without needing a
  manual page refresh. Hidden behind a feature flag.

• bulkProcessing: initial compound alerts component (#7179)

  Adds a reusable bulk processing alerts system for BulkTranscriptionModal
  and BulkTranslationModal using a custom hook pattern for future
  reusability in single processing views.

• bulkProcessing: no source alert (#7183)

  📣 Summary

  Implemented "No Source" alert for bulk transcription and translation
  modals, which warns users when submissions are missing audio files
  (transcription) or transcripts (translation).

  💭 Notes

  Changes here:

  Alert evaluator

  • alertEvaluators.ts:
  • Implemented evaluateNoSource() that checks for missing audio
    attachments (transcription) or missing transcripts (translation)
  • alertEvaluators.tests.ts:
  • Added 11 test cases covering both transcription and translation
    scenarios

  👀 Preview steps

  1. Open any form with submissions that have audio questions → Project →
     Data → Table
  2. Select multiple submissions
  3. Test transcription alerts:
  4. Click header column dropdown "Transcribe selected…" option in audio
     column
  5. Expected: Alert shows "{X} submissions are missing audio file and
     will be ignored" for submissions without audio attachments
  6. Submissions with missing audio should be excluded from the count
  7. Test translation alerts:
  8. First, transcribe some submissions (so they have transcripts)
  9. Select mix of transcribed and non-transcribed submissions
  10. Click header column dropdown "Translate selected…" option in
      transcription column
  11. Expected: Alert shows "{X} submissions are missing transcription and
      will be ignored" for submissions without transcripts
  12. Only transcribed submissions should be eligible for translation
  13. Verify deleted audio attachments are treated as missing (if you
      have any test submissions with deleted attachments)
  14. Verify multiple alerts look fine
  15. Select only submissions without transcripts
  16. Click header column dropdown "Transcribe selected…" option in audio
      column
  17. Expected: Alert shows "{x} submissions are missing audio file and
      will be ignored" and below alert "No submissions to process, see alerts
      above."

• bulkProcessing: already translated alert (#7185)

  Bulk translation modal now detects and skips submissions that already
  have translations in the selected language.

• bulkProcessing: add bulk-accept endpoint for NLP results (#7176)

  This PR implements a new POST
  /api/v2/assets/{uid_asset}/data/supplements/bulk/ endpoint that
  accepts transcription or translation results for multiple submissions in
  a single request.

  Users previously had to accept ASR/MT results one submission at a time.
  This PR adds a dedicated bulk-accept endpoint so the UI can approve all
  selected submissions in a single request.

  • BulkAcceptSerializer.accept() fetches all targeted
    SubmissionSupplement rows in one query, stamps _dateAccepted on the
    latest _versions[0] entry in memory, then flushes with bulk_update,
    no per-submission ORM round-trips.
  • Skips submissions with no supplement, missing action data, or a
    null-value version (in-progress / deleted); those are excluded from
    accepted_count.

• bulkProcessing: limit reached alert (#7188)

  Bulk transcription and translation modals now show an error alert when
  users have reached their quota limit (0 minutes/characters remaining).

• deleteAsset: migrate asset delete modal to Mantine (#7110)

  Deleting a project now uses the new Mantine ported confirmation modal
  flow while keeping the same safety checks as before.

• deleteAsset: handle API errors gracefully (#7191)

  API errors are handled properly now when deleting an asset.

• deleteProject: refactor single and multiple project deletion process (#7159)

  This PR enables the delete button quick action for MMO members, refactor
  the existing delete prompt modal to use mantine components and adds the
  checks and a blocker modal to inform the user if the selected project(s)
  cannot be deleted.

• frontend: mantineify AssetTagsModal (#7115)

  Modernize looks of "Edit tags" modal in My Library.

• frontend: enable filtering for hidden question type (#7123)

  Make it possible to filter hidden questions in Project → Data → Table.

• frontend: add Mantine notifications (#7120)

  Enabled Mantine notifications across the app and Storybook, and added a
  Design System story that demonstrates triggering a notification with
  notifications.show.

• frontend: introduce kobo-themed mantine focus styles (#7129)

  Improved keyboard focus visibility across Kobo by replacing inconsistent
  focus styling with a shared Kobo-themed focus ring for Mantine and
  legacy UI controls.

• frontend: update button and pill (#7152)

  Added Pill component wrapper with gray-light and amber-light variants,
  improved Button stories organization, and updated amber color palette.

• map: improve settings modal tabs logic (#7117)

  Map display settings modal now has a consistent and tested tab display
  behaviour. The order of the tabs is now enforced and kept the same.

• projectSettings: enable delete button for MMO members (#7192)

  Now the delete project button in project settnigs is always visible to
  MMO members

• projectViews: creating Project Views with organizations besides countries (#7138)

  Added the capability for server admins to define Project Views based on
  organization uid's (Teams), while continuing to support the existing
  country-based filtering approach.

  • Introduced uid_organizations field to ProjectView to store
    comma-separated organization uids, defaulting to * (that represents "all
    teams") and added this field to the serializer class, the admin list
    display
  • Implemented get_uid_organizations() method for processing and
    stripping white spaces from the UID list.
  • Updated get_project_view_user_permissions_for_asset logic to include
    an intersecting query matching the asset owner's organization against
    the ProjectView.uid_organizations list.
  • Refactored _get_regional_queryset to evaluate and filter the query
    sets depending on whether the asset owner or the target user is a member
    of any of the specified organizations (using
    organizations_organization). If uid_organizations contains *, it behaves
    like the original logic without enforcing organizational filtering
  • Included basic unit tests for project views filters and permissions
    code

• search: switch relational-lookup validation from blacklist to allowlist for q search (#7243)

  Implemented the allowlist approach for the q search endpoint to replace
  the old blacklist method plus super users bypass the allowlist

  Changes included in this PR:

  • Added ALLOWED_LOOKUP_FIELDS and DENIED_LOOKUP_FIELDS to
    kpi/constants.py mapping out which explicitly allowed fields are
    available per model (used Gemini to find this list and included the
    prompt used)
  • Updated QueryParseActions._validate_field inside
    kpi/utils/query_parser/query_parser.py to stop using the previous
    blacklist and solely rely on the new allowlist mapping
  • Kept the native JSONField validation and Django transform parsing,
    meaning validation stops appropriately at a JSONField (leaving nested
    keys unprocessed for relation logic) and implicit lookup filters like
    __icontains bypass relationship checks
  • Extended the parse() method to accept a user argument and updated
    kpi/filters.py to pass request.user downstream to allow superusers
    to bypass the allowlist checks

• subsequences: add pendingReview flag to _supplementalDetails for unaccepted NLP results (#7182)

  Updates the data output transformation for the
  /api/v2/assets/{asset-uid}/data endpoint to surface a pendingReview: true flag in _supplementalDetails when a transcription or translation
  is completed but awaiting manual acceptance.

  Previously, the endpoint only returned data if it had a _dateAccepted
  timestamp. Because of this, the frontend couldn't detect when an
  automatic transcription was finished and ready for review.

  Additionally, this fixes a precedence issue: if a user generated a new
  auto-transcription on top of an older, accepted manual transcription,
  the API would still serve the old manual version. Now, the newest action
  takes precedence, accurately surfacing the pending review status so the
  UI can display the "Review" button.

  What changed:

  • Updated transform_data_for_output within the transcription and
    translation mixins.
  • The data endpoint now evaluates the most recently created version of
    an action, rather than just the most recently accepted one.
  • If the latest version lacks a _dateAccepted, the API returns
    pendingReview: True and omits the text value.
  • Retained the check to ensure that if the latest version is deleted
    (where value is explicitly None), it correctly returns an empty object.

• users: add last_project_activity field and backfill LRM (#7167)

  Internal only — no user-visible change.

  👷 Description for instance maintainers

  A new last_project_activity field on ExtraUserDetail tracks the most
  recent project-related activity for each user, including data
  submissions, submission edits, and asset creation or modification. The
  field is updated automatically on each relevant action and is throttled
  (default: 1 hour) to avoid write storms.

  A long-running migration (LRM 0026) backfills the field for existing
  users. Users with a recent login are skipped as they are already
  classified as active.

  This lays the groundwork for optimizing mass email filters and the
  Django admin user filters in the next release.

• assets: fix 500 when parent uid is wrong (#7168)

  Return an empty list instead of an error when an invalid parent id is
  passed to the asset list filter query.

• assets: do not create version on all changes (#7242)

  Only add entries to form history when the form content or name changes.

• bulkProcessing: update copy and fix aggressive css (#7186)

• bulkProcessing: fix bulk action re-processing after deletion and language switching (#7184)

  This PR fixes two bugs in the bulk transcription and translation feature
  that prevented users from re-processing audio submissions after they had
  already interacted with them.

  (): When a user switched the interface language and then tried
  to run a bulk transcription or translation on their submissions, the
  entire batch would fail with an error. This happened because the
  language-switching caused an internal mismatch that broke the bulk
  processing request before it even started.

  (): When a user transcribed a submission, then deleted that
  transcription (via the "Discard" button in the Single Processing view),
  and then tried to transcribe the same submission again using the same
  language, they would get a "400 - All submissions are already processed
  or currently being processed" error. The system was not correctly
  recognising that the previous transcription had been deleted and was
  treating the submission as if it was still processed.

  What was fixed? Both bugs have been resolved so that users can now:

  • Switch languages freely without disrupting bulk processing requests.
  • Delete a transcription and immediately re-transcribe the same
    submission in the same language without errors.

• dataTable: bulk processing stories not working (#7130)

• dataTable: filter input losing focus (#7148)

  Fixed filter inputs in the Data Table losing focus while typing, which
  caused keystrokes to be lost after query completion.

• editableForm: stale closure and cache mutability issues (#7139)

  Fixed two bugs in Form Builder: event listeners weren't cleaning up on
  unmount (memory leak), and the react-query cache was exposed to direct
  mutation from Backbone code.

• formMedia: improve multiple pending uploads handling (#7095)

  Form Media now keeps the loading indicator visible until all dropped
  files finish uploading, reducing confusing false-failure moments during
  multi-file uploads.

  This PR continues the Form Media modernization and fixes the key UX bug
  from .

  Previously, when users dropped multiple files, the loading spinner could
  disappear after early files finished, even while large files were still
  uploading. That made it look like upload had stopped, and users could
  retry too early and hit misleading duplicate-file errors.

  Now the spinner remains visible until every in-flight file upload
  settles.
  The PR also includes stronger story coverage for the multi-file case and
  a small cleanup pass to make behavior and tests more reliable.

• formbuilder: duplicate list name (#7158)

  Fixed a bug where adding the same select question from the Library
  multiple times made all copies share one choice list, so editing one
  changed all the others.

• library: use full metadata flow when creating project from template (#7161)

  Fixed bug where creating a project from a template in My Library skipped
  the metadata prompt, leaving required fields blank.

• map: include start-geopoint in Project → Data → Map UI (#7136)

  Map now shows submissions that only have start-geopoint data. Before
  this, the map stayed empty if your project collected locations via
  start-geopoint but didn't have any geopoint questions.

• mfa: enforce 2FA when an authenticated user logs in as another account (#7223)

  Signing in with a second account's credentials while already logged in
  now signs you into that account (prompting for 2FA when enabled) instead
  of silently ignoring the new credentials.

  Previously, if you were already logged in and submitted a different
  account's credentials on the login form, the new login was silently
  discarded. You stayed logged in as your original account and never saw a
  2FA prompt. Now the stale session is dropped so the new login completes
  normally, including its two-factor challenge.

• myLibrary: popover menu items not working 2nd time (#7163)

  Fix issue with My Library "…" ("More actions") dropdown options not
  working after trying to use them 2nd time.

• nlp: remove region selector from translation step (#7174)

  Remove region selector from the translate step in the single processing
  view.

• nlp: add configurable Google STT language code overrides and improve batch error handling (#7248)

  Adds configurable language code overrides for Google Speech-to-Text v2
  (e.g. mapping Swahili to auto) and improves batch transcription error
  handling by surfacing per-file Google API errors instead of generic
  missing-output failures.

  Problem

  Google Speech-to-Text v2 currently fails to transcribe Swahili when
  using the standard language codes (sw or sw-KE), even though the
  language is supported. However, using auto allows Google to correctly
  detect and transcribe the audio.

  Additionally, batch transcription failures could sometimes be reported
  as "No transcription output written to GCS" even though Google had
  already returned a more specific per-file error.

  Solution

  This PR introduces configurable language code overrides for Google
  Speech-to-Text and improves error handling for batch transcription
  responses.

  Changes

  • Added a configurable ASR_LANGUAGE_CODE_OVERRIDES Constance setting
    with environment variable support.
  • Added parsing for comma-separated language mappings (e.g.
    sw:auto,sw-KE:auto).
  • Applied language overrides only within the Google Speech-to-Text
    integration before sending requests to Google, keeping Kobo language
    definitions unchanged.
  • Added informational logging whenever a configured language override is
    applied.
  • Improved Speech-to-Text v2 batch error handling by detecting per-file
    errors returned by Google before attempting to read output files from
    Cloud Storage.
  • Added clearer logging and user-facing error messages for per-file
    transcription failures.

  👷 Description for instance maintainers

  This PR introduces a default language-code override for Google
  Speech-to-Text:

  • sw -> auto
  • sw-KE -> auto

  This workaround is enabled by default because Google Speech-to-Text v2
  currently fails to transcribe Swahili when explicitly using sw or
  sw-KE, while automatic language detection (auto) succeeds.

  The default can be overridden or disabled by setting the
  CONSTANCE_ASR_LANGUAGE_CODE_OVERRIDES environment variable (or by
  updating the Constance setting after deployment).

  This workaround is intended to be temporary and should be removed once
  Google resolves the underlying Speech-to-Text issue.

• reports: general color override doesn't work for single report with custom type (#7118)

  Fixed report color behavior so chart colors update consistently across
  chart types and still inherit global color changes after per-question
  chart-type overrides.

• reports: disable restricted report actions (#7162)

  Disables custom report buttons when the user doesn't have the correct
  permissions.

• storybook: unreliable asset tags modal test (#7149)

  Fixes AssetTagsModal storybook test by removing check for modal being
  removed from the dom.

• submissions: reject submissions with invalid xml character (#6385)

  Reject form submissions containing invalid or invisible XML characters,
  to prevent data corruption when responses are later viewed or edited.

  Some clients can submit XML content containing characters outside the
  range permitted by the XML standard (invisible control characters,
  unpaired surrogates, etc.). These submissions could previously be stored
  as-is, only to later fail when opened or edited in Enketo, risking data
  loss or broken records.

  This adds server-side validation (validate_xml_chars) that rejects such
  submissions upfront with a clear 400 error message, instead of letting
  them silently corrupt data down the line. Only characters permitted by
  the XML standard are accepted.

  This validation is kept as a safety net for any client (not just Enketo)
  that might still submit disallowed characters, now that Enketo itself
  has been fixed to strip them before submission.

• supplement: add UUID query fallback to supplement view (#7222)

  Use the fallback query in the supplement view to avoid throwing a 404
  error for submissions that didn't get the rootUuid backfilled in mongo.

  In kpi/views/v2/data.py, the method DataViewSet.supplement I
  replaced the strict meta/rootUuid MongoDB query with the method
  _get_submission_by_id_or_root_uuid that searches both meta/rootUuid
  and meta/instanceID as a fallback. Also added logic to safely inject
  the matched uuid into submission['meta/rootUuid'] if it was missing.

• tests: fix failing google transcribe test (#7160)

  Fix failing google transcribe test

• migration conflicts (7cf61a5)

• frontend: make FormMedia tests more stable (#7112)
• subsequences: fix bad Constance setting after merge (3694aaf)

• assetTagsModal: reorganise files (#7132)
• assetTagsModal: make stories more stable (#7146)
• deleteAccountBanner: make stories more stable (#7144)
• formLanguagesManager: reorganise files (#7133)
• formMap: change storybook example coordinates to make test more robust (#7196)
• formMap: hide map tiles to avoid uncontrollable chromatic test failures (#7198)
• formMedia: tsify, functional component, add stories (#7093)
• formMedia: orvalify component (#7135)
• frontend: remove PopoverMenu in favor of Mantine Menu (#7165)
• libraryUploadModal: reorganise files (#7131)
• massEmails: simplify inactive/active user queries (#7170)

  Internal only — no user-visible change.

  👷 Description for instance maintainers

  Builds on #7167 (adds ExtraUserDetail.last_project_activity) and
  simplifies the mass email and Django admin user activity queries to use
  that field directly, replacing the previous multi-table queries on
  Instance and Asset that could time out on large deployments.

  The get_inactive_users and get_active_users functions now each issue
  a single query using last_project_activity, which is kept up to date
  automatically on every submission, submission edit, asset creation, and
  asset modification, for both the actor and the form owner.

• migrations: merge bulkaction and google region migrations (2ee111d)

• SSO Provisioning (#7141)

This branch contains all the code contributions to the Automatic SSO
Provisioning (account creation) project.