kobotoolbox/kpi 2.026.30

on GitHub

KMLExport: route KML exports through KPI export tasks (#7236)

KML exports now run through KPI export tasks and no longer use the
legacy OpenRosa KML export endpoint.

TOS: allows admins to force TOS reaccpetance (#7235)

Add an admin action to require all users to reaccept the Terms of
Service.

Adds an admin action to the SitewideMessage admin. When triggered, as
long as there is a terms of service sitewide message present, all users
will be prompted to accept the TOS regardless of whether they have
already done so.

bulkProcessing: use proper xpath for bulk translation (#7207)

Fixed bulk translation requests that were failing due to incorrect
question path being sent to the backend.

bulkProcessing: display review button (#7203)

Display "Review" button in table cells for automatic transcripts and
translations that haven't been accepted yet. Works both for bulk and
single requests.

bulkProcessing: review button better navigation (#7204)

Clicking the Review button on a translation now opens directly to that
specific language instead of defaulting to the most recent one.

bulkProcessing: approve all modal (#7209)

Added "Approve all selected" button to transcript and translation column
headers for bulk-approving ASR/MT results.

bulkProcessing: add audio duration estimate to bulk transcription modal (#7177)

Adds an estimated total time for all selected submissions' audio
attachments in the bulk transcription modal.

bulkProcessing: ongoing job conflict alert (#7181)

Bulk transcription and translation now warn you when you try to process
submissions that are already being processed by another job.

bulkProcessing: poll activity logs if bulk processing is in progress (#7205)

Activity log now automatically updates bulk processing counters while
jobs are running.

bulkProcessing: already transcribed alert (#7231)

Bulk Transcription now warns correctly when selected audio files are
already transcribed, including an accurate total duration, and those
files are filtered out before processing.

bulkProcessing: near limit alert (#7232)

Adds a new Near Limit error alert to bulk transcription and bulk
translation modals, shown when a user still has quota left but not
enough to process all selected submissions.

bulkProcessing: bulk job alert nlp view (#7241)

Display alerts in Single Processing view when a bulk processing job is
already working on the same submission. Also disables conflicting UI
until job is finished.

dataTable: improve repeat group responses handling (#7226)

Data Table repeat-group handling was stabilized for nested structures:
values no longer disappear for valid nested repeat paths, andrepeat
answers are (still) shown as a simple flat comma-separated list.

longRunningMigrations: reprocess XForms wrongly tagged by 0027 (#7273)

Some forms were skipped by a background data-repair task after it hit
bugs that have since been fixed. This makes the task run over those
forms again so they get repaired.

👷 Description for instance maintainers

Earlier bugs made long-running migration 0027 tag some forms as failed
(kobo-root-uuid-failed-0027) and skip them for good, on a task timeout
or on an old submission it could not clean. Those bugs are fixed in a
separate change.

This adds a data migration that removes that tag from every affected
form and resets migrations 0027 and 0028 back to pending, so they run
again over the previously skipped forms. It runs at deploy time, before
the workers restart. No database schema change and no new table.

Deploy order matters: the 0027 fixes must be present before this
migration runs, otherwise the same forms get tagged again. Ship this
together with, or after, those fixes.

projectViews: show organization IDs in the admin organization selector (#7331)

Organizations are now shown with their ID in the Django admin, so
organizations that share the same name can be told apart.

Many organizations share a name while being completely separate teams.
Until now the Organizations selector of a Project View listed only
names, so there was no way to know which entry was the intended one.
Organizations are now displayed as Name (org_id) everywhere the Django
admin shows one: in the Organizations suggestions, in the entries
already selected, and in the other organization pickers.

qualitativeAnalysis: add QATagTracker model for QA tag autocomplete (#7195)

This PR introduces the QATagTracker model to support autocomplete
suggestions for QA tag-type questions.

This PR adds the QATagTracker model to the subsequences app as the
database foundation for the autocomplete-for-QA-tag-questions feature.
Each row represents a unique tag value used for a specific QA question
on a specific asset.

The model includes a foreign key to Asset, a question_uuid field
identifying the QA question (stored as a plain char field since QA
questions are not persisted as database objects), and a value field
holding the tag string. A unique constraint on (asset, question_uuid,
value) ensures no duplicate tags are stored per question.

qualitativeAnalysis: track QA tags on supplement save (#7199)

This PR implements tag tracking for qualTags type QA questions by
creating QATagTracker rows whenever tags are saved via the supplement
PATCH endpoint.

This PR adds the logic to populate QATagTracker on every successful
supplement save. When the manual_qual action processes a qualTags
question, SubmissionSupplement._sync_qual_tag_trackers() is called
after the supplement data is committed. It looks up the question type
from the QuestionAdvancedFeature params, and if the type is
qualTags, it bulk-creates tracker rows one per tag value using
ignore_conflicts=True so duplicate tags across submissions are
silently skipped without any additional query overhead.

qualitativeAnalysis: add list endpoint for tag trackers (#7225)

This PR adds a list endpoint for tag trackers, nested under the relevant
asset and QA question:

GET /api/v2/assets/{uid_asset}/qual-questions/{uid_qa_question}/tags/

The endpoint returns previously tracked tag values for a given QA tags
question, intended to power frontend autocomplete suggestions when
submitting answers to QA questions.

• The endpoint uses AdvancedSubmissionPermission, since only users who
  can submit answers to QA questions should be able to see this data.
• Results are filtered by asset + question_uuid and ordered by
  value.

qualitativeAnalysis: add suggestions to qualitative analysis tags questions (#7233)

The tags question in qualitative analysis now displays responses from
previous answers to the same question in a dropdown

subsequences: background polling for async Google translations (#7217)

This PR enables background polling support for asynchronous automated
Google Translate operations.

Enables allow_async=True on AutomaticGoogleTranslationAction, so
long-running translation batch jobs now reuse the same
poll_run_external_process Celery task already used for automatic
transcription, instead of relying on the user to manually re-request the
translation after a timeout.

GoogleTranslationService now returns status: in_progress (instead of
status: failed with the SYNC_RETRY_LATER_ERROR message) whenever the
batch translation job is still running or a transient Google
infrastructure error occurs while starting/polling it. The Google
operation reference is preserved in these cases so the next poll resumes
the same job rather than starting a duplicate one.

KMLExport: add simplekml to requirements (#7245)

Fixes deployment issue where the build failed due to a missing python
dependency required for KML export.

analytics: update gtag function declaration (#7318)

api: return generic messages instead of raw exception text (#7338)

Error messages shown when something can't be saved are now short and
readable instead of long technical text.

In a few places — setting up two-factor authentication, turning on
automatic transcription or translation for a question, and saving a
setting that contains JSON — KoboToolbox used to show a long, technical
error message when something went wrong. Those messages now read as a
short plain sentence, and they can be translated. The technical detail
is still recorded on the server, so support can still investigate.

auth: handle user names from IdP and metadata refactor (#7215)

Map standard SCIM name attributes (givenName, familyName, and formatted)
natively to Kobo user fields (User.first_name, User.last_name, and
ExtraUserDetail.data['name']) during SCIM provisioning.

Previously, Kobo's SCIM implementation did not natively capture standard
name payload attributes from the IdP when creating or updating users.
This resulted in new users appearing with empty full names inside the
Kobo frontend. This PR resolves the issue by intercepting the standard
SCIM core schema attributes for names and mapping them to their
appropriate Kobo storage locations:

• name.givenName → User.first_name
• name.familyName → User.last_name
• name.formatted → ExtraUserDetail.data['name']

• Refactored the core extraction logic inside
  kobo/apps/kobo_scim/utils.py. Created a helper function
  get_scim_value() that parses both nested JSON dictionaries and SCIM
  extension URN paths
• Unified Metadata Application: Updated apply_scim_user_metadata() to
  natively process standard SCIM core fields alongside the configurable
  custom metadata mappings (USER_METADATA_FIELDS). Because this utility
  handles all mapping, POST, PUT, and PATCH API requests automatically
  receive this behavior without duplicating code in the view layer
• Updated ScimUserSerializer.get_name() to fetch the formatted name
  from ExtraUserDetail so the SCIM API returns the correct data. It safely
  falls back to concatenating first_name and last_name if formatted is
  omitted

bulkProcessing: skip already-accepted versions in bulk-accept (#7208)

The bulk-accept endpoint (POST /api/v2/assets/{uid_asset}/data/supplements/bulk/) was re-stamping
_dateAccepted and counting already-accepted submissions on every call,
making accepted_count misleading. Added a guard in
BulkAcceptSerializer.accept() to skip versions where _dateAccepted
is already set.

Calling the bulk-accept endpoint multiple times for the same submission
always returned accepted_count: 1 (or more), even though no new
acceptance action was performed. The caller had no way to distinguish
"just accepted" from "already accepted".

This PR adds an early continue in the per-supplement loop when
latest_version.get(_dateAccepted) is already truthy. Already-accepted
versions are now silently skipped _dateAccepted is not modified and
the supplement is excluded from to_update, so accepted_count
correctly reflects only newly accepted records.

bulkProcessing: keep activity log progress in sync with bulk items (#7337)

This PR fixes bulk transcription and translation jobs reporting stale
progress on the project activity page.

Problem

The data table and the project activity page report bulk processing
progress from two different sources. The data table reads the bulk
action item rows directly, so it is always current. The activity page
reads a snapshot stored on the project history log, which is written
when the job starts and refreshed only as a side effect of a periodic
Celery task. That made the activity page only as fresh as the periodic
task.

Solution

The history log is now refreshed as each submission reaches a terminal
state, rather than waiting for the next poll, so the activity page and
the data table move together. Because several submissions can finish at
once and refreshing the log is a read-modify-write of a single JSON
field, the refresh takes a row lock so concurrent updates cannot be
lost.

constance: add new setting to fieldsets (#7255)

Fixes an error that was preventing saves to Constance.

constance: add new default (#7266)

Fixes a bug that was preventing constance from being updated when there
was no initial TOS update date.

formBuilder: translate entire validation logic string (#7272)

Allow validation logic description in formbuilder to be translatable.

gallery: heic images not being displayed (#7206)

HEIC images now show up in the Project → Data → Gallery view.

import: forbid bad names in import (#7227)

Forbid uploading assets with bad node names

Previously we allowed users to upload assets with spaces or other
non-xml-compliant characters in the node names, then sanitized those
names when we deployed, resulting in a mismatch between the asset
content and what was in submissions and breaking the data table in some
cases.

import: forbid duplicate names on import (#7240)

Do not allow users to import XLS files with duplicate names.

Previously, users were allowed to import XLS files with duplicate names
that were then renamed on deploy. Update the import process to no longer
allow this.

longRunningMigrations: convert legacy googlets to automatic_google_transcription in LRM 0024 (#7228)

Fixes a background data upgrade so that older automatic-transcription
results are carried into the current format instead of stopping the
upgrade for an entire server.

The background task that upgrades legacy qualitative-analysis / NLP data
stopped with an error on some servers when a project still referenced a
pre-migration transcription result. It now converts those legacy results
into the current format and finishes the upgrade.

longRunningMigrations: persist LRM 0027/0028 cursor (#7312)

Fixed a background data-repair job that could stall without making
progress after a restart.

Two background migration jobs (LRM 0027 and 0028) backfill a stable
identifier (root_uuid) for old submissions. They kept track of how far
they'd gotten with an in-memory position that reset to the very
beginning every time the job restarted, so a restart could waste a lot
of time re-checking records it had already confirmed didn't need any
work, instead of making forward progress. This position is now saved
outside the job, so a restart picks up where it left off.

myLibrary: submenu size (#7213)

Fixes "Move to" (collection) submenu to adjust size based on number of
items in it.

pairedData: return null source for deleted parent projects (#7218)

When a source project used for a connected (paired) dataset is deleted,
it is now clearly reported as deleted rather than still appearing as a
normal source.

Connected projects pull data from a "source" project. If that source
project was deleted, the paired-data endpoint kept listing it as if it
were a live source, which the connected-projects UI couldn't reliably
detect. The deleted source's source link is now returned as null,
giving the frontend a dependable signal.

parsers: preserve raw filenames beyond 255 chars (#7234)

Fixed a server error that blocked KoboCollect submissions for projects
with very long names.

Submitting a form from KoboToolbox Collect could fail with a server
error when the project name was very long. Submissions now succeed
regardless of name length, so no data is lost.

storybook: missing settings causing crash (#7263)

subsequences: make translation _dependency optional in schema (#7262)

Corrects the data API documentation so it no longer claims that every
translation always has a linked source transcript.

In the API, each translation records which transcript it was based on.
Deleted translations don't have one. The documentation wrongly listed
that link as always present — it's now marked optional, matching what
the API actually returns.

supplement: include null value for deleting automatic transcription/translation (#7239)

API-schema fix

The API reference for automatic transcription and translation now
documents that a result can be deleted by sending an empty (null) value.
This only affects the API documentation and generated API client; the
delete itself already worked.

tests: remove unused parameter (#7251)

versions: allow date_deployed to be false in version responses (#7249)

Corrects the deployment date the API reports for form versions that were
never deployed.

In a form's version history, versions that were never deployed have no
deployment date. The API already returns this correctly, but its
published type description wrongly claimed the deployment date is always
a date. This corrects the description so the value is understood as
either a date or "not deployed" — matching what the API actually
returns. There is no visible change in the app.

xlsForm: keep leading "=" in labels as text instead of a formula (#7282)

Question and choice labels that start with an equals sign (e.g. =foo)
now display and export correctly instead of turning into 0 or a
spreadsheet error.

If a label began with =, it was mistaken for a spreadsheet formula.
The deployed form showed 0 in place of the label in Enketo, and
exporting the form to XLSForm produced an error (Err:520) where the
label should be. Labels (and other cells) starting with = are now kept
as plain text everywhere.

asset: fetch permissions once in Asset detail endpoint (#7238)

Opening a single project or collection is a bit faster — the server does
less redundant work per request.

When you open a project or collection, the server was preparing its
permission information twice. It now does that once, so detail pages
load slightly faster. Nothing changes in what you see.

longRunningMigrations: avoid full index scan in LRM 0027 xform paging (#7333)

Speeds up an internal data migration that backfills missing submission
root_uuids, so it no longer times out on projects with a very large
number of submissions.

👷 Description for instance maintainers

Long-running migration 0027 (root_uuid backfill) pages through XForms
by repeatedly finding the next batch of distinct xform_ids that still
have Instance rows with a null root_uuid. That pagination query used
a plain SELECT DISTINCT xform_id ... ORDER BY xform_id LIMIT, relying
on PostgreSQL to deduplicate as it scans.

PostgreSQL has no native loose ("skip") index scan: it has to walk every
matching index entry in order, including duplicates, before it can move
on to the next distinct value. When a single XForm has millions of
Instance rows with a null root_uuid, the scan has to read all of
them just to advance past that one XForm, even though the query only
asks for 100 distinct ids.

In production this was confirmed with EXPLAIN, the planner estimated
million of rows to walk to produce the next 100 distinct ids. In
practice, the query got killed by PostgreSQL's statement_timeout
(4200s in Celery workers) with canceling statement due to statement timeout, stalling the migration on that page of XForms.

Rewrote the pagination query as a recursive CTE that seeks directly to
the next distinct xform_id at each step (xform_id > current ORDER BY xform_id LIMIT 1), instead of scanning through every duplicate row.
Verified against production data: the same lookup that used to time out
at 4200s now returns in under 1s.

Upgraded Storybook from v9 to v10 for compatibility with modern tooling
and Node 20.19+. And hopes of making it behave better.

👷 Description for instance maintainers

This upgrade requires Node 20.19.0 or newer (was 20.18.1). The .nvmrc
file has been updated, so nvm use will switch to the correct version.
Although the version bump was due to Storybook (code testing), the
version bump is project wide, so it influences the production code
building too.

frontend: use orval to generate MSW mocks (#7219)

Migrates from manually maintained API mocks to Orval-generated ones for
type-safe mocking in Storybook stories and tests. Ensures OpenAPI schema
is the single source of truth for both API types and mock data.

storybook: fix DeleteAccountBanner tests (#7276)

storybook: fix processing column story (#7287)

Pins an override version for the minimist transitive dependency to avoid
security issues from older versions pinned by older, umaintained direct
dependencies.

fonts: replace webfonts-generator (#7265)

frontend: use Links for navigation (#7257)

Updated navigation-related buttons to use route links for better UX.

projectDownloads: migrate components to functional (#7247)

projectSettings: tsify (#7190)

projectSettings: split files (#7193)

settings: standardize Constance env overrides to CONSTANCE_ prefix (#7230)

🔗 Related PRs

Part of a 3-repo Constance env-var standardization:

• kobotoolbox/kobo-install#279 — same rename in the kobo-install env
  template
• kobotoolbox/kobo-docker#373 — same rename in the kobo-docker
  default env file

Backward compatibility is preserved (see below), so these can land
independently — the deploy-repo PRs simply complete the migration to the
new names.

Standardize all Constance override env vars on a single CONSTANCE_
prefix.

The env vars that override Constance defaults used three conventions
(bare, KOBO_, CONSTANCE_). This unifies them on CONSTANCE_,
matching the existing CONSTANCE_ASR_MT_GOOGLE_* precedent.

frontend: delete leftover file (#7214)

frontend: remove leftover $ characters after #6987 (#7274)

frontend: commit msw build (#7319)

openapi: type data_sharing in Open API assetsPartialUpdate request model (#7252)

Remodeled the data_sharing property properly in AssetPatchRequest and
cleaned up boilerplate code. Also updated connectProject.tsx to use
the new geneated model.

• Replaced the drf-spectacular schema extensions with natively supported
  DRF discriminated unions using a PolymorphicProxySerializer that
  allows the backend to generate oneOf blocks from DRF serialization
  structures. Remodeled the data_sharing property properly in
  AssetPatchRequest.
• Deleted AssetCreateRequestSerializerExtension,
  AssetPatchRequestSerializerExtension, and
  BulkPayloadSerializerExtension from
  kpi/schema_extensions/v2/assets/extensions.py. This reduces
  boilerplate and prevents future disconnects between the codebase logic
  and the OpenAPI JSON mapping
• With the correct model for PatchedAssetPatchRequest (including the
  data_sharing: { enabled, fields } structure), we removed the explicit
  as any typing workaround in connectProjects.tsx

pull transifex translations for 2.026.30 (fbc8572)