piveau/hub/piveau-hub-repo 4.6.0

on GitLab

Minor release: it adds ETags for custom and DCAT resources and the configuration PIVEAU_HUB_CANONICAL_HASH_TIMEOUT, and replaces the record checksum algorithm. The first PUT of an unchanged resource after the upgrade replaces its stored checksum. Rolling back to 4.5.2 after that makes every resource touched in between do one full update (rewrite, reindex, system pipes) on its next harvest, and a rolling upgrade with mixed versions rewrites checksums back and forth until the rollout ends. Large payloads come close to the default hash budget of 5 s (the largest bundled vocabulary takes 3.9 s, and the budget is wall-clock time, so CPU contention can push borderline payloads over it): raise PIVEAU_HUB_CANONICAL_HASH_TIMEOUT for very large catalogues.

Added:

• GET and HEAD of a custom resource (/customresources/{type}/{id}, and GET /customresources/{type}/origin) send the checksum of its catalog record as a weak ETag (W/"<checksum>"), so clients can tell whether a resource changed since they read it. It identifies the stored version of the resource and is the same for every serialization. The checksum is read from the same graph as the returned content. It changes when a PUT stores different content and stays the same for an identical PUT (304). The first PUT after the upgrade that replaces an old checksum changes it once, also with an identical payload; the same holds for the ETag of DCAT resources. The response also sends Access-Control-Expose-Headers: ETag. Conditional requests (If-Match, If-None-Match) are not evaluated
  
• GET and HEAD of a DCAT resource (/resources/{resourceId}, and /catalogues/{catalogueId}/resources/origin) send a weak ETag (W/"<value>") whose value is the SHA-256 of the record checksum and dct:modified, joined by |. It identifies the stored version of the resource and is the same for every serialization. It changes when a PUT stores different content and when translations change, and stays the same when the resolve job rewrites references. The response also sends Access-Control-Expose-Headers: ETag. Conditional requests are not evaluated
  
• Configuration PIVEAU_HUB_CANONICAL_HASH_TIMEOUT: time budget in ms (default 5000, at least 1) for hashing a payload into its record checksum. A value below 1 or a non-numeric value stops the start. Hashing a vocabulary uses a fixed budget of 60 s
  

Changed:

• Record checksums are now RDF Dataset Canonicalization (RDFC-1.0) plus SHA-256, stored with spdx:algorithm spdx:checksumAlgorithm_sha256 (requires piveau-utils 12.0.21). A payload that cannot be canonicalised within the budget, or that contains RDF 1.2 triple terms, is rejected with 400 and not stored. This applies to datasets, dataset drafts, DCAT resources, custom resources and PUT /vocabularies
  
• On the first PUT after the upgrade, an unchanged dataset, DCAT resource or custom resource whose stored checksum is an old one only gets its checksum replaced: dct:modified stays, and there is no reindex, no system pipe and no Keycloak call. The response is 304 as before. The vocabulary install job only replaces the stored checksum of an unchanged vocabulary and does not reinstall it; PUT /vocabularies/{vocabularyId} stores and reindexes the vocabulary on every call, as before. Three cases still get one full update: DCAT resources that were only created and never updated, custom resources stored before 4.5.2, and, with access control v2, datasets created without a requested publication status and access level and never updated since, because their stored checksum was computed over a different model state. The last case also gets that one update without the upgrade, on its first identical PUT after creation. In that one transition PUT, a real change that collides under the old checksum is still not detected, as before
  
• Update indexing library to 1.0.37
  
• Update piveau-utils to 12.0.22. Large graph writes to Virtuoso no longer fail at random with "Connection was closed", and failed triplestore queries report the triplestore's error message instead of only the HTTP status
  

Fixed:

• The job scheduler now always starts. Before, it only started with a non-empty PIVEAU_HUB_REPO_SCHEDULER_CONFIG; without it, jobs scheduled by actions with asJob, by the event action and by the shell commands index and resolve were dropped, while the action still answered scheduled
  
• A PUT that only moved values between blank nodes, for example swapped dates of two periods, returned 304 and was not stored, because the old checksum ignored how values were spread over blank nodes
  
• A catalog record with several checksums no longer makes a PUT of a DCAT resource delete and recreate the resource. The record readers now use one row per record
  
• With access control v2, read and delete requests for a dataset that does not exist now return 404. Before, they failed with HTTP 500 ("incomplete RBAC context"). This covers /datasets/{datasetId} (also with a file extension), its record, metrics and identifier eligibility, and /catalogues/{catalogueId}/datasets/origin with an unknown originalId
  
• Polling a running index action could already report its terminal status (finished, completedWithErrors, error) while the action still held its single-job guard. A new start in that moment returned the old reference instead of starting a new job. A running action now reports running until its final result is stored. This also fixes the flaky ActionJobRegistryTest
  
• Shell command removeDups declares the dcat, dcterms and foaf prefixes in its queries. Since 4.3.1 it failed with HTTP 400 on every Virtuoso that has no dcat namespace declaration. The translation job query now also declares foaf and xsd instead of relying on the prefixes Virtuoso predefines
  
• Shell command removeDups writes triplestore errors to the shell and continues with the next catalogue
  
• Shell command removeDups binds each duplicated identifier as an escaped query parameter. Before, the identifier from the harvested metadata went into the query as it was, so an identifier with " or \ failed the check of its catalogue
  
• A shell command that fails now writes Command failed: <message> and ends. Before, the error was only logged as an unhandled coroutine exception and the shell never returned
  

Documentation:

• Action API: the event action and its finished event, which schedules the resolve job for the catalogue and is sent by piveau-consus-exporting-hub >= 8.1.0 after DCAT identifier lists. The resolve action had no description: parameters, resolved properties, results and limits (no custom resources, no reindexing)
  
• Jobs: lists the valid job types and adds a recurring resolve example
  
• Jobs and OpenAPI: removed the includeResources option from the indexing notes; it was removed in 4.5.0
  

Full changelog