ChangeLog

Unreleased

6.0.0 (2026-09-23)

Breaking: metrics v2, not compatible with v1 metrics. Catalogue scores are aggregated from the pv:finalScore values written by score 4.0.0, and the MongoDB documents and API payloads use the v2 model (metricsVersion 2.0.0); v1 data is not migrated, so clear the metrics at cutover. Requires score 4.0.0, annotator 3.0.0 and accessibility 3.0.0.

Added:

• Every entity block in a detail payload now names the resource it describes, by the URI the triplestore knows it by. distributions[].distribution and dataServices[].dataService already did; dataset.dataset is new, and the standalone-data-service payload (M§17.2) now writes the dataService field its OpenAPI schema had always declared. Catalogue documents gain info.catalogue beside info.id. The path segment these endpoints are addressed by is a portal-local slug — https-data-gov-lt-datasets-509-, not http://data.europa.eu/88u/dataset/https-data-gov-lt-datasets-509- — so a consumer joining a metrics document back to the RDF previously had to reconstruct the URI from the slug and a base URI it had to know out of band. Dataset and data-service details are built live from the triplestore, so those three fields appear as soon as the service is deployed; info.catalogue is served from Mongo and so appears per catalogue only after that catalogue's next successful refresh. Purely additive — no existing field changes name, type or value.
  

Fixed:

• The refresh sweep now sees every eligible catalogue. CatalogueInfos.sparql joined dcat:dataset into its basic graph pattern and cross-multiplied five OPTIONALs over multilingual dct:title/dct:description, so one catalogue with 25 language titles and 25 descriptions emitted 625 rows. Against staging that was 95,489 ms returning 50,000 rows — Virtuoso's default result cap, so the list was truncated after roughly 80 catalogues. The 2026-08-05 sweep refreshed 81 of 216 eligible catalogues; the other 135 had never had a metrics document written, silently and without an error. It is replaced by CatalogueList.sparql, which expresses the eligibility rules as FILTER EXISTS/NOT EXISTS and returns one row per catalogue (216 rows, 221 ms). The eligibility semantics are not identical, in two ways, though measurement shows neither changes the outcome on current data. The old rules all sat inside a single GRAPH ?g, so a catalogue's type, its dcat:dataset links and its visibility had to share one graph; the replacement uses independent graph variables and matches across graphs — both forms return the same 216 catalogues, consistent with a catalogue's triples living in its own named graph. And a catalogue asserting edp:visibility edp:hidden alongside another visibility value used to survive (OPTIONAL + !BOUND(?v) || ?v != edp:hidden yielded a second, non-hidden solution that DISTINCT kept) whereas FILTER NOT EXISTS now excludes it — currently unexercised, since only 2 catalogues carry edp:visibility at all, both assert hidden, and none carries a second value. Both changes are intended; they are recorded because they are real differences that could surface on other data, not because they alter today's result.
  
• info.spatial and info.type no longer depend on result-set order. The previous handler wrote the whole info object once per cross-product row, keeping whichever arrived last, so the stored value could differ between two refreshes of unchanged data. Repeated properties now resolve to the lexicographically smallest value.
  
• POST /admin/refresh/catalogues/{id} now answers immediately instead of blocking on a ~95 s query. It validated its path parameter by calling listCatalogues(), and only sent the 202 once that returned — so behind a proxy with a 30 s timeout the caller got a gateway timeout while the refresh had in fact started. listCatalogues() runs CatalogueInfos.sparql, which walks every catalogue under an unbound GRAPH ?g and cross-multiplies five OPTIONALs over multilingual dct:title/dct:description: measured against staging at 95,489 ms returning 50,000 rows, 50,000 being Virtuoso's default result cap, so the result was also truncated. Validation now uses a bound-graph existence check (CatalogueExists.sparql, ~140 ms). The countries and global branches were never affected — they validate against an in-memory map, or not at all.
  
• A single-catalogue refresh no longer overwrites a catalogue's stored info with null. The info cache is populated by listCatalogues(), which only a full sweep now calls, so getCatalogueInfo can legitimately return null on a process that has not swept yet; the stored value is left for the next sweep instead of being erased.
  
• The reachability endpoint returns the failing URL again, with its status code and check timestamp, under v1's field names — the URL had been absent from the response entirely since the v2 rebuild, so the MQA distributions page showed "No Access URL available" for every entry
  
• Each reachability entry is now one row per failing probed URL, not one row per (distribution, metric). As a side effect, four fields present on the older v2 shape are gone from the response: metric, the statusCodes array, method and description. Of those, dct:description — the only human-readable failure text this pipeline produces (e.g. "Not Found", or a timeout message) — is still written by piveau-metrics-accessibility's probe records; it is simply not read by either ReachabilityErrors.sparql or its handler, so it never reaches this endpoint's response
  
• The reachability endpoint has a documented response schema; it previously had none, which is how a missing field went unnoticed
  
• Rewrote all 15 scope queries ({All,Catalogue,Country} × FinalScore/Counts/{Dataset,Distribution,DataService}Coverage) to bind the entity from its scope via the annotator's dqv:hasQualityMetadata anchor, instead of searching for the version marker under an unbound GRAPH ?g. The old shape had to enumerate every measurement in the store before narrowing to a scope: against the staging triplestore it took a constant ~120 s regardless of catalogue size and exceeded Virtuoso's anytime limit, returning HTTP 206 partial results — FinalScore came back empty for every catalogue, Coverage came back with plausible-but-wrong counts. Measured on the same store, a full estat refresh (9,785 datasets / 40,200 distributions) now takes 14.6 s across all five queries where each previously took ~120 s and failed. Requires re-annotated data — see docs/DATA-CONTRACT.adoc §1.3.
  
• A refresh no longer aborts on the first failing document. collectIterative previously propagated a
  per-document failure through its composition chain, so one unreachable catalogue — or one triplestore
  query exceeding Virtuoso's anytime limit and returning HTTP 206, which TripleStore.select raises as
  a TripleStoreException — silently cost every remaining catalogue and country its update. Failures are
  now recorded and skipped, and each scope logs a summary of how many documents succeeded, how many
  failed, and the first few failing ids.
  

Changed:

• Breaking: info.title and info.description are now language maps ({"en": "…", "de": "…"}) rather than single strings, on catalogue and country documents alike. DCAT-AP allows one literal per language and the catalogue keeps all of them; the previous shape kept exactly one, chosen by result-set order. Agreed with the UI team. A literal carrying no language tag is keyed und (BCP-47 "undetermined"), verified to appear on zero existing literals so it cannot merge with real data — and deliberately not mapped to a real language, since 94 of the 96 catalogues with an untagged title also carry a tagged one that guessing would overwrite. Two catalogues (taxud, trade) have only an untagged title, so und may be the sole key; a consumer wanting one display string should try its preferred language, then und, then any remaining key. Stored documents self-heal — info is written with $set on every refresh — but between deploying and the first completed sweep, GET /catalogues returns a mix of old strings and new objects.
  
• getCatalogueInfo(id) queries the triplestore live; the catalogueInfos warm-up map is deleted. That map was populated only as a side effect of listCatalogues(), which is why a single-catalogue refresh on a process that had not yet swept could write null over stored info. The guard added for that is removed along with the coupling that required it.
  
• Counts now counts entities carrying the v2 quality-metadata anchor, i.e. the entities the annotator actually evaluated, rather than entities carrying at least one measurement inside a v2-marked graph. On the staging store the two differ by ~0.04% (entities evaluated whose every metric was excluded or unresolvable). The data-service count remains the distinct union of services embedded in the scope's distributions and services listed directly on the catalogue — counted with UNION + COUNT DISTINCT, not as two counts summed, so a service reachable both ways is counted once.
  
• v2 aggregation model: catalogue/scope score is now a single Method-1 average of
  pv:finalScore (raw 0–7.5 decimal), replacing the v1 five-dimension scores; per-metric
  coverage percentages (Dataset/Distribution/DataService blocks) replace the ~20
  per-indicator percentages. Only metrics graphs marked dct:conformsTo <https://piveau.eu/ns/metrics/2.0.0> are read (hard cutover, no v1/v2 dual read path).
  Runs on Vert.x 5. See docs/DATA-CONTRACT.adoc and docs/ADR-005-v2-aggregation-model.adoc.
  
• Logging/error-handling alignment: boot-time config redaction via allow-list (ConfigLog),
  uniform JSON error responses (400/401/403/404/409/500), dedicated piveau.access request
  logging, tracing with trace_id correlation (X-Trace-Id header + 500 body trace_id), and
  thread-safe date handling (java.time) on request and refresh paths.
  

5.8.3 (2026-03-02)

Fixed:

• Most SPARQL queries regarding ORDER BY for aggregated results for better compliancy
  

5.8.2 (2025-10-31)

Fixed:

• metrics history threw an error because of an invalid database query
  

5.8.1 (2025-04-02)

Fixed:

• Changelog update & readme typo fixes
  

5.8.0 (2025-04-02)

Added:

• Option to protect all endpoints from unauthorized access. this can be anabled with the environment variable
  PROTECT_METRICS_ENDPOINTS_WITH_AUTH
  

Changed:

• The header for using the API key was changed from Authorization to X-API-Key.
  

5.7.0 (2025-02-13)

Changed:

• Counting the datasets instead of the values in some distribution queries
  

5.6.17 (2025-02-09)

Changed:

• Admin refresh API for single refreshes (now scope is required)
  
• Overall score is calculated from the dimensions, not via SPARQL query
  

5.6.16 (2025-02-07)

Added:

• Two additional fields accessUrlAccessibility and downloadUrlAccessibility
  

Changed:

• Revert the revert again :/ But only count scored distribution
  

5.6.15 (2025-02-06)

Changed:

• Revert last change (makes no sense, it causes values with more than 100%)
  

5.6.14 (2025-02-06)

Changed:

• Percentage of status codes now based on total count of datasets
  

5.6.13 (2025-02-04)

Fixed:

• Total count when no "yes" values available
  

5.6.12 (2025-02-03)

Fixed:

• NPE because of missing value in status code measurement of distributions
  

5.6.11 (2024-07-13)

Changed:

• Exclude empty catalogues from calculation
  

5.6.10 (2024-07-13)

Changed:

• Loading quartz scheduler optionally and configurable
  
• Queries for distribution measurements are now limited only to scored distributions
  

Fixed:

• Issue with issued and modified scoring
  

5.6.9 (2024-01-31)

Added:

• Missing H2 DB table file and quartz properties
  

5.6.8 (2023-12-27)

Changed:

• Refactored single dataset metrics calculation
  

5.6.7 (2023-12-09)

Changed:

• Adjusted log output levels
  
• Dump lib dependencies
  

5.6.6 (2023-11-08)

Added

• support for scheduling refresh of all
  

???? Missing changelogs

5.5.0 (2022-11-01)

Fixed:

• code generation for vertx
  

Added:

• support for linting results in distributions
  

5.4.1 (2022-11-01)

Changed:

• caching the query results for violations, with an expiry check
  

Fixed:

• Counting Query for violations and violations query itself are now using the same clauses
  
• Normalization of piveau id
  

5.4.0 (2022-07-04)

Changed:

• Moved date modified from distribution to dataset
  

5.3.6 (2022-04-27)

Changed:

• Lock for refresh, preventing from multiple runs in parallel
  

Fixed:

• Use getProperty instead of getRequiredProperty to get rid of exceptions
  

5.3.5 (2022-02-25)

Changed:

• Queries using sub selects for better performance
  

5.3.4 (2022-02-24)

Added:

• Rudiment support of Accept-Language header
  

Changed:

• Dataset metrics handling
  

Fixed:

• Bug in refresh promise complete
  

5.3.3 (2022-02-04)

Fixed:

• Issue #1957: 404 when graph was found instead of when not found
  

5.3.2 (2022-01-13)

Changed:

• MongoDB configuration
  

5.3.2 (2022-01-26)

Added

• Server in OpenApi to fix API call examples, see https://github.com/Redocly/redoc/issues/1172
  

5.3.1 (2022-01-12)

Changed:

• Lib dependencies
  

5.3.0 (2021-10-17)

Fixed:

• Hard coded dcat-ap schema context
  

Changed:

• Complete queries rework
  

5.2.1 (2021-10-15)

Changed:

• Use Vert.x 4 router builder
  
• use actual api key security handler
  

5.2.0 (2021-10-09)

Added:

• Extended DCAT-AP schema configuration
  

Removed:

• 404 error log output
  

5.1.1 (2021-09-29)

Fixed:

• Some log outputs
  

5.1.0 (2021-06-15)

Fixed:

• Jena lib update with bug fixes
  

5.0.0 (2021-06-07)

Changed:

• Query metrics with reference from record
  

Fixed:

• Required property disarmed
  

4.0.6 (2021-03-18)

Changed:

• Status error count query split in two queries
  

Fixed:

• Dataset quality calculation reliability
  
• Dataset metrics "tokenizer"
  

4.0.5 (2021-03-18)

Changed:

• Counting error codes is disarmed due to huge performance issues. Returns always 0
  

4.0.4 (2021-03-17)

Fixed:

• SPARQL queries
  

4.0.3 (2021-03-16)

Changed:

• Eventbus timeouts increased
  
• Small query optimizations
  

4.0.2 (2021-03-05)

Added:

• Average score for each dimension
  
• Dimension score fields to OpenApi
  

Changed:

• Get average scores from latest, not history anymore
  

Fixed

• Dimension scores are returned when filtering for that dimension, the score or both
  

3.1.1 (2020-10-26)

Changed:

• Return -1.0 when dqv result is empty
  

3.1.0 (2020-10-23)

Changed:

• Almost all queries to select latest or historic metrics graphs
  

3.0.2 (2020-09-17)

Fixed:

• Single metric refresh checks for valid catalogue id
  
• The current endpoint will now return the last value, when there are multiple for the last day
  

3.0.1 (2020-09-10)

Added

• Debug output
  

Removed

• Some stupid log output
  

3.0.0 (2020-09-xx)

Added

• Licence
  

Changed:

• Updated README to align with template used in other services
  

Removed

• Hot-deployment scripts
  

2.2.0 (2020-07-01)

Added:

• REST API for refreshing single catalogue
  
• REST API for deleting single catalogue metrics
  

2.1.0 (2020-06-15)

Added:

• Warnings in SHACL results
  
• Severity in SHACL violation reports
  

2.0.1 (2020-05-29)

Fixed:

• Missing time values for status code errors
  

2.0.0 (2020-05-25)

Changed:

• Almost everything, except the API itself
  

Added:

Changed:

• dateIssuedAvailability is calculated only on Datasets
  
• Apikey now must be set via environment variables
  

Removed:

Fixed:

• Queries for violations