medizininformatik-initiative/aether v1.5.2

on GitHub

This patch release reduces the memory use of the flattening step and keeps the precision of FHIR decimals. The flattening step builds one Provenance index for each Bundle, not one index for all Bundles, and keeps pending batches as raw JSON. For large TORCH extractions, the peak heap no longer grows with the input size. The dimp and validation steps keep the original text of JSON numbers, so 1.50 stays 1.50. The release also fixes a data race on the progress bar and the duration of finished jobs in pipeline status. It updates the Go toolchain to v1.27.3, TORCH to v1.0.3, fhir-flattener to v0.1.0-alpha.10 and fhir-pseudonymizer to v3.1.2.

What's Changed

Breaking Changes

• flattening routes entries only with the Provenance of the same Bundle — The step reads each input file one time and builds the Provenance index for each Bundle. It does not build an index of all Bundles first. For 25 M references, that index used about 7 GB of memory. Now the peak heap does not change with the input size, and the step is about 37 % faster. TORCH output does not change, because TORCH writes each Provenance in the same Bundle as its targets. For input from local_import or http_import with a different layout: a resource outside a Bundle, or a resource that only a Provenance of a different Bundle targets, goes to no group. #1047 (closes #1044)

Bug Fixes

• Keep flattening batches as raw JSON to limit memory — Each pending batch is compact raw JSON, not decoded maps. The heap of a batch drops from about 7× to about 1.4× its JSON size. With the default batch_size_mb: 500, the old batch could use about 2.5 GB of heap. #1045 (closes #1043)
• Keep the text of JSON numbers in the dimp and validation steps — The decoders keep json.Number, so numbers do not go through float64. 1.50, 0.000000120, 100.0 and integers with more than 15 digits stay unchanged. #1049 (closes #1048)
• Make the progress bar counter safe for concurrent use — The counter is an atomic.Int64. Before, concurrent validation chunks caused a data race and lost increments. CI now runs the tests with -race. #1037 (closes #1036)
• Show a fixed duration for a finished job in pipeline status — For a completed, failed, stopped or waiting job, the duration is the time from the job creation to the last state write. Before, the duration increased at each query. #1014 (closes #1012)

Internal & CI

• Run only the tests that cover the mutated line in mutation testing — A go wrapper for gremlins runs only the covering tests for each mutant. make mutation-diff for one function takes 35–44 s instead of 77 s. #1021 (closes #1013)
• Keep the TORCH test data in the repository — CI does not download the test data from an external server anymore. #1017 (closes #1016)

Dependency Updates

• Go: toolchain directive v1.27.3 #1006, aws-sdk-go-v2 monorepo #987 #1023 #1035 #1039, aws-sdk-go-v2/service/s3 v1.114.4 #1018, github.com/klauspost/compress v1.20.3 #992, golang.org/x/text v0.43.0 #986, golang.org/x/vuln v1.9.0 #988
• Services: fhir-flattener v0.1.0-alpha.10 #1051, TORCH v1.0.3 #993, fhir-pseudonymizer v3.1.2 #1000 #1031, vfps v1.26.0 #990 #995 #997 #999 #1004 #1009, mii-fhir-validator 0.0.1-alpha.9 #1008 #1022, Blaze v1.12.1 #1025 #1028, postgres 18.6 digest #1011 #1019 #1046, SeaweedFS v4.51 #1030 #1032, debian:bookworm-slim digest #1010 #1041, nginx-unprivileged 1.31-alpine digest #991 #996 #1005
• Tooling: golangci-lint v2.15.0 #1002, Vale v3.25.1 #1024, step-security/harden-runner v2.22.3 #1007 #1038, github/codeql-action #998 #1015, docker/login-action #1027, ossf/scorecard-action v2.5.0 #994, fhir-ontology-generator v5.1.0 #1003
• Docs site: Node v26 #1040, Vue v3.5.44 #1001, Vite v8.3.5 #989 #1029

Full Changelog