medizininformatik-initiative/aether v0.16.0

on GitHub

This release hardens Bundle splitting in the DIMP step: the partitioner now computes the exact Bundle wrapper size, counts entry separators, and passes an empty Bundle through unchanged, so chunks no longer overshoot the size threshold. Flattening becomes more robust — aether now requests NDJSON from the fhir-flattener and builds the CSV itself, which fixes parse failures on free text, and a failed CSV write is kept as a .partial file instead of a truncated result. A new install-dev.sh installs a build from a pull request or from main in one step. CI gains Go native fuzz targets, a compatibility test against the pinned fhir-ontology-generator flattening release, and per-PR binary builds for all platforms.

What's Changed

Features

• feat: add install-dev.sh to install a PR or main build — ./install-dev.sh 663 installs the binary built for pull request 663; ./install-dev.sh main (or no argument) installs the newest main build. (closes #665) #666

Bug Fixes

• fix(dimp): compute the exact Bundle wrapper size when partitioning — the partitioner used an estimate of the wrapper size, so a chunk could exceed the configured threshold. It now measures the real serialized wrapper. (closes #659) #661
• fix(dimp): count entry separators when partitioning a Bundle — the , separators between entries were missing from the size calculation, another source of oversized chunks. (closes #649) #657
• fix(dimp): return an empty Bundle unchanged when splitting — SplitBundle no longer produces a chunk with an empty entry array for a Bundle without entries. (closes #650) #658
• fix(flattening): keep failed CSV output as .partial and report write errors — a batch that fails mid-write no longer leaves a file that looks complete; the output keeps a .partial suffix and the error is reported. (closes #669) #671
• fix(flattening): emit only one warning for a deprecated parent field — loading a lookup table with authored parent fields logged one warning per entry; it now logs one warning per file. (closes #672) #673
• fix(config): reject ISO 8601 durations that overflow time.Duration — an overflowing duration in the config now fails validation instead of wrapping around. (closes #647) #656
• fix(compression): match the .zst suffix case insensitively when stripping — a file named .ZST/.Zst is now recognized as compressed. (closes #648) #655

Internal & CI

• refactor(flattening): request NDJSON from the flattener instead of CSV — the chain had two CSV dialects (Spark CSV from fhir-flattener, RFC 4180 in aether), so free text with an embedded quote stopped the step. aether now receives NDJSON, maps values by column name, and writes the CSV once. (closes #668) #670
• test(fuzz): add Go native fuzz targets over the code that reads untrusted input. (closes #646) #651
• test(compat): add a compatibility test against the pinned fhir-ontology-generator flattening release — a new CI job loads the released flatteningLookup.json and builds a ViewDefinition for every profile in it. (closes #645) #654
• ci: build the binary for all platforms on every pull request. (closes #644) #652
• ci: run in-package tests under coverage. (closes #674) #675
• ci: set Go up in the e2e-test job. (closes #662) #663
• ci(docs-deploy): scope contents: write to the deploy job. (closes #641) #643

Dependency Updates

• Go modules: golang.org/x/text v0.39.0 #642
• Docker images: miracum/vfps v1.10.2 #653; debian:bookworm-slim #676; postgres #677
• GitHub Actions: dorny/paths-filter #678

Full Changelog