This release adds several new pipeline steps—wait, compression, flattening, and send—along
with improved configuration flexibility and documentation updates.
What's Changed
New Pipeline Steps
- Wait Step — Pause the pipeline for manual data inspection and modification #73
- Creates a
{step}_waitdirectory with copied files for review - Resume with
aether pipeline continue <job-id>
- Creates a
- Flattening Step — Transform FHIR NDJSON to CSV using fhir-flattener #6 #24
- Builds ViewDefinitions from CRTDL attribute groups and lookup tables
- Outputs CSV files to
jobs/<job-id>/csv/
- Send Step — Transfer pipeline results to downstream systems #124 #125
- DSF mode: prepares DocumentReference + Binary resources for DSF transfer
- FHIR server mode: uploads resources directly to a target FHIR server
Improvements
- Zstd compression for pipeline output files #87
- Configurable levels: fastest, default, better, best
- Auto-detects compressed files on read for backward compatibility
- Configure
local_importdirectory via config file with CLI override #140 - Store ViewDefinitions in job directory for debugging #139
- Support
columnat ViewDefinition level in lookup tables #142 pipeline continuenow loops through all remaining steps #112
Documentation
- Simplify documentation for non-developer audience
- Link feature cards to their documentation pages #76
- Align contributing docs with OpenSSF best practices #85
- Add OpenSSF Best Practices badge
- Fix redundant DIMP Pseudonymization naming #83
- Update documentation to match current implementation #138
- Add dark-mode compatible logo #95
Security & CI
- Add step-security/harden-runner to all CI workflow jobs
- Apply least privilege principle to workflow permissions
- Update cosign to use
--bundleflag for keyless signing