ehrdata 0.5.0

0.5.0 🌈 Release

on Python PyPI

ehrdata 0.5.0 adds readers for event tables, MEDS and FHIR R4, annotation and aggregation of medical codes, rebin for coarser time intervals, and five real, openly downloadable datasets. It also makes the PhysioNet loaders return analysis-ready data and fixes many bugs in OMOP setup and in writing to and reading from .h5ed and .zarr files.

Added

• ehrdata.rebin aggregates the time axis of an EHRData into coarser intervals, for instance hourly into 6-hourly ones, with the aggregation_strategy of the loaders. It applies to .X and every layer with a time axis and updates the interval offsets in .tem. (#316) @Zethson
• ehrdata.io.from_events bins a table of events, one row per code recorded for a subject at a time as in the Medical Event Data Standard (MEDS), into an EHRData with a time axis. It reads pandas, pyarrow, DuckDB relations and parquet files, starts each subject's intervals at an anchor time from obs or at the subject's first event, counts the events of codes without numeric values, and returns a dense array or a sparse.COO. (#313) @Zethson
• ehrdata.io.read_meds reads a MEDS dataset: it bins the events with ehrdata.io.from_events, selects subjects by split, stores the subject splits and the static events in obs and the code descriptions in var. ehrdata.io.write_meds writes an EHRData as a MEDS dataset, with each value as an event at the start of its interval. (#314) @Zethson
• ehrdata.annotate_codes annotates variables named by codes like ICD10CM/I21.0, ICD9CM/410.01, ATC/C07AB02, or MEDS MIMIC-IV codes like DIAGNOSIS//ICD//10//I210 with their vocabulary, code, description, ICD-9-CM and ICD-10 chapter, category and diagnosis codes, and ATC levels, without any licensed tables. With an OMOP CDM database, it takes the descriptions, the ICD source codes that map to SNOMED CT and other standard concepts, and the ATC levels of drugs in other vocabularies, such as RxNorm, from the concept, concept_relationship and concept_ancestor tables. ehrdata.aggregate_codes aggregates the variables to one of these levels, for 2D and 3D data. (#317) @Zethson
• ehrdata.io.read_fhir reads FHIR R4 Bulk Data NDJSON exports, optionally compressed, and Bundles: it bins Observations, Conditions, medications, Procedures, Encounters and Immunizations with ehrdata.io.from_events, codes them as LOINC/8480-6 like OMOP vocabularies, stores times in UTC, and stores the Patient resources in obs and the code descriptions and units in var. (#315) @Zethson
• The Reading EHR data tutorial reads small synthetic examples with every reader, ehrdata.io.from_pandas, ehrdata.io.from_events, ehrdata.io.read_meds, ehrdata.io.read_fhir, ehrdata.io.read_h5ed, and ehrdata.io.read_zarr, and annotates the codes with ehrdata.annotate_codes. (#319) @Zethson
• ehrdata.dt.eicu_crd, ehrdata.dt.mimic_iv_meds, ehrdata.dt.pbcseq, ehrdata.dt.heart_failure, and ehrdata.dt.heart_disease load real, openly downloadable clinical datasets.
  The demo of the eICU Collaborative Research Database holds vital signs and laboratory measurements of 2520 ICU stays since ICU admission, the MIMIC-IV demo in MEDS all events of 100 patients since their first hospital admission, and the Mayo Clinic primary biliary cholangitis trial the repeated visits of 312 patients.
  The UCI heart failure and heart disease datasets are static cohorts of 299 and 920 patients.
  Like ehrdata.dt.physionet2012, they hold the labels of documented codes as categories, documented placeholders as missing values, binary outcomes as 0 or 1, and the start of every interval in tem["time_value"]. (#322) @Zethson
• ehrdata.io.read_meds takes anchor_code to start the intervals of each subject at its first event of this code, such as a hospital admission, instead of at its first event, which in MEDS datasets is often the birth. (#322) @Zethson
• ehrdata.harmonize_missing_values and ehrdata.infer_feature_types support sparse.COO arrays. For a numeric sparse.COO layer, harmonize_missing_values turns the implicit fill value 0 into np.nan, except for the variables passed as vars, whose 0 values stay real values. (#306) @sueoglu

Changed

• Breaking: ehrdata.dt.physionet2012 and ehrdata.dt.physionet2019 now return analysis-ready data, which changes the dataset contents. Gender (and ICUType of PhysioNet 2012) hold the labels of their documented codes as categories, such as "female" and "male", and the -1 that PhysioNet 2012 uses for missing values, for instance in Height, Survival, DiasABP, NIDiasABP and Weight, are missing values. Outcomes such as In-hospital_death and SepsisLabel stay 0 or 1. A numeric time_value column in .tem holds the start of every interval in interval_length_unit for plotting, and ehrdata.rebin keeps it. (#318) @Zethson
• ehrdata now requires anndata>=0.13.1, the first release providing everything EHRData builds on: a 3D .X, .X unified into layers[None], and pydata-sparse COO arrays in memory. The last ehrdata still tolerated anndata<0.13, while already supporting it. (#277) @eroell
• ehrdata.io.omop.setup_variables and ehrdata.io.omop.setup_interval_variables with aggregation_strategy="last" (or "first") no longer return a missing value when the last (or first) record of an interval has no value in the kept field. The aggregation now considers only the records that do carry a value, so the last (or first) observed value is kept. Consequently, records without a value no longer contribute a unit to the unit report, and is_present of the long format table marks the intervals in which a value was observed. (#298) @eroell

Fixed

• ehrdata.io.write_h5ed and ehrdata.io.write_zarr now write the datetime, timedelta and object columns of .obs, .var and .tem: datetimes and timedeltas as ISO 8601 strings, and object columns as a numeric dtype where their values allow it and as strings otherwise, with missing values staying missing. Integers and booleans among them stay exact, in a nullable dtype if values are missing, and strings next to other values stay strings, keeping codes like 007 intact. Previously, ehrdata.io.write_h5ed failed on such columns, so that the output of ehrdata.io.omop.setup_obs could not be written at all, as the OMOP tables bring datetime columns, and object columns for the fields a database read finds empty; ehrdata.io.write_zarr failed on datetime columns and wrote object columns as strings, with "None" for missing values. Nullable float columns are now written too. (#303) @eroell
• ehrdata.io.write_zarr with convert_strings_to_categoricals=True no longer converts the string columns of the written object's own .tem to categorical dtype; the conversion now happens on the copy that is written. (#303) @eroell
• ehrdata.io.write_h5ed and ehrdata.io.write_zarr now cast every object layer to a numeric dtype, or to strings if that fails, also when X is not of object dtype or is None, as for the layers ehrdata.io.omop.setup_variables builds. Previously, ehrdata.io.write_h5ed failed on such a layer if it held anything but strings, and ehrdata.io.write_zarr wrote it as strings, with "None" for missing values. Missing values of an object X or layer cast to strings are now written as "nan", which reading harmonizes to np.nan. The arrays are cast while writing, without copying the EHRData, so that the layers of one read in backed mode can be written. (#304) @eroell
• ehrdata.io.read_h5ed and ehrdata.io.read_zarr with cast_variables_to_float=True, the default, now cast the variables of each non-numeric layer to float. Previously, they silently replaced every non-numeric layer by a copy of X, and failed if X was None or a scipy sparse array. (#304) @eroell
• ehrdata.io.omop.setup_variables and ehrdata.io.omop.setup_interval_variables with aggregation_strategy="last" (or "first") are now deterministic when several records share the same person, concept and datetime. (#330) @eroell
• ehrdata.annotate_codes no longer reads names like BMI (kg/m2) as the code m2) of the vocabulary BMI (kg: only names whose part before the / has no spaces or parentheses are codes. (#324) @Zethson
• ehrdata.dt.mimic_iv_meds drops the codes without any event in the intervals, which were all-missing variables: with the defaults, it now returns 4253 instead of 7033 codes. (#325) @Zethson
• ehrdata.io.write_h5ed now writes the nullable string columns that ehrdata.annotate_codes adds to .var, as ehrdata.io.write_zarr already did. It previously failed whenever such a column was not converted to a categorical because its values are unique, for instance on subsets of a few variables. (#326) @Zethson
• ehrdata.infer_feature_types skips features without any value with a warning and leaves their feature type missing instead of raising a ValueError, so that it, and every function that infers feature types on the fly, works on data with all-missing variables. (#327) @Zethson
• Subsetting an EHRData with arrays or lists on more than one axis, such as edata[obs_indices, var_names] or edata[:, var_names, time_indices], now selects every combination of the indices, like numpy.ix_, for every layer with a time axis and for a 3D .X, also for sparse.COO and dask arrays. Previously, the time axis was applied to the observations, or the indices were paired up, so that these layers got the wrong shape and EHRData.copy failed. (#323) @Zethson
• ehrdata.io.omop.setup_variables and ehrdata.io.omop.setup_interval_variables now restrict the variables to concept_ids, which they previously ignored. (#312) @Zethson
• ehrdata.io.omop.setup_variables with enrich_var_with_unit_info=True now merges the unit information on the concept id of each variable instead of its row position, which made unit_concept_id come out as 0, 1, 2, etc. (#312) @Zethson
• ehrdata.io.omop.setup_variables and ehrdata.io.omop.setup_interval_variables no longer bin data points after the inclusive end date of the observation table chosen in ehrdata.io.omop.setup_obs, such as observation_period_end_date, which changes the results wherever intervals extended past it. (#312) @Zethson
• ehrdata.io.omop.setup_connection, and with it dataset loaders such as ehrdata.dt.gibleed_omop, now replaces existing tables instead of failing when run twice on the same connection. (#312) @Zethson
• ehrdata.io.omop.setup_variables and ehrdata.io.omop.setup_interval_variables now make every interval exactly interval_length_number interval_length_unit long, where DuckDB previously turned offsets of 30 days or more into calendar months, so that e.g. 30-day intervals from 2100-01-01 had their second one start on 2100-02-01 instead of 2100-01-31. (#312) @Zethson
• ehrdata.io.to_pandas with format="long" no longer fails for a 3D sparse.COO array, or for an index of obs or tem that has a name, such as the tem of ehrdata.dt.physionet2012 and ehrdata.dt.physionet2019. (#314) @Zethson
• ehrdata.infer_feature_types no longer fails on 3D data when printing its overview, and ehrdata.feature_type_overview takes layer and counts categories across observations and timepoints. (#311) @Zethson
• In ehrdata.io.omop.setup_variables and ehrdata.io.omop.setup_interval_variables, the units are now carried along the aggregation: from the very data point that is kept for "last"/"first", and shared by the data points that are combined otherwise. Since combining values of different units is not meaningful, an aggregation_strategy other than "last", "first" and "count" now raises a NotImplementedError if a feature has more than one unit_concept_id. Data tables without a unit_concept_id are unaffected, which in setup_interval_variables is all of them but device_exposure and dose_era. (#300) @eroell
• ehrdata.io.omop.setup_variables and ehrdata.io.omop.setup_interval_variables now mark is_present of the long format table for the intervals in which a value was observed with every aggregation_strategy, not only with "last"/"first". (#300) @eroell
• ehrdata.io.omop.setup_variables and ehrdata.io.omop.setup_interval_variables with aggregation_strategy="std" no longer fail with Catalog Error: Scalar Function with name std does not exist!. The strategy is now mapped to DuckDB's stddev_samp. (#300) @eroell
• ehrdata no longer imports the storage type aliases (H5Array, H5Group, ZarrArray, ZarrGroup) from the private anndata.compat, which dropped them in anndata 0.13.3. (#293) @eroell
• ehrdata.dt.mimic_2 now downloads the raw data from the scverse example data mirror (https://exampledata.scverse.org/ehrapy/full_cohort_data.csv) instead of physionet, which is unreachable from some sandboxed environments. The data is unchanged. (#256) @eroell @Zethson
• ehrdata.dt.ehrdata_blobs now defaults layer to None and stores the generated time series tensor in .X. Previously it always wrote to .layers["tem_data"] regardless of the layer argument, so code that omitted layer silently analyzed a static 2D snapshot instead of the intended 3D time series. Pass layer= explicitly to keep the tensor in a named layer instead. (#284) @sueoglu

Maintenance

• Support duckdb 2.0. (#333) @eroell

Full Changelog: v0.4.0...v0.5.0