Release notes for sbmlsim 0.12.0

A release of the evaluation of a fit and of the first milestone of the uncertainty quantification: acceptance criteria next to the metrics, the evaluation by subsets of the data, one figure per fit mapping and better Bland-Altman plots; uncertain values with covariances, their propagation through simulations and probability statements about the predictions which do not depend on the number of draws. Every error of a fit is a standard deviation now, which changes the weights of data reported with a standard error.

Features

• Acceptance criteria of a fit (#325, #326, #365). The new module sbmlsim.fit.evaluation has the criteria of a data point, Fold(fold) (DV/fold <= IPRED <= DV*fold, for the points with a positive value and prediction) and Uncertainty(k) (|DV - IPRED| <= k SD, for the points with a standard deviation), the fold errors gmfe (10**mean(|log10(IPRED/DV)|)) and afe (the bias as a fold factor) and criteria_metrics. The metrics of a fit (summary_df, mappings_df, metrics.tsv, metrics_mappings.tsv) have the columns GMFE, AFE, within_1.25fold, within_1.5fold, within_1sd and within_2sd after the existing ones; FitMetrics.observables_df and metrics_observables.tsv are the metrics per kind and observable with the median standard deviation of its data, so the RMSE of an observable is read against the uncertainty of its data. The criterion of Guest et al. (2011) for ratios, e.g. the AUC ratios of drug-drug interactions, is guest_limits and within_guest as published (delta = 1); it is not applied to the absolute data points of a fit, where its limits would depend on the unit.
• The data points of a fit carry their observable, standard deviation, count and metadata (#356, #365). FitMetrics.datapoints_df and datapoints.tsv append the columns observable, observation (timecourse, value per simulation, values over a dimension), DV_SD, N (the count of the subjects) and one column per field of the metadata of the mappings (OptimizationProblem.mapping_metadata), empty where a mapping has no such field.
• The evaluation by subsets of the data (#356, #367). FitReport(groupings=[Grouping.by("tissue")]) groups the data points by the values of columns, e.g. the tissue or the health of the metadata, Grouping.of_subsets(name, Subset(label, filter), ...) by named subsets of the fit mappings selected with the filters a fit selects its data with. Every grouping adds a goodness of fit and a Bland-Altman figure with a row per group and a column per kind, a figure of the criteria per group and kind, metrics_groups.tsv and a section "Evaluation" of the report. FitDefinition.groupings sets the groupings of the reports of a fit, --group-by COLUMNS of fit_cli, report_cli and identifiability_cli replaces them.
• One figure per fit mapping (#356, #366). FitReport.plot_mapping draws the data and the simulation of every parameter set, linear and logarithmic, with the prediction at the data points as open markers, and the goodness of fit and the Bland-Altman plot of the points of the mapping, drawn as the summary figures draw them, with the metrics of the mapping (the number of points, GMFE, AFE, the percentages within twofold and within two standard deviations and the cost of the mapping with its share of the cost of the training data). The title has the kind of every mapping, training too, and below it the observable and the metadata of the mapping.
• Better Bland-Altman plots (#325, #368). The lines of twofold, ±log10(2), and the agreement of the points of a panel itself with the 95% confidence intervals of its bias and limits (Bland and Altman 1999), subset_agreement, for a panel of at least ten points: the validation panel, the panels of a grouping and the figure of a mapping. FitReport(error_bars=True) draws the standard deviations of the data, FitReport(color_by="tissue") colors the points by a column instead of their study. The box of a goodness of fit has GMFE, AFE and the percentages within twofold and within two standard deviations.
• The optimization level of the JIT is a setting (#331, #361). A model is compiled with LLJIT at a level of JitOptimization, O0 to O3: jit_optimization= of RoadrunnerSBMLModel, ExperimentRunner and OptimizationProblem, else the environment variable SBMLSIM_JIT_OPTIMIZATION. O2 stays the default; O0 compiles about four times faster (the dulaglutide model of pkdb_models in 1.1 s against 4.1 s), simulates up to a quarter slower and gives the results of MCJIT bitwise, e.g. for the runs of experiments which compile more than they simulate. The workers of a scan and of a fit compile at the level of the parent.
• The evaluation by subsets of the data, completed (#369). The metrics of a group have the denominators of their percentages, n_ratio and n_sd, and no RMSE over the units of several observables; the values which are not reported, empty or NR, are one group not reported; the figures of a grouping of many groups are pages of at most six rows; the section "Evaluation" has a tab per grouping with the headers of the criteria; FitReport checks the columns of its groupings and of color_by when it is created; a field of the metadata named like a column of the data points raises; within_guest checks its ratios.
• Uncertainty quantification, first milestone (#357, #190, #358, #375, #376, #377, #378, #379). sbmlsim.uncertainty: Uncertain values (a value with its SD, CV, range, interval or distribution; a positive value given with SD or CV is log-normal by default) wherever a change takes a value and as the prior of a fit parameter, simulated at their nominal values; Covariance (matrices and correlations, validated) for the dependence between them; propagate runs them through a simulation or scenarios with common random numbers on the scan core, with correlated quasi-random draws and an adaptive number of draws (GUM Supplement 1), and gives the summaries with their Monte Carlo errors, the contrasts between scenarios and the contributions of the uncertain values; probability statements about the predictions which do not depend on the number of draws (anytime-valid confidence sequences; decisions accepted, rejected, equivalent or undecided which more draws never reverse), several of them decided at the level of Bonferroni, and hypotheses which a propagation draws until they are decided. The page Uncertainty and examples/uncertainty_midazolam.py show it. The priors of a fit do not enter the optimization yet (MAP, #190), and tests of simulated studies are #380.
• Bands, quantiles and log axes for the steps of the integrator (#364). A band of a ragged scan, whose simulations output the steps of their integrator, is drawn: every draw is interpolated onto the time points of the figure (plot_times, PLOT_POINTS 500 points, dense where the integrator stepped densely, and the times of the changes), where a band of a ragged scan raised before. The quantiles of a band sort once (nan_quantile) instead of numpy.nanquantile, which took 87 of the 94 s of a sensitivity experiment of midazolam. A log axis shows at most LOG_DECADES (6) decades below its largest value (log_limits, both backends, limit_log_axes for figures_mpl), so the tiny first steps of the integrator do not stretch it over dozens of decades.

Fixes

• The margins of a figure do not grow with its number of panels (#323, #363). The margins of a matplotlib figure were the fractions of matplotlib of the whole figure while the panels have a size in inches: a figure of six rows had 3.3 in below its last row. They are the margins of a figure of one panel now.
• The labels of a figure saved without a tight box are not cut off (#362, #381). The margins of matplotlib are sized for its fonts of 10 pt, the label of the x axis of the fonts of sbmlsim ended below the figure, which figures_mpl(), a savefig without bbox_inches="tight" and show_mpl_figures cut off. A figure grows by what the ticks and labels of its panels need beyond its margins, at least 0.1 in; the panels keep their size. A label longer than its axis is wrapped at its words.
• The links of a report have slashes on Windows (#359). The links of the models, the code, the datasets and the figures of the page of an experiment had the backslashes of a path of Windows; they are paths with slashes, an absolute one a file URL.
• An error which the weights fill is no standard deviation of its point (#368). A zero or missing error of a data point is filled with the largest error of its mapping for the weights of the fit, as before; DV_SD, the criteria of the standard deviation and the error bars of the figures leave it out (OptimizationProblem.y_errors_measured).
• The results of LLJIT at DEFAULT are not the ones of MCJIT bitwise. The notes of 0.11.0 said so; they differ in the last bits, so a fit can take another path, e.g. the best cost of the hctz fit of the PK data 59.337223 against 59.337767. JitOptimization.O0 gives the results of MCJIT bitwise.
• No deprecated calls of pandas 3.1. The inplace keyword of DataFrame.rename and drop and float_precision="round_trip" of read_csv are deprecated in pandas 3.1; the comparison of results raised its warning. The tables of the PEtab SciML test suite are read with the parser of python, which reads every number as it was written.

Changes

• Every error of a fit is a standard deviation (#365). The problem keeps the count of every data point (FitData(count=...), an integer or a column of the dataset, now per row, OptimizationProblem.y_counts) and converts a standard error to a standard deviation, SE * sqrt(count); y_errors are standard deviations and y_errors_type is "SD" or None. The fits of data reported with a standard error change: the weights of ERROR_WEIGHTING, the default noise model of the PEtab likelihood and the error type of the PEtab export use the standard deviation, which is sqrt(count) times the standard error. A data point with a standard error and without a count raises a ValueError when the problem is initialized, naming the mapping and the rows.
• The residual figure of a fit mapping is removed (#366): FitReport.plot_fit and plot_fit_residual are plot_mapping, the files fit_<experiment>_<mapping>.<format> are not written, the figure of a mapping is <experiment>_<mapping>.<format>. FitReport.mapping_title states the kind of every mapping, [training] too.
• The legend of the data of a mapping is data (mean ± SD, N = 12), with the count of the subjects, instead of reference_data ± SD.
• The datasets of a run go into the folder datasets of the experiment (#359). A run wrote every dataset as <sid>_<dataset>.tsv next to the figures and results; it writes datasets/<sid>_<dataset>.tsv below the output path of the experiment now (DATASETS_DIR, SimulationExperiment.dataset_file), and the report links them there. An experiment without datasets gets no folder.