Release notes for sbmlsim 0.37.0
The engine core is the default engine of sbmlsim, and the three places which cost the most where a result leaves the simulator are gone: the netCDF file of a result, the interpolation of a ragged scan onto a grid and the conversion of its units.
Features
-
The engine core is the default engine (#608).
sbmlsim.engine.DEFAULT_ENGINEis the one place the default lives, and it isEngine.CORE: aSimulator, aSimulatorSettingsof a project, anExperimentRunnerand anOptimizationProblemwhich name no engine take the Rust SBML engine compiled to WebAssembly, which every install of sbmlsim brings (sbmlsim-core), where they took libroadrunner before. It leads every model of Benchmarks in a simulation and in a scan: on the HCTZ model one simulation takes 0.877 ms against roadrunner's 1.68 ms, a serial scan of 1000 points 0.760 s against 1.50 s and a parallel one 0.162 s against 0.237 s, and a scan runs its points on the threads of the engine in one python process instead of a pool of worker processes which loads the model again in each of them. The 40 repeats of thePKfit of the HCTZ example take 8.1 s against 14.1 s, to the same optimum.The engine integrates with the tolerance vector roadrunner computes but with its own build of CVODE, which chooses its own steps, so a result is roadrunner's within the integration and not to the last bit: the documented outputs of this release show it, 841 integrator steps of a tutorial simulation where roadrunner takes 840 and a half life of 6.837 h where roadrunner gives 6.834 h.
Simulator(engine="roadrunner")is the way back, and the differences of the four engines are in Engines.What the engine does not simulate it refuses by name (
UnsupportedModelError) instead of simulating it differently: a change before the initialization whose value is a formula the executor cannot fold without the model, a steady state after the end of a simulation, and the other gaps in the module docstring ofsbmlsim.simulator.executor_core. A model which hits one of them names roadrunner. -
The variables of a result are one array in its netCDF file (#608). HDF5 and h5netcdf cost about 2.5 ms to write and 3.4 ms to read per variable of a file whatever its size, which a result of hundreds of selections pays for in every one of them: the 190 selections of the HCTZ model over 200 points took 636 ms to write and 552 ms to read.
write_netcdfnow gathers the numeric variables of equal dimensions and dtype, fromSTACK_MIN(8) of them on, into one array over a dimension of their names, which takes 58 ms and 34 ms, 11 times faster to write and 16 times to read, and the file is a tenth smaller, since a chunk of the array holds what a chunk of a variable held. The cost was the variable and not the compression, which the same measurement without it shows (366 ms and 422 ms against 15 ms and 10 ms).A result of fewer than
STACK_MINvariables keeps them named, so a small file is still read as itself, and an array over a dimension of names is ordinary netCDF, so every reader reads a gathered file; what it does not read is one variable per selection by name.read_netcdfhands the variables back in the order they were written with their attributes, so a result read from a file is the one that was written, bitwise and variable for variable. -
A ragged scan is interpolated point by point (#608). The linear interpolation of a point onto a grid depends on the time points of that point alone, which every variable of the result shares, so
ScanResult.interpolatewalks the points and not the variables and computes the weights of a point once instead of once per variable: a scan of 200 points of the HCTZ model with its 191 variables on its integrator steps takes 407 ms against 3849 ms, 9.5 times faster, and holds one set of weights at a time whatever the number of points. The values are the ones ofsbmlsim.result.timecourse.interpolateof every point and variable on its own, bitwise. -
ScanResult.toassigns its conversions at once (#608). The converted variables and coordinates are assigned in one call each instead of one key at a time, which a dataset of many variables pays for in every one of them: 190 conversions take 1.7 ms against 94.9 ms, 56 times faster. -
A fit in the engine tests its step and its cost (#608, sbmlsim-core#57). The optimizer of the engine took the tolerance of its gradient and had no default for the tolerance of its step and of the change of its cost, the
xtolandftolofscipy.optimize.least_squares, so a run tested neither and spent its last iterations on the noise of the integrator: thePKfit of the HCTZ example took 1800 evaluations where the optimizer in python took 796.gaps_of_a_fitnow passes both, from the arguments of the fit or else fromdefault_toleranceof the relative tolerance of the integrator as a fit in python gets them, and the same fit takes 853 evaluations, the optimum unchanged. -
A formula before the initialization is folded for any target (#608). The executor of the engine core folded a formula of a parameter only and refused one of a compartment or of a species, so a PEtab v2 condition which estimates a concentration, e.g.
[S] = S0, could not be fitted on the engine. The fold gives the number the formula has, and the engine then applies that number with the bookkeeping of the volumes and the amounts of its initialization, exactly as it applies a number the pre-initialization states for the same target, so the answer is roadrunner's; a formula which reads a symbol the pre-initialization sets as anything but a parameter stays refused, since the bookkeeping and not the pre-initialization gives such a symbol its value. -
The service shares the run of a panel on the engine core (#608).
serve.runner.SHARED_ENGINESis roadrunner and the engine core: the integrator of each takes its steps whatever a scan selects, measured on a grid and on the steps of the integrator, so the panels of a page which share aPanelSimulationshare one run and an answer read out of it is the answer of a run of the panel alone, bitwise. Without it every panel of a page would have run its own scan once the engine core became the default. -
ScanResult.summaryreduces the mean and the standard deviation once (#608).cvis the standard deviation over the mean and reduced both again, so the default statistics reduced the dataset seven times for five statistics. They are reduced once andcvdivides them: the default summary of 190 variables over 200 points is 1.6 to 1.8 times faster and its values are identical.
Fixes
-
A profile of a fit of one parameter (#608). Every point of such a profile holds the one parameter of the fit and has none left to fit again, which the optimizer of the engine refuses;
gaps_of_the_profilesreports it as the gapparameters, so the analysis runs in python, which evaluates the cost at the point. A fit of one parameter is the usual shape of a toy problem and of a test. -
A project passes the engine of its simulator to its fits when it differs from the default engine and not when it differs from roadrunner (
Project.load_fit_definitions), so aFitDefinitionwithout an engine stays without one on a project of the default engine.
Fixes of the engine core
-
The bindings outlive the thread which imported them (#608). The python bindings of sbmlsim-core do not outlive the thread which imported them: a process which imported them on a thread which then exited crashed with a segmentation fault when another thread compiled a model or simulated with it, while bindings whose thread is alive are used from any thread. The service answers the panels of a model in the threads of its server, which come and go, so with the engine core as the default the first request imported them on a thread that was reaped before the next request came, and the second request killed the server.
sbmlsim.engine.import_corenow imports them on a thread which parks and lives as long as the process whenever the importing thread is not the main thread, so every later thread finds them valid;tests/model/test_model_core.py::test_the_bindings_outlive_the_thread_which_imported_themis the regression test, which exits with SIGSEGV without the fix. -
A model is compiled once however many threads ask for it (#608).
sbmlsim.model.model_core.COMPILEDwas a plain dict andruntime()afunctools.cache, so two threads which asked for one SBML both compiled it and both could create a runtime. The cache is now filled under a lock per key, so one SBML is compiled once while two different ones still compile at the same time, and the runtime is created once. -
An optional import no longer hands out a module which is still initializing (#608).
sbmlsim.engine._import_optionalreadsys.modulesand returned whatever it found; python puts a module there before it executes its body, so a thread which asked for the bindings while another thread imported them for the first time got a module whose attributes did not exist yet (AttributeError: partially initialized module 'sbmlsim_core' ... has no attribute 'CompileError'). It now returns a module ofsys.modulesonly once its spec says it is initialized, and otherwise goes throughimportlib.import_module, which holds the lock of the module. This covers COPASI and AMICI as well. -
The provenance and the methods of a fit name the default engine (#608).
fit.provenance.collect_versions,fit.provenance.fit_provenanceandfit.methods._integratordefaulted their engine to roadrunner, so a record written without one, e.g. therun.jsonof the run layer, recorded libroadrunner where the run used the engine core. They default toDEFAULT_ENGINE.
The parameter fitting of the engine core
The fit of a problem in the engine (the contract fit/1, sbmlsim.fit.fit_core) now covers everything a fit itself can express. The measurement over the PEtab v2 test suite, which is the new test tests/fit/test_petab_testsuite.py::test_the_engine_fits_the_case_as_the_problem_computes_it (one test per case, marker petab_testsuite): of its 31 cases 30 are a fit of the engine, and the residuals of every one of them are the residuals of the problem bitwise (numpy.testing.assert_array_equal, not a tolerance). The one which is not has parameters which have a prior and no bounds of their own, which no optimizer of bounds can search; that is a gap of the problem and not of the engine.
-
A measurement at the time of a change (sbmlsim-core#58, released as 0.16.0). A simulation has two states at the time of a change, the one before it and the one after it, and
data_at_changesays which a data point is compared with. The curve of a request now carries one flag per point (after_change), so the engine reads the row the fit asks for and the gapdata_at_changeis gone.CurveRequestof the engine has nodeny_unknown_fields, so a 0.15.0 engine would ignore the field and compute other residuals:reads_after_changecompares the installed bindings withAFTER_CHANGE_VERSIONand keeps the documented gap on older ones rather than sending a field they drop. -
An observable which is a formula of the selections is written into the model of the fit as a global parameter of an assignment rule (
sbmlsim.fit.observable_sbml), so the compiler of the engine reads it as it reads the math of the model, the engine selects it as an ordinary column and its forward sensitivities are the ones of a column. Nothing of the math is reimplemented and nothing is evaluated twice: the formula is written as the MathML of SBML byformula_mathmlfrom the parse tree ofcompile_formula. The model of the user is not touched; the model of the fit is loaded from its description, so it keeps the added parameters a PEtab parameter of the fit is.A selection of roadrunner is not an identifier of SBML, and a species in the math of SBML is its concentration unless it has
hasOnlySubstanceUnits, where it is its amount, so the amountSand the concentration[S]each need the size of the compartment the other way round. Only the brackets of a concentration are rewritten as text, which is unambiguous; every other symbol is carried by thenamesofformula_mathml, and a symbol which is no identifier of the model becomes a helper parameter ofS * corS / c. The placeholders of a PEtab observable become parameters of their values, a number a constant one and a formula one of a rule.tests/fit/test_observable_sbml.pychecks every row of that table against the observable of the problem bitwise, on a model which has a concentration species, a substance-only species and a compartment whose size is not 1.A placeholder whose value differs between the points of one curve makes the mapping one curve per run of consecutive points which share their values, in the order of the points, so the residuals stay the residuals of the problem in their places.
-
gaps_ofasks the engine. Its own checks are the ones of the fit, of its data and of its parameters; what is left is everything the engine says about the simulation of a condition, and the engine is the authority on that._gaps_of_the_requestbuilds the request and reports what it refuses as the gapunsupported, so a problem the engine would not simulate is fitted in python instead of failing. Three cases of the suite (0020, 0023 and 0026) did fail that way before:gaps_ofcalled them clean and the fit then raised._gaps_of_the_boundsmirrorscheck_parametersof the engine for the same reason.The cases which are not a fit of the engine are recorded with the code of their gap in
ENGINE_GAPS, so the list cannot drift. It is one case now, the one whose parameters have a prior and no bounds. -
A condition whose value is an estimated parameter (sbmlsim-core#59, released as 0.17.0). A value of a request of
simulation/1is a number, so a PEtab v2 condition which assigns an entity an estimated parameter, e.g.[B] = initial_Bor[A] = initial_A1 + initial_A2, could not be a condition of a fit: the number folded when the request was built holds that value where the fit started while the optimizer moves the parameter, which is why the executor refuses such a formula. A condition offit/1now carries those values asderived, each a linear formconstant + Σ factor * parameterwhich the engine evaluates at the values the optimizer is at and sets on its target before every simulation, as a varying target of a prepared request already is.linear_formreads the form off the formula with sympy and_derived_oftakes the assignment out of the plan; a formula which is no linear form over the parameters of the fit alone keeps its gap, andreads_derivedcompares the installed bindings withDERIVED_VERSIONasafter_changedoes, since the condition has nodeny_unknown_fieldseither.A fit whose conditions carry derived values takes the difference quotients of its optimizer for its jacobian: a derived value follows more than one parameter in general, which the one parameter per varying target of a block of the forward sensitivities cannot carry.
The test suites run on the engine core
The SBML Test Suite, the PEtab v2 test suite and the PEtab benchmark collection now run on the engine core, the default engine, and roadrunner is the reference they are measured against. They are the check of what sbmlsim simulates correctly, so they follow the engine sbmlsim simulates with. The whole benchmark collection passes, 35 of 35 problems.
-
The SBML Test Suite: 1754 of 1789 cases pass on the engine core, where roadrunner passes 1576. The 35 cases the engine core does not pass are a strict subset of the 213 roadrunner does not: there is no case which passes on roadrunner and fails on the engine core, and no case the engine reads and then simulates outside the tolerances of the suite. All 35 are the 35 cases of a fast reaction, which neither engine compiles. The 178 cases the engine core adds are 123 of an algebraic rule (SUNDIALS IDA since sbmlsim-core 0.10.0), 50 of a delay (an integrator of its own since 0.11.0) and the 5 of a
stoichiometryMathroadrunner cannot integrate.run_case(case, engine=),run_suite(..., engine=),write_submission(..., engine=)andscripts/testsuite.py --enginename the engine, the baseline and the manifest of the submission record it, andtests/data/testsuite_baseline.jsonis the baseline of the engine core. -
A pre-initialization formula reads a constant parameter of the model. The executor of the engine core folded a formula whose symbols the same pre-initialization sets to numbers, or sets none of, and refused the mixed case: a symbol it does not set could follow one it does through a rule or an initial assignment, so the value of the model as loaded would be stale. That is too strict for a constant parameter, the variable of no rule and of no initial assignment, which nothing a pre-initialization sets can move: it is folded now whatever else is set (
executor_core._constant_parameter). The parameters of a PEtab problem are such parameters, so a condition like[A] = initial_A1 + initial_A2simulates on the engine;test_preinit_formulas_read_what_they_read_in_roadrunnercompares every folded formula with roadrunner, which is what holds the fold to its values. -
A value which follows an estimated parameter is no longer folded silently (#609). Folding such a value into a number holds it where the fit started while the optimizer moves the parameter, so the engine would answer the residuals of another problem without an error. Two cases of the PEtab suite did exactly that before, where the pre-initialization set no other number and the fold read the value of the model as loaded.
_gaps_of_the_valuesnow reports every value which reads a parameter of the fit and is not carried as a derived value, so such a problem is fitted in python instead of being answered wrongly. -
A formula at a change is folded for a model with events (#610, sbmlsim-core#19, released as 0.18.0). The executor refused every formula of a change of a model with an event of its own, since a request answers the state of a time after the events of that time have fired while the executor of roadrunner reads it before: measured on a model whose event adds 10 to a state at the time 5, roadrunner folds 0.0067 and the request answered 10.0067, a factor of 1484 in that column with no error and no warning. A request now carries
before_events, which is exactly that state: the engine integrates to the time of the change and does not fire the events there, so the last row of the probe is the state roadrunner folds (probe_before_events). The refusal is gone, for any formula and any model, and the cases 0016 and 0030 of the PEtab test suite are fits of the engine.A probe which ended one unit in the last place before the time of the change was tried first and does not work: the engine fires an event of a time for an end within about 1e-9 of it, far wider than the unit in the last place CVODE resolves (measured: probes at 1e-15, 1e-14, 1e-13, 1e-12 and 1e-11 before the time all answer the state after the event, only one at 1e-7 answers the state before it), so no probe can tell the two states apart and the field had to go into the contract.
-
The steady state after the end of a simulation runs on the engine, for every model (#610, sbmlsim-core#60, released as 0.18.0). The output at the time
infwas refused outright, since thesteady_stateof a request is the pre-equilibration before the start. It is the fieldsteady_state_afterof the same request now: the engine continues from the state the timecourse of that request ended in, with the triggers of its events as the timecourse left them, and the row of the steady state is the last of the output. The whole PEtab benchmark collection now runs on the engine core, 35 of 35 problems, whereBlasi_CellSystems2016hit this gap.Carrying the triggers is what the field is for. A second request which carried the state the timecourse ended in through a pre-initialization was tried first: it initializes the model, so an event whose trigger that state makes true fires again, where roadrunner, continuing its own model, does not - case 0023 of the PEtab test suite, whose event adds 1000 to its state at the threshold its steady state is past, answered 2500 against roadrunner's 1500. The
fromof the field, the time the integration to the steady state starts at, is the end of the timecourse for a model whose math reads the time or whose events have a delay, where roadrunner continues at the clock of its model, and 0 for every other one, which is what the executor of roadrunner passes, so the two agree whatever the model. Nothing of it is refused any more, and case 0023 is a fit of the engine. -
The start of a request is clamped into its bounds. A nominal value of a PEtab problem may be outside the bounds of its parameter, which PEtab reports and answers by starting at the bound; the engine refuses such a start. A start inside its bounds is left as it is, so this changes nothing for any other problem.
-
30 of the 31 cases of the PEtab v2 test suite are a fit of the engine, each of them bitwise the residuals of the problem. The test is new in this release, and the three steps of it moved the count: 23 when the engine became the default, 27 with the derived values of a condition and 30 with the two fields above. The one case which is not has parameters with a prior and no bounds of their own, which no optimizer of bounds can search.
Known gaps of the default engine
What the engine core does not simulate it refuses by name (UnsupportedModelError) rather than simulating it differently, so these are errors and not wrong numbers. Simulator(engine="roadrunner"), SimulatorSettings(engine="roadrunner") of a project or OptimizationProblem(engine=...) is the way around each of them, and the module docstring of sbmlsim.simulator.executor_core is the full list.
- A value before the initialization which a plan sets by a formula the executor cannot evaluate without the model: a request carries numbers, and no request answers the values before the initialization, so a formula which reads a symbol that same pre-initialization sets as anything but a constant parameter stays refused.
- A formula at a change which reads a symbol that is no output of the engine, i.e. neither a parameter, a compartment nor the amount
Sor the concentration[S]of a species.compile_simulationrefuses such a formula already, so only a plan built by hand reaches it. - The selections of roadrunner which are no entity of the model, e.g. the rate of change
C'oreigenReal(C).
The two gaps the notes above describe, the steady state after the end of a simulation and the formula at a change of a model with events, are closed by sbmlsim-core 0.18.0 and are no gaps of this release.
Measured and not changed
-
The windows of the FAST intervals. The bootstrap windows are 97 to 99 % of a FAST analysis, and transforming the windows of a parameter together instead of one at a time was 0.6 times as fast: the transpose of the windows into one array costs more than the per-call overhead it saves, and the transforms of the loop are already near the minimum work.
-
The jacobian of the forward sensitivities of the engine for the speed of a Fisher information: 1.12 times the central differences of python, where the accuracy is the reason to use it and the speed is not.
-
The per-point request building of a scan of the engine core: 6.3 µs per point against 656 µs of the engine itself.
Dependencies
sbmlsim-core>=0.18.0, the release whose request ofsimulation/1carries thebefore_eventsof its end (sbmlsim-core#19) and itssteady_state_after(sbmlsim-core#60); 0.17.0 brought thederivedvalues of a condition offit/1(sbmlsim-core#59), 0.16.0 theafter_changeof its measurements (sbmlsim-core#58) and 0.15.0 thestep_toleranceand thecost_toleranceof its optimizer (sbmlsim-core#57).SimulationRequestdenies unknown fields, so an older engine raises aValueErrorfor a request of the new fields rather than dropping them silently, which is why the floor is raised instead of the fields being gated asafter_changeandderivedare.