nipreps/fmriprep 1.0.0-rc12

on GitHub

Release notes

This release is almost completely devoted to fix the memory-related issues that kept appearing in the issue tracker (#164, #225, #657, #688, #766, #799, #801, #832, #833, #836).

We found a snow-ball problem that may affect any workflow based on Nipype, and that particularly affected FMRIPREP for several reasons. The memory explosion typically occurred after a large number of nodes of the execution graph had been run, and was definitely triggered with a memory-hungry process finished at that point in time. Briefly,

1. The workflow building of FMRIPREP is heavyweight routine, and the garbage collector had some trouble to clear-up its debris.
2. With that extra memory reserved, FMRIPREP called the run() member of the main workflow which uses a multiprocessing.Pool. By default, --nthread workers were spun up using the fork system call. Subprocess generated this way allocate as much memory as the parent had allocated. Only some was free, and this meant around 350MB per worker in the case of FMRIPREP.
3. Nipype uses subprocess.Popen to call the third party software (FSL, AFNI, FreeSurfer, etc.). Again, Popen calls try to allocate as much memory as their parent (the workers, or the main thread for nodes with run_without_submitting=True).
4. The workers and the main thread kept growing slowly until the memory explosion condition happened. From that point on, memory grew linearly with running time.

Changes in this release first address two targets:

• Try to keep the main thread of FMRIPREP as lightweight as possible:
  • Isolating as much as possible the workflow creation wrapping it with a subprocess.
  • Removing run_without_submitting=True from all nodes that opened images (which are memory-hungry).
  • When --nthreads 1 use Nipype MultiProc (as if parallel processing were required) for the reasons below.
• Try to keep the workers as thin as possible:
  • Using multiprocessing in forkserver mode, which prevent subprocess from allocating as much memory as the parent.
  • Clean-up memory as often as possible (forcing garbage collector calls, explicitly deleting objects, etc.)
  • Restart the workers as often as possible (currently, after running just one node)
  • Reduce FoV of images whenever possible.

All these efforts also required some changes to Nipype, that will be released with their version 0.14.0.

CHANGES

• [ENH] Use Nipype MultiProc even for sequential execution (#856)
• [REF] More memory annotations and considerations (#816)
• [FIX] Controlling memory explosion (#854)
• [WRAPPER] Mount nipype repositories as niworkflows submodule (#834)
• [FIX] Reduce image loads in local memory (#839)
• [ENH] Always sync qforms, refactor error messaging (#851)