Pydra

Pydra is a lightweight dataflow engine written in Python. Although designed to succeed Nipype in order to address the needs of the neuroimaging community, Pydra can be used for analytics in any scientific domain. Pydra facilitates the design of reproducible, scalable and robust workflows that can link diverse processing tasks implemented as shell commands or Python functions.

Key features:

• Combine diverse tasks (Python functions or shell commands) into coherent, robust workflows

• Dynamic workflow construction using Python code (see Dynamic construction)

• Concurrent execution on choice of computing platform (e.g. workstation, SLURM, SGE, Dask, etc...)

• Map-reduce-like semantics (see Splitting and combining)

• Global caching to reduce recomputation (see Caches and hashes)

• Tasks can be executed in separate software environments, e.g. containers (see Software environments)

• Strong type-checking, including file types, before execution (see Typing and file-formats)

See Design philosophy for more details on the rationale behind Pydra's design.

Installation

Pydra is implemented purely in Python and has a small number of dependencies It is easy to install via pip for Python >= 3.11 (preferably within a virtual environment):

$ pip install pydra

Pre-designed tasks are available under the pydra.tasks.* namespace. These tasks are typically implemented within separate packages that are specific to a given shell-command toolkit, such as FSL (pydra-fsl), AFNI (pydra-afni) or ANTs (pydra-ants), or a collection of related tasks/workflows, such as Niworkflows (pydra-niworkflows). Pip can be used to install these extension packages as well:

$ pip install pydra-fsl pydra-ants

Of course, if you use Pydra to execute commands within non-Python toolkits, you will need to either have those commands installed on the execution machine, or use containers to run them (see Software environments).

Tutorials and notebooks

The following tutorials provide a step-by-step guide to using Pydra. They can be studied in any order, but it is recommended to start with Getting started and step through the list from there.

The tutorials are written in Jupyter notebooks, which can be downloaded and run locally or run online using the button within each tutorial.

If you decide to download the notebooks and run locally, be sure to install the necessary dependencies (ideally within a virtual environment):

$ pip install -e /path/to/your/pydra[tutorial]

Execution

Learn how to execute existing tasks (including workflows) on different systems

• Getting started

• Advanced execution

• Troubleshooting

Design

Learn how to design your own tasks, wrapped shell commands or Python functions, or workflows,

• Python-tasks

• Shell-tasks

• Workflows

• Canonical task form

Examples

The following comprehensive examples demonstrate how to use Pydra to build and execute complex workflows

• T1w MRI preprocessing

• General Linear Model (GLM)

How-to Guides

The following guides provide step-by-step instructions on how to

• Create a task package

• Port interfaces from Nipype

Reference

See the full reference documentation for Pydra

• API

• Index

• Module Index

• Glossary