Warning: This document is for an old version of Pydra: A simple dataflow engine with scalable semantics. The main version is master.

pydra.engine.task module

Implement processing nodes.

Notes:

  • Environment specs

    1. neurodocker json

    2. singularity file+hash

    3. docker hash

    4. conda env

    5. niceman config

    6. environment variables

  • Monitors/Audit

    1. internal monitor

    2. external monitor

    3. callbacks

  • Resuming

    1. internal tracking

    2. external tracking (DMTCP)

  • Provenance

    1. Local fragments

    2. Remote server

  • Isolation

    1. Working directory

    2. File (copy to local on write)

    3. read only file system

  • Original implementation

class pydra.engine.task.ContainerTask(container_info=None, *args, **kwargs)

Bases: ShellCommandTask

Extend shell command task for containerized execution.

bind_paths()

Get bound mount points

Returns:

mount points – mapping from local path to tuple of container path + mode

Return type:

dict

binds(opt)

Specify mounts to bind from local filesystems to container and working directory.

Uses py:meth:bind_paths

container_check(container_type)

Get container-specific CLI arguments.

class pydra.engine.task.DockerTask(container_info=None, *args, **kwargs)

Bases: ContainerTask

Extend shell command task for containerized execution with the Docker Engine.

property container_args

Get container-specific CLI arguments, returns a list if the task has a state

init = False
class pydra.engine.task.FunctionTask(func: Callable, audit_flags: AuditFlag = AuditFlag.NONE, cache_dir=None, cache_locations=None, input_spec: SpecInfo | BaseSpec | None = None, cont_dim=None, messenger_args=None, messengers=None, name=None, output_spec: SpecInfo | BaseSpec | None = None, rerun=False, **kwargs)

Bases: TaskBase

Wrap a Python callable as a task element.

class pydra.engine.task.ShellCommandTask(container_info=None, *args, **kwargs)

Bases: TaskBase

Wrap a shell command as a task element.

property cmdline

Get the actual command line that will be submitted Returns a list if the task has a state.

property command_args

Get command line arguments

input_spec = None
output_spec = None
class pydra.engine.task.SingularityTask(container_info=None, *args, **kwargs)

Bases: ContainerTask

Extend shell command task for containerized execution with Singularity.

property container_args

Get container-specific CLI arguments.

init = False
pydra.engine.task.split_cmd(cmd: str)

Splits a shell command line into separate arguments respecting quotes

Parameters:

cmd (str) – Command line string or part thereof

Returns:

the command line string split into process args

Return type:

str