Warning: This document is for the development version of Pydra: A simple dataflow engine with scalable semantics. The main version is master.

pydra.engine.specs module

Task I/O specifications.

class pydra.engine.specs.BaseSpec

Bases: object

The base dataclass specs for all inputs and outputs.

check_fields_input_spec()

Check fields from input spec based on the medatada.

e.g., if xor, requires are fulfilled, if value provided when mandatory.

check_metadata()

Check contained metadata.

collect_additional_outputs(inputs, output_dir, outputs)

Get additional outputs.

copyfile_input(output_dir)

Copy the file pointed by a File input.

property hash

Compute a basic hash for any given set of fields.

retrieve_values(wf, state_index=None)

Get values contained by this spec.

template_update()

Update template.

class pydra.engine.specs.ContainerSpec(*, executable: str | List[str], args: str | List[str] | None = None, image: File | str, container: File | str | None, container_xargs: List[str] | None = None)

Bases: ShellSpec

Refine the generic command-line specification to container execution.

container: File | str | None

The container.

container_xargs: List[str] | None

image: File | str

The image to be containerized.

class pydra.engine.specs.Directory

Bases: object

An os.pathlike object, designating a folder.

class pydra.engine.specs.DockerSpec(*, executable: str | List[str], args: str | List[str] | None = None, image: File | str, container_xargs: List[str] | None = None, container: str = 'docker')

Bases: ContainerSpec

Particularize container specifications to the Docker engine.

container: str

The container.

class pydra.engine.specs.File

Bases: object

An os.pathlike object, designating a file.

class pydra.engine.specs.FunctionSpec

Bases: BaseSpec

Specification for a process invoked from a shell.

check_metadata()

Check the metadata for fields in input_spec and fields.

Also sets the default values when available and needed.

class pydra.engine.specs.LazyField(node, attr_type)

Bases: object

Lazy fields implement promises.

get_value(wf, state_index=None)

Return the value of a lazy field.

class pydra.engine.specs.MultiInputFile

Bases: MultiInputObj

A ty.List[File] object, converter changes a single file path to a list

class pydra.engine.specs.MultiInputObj

Bases: object

A ty.List[ty.Any] object, converter changes a single values to a list

classmethod converter(value)

class pydra.engine.specs.MultiOutputFile

Bases: MultiOutputObj

A ty.List[File] object, converter changes an 1-el list to the single value

class pydra.engine.specs.MultiOutputObj

Bases: object

A ty.List[ty.Any] object, converter changes an 1-el list to the single value

classmethod converter(value)

class pydra.engine.specs.Result(*, output: Any | None = None, runtime: Runtime | None = None, errored: bool = False)

Bases: object

Metadata regarding the outputs of processing.

errored: bool

get_output_field(field_name)

Used in get_values in Workflow

Parameters:

field_name (str) – Name of field in LazyField object

output: Any | None

runtime: Runtime | None

class pydra.engine.specs.Runtime(*, rss_peak_gb: float | None = None, vms_peak_gb: float | None = None, cpu_peak_percent: float | None = None)

Bases: object

Represent run time metadata.

cpu_peak_percent: float | None

Peak in cpu consumption.

rss_peak_gb: float | None

Peak in consumption of physical RAM.

vms_peak_gb: float | None

Peak in consumption of virtual memory.

class pydra.engine.specs.RuntimeSpec(*, outdir: str | None = None, container: str | None = 'shell', network: bool = False)

Bases: object

Specification for a task.

From CWL:

InlineJavascriptRequirement
SchemaDefRequirement
DockerRequirement
SoftwareRequirement
InitialWorkDirRequirement
EnvVarRequirement
ShellCommandRequirement
ResourceRequirement

InlineScriptRequirement

container: str | None

network: bool

outdir: str | None

class pydra.engine.specs.ShellOutSpec(*, return_code: int, stdout: File | str, stderr: File | str)

Bases: object

Output specification of a generic shell process.

collect_additional_outputs(inputs, output_dir, outputs)

Collect additional outputs from shelltask output_spec.

generated_output_names(inputs, output_dir)

Returns a list of all outputs that will be generated by the task. Takes into account the task input and the requires list for the output fields. TODO: should be in all Output specs?

return_code: int

The process’ exit code.

stderr: File | str

The process’ standard input.

stdout: File | str

The process’ standard output.

class pydra.engine.specs.ShellSpec(*, executable: str | List[str], args: str | List[str] | None = None)

Bases: BaseSpec

Specification for a process invoked from a shell.

args: str | List[str] | None

check_metadata()

Check the metadata for fields in input_spec and fields.

Also sets the default values when available and needed.

executable: str | List[str]

retrieve_values(wf, state_index=None)

Parse output results.

class pydra.engine.specs.SingularitySpec(*, executable: str | List[str], args: str | List[str] | None = None, image: File | str, container_xargs: List[str] | None = None, container: str = 'singularity')

Bases: ContainerSpec

Particularize container specifications to Singularity.

container: str

The container.

class pydra.engine.specs.SpecInfo(*, name: str, fields: List[Tuple] = _Nothing.NOTHING, bases: Sequence[Type[BaseSpec]] = _Nothing.NOTHING)

Bases: object

Base data structure for metadata of specifications.

bases: Sequence[Type[BaseSpec]]

Keeps track of specification inheritance. Should be a tuple containing at least one BaseSpec

fields: List[Tuple]

List of names of fields (can be inputs or outputs).

name: str

A name for the specification.

class pydra.engine.specs.TaskHook(*, pre_run_task: ~typing.Callable = <function donothing>, post_run_task: ~typing.Callable = <function donothing>, pre_run: ~typing.Callable = <function donothing>, post_run: ~typing.Callable = <function donothing>)

Bases: object

Callable task hooks.

post_run: Callable

post_run_task: Callable

pre_run: Callable

pre_run_task: Callable

reset()

pydra.engine.specs.attr_fields(spec, exclude_names=())

pydra.engine.specs.attr_fields_dict(spec, exclude_names=())

pydra.engine.specs.donothing(*args, **kwargs)

pydra.engine.specs.path_to_string(value)

Convert paths to strings.