Warning: This document is for the development version of Pydra: A simple dataflow engine with scalable semantics. The main version is master.

pydra.utils.profiler module

Utilities to keep track of performance and resource utilization.

class pydra.utils.profiler.ResourceMonitor(pid, interval=5, logdir=None, fname=None)

Bases: Thread

A thread to monitor a specific PID with a certain frequency to a file.

property fname

Get/set the internal filename.

run()

Core monitoring function, called by start().

stop()

Stop monitoring.

pydra.utils.profiler.get_max_resources_used(pid, mem_mb, num_threads, pyfunc=False)

Get the RAM and threads utilized by a given process.

Parameters:

• pid (integer) – the process ID of process to profile

• mem_mb (float) – the high memory watermark so far during process execution (in MB)

• num_threads (int) – the high thread watermark so far during process execution

Returns:

• mem_mb (float) – the new high memory watermark of process (MB)

• num_threads (float) – the new high thread watermark of process

pydra.utils.profiler.get_system_total_memory_gb()

Get the total RAM of the running system, in GB.

pydra.utils.profiler.log_nodes_cb(node, status)

Record node run statistics to a log file as json dictionaries.

Parameters:

• node (nipype.pipeline.engine.Node) – the node being logged

• status (string) – acceptable values are ‘start’, ‘end’; otherwise it is considered and error

Returns:

this function does not return any values, it logs the node status info to the callback logger

Return type:

None