Warning: This document is for the development version of Pydra: A simple dataflow engine with scalable semantics. The main version is master.

pydra.engine.helpers_file module

Functions ported from Nipype 1, after removing parts that were related to py2.

pydra.engine.helpers_file.copyfile(originalfile, newfile, copy=False, create_new=False, use_hardlink=True, copy_related_files=True)

Copy or link files.

If use_hardlink is True, and the file can be hard-linked, then a link is created, instead of copying the file.

If a hard link is not created and copy is False, then a symbolic link is created.

Copy options for existing files

  • symlink

    • to regular file originalfile (keep if symlinking)

    • to same dest as symlink originalfile (keep if symlinking)

    • to other file (unlink)

  • regular file

    • hard link to originalfile (keep)

    • copy of file (same hash) (keep)

    • different file (diff hash) (unlink)

Copy options for new files

  • use_hardlink & can_hardlink => hardlink

  • ~hardlink & ~copy & can_symlink => symlink

  • ~hardlink & ~symlink => copy

Parameters:

  • originalfile (str) – full path to original file

  • newfile (str) – full path to new file

  • copy (Bool) – specifies whether to copy or symlink files (default=False) but only for POSIX systems

  • use_hardlink (Bool) – specifies whether to hard-link files, when able (Default=False), taking precedence over copy

  • copy_related_files (Bool) – specifies whether to also operate on related files, as defined in related_filetype_sets

Return type:

None

pydra.engine.helpers_file.copyfile_input(inputs, output_dir)

Implement the base class method.

pydra.engine.helpers_file.copyfiles(filelist, dest, copy=False, create_new=False)

Copy or symlink files in filelist to dest directory.

Parameters:

  • filelist (list) – List of files to copy.

  • dest (path/files) – full path to destination. If it is a list of length greater than 1, then it assumes that these are the names of the new files.

  • copy (Bool) – specifies whether to copy or symlink files (default=False) but only for posix systems

Return type:

None

pydra.engine.helpers_file.ensure_list(filename)

Return a list given either a string or a list.

pydra.engine.helpers_file.fname_presuffix(fname, prefix='', suffix='', newpath=None, use_ext=True)

Manipulate path and name of input filename.

Parameters:

  • fname (str) – A filename (may or may not include path)

  • prefix (str) – Characters to prepend to the filename

  • suffix (str) – Characters to append to the filename

  • newpath (str) – Path to replace the path of the input fname

  • use_ext (bool) – If True (default), appends the extension of the original file to the output name.

Returns:

path – Absolute path of the modified filename

Return type:

str

Examples

>>> import pytest, sys
>>> if sys.platform.startswith('win'): pytest.skip()
>>> from pydra.engine.helpers_file import fname_presuffix
>>> fname = 'foo.nii.gz'
>>> fname_presuffix(fname,'pre','post','/tmp')
'/tmp/prefoopost.nii.gz'

Return a list of related files.

As defined in related_filetype_sets, for a filename (e.g., Nifti-Pair, Analyze (SPM), and AFNI files).

Parameters:

  • filename (str) – File name to find related filetypes of.

  • include_this_file (bool) – If true, output includes the input filename.

pydra.engine.helpers_file.hash_dir(dirpath, crypto=<built-in function openssl_sha256>, ignore_hidden_files=False, ignore_hidden_dirs=False, raise_notfound=True, precalculated=None)

Compute hash of directory contents.

This function computes the hash of every file in directory dirpath and then computes the hash of that list of hashes to return a single hash value. The directory is traversed recursively.

Parameters:

  • dirpath (str) – Path to directory.

  • crypto (:obj: function) – cryptographic hash functions

  • ignore_hidden_files (bool) – If True, ignore filenames that begin with ..

  • ignore_hidden_dirs (bool) – If True, ignore files in directories that begin with ..

  • raise_notfound (bool) – If True and dirpath does not exist, raise FileNotFound exception. If False and dirpath does not exist, return None.

Returns:

hash – Hash of the directory contents.

Return type:

str

pydra.engine.helpers_file.hash_file(afile, chunk_len=8192, crypto=<built-in function openssl_sha256>, raise_notfound=True, precalculated=None)

Compute hash of a file using ‘crypto’ module.

pydra.engine.helpers_file.is_container(item)

Check if item is a container (list, tuple, dict, set).

Parameters:

item (object) – Input object to check.

Returns:

output – True if container False otherwise.

Return type:

bool

pydra.engine.helpers_file.is_existing_file(value)

checking if an object is an existing file

pydra.engine.helpers_file.is_local_file(f)
pydra.engine.helpers_file.on_cifs(fname)

Check whether a file path is on a CIFS filesystem mounted in a POSIX host.

POSIX hosts are assumed to have the mount command.

On Windows, Docker mounts host directories into containers through CIFS shares, which has support for Minshall+French symlinks, or text files that the CIFS driver exposes to the OS as symlinks. We have found that under concurrent access to the filesystem, this feature can result in failures to create or read recently-created symlinks, leading to inconsistent behavior and FileNotFoundError errors.

This check is written to support disabling symlinks on CIFS shares.

pydra.engine.helpers_file.related_filetype_sets = [('.hdr', '.img', '.mat'), ('.nii', '.mat'), ('.BRIK', '.HEAD')]

List of neuroimaging file types that are to be interpreted together.

pydra.engine.helpers_file.split_filename(fname)

Split a filename into parts: path, base filename and extension.

Parameters:

fname (str) – file or path name

Returns:

  • pth (str) – base path from fname

  • fname (str) – filename from fname, without extension

  • ext (str) – file extension from fname

Examples

>>> pth, fname, ext = split_filename('/home/data/subject.nii.gz')
>>> pth
'/home/data'

>>> fname
'subject'

>>> ext
'.nii.gz'
pydra.engine.helpers_file.template_update(inputs, output_dir, state_ind=None, map_copyfiles=None)

Update all templates that are present in the input spec.

Should be run when all inputs used in the templates are already set.

pydra.engine.helpers_file.template_update_single(field, inputs, inputs_dict_st=None, output_dir=None, spec_type='input')

Update a single template from the input_spec or output_spec based on the value from inputs_dict (checking the types of the fields, that have “output_file_template)”