Utilities for transforming and validating data types

Given that many of the data transformations involve copying the data, they should ideally happen in a lazy manner to avoid memory issues.

Created on Tue Nov 3 21:14:25 2015

@author: Suhas Somnath, Chris Smith



Checks the datatype of the input HDF5 dataset and provides the appropriate function calls to convert it to a float

contains_integers(iter_int[, min_val])

Checks if the provided object is iterable (list, tuple etc.) and contains integers optionally greater than equal to the provided min_val

flatten_complex_to_real(dataset[, lazy])

Stacks the real values followed by the imaginary values in the last dimension of the given N dimensional matrix.

flatten_compound_to_real(dataset[, lazy])

Flattens the individual components in a structured array or compound valued hdf5 dataset along the last axis to form a real valued array.

flatten_to_real(ds_main[, lazy])

Flattens complex / compound / real valued arrays to real valued arrays


Returns a dictionary of the dtypes of each of the fields in the given structured array dtype


Gets the scale / exponent for a sequence of numbers.


Converts a sequence of iterables to a list of slice objects denoting sequences of consecutive numbers


Checks if the provided dtype is a complex dtype


Loads the provided object as a dask array (h5py.Dataset or numpy.ndarray

stack_real_to_complex(ds_real[, lazy])

Puts the real and imaginary sections of the provided matrix (in the last axis) together to make complex matrix

stack_real_to_compound(ds_real, compound_type)

Converts a real-valued dataset to a compound dataset (along the last axis) of the provided compound d-type

stack_real_to_target_dtype(ds_real, new_dtype)

Transforms real data into the target dtype


Checks the provided object to ensure that it is a valid dtype that can be written to an HDF5 file.

validate_list_of_strings(str_list[, parm_name])

This function is to be used when validating and cleaning a list of strings.

validate_single_string_arg(value, name)

This function is to be used when validating a SINGLE string parameter for a function.

validate_string_args(arg_list, arg_names)

This function is to be used when validating string parameters for a function.