hdf5storage.Marshallers

Module for the classes to marshall Python types to/from file.

write_object_array(f, data, options) Writes an array of objects recursively.
read_object_array(f, data, options) Reads an array of objects recursively.
TypeMarshaller() Base class for marshallers of Python types.
NumpyScalarArrayMarshaller()

Methods
PythonScalarMarshaller()

Methods
PythonStringMarshaller()

Methods
PythonNoneMarshaller()

Methods
PythonDictMarshaller()

Methods
PythonListMarshaller()

Methods
PythonTupleSetDequeMarshaller()

Methods

write_object_array

hdf5storage.Marshallers.write_object_array(f, data, options)[source]

Writes an array of objects recursively.

Writes the elements of the given object array recursively in the HDF5 Group options.group_for_references and returns an h5py.Reference array to all the elements.

Parameters:

f : h5py.File

The HDF5 file handle that is open.

data : numpy.ndarray of objects

Numpy object array to write the elements of.

options : hdf5storage.core.Options

hdf5storage options object.
Returns:

numpy.ndarray of h5py.Reference

A reference array pointing to all the elements written to the HDF5 file. For those that couldn’t be written, the respective element points to the canonical empty.
Raises:

TypeNotMatlabCompatibleError

If writing a type not compatible with MATLAB and options.action_for_matlab_incompatible is set to 'error'.

See also

read_object_array, hdf5storage.Options.group_for_references, h5py.Reference

read_object_array

hdf5storage.Marshallers.read_object_array(f, data, options)[source]

Reads an array of objects recursively.

Read the elements of the given HDF5 Reference array recursively in the and constructs a numpy.object_ array from its elements, which is returned.

Parameters:

f : h5py.File

The HDF5 file handle that is open.

data : numpy.ndarray of h5py.Reference

The array of HDF5 References to read and make an object array from.

options : hdf5storage.core.Options

hdf5storage options object.
Returns:

numpy.ndarray of numpy.object_

The Python object array containing the items pointed to by data.
Raises:

NotImplementedError

If reading the object from file is currently not supported.

See also

write_object_array, hdf5storage.Options.group_for_references, h5py.Reference

TypeMarshaller

class hdf5storage.Marshallers.TypeMarshaller[source]

Bases: object

Base class for marshallers of Python types.

Base class providing the class interface for marshallers of Python types to/from disk. All marshallers should inherit from this class or at least replicate its functionality. This includes several attributes that are needed in order for reading/writing methods to know if it is the appropriate marshaller to use and methods to actually do the reading and writing.

Subclasses should run this class’s __init__() first thing. Inheritance information is in the Notes section of each method. Generally, read, write, and write_metadata need to be overridden and the different attributes set to the proper values.

For marshalling types that are containers of other data, one will need to appropriate read/write them with the lowlevel functions lowlevel.read_data and lowlevel.write_data.

See also

hdf5storage.core.Options, h5py.Dataset, h5py.Group, h5py.AttributeManager, hdf5storage.lowlevel.read_data, hdf5storage.lowlevel.write_data

Attributes

python_attributes (set of str) Attributes used to store type information.
matlab_attributes (set of str) Attributes used for MATLAB compatibility.
types (list of types) Types the marshaller can work on.
python_type_strings (list of str) Type strings of readable types.
matlab_classes (list of str) Readable MATLAB classes.

Methods

get_type_string(data, type_string) Gets type string.
read(f, grp, name, options) Read a Python object from file.
write(f, grp, name, data, type_string, options) Writes an object’s metadata to file.
write_metadata(f, grp, name, data, ...) Writes an object to file.
python_attributes = {'Python.Type'}

Attributes used to store type information.

set of str

set of attribute names the marshaller uses when an Option.store_python_metadata is True.

matlab_attributes = {'H5PATH'}

Attributes used for MATLAB compatibility.

set of str

set of attribute names the marshaller uses when maintaing Matlab HDF5 based mat file compatibility (Option.matlab_compatible is True).

types = []

List of Python types that can be marshalled.

list of types

list of the types (gotten by doing type(data)) that the marshaller can marshall. Default value is [].

python_type_strings = []

Type strings of readable types.

list of str

list of the str that the marshaller would put in the HDF5 attribute ‘Python.Type’ to identify the Python type to be able to read it back correctly. Default value is [].

matlab_classes = []

MATLAB class strings of readable types.

list of str

list of the MATLAB class str that the marshaller can read into Python objects. Default value is [].

get_type_string(data, type_string)[source]

Gets type string.

Finds the type string for ‘data’ contained in python_type_strings using its type. Non-None ‘type_string` overrides whatever type string is looked up. The override makes it easier for subclasses to convert something that the parent marshaller can write to disk but still put the right type string in place).

Parameters:

data : type to be marshalled

The Python object that is being written to disk.

type_string : str or None

If it is a str, it overrides any looked up type string. None means don’t override.
Returns:

str

The type string associated with ‘data’. Will be ‘type_string’ if it is not None.

Notes

Subclasses probably do not need to override this method.

read(f, grp, name, options)[source]

Read a Python object from file.

Reads the Python object ‘name’ from the HDF5 Group ‘grp’, if possible, and returns it.

Parameters:

f : h5py.File

The HDF5 file handle that is open.

grp : h5py.Group or h5py.File

The parent HDF5 Group (or File if at ‘/’) that contains the object with the specified name.

name : str

Name of the object.

options : hdf5storage.core.Options

hdf5storage options object.
Returns:

data

The Python object ‘name’ in the HDF5 Group ‘grp’.
Raises:

NotImplementedError

If reading the object from file is currently not supported.

See also

hdf5storage.lowlevel.read_data

Notes

Must be overridden in a subclass because a NotImplementedError is thrown immediately.

write(f, grp, name, data, type_string, options)[source]

Writes an object’s metadata to file.

Writes the Python object ‘data’ to ‘name’ in h5py.Group ‘grp’.

Parameters:

f : h5py.File

The HDF5 file handle that is open.

grp : h5py.Group or h5py.File

The parent HDF5 Group (or File if at ‘/’) that contains the object with the specified name.

name : str

Name of the object.

data

The object to write to file.

type_string : str or None

The type string for data. If it is None, one will have to be gotten by get_type_string.

options : hdf5storage.core.Options

hdf5storage options object.
Raises:

NotImplementedError

If writing ‘data’ to file is currently not supported.

TypeNotMatlabCompatibleError

If writing a type not compatible with MATLAB and options.action_for_matlab_incompatible is set to 'error'.

See also

hdf5storage.lowlevel.write_data

Notes

Must be overridden in a subclass because a NotImplementedError is thrown immediately.

write_metadata(f, grp, name, data, type_string, options)[source]

Writes an object to file.

Writes the metadata for a Python object data to file at name in h5py.Group grp. Metadata is written to HDF5 Attributes. Existing Attributes that are not being used are deleted.

Parameters:

f : h5py.File

The HDF5 file handle that is open.

grp : h5py.Group or h5py.File

The parent HDF5 Group (or File if at ‘/’) that contains the object with the specified name.

name : str

Name of the object.

data

The object to write to file.

type_string : str or None

The type string for data. If it is None, one will have to be gotten by get_type_string.

options : hdf5storage.core.Options

hdf5storage options object.

Notes

The attribute ‘Python.Type’ is set to the type string. All H5PY Attributes not in python_attributes and/or matlab_attributes (depending on the attributes of ‘options’) are deleted. These are needed functions for writting essentially any Python object, so subclasses should probably call the baseclass’s version of this function if they override it and just provide the additional functionality needed. This requires that the names of any additional HDF5 Attributes are put in the appropriate set.

NumpyScalarArrayMarshaller

class hdf5storage.Marshallers.NumpyScalarArrayMarshaller[source]

Bases: hdf5storage.Marshallers.TypeMarshaller

Methods

get_type_string(data, type_string) Gets type string.
read(f, grp, name, options)
write(f, grp, name, data, type_string, options)
write_metadata(f, grp, name, data, ...)
python_attributes = {'Python.Type', 'Python.Shape', 'Python.Empty', 'Python.numpy.UnderlyingType', 'Python.numpy.Container', 'Python.Fields'}
matlab_attributes = {'H5PATH', 'MATLAB_class', 'MATLAB_empty', 'MATLAB_int_decode', 'MATLAB_fields'}
types = [np.ndarray, np.matrix, np.chararray, np.core.records.recarray, np.bool_, np.void, np.uint8, np.uint16, np.uint32, np.uint64, np.int8, np.int16, np.int32, np.int64, np.float16, np.float32, np.float64, np.complex64, np.complex128, np.bytes_, np.str_, np.object_]
python_type_strings = ['numpy.ndarray', 'numpy.matrix', 'numpy.chararray', 'numpy.recarray', 'numpy.bool_', 'numpy.void', 'numpy.uint8', 'numpy.uint16', 'numpy.uint32', 'numpy.uint64', 'numpy.int8', 'numpy.int16', 'numpy.int32', 'numpy.int64', 'numpy.float16', 'numpy.float32', 'numpy.float64', 'numpy.complex64', 'numpy.complex128', 'numpy.bytes_', 'numpy.str_', 'numpy.object_']
matlab_classes = ['logical', 'char', 'single', 'double', 'uint8', 'uint16', 'uint32', 'uint64', 'int8', 'int16', 'int32', 'int64', 'cell', 'canonical empty']

PythonScalarMarshaller

class hdf5storage.Marshallers.PythonScalarMarshaller[source]

Bases: hdf5storage.Marshallers.NumpyScalarArrayMarshaller

Methods

get_type_string(data, type_string) Gets type string.
read(f, grp, name, options)
write(f, grp, name, data, type_string, options)
write_metadata(f, grp, name, data, ...)
python_attributes = {'Python.Type', 'Python.Shape', 'Python.Empty', 'Python.numpy.UnderlyingType', 'Python.numpy.Container', 'Python.Fields'}
matlab_attributes = {'H5PATH', 'MATLAB_class', 'MATLAB_empty', 'MATLAB_int_decode'}
types = [bool, int, float, complex]
python_type_strings = ['bool', 'int', 'float', 'complex']
matlab_classes = []

PythonStringMarshaller

class hdf5storage.Marshallers.PythonStringMarshaller[source]

Bases: hdf5storage.Marshallers.NumpyScalarArrayMarshaller

Methods

get_type_string(data, type_string) Gets type string.
read(f, grp, name, options)
write(f, grp, name, data, type_string, options)
write_metadata(f, grp, name, data, ...)
python_attributes = {'Python.Type', 'Python.Shape', 'Python.Empty', 'Python.numpy.UnderlyingType', 'Python.numpy.Container', 'Python.Fields'}
matlab_attributes = {'H5PATH', 'MATLAB_class', 'MATLAB_empty', 'MATLAB_int_decode'}
types = [str, bytes, bytearray]
python_type_strings = ['str', 'bytes', 'bytearray']
matlab_classes = []

PythonNoneMarshaller

class hdf5storage.Marshallers.PythonNoneMarshaller[source]

Bases: hdf5storage.Marshallers.NumpyScalarArrayMarshaller

Methods

get_type_string(data, type_string) Gets type string.
read(f, grp, name, options)
write(f, grp, name, data, type_string, options)
write_metadata(f, grp, name, data, ...)
python_attributes = {'Python.Type', 'Python.Shape', 'Python.Empty', 'Python.numpy.UnderlyingType', 'Python.numpy.Container', 'Python.Fields'}
matlab_attributes = {'H5PATH', 'MATLAB_class', 'MATLAB_empty', 'MATLAB_int_decode'}
types = [builtins.NoneType]
python_type_strings = ['builtins.NoneType']
matlab_classes = []

PythonDictMarshaller

class hdf5storage.Marshallers.PythonDictMarshaller[source]

Bases: hdf5storage.Marshallers.TypeMarshaller

Methods

get_type_string(data, type_string) Gets type string.
read(f, grp, name, options)
write(f, grp, name, data, type_string, options)
write_metadata(f, grp, name, data, ...)
python_attributes = {'Python.Type', 'Python.Fields'}
matlab_attributes = {'H5PATH', 'MATLAB_class', 'MATLAB_fields'}
types = [dict]
python_type_strings = ['dict']
matlab_classes = []

PythonListMarshaller

class hdf5storage.Marshallers.PythonListMarshaller[source]

Bases: hdf5storage.Marshallers.NumpyScalarArrayMarshaller

Methods

get_type_string(data, type_string) Gets type string.
read(f, grp, name, options)
write(f, grp, name, data, type_string, options)
write_metadata(f, grp, name, data, ...)
python_attributes = {'Python.Type', 'Python.Shape', 'Python.Empty', 'Python.numpy.UnderlyingType', 'Python.numpy.Container', 'Python.Fields'}
matlab_attributes = {'H5PATH', 'MATLAB_class', 'MATLAB_empty', 'MATLAB_int_decode'}
types = [list]
python_type_strings = ['list']
matlab_classes = []

PythonTupleSetDequeMarshaller

class hdf5storage.Marshallers.PythonTupleSetDequeMarshaller[source]

Bases: hdf5storage.Marshallers.PythonListMarshaller

Methods

get_type_string(data, type_string) Gets type string.
read(f, grp, name, options)
write(f, grp, name, data, type_string, options)
write_metadata(f, grp, name, data, ...)
python_attributes = {'Python.Type', 'Python.Shape', 'Python.Empty', 'Python.numpy.UnderlyingType', 'Python.numpy.Container', 'Python.Fields'}
matlab_attributes = {'H5PATH', 'MATLAB_class', 'MATLAB_empty', 'MATLAB_int_decode'}
types = [tuple, set, frozenset, collections.deque]
python_type_strings = ['tuple', 'set', 'frozenset', 'collections.deque']
matlab_classes = []