Utilities for reading and validating HDF5 files. More...

Classes
struct	IterateChunks
	Iterate through an high-dimensional dataspace by chunk. More...

class	ReclaimVlsMemory
	Reclaim memory for HDF5's variable length strings. More...

class	Stream1dNumericDataset
	Stream a 1-dimensional HDF5 numeric dataset into memory. More...

class	Stream1dStringDataset
	Stream a 1-dimensional HDF5 string dataset into memory. More...

Functions
template<typename Type_ >
const H5::PredType &	as_numeric_datatype ()

bool	exceeds_integer_limit (const H5::IntType &itype, std::size_t precision, bool is_signed)

bool	exceeds_integer_limit (const H5::DataSet &data, std::size_t precision, bool is_signed)

bool	exceeds_integer_limit (const H5::Attribute &attr, std::size_t precision, bool is_signed)

bool	exceeds_float_limit (const H5::DataSet &data, std::size_t precision)

bool	exceeds_float_limit (const H5::Attribute &attr, std::size_t precision)

template<class Object_ >
std::string	get_name (const Object_ &obj)

bool	is_utf8_string (const H5::StrType &stype)

bool	is_utf8_string (const H5::DataSet &data)

bool	is_utf8_string (const H5::Attribute &attr)

std::vector< hsize_t >	mock_contiguous_chunks (const std::vector< hsize_t > &dimensions, hsize_t chunk_size)

std::string	read_scalar_string (const H5::DataSet &data)

std::string	read_scalar_string (const H5::Attribute &attr)

std::size_t	strnlen (const char *ptr, std::size_t max)

void	validate_scalar_string (const H5::DataSet &data)

void	validate_1d_strings (const H5::DataSet &data, hsize_t full_length)

void	validate_nd_strings (const H5::DataSet &data, const std::vector< hsize_t > &dimensions)

void	validate_scalar_string (const H5::Attribute &attr)

void	validate_1d_strings (const H5::Attribute &attr, hsize_t full_length)

Detailed Description

Utilities for reading and validating HDF5 files.

Function Documentation

◆ as_numeric_datatype()

template<typename Type_ >

const H5::PredType & ritsuko::hdf5::as_numeric_datatype ( )

Choose the HDF5 datatype object corresponding to a particular C++ numeric type.

Template Parameters

Type_ A numeric C++ type. This can be any of the fixed-width integers, float or double. Some of the non-fixed integer types are also supported.

Returns: Reference to a HDF5 datatype object.

◆ exceeds_float_limit() [1/2]

bool ritsuko::hdf5::exceeds_float_limit	(	const H5::Attribute &	attr,
		std::size_t	precision )

inline

Overload of exceeds_float_limit() that accepts a HDF5 attribute.

Parameters

attr	A HDF5 attribute. Its datatype may be of any class.
precision	Number of bits in the limiting float type.

Returns: Whether the attribute uses a datatype than cannot be represented by the limiting integer type. true is also returned for non-numeric attributes.

◆ exceeds_float_limit() [2/2]

bool ritsuko::hdf5::exceeds_float_limit	(	const H5::DataSet &	data,
		std::size_t	precision )

inline

Check if a HDF5 dataset's datatype could hold values beyond the range of a limiting (IEEE754-compliant) float type. This is used by validators to ensure that a dataset can be represented in memory by the limiting type.

Note that the limiting float type is assumed to be IEEE754-compliant. If the HDF5 datatype is not also IEEE754-compliant, it will be considered out-of-range regardless of its precision. This is necessary as non-IEEE754 floats could have an arbitrary split of bits between the exponent and significand, such that two float datatypes with the same number of bits could represent a different set of numbers. (Though this seems unlikely in practice, as all CPU-specific predefined float types in later HDF5 versions are already aliases of the IEEE types.)

This function may return false for integer datasets provided that the integer datatype is small enough. Specifically, it must be small enough so that all integer values can be exactly represented in the limiting float type. For double- and single- precision IEEE754 floats, this means that the integer must be no more than 53 and 24 bits, respectively.

Parameters

data	A HDF5 dataset. Its datatype may be of any class.
precision	Number of bits in the limiting float type.

Returns: Whether the dataset uses a datatype than cannot be represented by the limiting float type. true is also returned for non-numeric datasets.

◆ exceeds_integer_limit() [1/3]

bool ritsuko::hdf5::exceeds_integer_limit	(	const H5::Attribute &	attr,
		std::size_t	precision,
		bool	is_signed )

inline

Overload of exceeds_integer_limit() that accepts a HDF5 attribute.

Parameters

attr	A HDF5 attribute. Its datatype may be of any class.
precision	Number of bits in the limiting integer type, assuming 2's complement.
is_signed	Whether the limiting integer type is signed.

Returns: Whether the attribute uses a datatype than cannot be represented by the limiting integer type. This will also return true for non-integer attributes.

◆ exceeds_integer_limit() [2/3]

bool ritsuko::hdf5::exceeds_integer_limit	(	const H5::DataSet &	data,
		std::size_t	precision,
		bool	is_signed )

inline

Overload of exceeds_integer_limit() that accepts a HDF5 dataset.

Parameters

data	A HDF5 dataset. Its datatype may be of any class.
precision	Number of bits in the limiting integer type, assuming 2's complement.
is_signed	Whether the limiting integer type is signed.

Returns: Whether the dataset uses a datatype than cannot be represented by the limiting integer type. This will also return true for non-integer datasets.

◆ exceeds_integer_limit() [3/3]

bool ritsuko::hdf5::exceeds_integer_limit	(	const H5::IntType &	itype,
		std::size_t	precision,
		bool	is_signed )

inline

Check if a HDF5 datatype could hold values beyond the range of a limiting integer type. This is used by validators to ensure that a dataset can be represented in memory by the limiting type.

Parameters

itype	HDF5 integer datatype.
precision	Number of bits in the limiting integer type, assuming 2's complement.
is_signed	Whether the limiting integer type is signed.

Returns: Whether the datatype cannot be represented by the limiting integer type.

◆ get_name()

template<class Object_ >

std::string ritsuko::hdf5::get_name ( const Object_ & obj )

Get the name of a HDF5 object, usually for printing informative error messages.

Template Parameters

Object_ Type of HDF5 object, usually a Group, DataSet or Attribute.

Parameters

obj	A HDF5 object.

Returns: Name of the HDF5 object inside the file.

◆ is_utf8_string() [1/3]

bool ritsuko::hdf5::is_utf8_string ( const H5::Attribute & attr )

inline

Overload of is_utf8_string() that accepts a HDF5 attribute.

Parameters

attr	A HDF5 attribute. Its datatype may be of any class.

Returns: Whether the attribute holds strings that use UTF-8 (or ASCII) encoding. This returns false if its datatype's class is not string.

◆ is_utf8_string() [2/3]

bool ritsuko::hdf5::is_utf8_string ( const H5::DataSet & data )

inline

Overload of is_utf8_string() that accepts a HDF5 dataset.

Parameters

data	A HDF5 dataset. Its datatype may be of any class.

Returns: Whether the dataset holds strings that use UTF-8 (or ASCII) encoding. This returns false if its datatype's class is not string.

◆ is_utf8_string() [3/3]

bool ritsuko::hdf5::is_utf8_string ( const H5::StrType & stype )

inline

Check if a HDF5 string datatype is compatible with the UTF-8 encoding.

Note that this returns true even if the string datatype uses ASCII encoding, given that ASCII is a subset of UTF-8. As a result, this function is mostly performative as all valid HDF5 strings are encoded in either ASCII or UTF-8; nonetheless, we run these checks to be explicit and to protect against the future addition of more encodings.

Parameters

stype HDF5 string datatype.

Returns: Whether stype uses UTF-8 (or ASCII) encoding.

◆ mock_contiguous_chunks()

std::vector< hsize_t > ritsuko::hdf5::mock_contiguous_chunks	(	const std::vector< hsize_t > &	dimensions,
		hsize_t	chunk_size )

inline

Mock chunk dimensions for a contiguous dataset. This allows classes like IterateBlock to be applied to contiguous datasets.

Parameters

dimensions	Array of the dataset dimensions.
chunk_size	Size of the mock chunk, in terms of the number of dimensions.

Returns: Chunk dimensions. This is of the same length as dimensions. Each entry is no greater than the corresponding element of dimensions, unless the latter is zero in which case the former is set to 1.

◆ read_scalar_string() [1/2]

std::string ritsuko::hdf5::read_scalar_string ( const H5::Attribute & attr )

inline

Parameters

attr	A HDF5 attribute. It is assumed that this attribute is scalar and that its datatype is of the string class.

Returns: The attribute as a string.

◆ read_scalar_string() [2/2]

std::string ritsuko::hdf5::read_scalar_string ( const H5::DataSet & data )

inline

Parameters

data	A HDF5 dataset. It is assumed that this dataset is scalar and that its datatype is of the string class.

Returns: String containing the contents of the sole dataset entry.

◆ strnlen()

std::size_t ritsuko::hdf5::strnlen	(	const char *	ptr,
		std::size_t	max )

inline

Get the length of a fixed-size string, either by searching for the first null terminator or by reaching the max length.

Parameters

ptr	Pointer to an array of characters, possibly containing a C-style string.
max	Maximum length of the array referenced by `ptr`.

Returns: The number of characters to the first occurence of the null terminator or max, depending on which is smaller.

◆ validate_1d_strings() [1/2]

void ritsuko::hdf5::validate_1d_strings	(	const H5::Attribute &	attr,
		hsize_t	full_length )

inline

Check that a 1-dimensional string attribute is valid. Currently, this involves checking that there are no NULL entries for variable-length string datatypes. For fixed-width string attributes, this function is a no-op.

Parameters

attr	Handle to the HDF5 string attribute. It is assumed that this attribute is 1-dimensional. It is also assumed that its datatype is of the string class.
full_length	Length of the attribute, i.e., the extent of its sole dimension.

◆ validate_1d_strings() [2/2]

void ritsuko::hdf5::validate_1d_strings	(	const H5::DataSet &	data,
		hsize_t	full_length )

inline

Check that a 1-dimensional string dataset is valid. Currently, this involves checking that there are no NULL entries for variable-length string datatypes. For fixed-width string datasets, this function is a no-op.

Parameters

data	A HDF5 dataset. It is assumed that this dataset is 1-dimensional. It is also assumed that its datatype is of the string class.
full_length	Length of the dataset, i.e., the extent of its sole dimension.

◆ validate_nd_strings()

void ritsuko::hdf5::validate_nd_strings	(	const H5::DataSet &	data,
		const std::vector< hsize_t > &	dimensions )

inline

Check that an N-dimensional string dataset is valid. Currently, this involves checking that there are no NULL entries for variable-length string datatypes. For fixed-width string datasets, this function is a no-op.

Parameters

data	A HDF5 dataset. It is assumed that this dataset has at least 1 dimension. It is also assumed that its datatype is of the string class.
dimensions	Dimensions of the dataset.

◆ validate_scalar_string() [1/2]

void ritsuko::hdf5::validate_scalar_string ( const H5::Attribute & attr )

inline

Check that a scalar string attribute is valid. Currently, this involves checking that there are no NULL entries for variable-length string datatypes. For fixed-width string attributes, this function is a no-op.

Parameters

attr	A HDF5 attribute. It is assumed that this attribute is scalar. It is also assumed that its datatype is of the string class.

◆ validate_scalar_string() [2/2]

void ritsuko::hdf5::validate_scalar_string ( const H5::DataSet & data )

inline

Check that a scalar string dataset is valid. Currently, this involves checking that there are no NULL entries for variable-length string datatypes. For fixed-width string datasets, this function is a no-op.

Parameters

data	A HDF5 dataset. It is assumed that this dataset is scalar. It is also assumed that its datatype is of the string class.

Classes

Functions

Detailed Description

Function Documentation

◆ as_numeric_datatype()

◆ exceeds_float_limit() [1/2]

◆ exceeds_float_limit() [2/2]

◆ exceeds_integer_limit() [1/3]

◆ exceeds_integer_limit() [2/3]

◆ exceeds_integer_limit() [3/3]

◆ get_name()

◆ is_utf8_string() [1/3]

◆ is_utf8_string() [2/3]

◆ is_utf8_string() [3/3]

◆ mock_contiguous_chunks()

◆ read_scalar_string() [1/2]

◆ read_scalar_string() [2/2]

◆ strnlen()

◆ validate_1d_strings() [1/2]

◆ validate_1d_strings() [2/2]

◆ validate_nd_strings()

◆ validate_scalar_string() [1/2]

◆ validate_scalar_string() [2/2]