asdf/core/ndarray.h

Minimalistic prototype implementation of the core/ndarray-1.1.0 schema

Support is not yet fully complete. What is implemented:

  • ASDF internal block sources

  • All int and most float data types (other data types can be read but are not fully implemented)

  • shape, byteorder, and offset

  • strides are partially supported

It can provide direct access to the raw data of an ndarray (via a void *); users must use the metadata provided in the asdf_ndarray_t struct to interpret the data. However, data can also be copied as tiles using the asdf_ndarray_read_tile_ndim and asdf_ndarray_read_tile_2d functions.

What is not yet supported:

  • Shape containing ‘*’

  • Reading complex64 or complex128 datatypes

  • Reading string datatypes (ascii or ucs4)

  • Reading structured datatypes (the datatypes are parsed but there is are no APIs yet for interpreting structured array data)

  • Reading arbitrarily strided data

  • Masks are not parsed or used at all, whether simple mask values or mask arrays (though if present a warning is logged indicating lack of support)

The current limitations are purely artificial–it is so that we can rapidly develop the minimal viable product needed to make ASDF ndarray data available in common use cases.

Complete ndarray support will follow gradually.

Types

enum asdf_ndarray_err_t

Error codes returned by some functions that read ndarray data

enumerator ASDF_NDARRAY_OK = 0

Indicates that the ndarray was read successfully

enumerator ASDF_NDARRAY_ERR_OUT_OF_BOUNDS

Return value indicating that an attempt was made to read beyond the bounds of the ndarray

enumerator ASDF_NDARRAY_ERR_OOM

A memory error occurred (typically out-of-memory)

enumerator ASDF_NDARRAY_ERR_INVAL

An invalid argument was passed to the function (e.g. incorrect number of shape dimensions)

enumerator ASDF_NDARRAY_ERR_OVERFLOW

An overflow occurred, particularly when reading integer values

enumerator ASDF_NDARRAY_ERR_CONVERSION

Some ndarray data elements could not be converted to the requested output datatype

struct asdf_ndarray_t

Public definition of the asdf_ndarray_t type

This is the main object through which ndarrays are used. They can be retrieved via asdf_get_ndarray and asdf_value_as_ndarray. The library allocates memory for this data structure which must be freed by the user with asdf_ndarray_destroy when no-longer needed.

For convenience some basic fields are made public for now, though this may not be ABI-stable in future releases.

size_t source

The index of the binary block containing the ndarray data

uint32_t ndim

The number of dimensions of the array

const uint64_t *shape

The shape of the array, itself an array of size .ndim

asdf_datatype_t datatype

The datatype of the array as represented by asdf_datatype_t

asdf_byteorder_t byteorder

The byteorder of the array data where applicable

uint64_t offset

Optional offset into the binary block where the array data begins

const int64_t *strides

Optional strides to use when iterating/index array data (an array of size .ndim giving the stride for each dimension)

Getting ndarrays

asdf_value_err_t asdf_get_ndarray(asdf_file_t *file, const char *path, asdf_ndarray_t **out)

Get an asdf_ndarray_t* out of the ASDF tree

Parameters:
Returns:

ASDF_VALUE_OK if the value exists and is an ndarray, otherwise ASDF_VALUE_ERR_NOT_FOUND or ASDF_VALUE_ERR_TYPE_MISMATCH.

asdf_value_err_t asdf_set_ndarray(asdf_file_t *file, const char *path, asdf_ndarray_t *ndarray)

Set an asdf_ndarray_t* to a path in the ASDF tree

This is used to add a new ndarray to the file at a given path. If the ndarray has data allocated for it and is set to internal storage (see asdf_ndarray_storage_set, a new binary block will be allocated for its data when writing the file.

Parameters:
Returns:

ASDF_VALUE_OK if the value exists and is an ndarray, otherwise an error if an inconsistency in the asdf_ndarray_t struct prevents serialization.

asdf_value_err_t asdf_value_as_ndarray(asdf_value_t *value, asdf_ndarray_t **out)

Cast a generic asdf_value_t* as an ndarray value, if possible

Parameters:
Returns:

ASDF_VALUE_OK if the value is an ndarray, otherwise ASDF_VALUE_ERR_TYPE_MISMATCH.

void asdf_ndarray_destroy(asdf_ndarray_t *ndarray)

Release datastructures and memory allocated for an asdf_ndarray_t

Parameters:

Data access

const void *asdf_ndarray_data(asdf_ndarray_t *ndarray, size_t *size)

Return a pointer to the ndarray’s element data

If the array is stored in a compressed block a buffer containing the decompressed data is returned (see also Compression). The data is presented in the array’s source datatype and byte order; use asdf_ndarray_read_all or the asdf_ndarray_read_tile_* functions to read it converted to a host datatype and native byte order.

The returned pointer is owned by ndarray and remains valid until the ndarray (or its file) is destroyed; the caller must not free it.

Parameters:

  • ndarray – An asdf_ndarray_t*

  • size – If non-NULL, receives the size of the data in bytes

Returns:

A pointer to the (decompressed) data, or NULL on error – for example if the array has no associated data block (such as an inline array) or if decompression failed

const void *asdf_ndarray_data_raw(asdf_ndarray_t *ndarray, size_t *size)

Return a pointer to the raw ndarray data, without decompressing it

For arrays stored in a compressed block this returns the still-compressed bytes exactly as they appear in the file. For uncompressed arrays it is equivalent to asdf_ndarray_data.

As with asdf_ndarray_data, the returned pointer is owned by ndarray and must not be freed by the caller.

Parameters:

  • ndarray – An asdf_ndarray_t*

  • size – If non-NULL, receives the size of the raw data in bytes

Returns:

A pointer to the raw data, or NULL on error

uint64_t asdf_ndarray_size(const asdf_ndarray_t *ndarray)

Return the total number of elements (not bytes) in the ndarray

Parameters:
Returns:

Total number of elements in the array (just the product of its shape)

uint64_t asdf_ndarray_nbytes(const asdf_ndarray_t *ndarray)

Return the total number of bytes in the ndarray data

Parameters:
Returns:

The byte size of the ndarray (this is just its size times the datatype nbytes)

Allocating data buffers

void *asdf_ndarray_data_alloc(asdf_ndarray_t *ndarray)

Allocate heap memory large enough to store the data for the ndarray

Every call to this function should have a corresponding asdf_ndarray_data_dealloc to free the allocated memory when it is no longer needed (such as after writing the file). The memory is not automatically freed.

Parameters:
Returns:

A void* to the allocated memory or NULL if the memory could not be allocated; subsequent calls on the same ndarray will return the same memory

void *asdf_ndarray_data_alloc_temp(asdf_file_t *file, asdf_ndarray_t *ndarray)

Allocate a temporary data buffer for an ndarray to be written to a file, with automatic cleanup after the write completes.

Like asdf_ndarray_data_alloc but the allocated memory is freed automatically after asdf_write_to (or asdf_close) is called. Extension authors should use this instead of asdf_ndarray_data_alloc when building ndarrays inside a serialize callback.

Parameters:

  • file – The asdf_file_t* to register the cleanup with

  • ndarray – An asdf_ndarray_t* whose shape and datatype are already set

Returns:

A void* to the zero-initialized buffer, or NULL on OOM

void asdf_ndarray_data_dealloc(asdf_ndarray_t *ndarray)

Free ndarray data allocated with asdf_ndarray_data_alloc

If the ndarray never had data allocated this is a no-op but does produce a debug log message if logging is enabled.

Parameters:

Storage and compression

int asdf_ndarray_compression_set(asdf_ndarray_t *ndarray, const char *compression)

Set the compression method to use for the ndarray data when writing

See also asdf_block_compression_set for which this is a shortcut (applies to the block created for holding this ndarray’s data).

Parameters:

  • ndarray – An asdf_ndarray_t* handle

  • compression – String representing the compressor to use (e.g. “bzp2”) if any, or NULL or the empty string to set no compression

Returns:

Non-zero if the compression could not be set (e.g. invalid/unknown compressor); use asdf_error to check the error code

asdf_array_storage_t asdf_ndarray_storage(asdf_ndarray_t *ndarray)

Return the storage mode that will be used when the ndarray is written.

If the ndarray was read from a file this reflects how it was originally stored. For a newly constructed ndarray this reflects whatever was last passed to asdf_ndarray_storage_set, or ASDF_ARRAY_STORAGE_INTERNAL if the storage was never explicitly set.

Parameters:
Returns:

The asdf_array_storage_t for this ndarray.

void asdf_ndarray_storage_set(asdf_ndarray_t *ndarray, asdf_array_storage_t storage)

Set the storage mode used when the ndarray is written.

ASDF_ARRAY_STORAGE_INLINE serializes the data as a nested YAML sequence under the data key. A warning is logged if the number of elements exceeds the configured threshold (see asdf_emitter_cfg_t.inline_ndarray_warning_thresh).

ASDF_ARRAY_STORAGE_INTERNAL writes the data in a binary block (the default when no storage mode is set).

ASDF_ARRAY_STORAGE_EXTERNAL is not yet supported; calling this function with that value logs an error and leaves the storage mode unchanged.

Parameters:
asdf_block_t *asdf_ndarray_block(asdf_ndarray_t *ndarray)

Get the pointer to the open asdf_block_t associated with the ndarray

This is mostly for debugging/low-level inspection and is not needed for typical use cases.

Parameters:
Returns:

A constant pointer to the asdf_block_t structure representing the binary block underlying the array, if any. This will be NULL, for example, if the ndarray used inline data.

Reading array data

asdf_ndarray_err_t asdf_ndarray_read_all(asdf_ndarray_t *ndarray, asdf_scalar_datatype_t dst_t, void **dst)

Read the full ndarray, copying into the provided buffer (or allocating a destination buffer if dst = NULL)

This is like asdf_ndarray_read_tile_ndim but with a default “tile” size of the full array. Like asdf_ndarray_read_tile_ndim it will also convert the data to the host native byte order if necessary, and can convert it to a different numeric type than the source array.

Parameters:

  • ndarray – The asdf_ndarray_t* handle to the ndarray

  • dst_t – An asdf_scalar_datatype_t to convert to, or ASDF_DATATYPE_SOURCE to keep the original source datatype

  • dst – Pointer to a destination void* already allocated to receive the exact number of bytes in the source ndarray, or NULL to indicate that a buffer should be allocated. In the latter case the caller is responsible for freeing the allocated buffer.

Returns:

An asdf_ndarray_err_t; either ASDF_NDARRAY_OK if the data read successfully; otherwise the relevant error code.

asdf_ndarray_err_t asdf_ndarray_read_tile_ndim(asdf_ndarray_t *ndarray, const uint64_t *origin, const uint64_t *shape, asdf_scalar_datatype_t dst_t, void **dst)

Read tiles of up to N-dimensions out of N-D arrays

Tiles can be slices of any number of dimenions <= N and of any shape so long as they don’t go past the bounds of the array (otherwise ASDF_NDARRAY_ERR_OUT_OF_BOUNDS is returned).

Parameters:

  • ndarray – The asdf_ndarray_t* handle to the ndarray

  • origin – The indices of the first pixel of the tile–an array of size ndim

  • shape – The shape of the tile to read–an array of size ndim

  • dst_t –

    The output datatype, if conversion from the source array’s datatype to the output datatype is possible

    Currently, if no conversion is possible it will just copy the tile data without conversion–this may change in the future to become an error. You can pass the special value ASDF_DATATYPE_SOURCE to indicate that the output datatype is the source datatype.

  • dst – Pointer to a destination void* already allocated to receive the exact number of bytes in the output tile based on shape and datatype, or NULL to indicate that a buffer should be allocated. In the latter case the caller is responsible for freeing the allocated buffer.

Returns:

An asdf_ndarray_err_t; either ASDF_NDARRAY_OK if the data read successfully; otherwise the relevant error code.

asdf_ndarray_err_t asdf_ndarray_read_tile_2d(asdf_ndarray_t *ndarray, uint64_t x, uint64_t y, uint64_t width, uint64_t height, const uint64_t *plane_origin, asdf_scalar_datatype_t dst_t, void **dst)

Like asdf_ndarray_read_tile_ndim but with conveniences for the common 2-D case

Parameters:

  • ndarray – The asdf_ndarray_t* handle to the ndarray

  • x – The x coordinate of the tile origin

  • y – The y coordinate of the tile origin

  • width – The width of the tile in the x direction

  • height – The height of the tile in the y direction

  • plane_origin – If the source array is greater than 2-dimensional, the ndim - 2 array of plane coordinates–may be NULL if either the source array is 2-D or otherwise the outer-most plane is used

  • dst_t –

    The output datatype, if conversion from the source array’s datatype to the output datatype is possible

    Currently, if no conversion is possible it will just copy the tile data without conversion–this may change in the future to become an error. You can pass the special value ASDF_DATATYPE_SOURCE to indicate that the output datatype is the source datatype.

  • dst – Pointer to a destination void* already allocated to receive the exact number of bytes in the output tile based on shape and datatype, or NULL to indicate that a buffer should be allocated. In the latter case the caller is responsible for freeing the allocated buffer.

Returns:

An asdf_ndarray_err_t; either ASDF_NDARRAY_OK if the data read successfully; otherwise the relevant error code.

asdf_ndarray_at(ndarray, type, ...)

Read the element at the given indices, converted to type

type must be one of the C scalar types named by the asdf_ndarray_read_<type>_at functions. The number of indices is taken from the number of arguments, and must equal the array’s ndim. At least one index is required:

double value = asdf_ndarray_at(ndarray, double, 3, 7);

Errors are not reported: on any error the value is zero, which is indistinguishable from an element whose value is zero. Use asdf_ndarray_at_err where that matters.

This is a macro and is unavailable in C++; use the asdf_ndarray_read_<type>_at functions instead.

Parameters:

  • ndarray – The asdf_ndarray_t* handle to the ndarray

  • type – The C scalar type to convert the element to

  • ... – The indices of the element to read

Returns:

The element converted to type, or zero on error

asdf_ndarray_at_err(ndarray, type, err, ...)

Like asdf_ndarray_at but reporting errors through err

The indices are given as a variadic argument list after err:

asdf_ndarray_err_t err = ASDF_NDARRAY_OK;
double value = asdf_ndarray_at_err(ndarray, double, &err, 3, 7);
Parameters:

  • ndarray – The asdf_ndarray_t* handle to the ndarray

  • type – The C scalar type to convert the element to

  • err – An asdf_ndarray_err_t* receiving the error code, or NULL

  • ... – The indices of the element to read

Returns:

The element converted to type, or zero on error

asdf_ndarray_err_t asdf_ndarray_read_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_scalar_datatype_t dst_t, void *dst)

Read a single element of the ndarray, converting it as asdf_ndarray_read_all does

The element is copied into dst, which must have room for one value of dst_t. dst need not be aligned for its type.

For C code the asdf_ndarray_at and asdf_ndarray_at_err macros are more convenient.

Note

If reading many elements of an array in a loop this call is inefficient; it is better to read multiple elements in a single pass using asdf_ndarray_read_tile_ndim, asdf_ndarray_read_tile_2d or asdf_ndarray_read_all depending on the use case. This method can still be useful for a few incidental single-element reads.

indices must have ndim entries; that many are read from it.

Parameters:
Returns:

An asdf_ndarray_err_t; ASDF_NDARRAY_OK if the element was read successfully, ASDF_NDARRAY_ERR_OUT_OF_BOUNDS if any index is outside the array’s shape; otherwise the relevant error code.

int8_t asdf_ndarray_read_int8_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

Read a single element of the ndarray, converted to the named type

These are the same as asdf_ndarray_read_at with dst_t fixed by the function name, except that the element is returned rather than copied into dst. On error the returned value is zero and, if err is not NULL, the error code is stored in it.

The asdf_ndarray_at and asdf_ndarray_at_err macros select among these automatically and are usually easier to read. They are unavailable in C++, which must call these directly.

indices must have ndim entries; that many are read from it.

Parameters:

  • ndarray – The asdf_ndarray_t* handle to the ndarray

  • indices – The indices of the element to read–an array of size ndim

  • err – Optional asdf_ndarray_err_t* receiving the error code, or NULL to ignore errors

Returns:

The element converted to the function’s type, or zero on error

uint8_t asdf_ndarray_read_uint8_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at

int16_t asdf_ndarray_read_int16_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at

uint16_t asdf_ndarray_read_uint16_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at

int32_t asdf_ndarray_read_int32_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at

uint32_t asdf_ndarray_read_uint32_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at

int64_t asdf_ndarray_read_int64_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at

uint64_t asdf_ndarray_read_uint64_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at

_Float16 asdf_ndarray_read_float16_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at; declared only if ASDF_HAVE_FLOAT16

float asdf_ndarray_read_float32_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at

double asdf_ndarray_read_float64_at(asdf_ndarray_t *ndarray, const uint64_t *indices, asdf_ndarray_err_t *err)

See asdf_ndarray_read_int8_at