Awkward Array: Awkward Array

Loading...
Searching...
No Matches
Awkward Array

Awkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms.

Arrays are dynamically typed, but operations on them are compiled and fast. Their behavior coincides with NumPy when array dimensions are regular and generalizes when they’re not.

## Documentation

- Main page (quickstart)
- C++ API reference: this site
- Python API reference
- GitHub repository

## Navigation

The Awkward Array project is divided into 3 layers with 5 main components.

The high-level interface and Numba implementation are described in the Python API reference, though the Numba implementation is considered an internal detail (in the `_connect` submodule, which starts with an underscore and is therefore private in Python).

This reference describes the

- C++ classes: in namespace awkward (often abbreviated as "ak"), which are compiled into libawkward.so (or dylib or lib).
- pybind11 interface: no namespace, but contained entirely within the python directory, which are compiled into awkward._ext for use in Python.
- CPU kernels: no namespace, but contained entirely within the kernels directory, which are compiled into libawkward-cpu-kernels.so (or dylib or lib). This library is fully usable from any language that can call functions through FFI.
- GPU kernels: FIXME! (not implemented yet)

### Array layout nodes

Awkward Arrays are a tree of layout nodes nested within each other, some of which carry one-dimensional arrays that get reinterpreted as part of the structure. These descend from the abstract base class ak::Content and are reflected through pybind11 as ak.layout.Content and subclasses in Python.

- ak::Content: the abstract base class.
- ak::EmptyArray: an array of unknown type with no elements (usually produced by ak::ArrayBuilder, which can't determine type at a given level without samples).
- ak::RawArrayOf<T>: a one-dimensional array of any fixed-bytesize C++ type `T`. RawArray.h is a header-only implementation and ak::RawArrayOf<T> are not used anywhere else in the Awkward Array project, Python in particular. This node type only exists for pure C++ dependent projects.
- ak::NumpyArray: any NumPy array (e.g. multidimensional shape, arbitrary dtype), though usually only one-dimensional arrays of numbers. Contrary to its name, ak::NumpyArray can be used in pure C++, without dependence on pybind11 or Python. The NumPy array is described in terms of `std::vectors` and a `std::string`.
- ak::RegularArray: splits its nested content into equal-length lists.
- ak::ListArrayOf<T>: splits its nested content into variable-length lists with full generality (may use its content non-contiguously, overlapping, or out-of-order).
- ak::ListOffsetArrayOf<T>: splits its nested content into variable-length lists, assuming contiguous, non-overlapping, in-order content.
- ak::RecordArray: represents a logical array of records with a "struct of arrays" layout in memory.
- ak::Record: represents a single record (a subclass of ak::Content in C++, but not Python).
- ak::IndexedArrayOf<T, ISOPTION>: rearranges and/or duplicates its content by lazily applying an integer index.
- ak::ByteMaskedArray: represents its content with missing values with an 8-bit boolean mask.
- ak::BitMaskedArray: represents its content with missing values with a 1-bit boolean mask.
- ak::UnmaskedArray: specifies that its content can contain missing values in principle, but no mask is supplied because all elements are non-missing.
- ak::UnionArrayOf<T, I>: interleaves a set of arrays as a tagged union, can represent heterogeneous data.
- ak::VirtualArray: generates an array on demand from an ak::ArrayGenerator or a ak::SliceGenerator and optionally caches the generated array in an ak::ArrayCache.
- ak::None: represents a missing value that will be converted to `None` in Python (a subclass of ak::Content in C++).

The ak::Record, ak::None, and ak::NumpyArray with empty shape are technically ak::Content in C++ even though they represent scalar data, rather than arrays. (This can be checked with the isscalar method.) This is because they are possible return values of methods that would ordinarily return ak::Contents, so they are subclasses to simplify the type hierarchy. However, in the Python layer, they are converted directly into Python scalars, such as ak.layout.Record (which isn't an ak.layout.Content subclass), Python's `None`, or a Python number/bool.

Most layout nodes contain another content node (ak::RecordArray and ak::UnionArrayOf<T, I> can contain more than one), thus forming a tree. Only ak::EmptyArray, ak::RawArrayOf<T>, and ak::NumpyArray cannot contain a content, and hence these are leaves of the tree.

Note that ak::PartitionedArray and its concrete class, ak::IrregularlyPartitionedArray, are not ak::Content because they cannot be nested within a tree. Partitioning is only allowed at the root of the tree.

Iterator for layout nodes: ak::Iterator.

Index for layout nodes: integer and boolean arrays that define the shape of the data structure, such as boolean masks in ak::ByteMaskedArray, are not ak::NumpyArray but a more constrained type called ak::IndexOf<T>.

Identities for layout nodes: ak::IdentitiesOf<T> are an optional surrogate key for certain join operations. (Not yet used.)

### High-level data types

This is the type of data in a high-level ak.Array or ak.Record as reported by ak.type. It represents as much information as a data analyst needs to know (e.g. the distinction between variable and fixed-length lists, but not the distinction between ak::ListArrayOf<T> and ak::ListOffsetArrayOf<T>).

- ak::Type: the abstract base class.
- ak::ArrayType: type of a non-composable, high-level ak.Array, which includes the length of the array.
- ak::UnknownType: a type that is not known because it is represented by an ak::EmptyArray.
- ak::PrimitiveType: a numeric or boolean type.
- ak::RegularType: lists of a fixed length; this `size` is part of the type description.
- ak::ListType: lists of unspecified or variable length.
- ak::RecordType: records with named fields or tuples with a fixed number of unnamed slots. The fields/slots and their types are part of the type description.
- ak::OptionType: data that may be missing.
- ak::UnionType: heterogeneous data selected from a short list of possibilities.

All concrete ak::Type subclasses are composable except ak::ArrayType.

### Low-level array forms

This is the type of a ak::Content array expressed with low-level granularity (e.g. including the distinction between ak::ListArrayOf<T> and ak::ListOffsetArrayOf<T>). There is a one-to-one relationship between ak::Content subclasses and ak::Form subclasses, and each ak::Form maps to only one ak::Type.

- ak::Form: the abstract base class.
- ak::EmptyForm for ak::EmptyArray.
- ak::RawForm for ak::RawArrayOf<T>.
- ak::NumpyForm for ak::NumpyArray.
- ak::RegularForm for ak::RegularArray.
- ak::ListForm for ak::ListArrayOf<T>.
- ak::ListOffsetForm for ak::ListOffsetArrayOf<T>.
- ak::RecordForm for ak::RecordArray.
- ak::IndexedForm for ak::IndexedArrayOf<T, ISOPTION> with `ISOPTION = false`.
- ak::IndexedOptionForm for ak::IndexedArrayOf<T, ISOPTION> with `ISOPTION = true`.
- ak::ByteMaskedForm for ak::ByteMaskedArray.
- ak::BitMaskedForm for ak::BitMaskedArray.
- ak::UnmaskedForm for ak::UnmaskedArray.
- ak::UnionForm for ak::UnionArrayOf<T, I>.
- ak::VirtualForm for ak::VirtualArray.

### ArrayBuilder structure

The ak.ArrayBuilder is an append-only array for generating data backed by ak.layout.ArrayBuilder (layout-level ArrayBuilder) and ak::ArrayBuilder (C++ implementation).

ak::ArrayBuilder is the front-end for a tree of ak::Builder instances. The structure of this tree indicates the current state of knowledge about the type of the data it's being filled with, and this tree can grow from any node. Types always grow in the direction of more generality, so the tree only gets bigger.

Here is an example that illustrates how knowledge about the type grows.

b = ak.ArrayBuilder()
# fill commands # as JSON # current array type
b.begin_list() # [ # 0 * var * unknown (initially, the type is unknown)
b.integer(1) # 1, # 0 * var * int64
b.integer(2) # 2, # 0 * var * int64
b.real(3) # 3.0 # 0 * var * float64 (all the integers have become floats)
b.end_list() # ], # 1 * var * float64
b.begin_list() # [ # 1 * var * float64
b.end_list() # ], # 2 * var * float64
b.begin_list() # [ # 2 * var * float64
b.integer(4) # 4, # 2 * var * float64
b.null() # null, # 2 * var * ?float64 (now the floats are nullable)
b.integer(5) # 5 # 2 * var * ?float64
b.end_list() # ], # 3 * var * ?float64
b.begin_list() # [ # 3 * var * ?float64
b.begin_record() # { # 3 * var * ?union[float64, {}]
b.field("x") # "x": # 3 * var * ?union[float64, {"x": unknown}]
b.integer(1) # 1, # 3 * var * ?union[float64, {"x": int64}]
b.field("y") # "y": # 3 * var * ?union[float64, {"x": int64, "y": unknown}]
b.begin_list() # [ # 3 * var * ?union[float64, {"x": int64, "y": var * unknown}]
b.integer(2) # 2, # 3 * var * ?union[float64, {"x": int64, "y": var * int64}]
b.integer(3) # 3 # 3 * var * ?union[float64, {"x": int64, "y": var * int64}]
b.end_list() # ] # 3 * var * ?union[float64, {"x": int64, "y": var * int64}]
b.end_record() # } # 3 * var * ?union[float64, {"x": int64, "y": var * int64}]
b.end_list() # ] # 4 * var * ?union[float64, {"x": int64, "y": var * int64}]
ak.to_list(b.snapshot())
# [[1.0, 2.0, 3.0], [], [4.0, None, 5.0], [{'x': 1, 'y': [2, 3]}]]
awkward::ArrayBuilder
User interface to the Builder system: the ArrayBuilder is a fixed reference while the Builder subclas...
Definition: ArrayBuilder.h:23

The ak::Builder instances contain arrays of accumulated data, and thus store both data (in these arrays) and type (in their tree structure). The hierarchy is not exactly the same as ak::Content/ak::Form (which are identical to each other) or ak::Type, since it reflects the kinds of data to be encountered in the input data: mostly JSON-like, but with a distinction between records with named fields and tuples with unnamed slots.

- ak::Builder: the abstract base class.
- ak::UnknownBuilder: the initial builder; a builder for unknown type; generates an ak::EmptyArray (which is why we have that class).
- ak::BoolBuilder: boolean type; generates a ak::NumpyArray.
- ak::Int64Builder: 64-bit integer type; generates a ak::NumpyArray. Appending integer data to boolean data generates a union; it does not promote the booleans to integers.
- ak::Float64Builder: 64-bit floating point type; generates a ak::NumpyArray. Appending floating-point data to integer data does not generate a union; it promotes integers to floating-point.
- ak::StringBuilder: UTF-8 encoded string or raw bytestring type; generates a ak::ListOffsetArrayOf<T> with parameter `"__array__"` equal to `"string"` or `"bytestring"`.
- ak::ListBuilder: list type; generates a ak::ListOffsetArrayOf<T>.
- ak::OptionBuilder: option type; generats an ak::IndexedArrayOf<T, ISOPTION> with `ISOPTION = true`.
- ak::RecordBuilder: record type with field names; generates a ak::RecordArray with a non-null ak::util::RecordLookup.
- ak::TupleBuilder: tuple type without field names; generates a ak::RecordArray with a null ak::util::RecordLookup.
- ak::UnionBuilder: union type; generates a ak::UnionArrayOf<T, I>.
- ak::IndexedBuilder: indexed ak::Content; inserts an existing array node into the new array under construction, referencing its elements with an ak::IndexedArrayOf<T, ISOPTION>. This way, complex structures can be included by reference, rather than by copying.

Options for building an array: ak::ArrayBuilderOptions are passed to every ak::Builder in the tree.

Buffers for building an array: ak::GrowableBuffer<T> is a one-dimensional array with append-only semantics, used both for buffers that will become ak::IndexOf<T> and buffers that will become ak::NumpyArray. It works like `std::vector` in that it replaces its underlying storage at logarithmically frequent intervals, but unlike a `std::vector` in that the underlying storage is a `std::shared_ptr<void*>`.

After an ak::ArrayBuilder is turned into a ak::Content with snapshot, the read-only data and its append-only source share buffers (with `std::shared_ptr<void*>`). This makes the snapshot operation fast and capable of being called frequently, and the `std::shared_ptr` manages the lifetime of buffers that stay in scope because a ak::Content is using it, even if a ak::GrowableBuffer<T> is not (because it reallocated its internal buffer).

Array building is not as efficient as computing with pre-built arrays because the type-discovery makes each access a tree-descent. Array building is also an exception to the rule that C++ implementations do not touch array data (do not dereference pointers in ak::RawArrayOf<T>, ak::NumpyArray, ak::IndexOf<T>, and ak::IdentitiesOf<T>). The ak::Builder instances append to their ak::GrowableBuffer<T>, which are assumed to exist in main memory, not a GPU.

### Reducers

Reducer operations like ak.sum and ak.max are implemented as a group for code reuse. The most complex part is how elements from variable-length lists are grouped when `axis != -1` (see ak::Content::reduce_next). The problem of actually computing sums and maxima is comparatively simple.

Each reducer has a class that implements the sum, max, etc.

- ak::Reducer: the abstract base class.
- ak::ReducerCount: for ak.count.
- ak::ReducerCountNonzero: for ak.count_nonzero.
- ak::ReducerSum: for ak.sum.
- ak::ReducerProd: for ak.prod.
- ak::ReducerAny: for ak.any.
- ak::ReducerAll: for ak.all.
- ak::ReducerMin: for ak.min.
- ak::ReducerMax: for ak.max.
- ak::ReducerArgmin: for ak.argmin.
- ak::ReducerArgmax: for ak.argmax.

### Slices

Many Python objects can be used as slices; they are each converted into a specialized C++ type before being passed to ak::Content::getitem.

- ak::Slice: represents a Python tuple of slice items, which selects multiple dimensions at once. A non-tuple slice in Python is wrapped as a single-item ak::Slice in C++.
- ak::SliceItem: abstract base class for non-tuple slice items.
- ak::SliceAt: represents a single integer as a slice item, which selects an element.
- ak::SliceRange: represents a Python `slice` object with `start`, `stop`, and `step`, which selects a range of elements.
- ak::SliceField: represents a single string as a slice item, which selects a record field or projects across all record fields in an array.
- ak::SliceFields: represents an iterable of strings as a slice item, which selects a set of record fields or projects across a set of all record fields in an array.
- ak::SliceNewAxis: represents `np.newaxis` (a.k.a. `None`), which inserts a new regular dimension of length 1 in the sliced output.
- ak::SliceEllipsis: represents `Elipsis` (a.k.a. `...`), which skips enough dimensions to put the rest of the slice items at the deepest possible level.
- ak::SliceArrayOf: represents an integer or boolean array (boolean arrays are converted into integers early), which do general element-selection (rearrangement, filtering, and duplication).
- ak::SliceJaggedOf: represents a one level of jaggedness in an array as a slice item.
- ak::SliceMissingOf: represents an array with missing values as a slice item.

Note: ak::SliceGenerator is not a ak::SliceItem; it's an ak::ArrayGenerator. (A case of clashing naming conventions.)

### Conversion to JSON

The following classes hide Awkward Array's dependence on RapidJSON, so that it's a swappable component. Header files do not reference RapidJSON, but the implementation of these classes do: ak::ToJson, ak::ToJsonString, ak::ToJsonPrettyString, ak::ToJsonFile, ak::ToJsonPrettyFile.

### CPU kernels and GPU kernels

The kernels library is separated from the C++ codebase by a pure C interface, and thus the kernels could be used by other languages.

The CPU kernels are implemented in these files:

- kernels/getitem: kernels to implement ak::Content::getitem.
- kernels/identities: kernels to implement ak::IdentitiesOf<T>.
- kernels/reducers: kernels to implement ak::Content::reduce_next.
- kernels/operations: kernels to implement all other operations.

The GPU kernels follow exactly the same interface, though a different implementation.

FIXME: kernel function names and arguments should be systematized in some searchable way.