OpenVDB: PagedArray< ValueT, Log2PageSize > Class Template Reference

OpenVDB 3.2.0

Classes | Public Types | Public Member Functions | Static Public Member Functions | Friends | List of all members
PagedArray< ValueT, Log2PageSize > Class Template Reference

Concurrent page-based linear data structure with O(1) random access and std-compliant iterators. It is primarily intended for applications that involve multi-threading of dynamically growing linear arrays with fast random access. More...

`#include <PagedArray.h>`

## Classes
class | ConstIterator
struct | Fill
class | Iterator
class | Page
class | ValueBuffer

## Public Types
typedef ValueT | ValueType

## Public Member Functions
| PagedArray ()
| Default constructor. More...
| ~PagedArray ()
| Destructor removed all allocated pages. More...
size_t | push_back (const ValueType &value)
| Thread safe insertion, adds a new element at the end and increases the container size by one. More...
size_t | push_back_unsafe (const ValueType &value)
| Slightly faster then the thread-safe push_back above. More...
ValueType | pop_back ()
| Returns the last element, decrements the size by one. More...
void | shrink_to_fit ()
| Reduce the page table to fix the current size. More...
ValueType & | operator[] (size_t i)
| Return a reference to the value at the specified offset. More...
const ValueType & | operator[] (size_t i) const
| Return a const-reference to the value at the specified offset. More...
void | fill (const ValueType &v)
| Set all elements to the specified value. More...
void | resize (size_t size)
| Resize this array to the specified size. More...
void | resize (size_t size, const ValueType &v)
| Resize this array to the specified size and set all elements to the specified value. More...
size_t | size () const
| Return the number of elements in this array. More...
size_t | capacity () const
| Return the maximum number of elements that this array can contain without allocating more memory pages. More...
size_t | freeCount () const
| Return the number of additional elements that can be added to this array without allocating more memory pages. More...
size_t | pageCount () const
| Return the number of allocated memory pages. More...
size_t | memUsage () const
| Return the memory footprint of this array in bytes. More...
bool | isEmpty () const
| Return true if the container contains no elements. More...
bool | isPartiallyFull () const
| Return true if the page table is partially full, i.e. the last non-empty page contains less than pageSize() elements. More...
void | clear ()
| Removes all elements from the array and delete all pages. More...
Iterator | begin ()
| Return a non-const iterator pointing to the first element. More...
Iterator | end ()
| Return a non-const iterator pointing to the past-the-last element. More...
ConstIterator | cbegin () const
| Return a const iterator pointing to the first element. More...
ConstIterator | cend () const
| Return a const iterator pointing to the past-the-last element. More...
void | sort ()
| Parallel sort of all the elements in ascending order. More...
void | invSort ()
| Parallel sort of all the elements in descending order. More...
template<typename Functor >
void | sort ()
| Parallel sort of all the elements based on a custom functor with the api: More...
void | merge (PagedArray &other)
| Transfer all the elements (and pages) from the other array to this array. More...
void | print (std::ostream &os=std::cout) const
| Print information for debugging. More...

## Static Public Member Functions
static size_t | pageSize ()
| Return the number of elements per memory page. More...
static size_t | log2PageSize ()
| Return log2 of the number of elements per memory page. More...

## Friends
class | ValueBuffer

## Detailed Description

### template<typename ValueT, size_t Log2PageSize = 10UL>
class openvdb::v3_2_0::util::PagedArray< ValueT, Log2PageSize >

Concurrent page-based linear data structure with O(1) random access and std-compliant iterators. It is primarily intended for applications that involve multi-threading of dynamically growing linear arrays with fast random access.

Note
  Multiple threads can grow the page-table and push_back new elements concurrently. A ValueBuffer provides accelerated and threadsafe push_back at the cost of potentially re-ordering elements (when multiple instances are used).

This data structure employes contiguous pages of elements (like a std::deque) which avoids moving data when the capacity is out-grown and new pages are allocated. The size of the pages can be controlled with the Log2PageSize template parameter (defaults to 1024 elements of type ValueT).

There are three fundamentally different ways to insert elements to this container - each with different advanteges and disadvanteges.

The simplest way to insert elements is to use PagedArray::push_back e.g.

PagedArray<int> array;
for (int i=0; i<100000; ++i) array.push_back(i);

or with tbb task-based multi-threading

struct Functor1 {
Functor1(int n, PagedArray<int>& _array) : array(&_array) {
tbb::parallel_for(tbb::blocked_range<int>(0, n, PagedArray<int>::pageSize()), *this);
}
void operator()(const tbb::blocked_range<int>& r) const {
for (int i=r.begin(), n=r.end(); i!=n; ++i) array->push_back(i);
}
PagedArray<int>* array;
};
PagedArray<int> array;
Functor1 tmp(10000, array);

PagedArray::push_back has the advantage that it's thread-safe and preserves the ordering of the inserted elements. In fact it returns the linear offset to the added element which can then be used for fast O(1) random access. The disadvantage is it's the slowest of the three different ways of inserting elements.

The fastest way (by far) to insert elements is to use one (or more) instances of a PagedArray::ValueBuffer, e.g.

PagedArray<int> array;
PagedArray<int>::ValueBuffer buffer(array);
for (int i=0; i<100000; ++i) buffer.push_back(i);
buffer.flush();

or

PagedArray<int> array;
{//local scope of a single thread
PagedArray<int>::ValueBuffer buffer(array);
for (int i=0; i<100000; ++i) buffer.push_back(i);
}

or with tbb task-based multi-threading

struct Functor2 {
Functor2(int n, PagedArray<int>& array) : buffer(array) {
tbb::parallel_for(tbb::blocked_range<int>(0, n, PagedArray<int>::pageSize()), *this);
}
void operator()(const tbb::blocked_range<int>& r) const {
for (int i=r.begin(), n=r.end(); i!=n; ++i) buffer.push_back(i);
}
mutable typename PagedArray<int>::ValueBuffer buffer;
};
PagedArray<int> array;
Functor2 tmp(10000, array);

or with tbb Thread Local Storage for even better performance (due to fewer concurrent instantiations of partially full ValueBuffers)

struct Functor3 {
typedef tbb::enumerable_thread_specific<PagedArray<int>::ValueBuffer> PoolType;
Functor3(size_t n, PoolType& _pool) : pool(&_pool) {
tbb::parallel_for(tbb::blocked_range<int>(0, n, PagedArray<int>::pageSize()), *this);
}
void operator()(const tbb::blocked_range<int>& r) const {
PagedArray<int>::ValueBuffer& buffer = pool->local();
for (int i=r.begin(), n=r.end(); i!=n; ++i) buffer.push_back(i);
}
PoolType* pool;
};
PagedArray<int> array;
PagedArray<int>::ValueBuffer exemplar(array);//dummy used for initialization
Functor3::PoolType pool(exemplar);//thread local storage pool of ValueBuffers
Functor3 tmp(10000, pool);
for (Functor3::PoolType::iterator i=pool.begin(); i!=pool.end(); ++i) i->flush();

This technique generally outperforms PagedArray::push_back, std::vector::push_back, std::deque::push_back and even tbb::concurrent_vector::push_back. Additionally it is thread-safe as long as each thread has it's own instance of a PagedArray::ValueBuffer. The only disadvantage is the ordering of the elements is undefined if multiple instance of a PagedArray::ValueBuffer are employed. This is typically the case in the context of multi-threading, where the ordering of inserts are undefined anyway. Note that a local scope can be used to guarentee that the ValueBuffer has inerted all its elements by the time the scope ends. Alternatively the ValueBuffer can be explicitly flushed by calling ValueBuffer::flush.

The third way to insert elements is to resize the container and use random access, e.g.

PagedArray<int> array;
array.resize(100000);
for (int i=0; i<100000; ++i) array[i] = i;

or in terms of the random access iterator

PagedArray<int> array;
array.resize(100000);
for (PagedArray<int>::Iterator i=array.begin(); i!=array.end(); ++i) *i = i.pos();

While this approach is both fast and thread-safe it suffers from the major disadvantage that the problem size, i.e. number of elements, needs to be known in advance. If that's the case you might as well consider using std::vector or a raw c-style array! In other words the PagedArray is most useful in the context of applications that involve multi-threading of dynamically growing linear arrays that require fast random access.

## Member Typedef Documentation

## § ValueType

typedef ValueT ValueType

## Constructor & Destructor Documentation

## § PagedArray()

PagedArray | ( |  | ) |
| inline

Default constructor.

## § ~PagedArray()

~PagedArray | ( |  | ) |
| inline

Destructor removed all allocated pages.

## Member Function Documentation

## § begin()

Iterator begin | ( |  | ) |
| inline

Return a non-const iterator pointing to the first element.

## § capacity()

size_t capacity | ( |  | ) | const
| inline

Return the maximum number of elements that this array can contain without allocating more memory pages.

## § cbegin()

ConstIterator cbegin | ( |  | ) | const
| inline

Return a const iterator pointing to the first element.

## § cend()

ConstIterator cend | ( |  | ) | const
| inline

Return a const iterator pointing to the past-the-last element.

Warning
  Itrator does not point to a valid element and should not be dereferenced!

## § clear()

void clear | ( |  | ) |
| inline

Removes all elements from the array and delete all pages.

Warning
  Not thread-safe!

## § end()

Iterator end | ( |  | ) |
| inline

Return a non-const iterator pointing to the past-the-last element.

Warning
  Iterator does not point to a valid element and should not be dereferenced!

## § fill()

void fill | ( | const ValueType & | v | ) |
| inline

Set all elements to the specified value.

## § freeCount()

size_t freeCount | ( |  | ) | const
| inline

Return the number of additional elements that can be added to this array without allocating more memory pages.

## § invSort()

void invSort | ( |  | ) |
| inline

Parallel sort of all the elements in descending order.

## § isEmpty()

bool isEmpty | ( |  | ) | const
| inline

Return true if the container contains no elements.

## § isPartiallyFull()

bool isPartiallyFull | ( |  | ) | const
| inline

Return true if the page table is partially full, i.e. the last non-empty page contains less than pageSize() elements.

When the page table is partially full calling merge() or using a ValueBuffer will rearrange the ordering of existing elements.

## § log2PageSize()

static size_t log2PageSize | ( |  | ) |
| inlinestatic

Return log2 of the number of elements per memory page.

## § memUsage()

size_t memUsage | ( |  | ) | const
| inline

Return the memory footprint of this array in bytes.

## § merge()

void merge | ( | PagedArray< ValueT, Log2PageSize > & | other | ) |

Transfer all the elements (and pages) from the other array to this array.

Note
  The other PagedArray is empty on return.

Warning
  The ordering of elements is undefined if this page table is partially full!

## § operator[]() [1/2]

ValueType& operator[] | ( | size_t | i | ) |
| inline

Return a reference to the value at the specified offset.

Note
  This random access has constant time complexity.

Warning
  It is assumed that the i'th element is already allocated!

## § operator[]() [2/2]

const ValueType& operator[] | ( | size_t | i | ) | const
| inline

Return a const-reference to the value at the specified offset.

Note
  This random access has constant time complexity.

Warning
  It is assumed that the i'th element is already allocated!

## § pageCount()

size_t pageCount | ( |  | ) | const
| inline

Return the number of allocated memory pages.

## § pageSize()

static size_t pageSize | ( |  | ) |
| inlinestatic

Return the number of elements per memory page.

## § pop_back()

ValueType pop_back | ( |  | ) |
| inline

Returns the last element, decrements the size by one.

Consider subsequnetly calling shrink_to_fit to reduce the page table to match the new size.

Note
  Calling this method on an empty containter is undefined (as is also the case for std containers).

Warning
  If values were added to the container by means of multiple ValueBuffers the last value might not be what you expect since the ordering is generally not perserved. Only PagedArray::push_back preserves the ordering (or a single instance of a ValueBuffer).

## § print()

void print | ( | std::ostream & | os = `std::cout` | ) | const
| inline

Print information for debugging.

## § push_back()

size_t push_back | ( | const ValueType & | value | ) |
| inline

Thread safe insertion, adds a new element at the end and increases the container size by one.

Note
  Constant time complexity. May allocate a new page.

## § push_back_unsafe()

size_t push_back_unsafe | ( | const ValueType & | value | ) |
| inline

Slightly faster then the thread-safe push_back above.

Note
  For best performance consider using the ValueBuffer!

Warning
  Not thread-safe!

## § resize() [1/2]

void resize | ( | size_t | size | ) |
| inline

Resize this array to the specified size.

Note
  This will grow or shrink the page table.

Warning
  Not thread-safe!

## § resize() [2/2]

void resize | ( | size_t | size,
|  | const ValueType & | v
| ) |  |
| inline

Resize this array to the specified size and set all elements to the specified value.

Warning
  Not thread-safe!

## § shrink_to_fit()

void shrink_to_fit | ( |  | ) |

Reduce the page table to fix the current size.

Warning
  Not thread-safe!

## § size()

size_t size | ( |  | ) | const
| inline

Return the number of elements in this array.

## § sort() [1/2]

void sort | ( |  | ) |
| inline

Parallel sort of all the elements in ascending order.

## § sort() [2/2]

void sort | ( |  | ) |
| inline

Parallel sort of all the elements based on a custom functor with the api:

bool operator()(const ValueT& a, const ValueT& b)

which returns true if a comes before b.

## Friends And Related Function Documentation

## § ValueBuffer

friend class ValueBuffer
| friend

The documentation for this class was generated from the following file:

- PagedArray.h