Skip to content

PinnedMemory

#include <qbruntime/pinned_memory.h>

template <typename T>
class mobilint::PinnedMemory

A buffer of NPU-accessible pinned (physically contiguous) memory.

A PinnedMemory wraps a host-mapped buffer that is pinned in device memory so the NPU can read from / write to it directly, avoiding extra copies during inference. Instances are created through Accelerator::allocatePinnedMemory and consumed by Model::inferPinnedMemory.

Note

This is an experimental API and is only supported on Regulus hardware.

Note

Destroying the last handle to a buffer (or destroying the owning Accelerator) frees the buffer and the relocation state derived from it. This must not happen while a Model::inferPinnedMemory call that uses the buffer is in flight on another thread; doing so is undefined behavior.

Warning

Model::inferPinnedMemory does not take ownership of the buffers passed to it. The caller must keep every PinnedMemory handle given as an input or output alive for the entire duration of the call. Do not destroy the last handle to such a buffer, and do not destroy the owning Accelerator, from another thread while the call is running — the NPU may still be reading from / writing to the buffer, so freeing it mid-flight is undefined behavior. This is the caller's responsibility; the runtime does not synchronize the buffer's lifetime against an in-flight inference.

The object has shared ownership of the underlying buffer: copying produces another handle to the same buffer (no data is duplicated), and the buffer is released only once the last handle is destroyed, or when the owning Accelerator is destroyed first (in which case the buffer is reclaimed and the handles become unavailable). Use isAvailable() to check whether the buffer is still valid before use.

Template parameters

  • T — The element type of the buffer (uint8_t or float).

Member functions

Member Description
PinnedMemory ×3
operator= ×2
data Returns a pointer to the host-mapped buffer.
shape Returns the shape of the buffer.
size Returns the size of the buffer in bytes.
isAvailable Checks whether the buffer is allocated and still usable.

PinnedMemory

PinnedMemory()=delete
PinnedMemory(const PinnedMemory &other)
PinnedMemory(PinnedMemory &&other) noexcept

operator=

PinnedMemory & operator=(const PinnedMemory &other)
PinnedMemory & operator=(PinnedMemory &&other) noexcept

data

T * data() const

Returns a pointer to the host-mapped buffer.

Returns

A pointer to the buffer, or nullptr if the buffer is not available.

shape

const std::vector<int64_t> & shape() const

Returns the shape of the buffer.

Returns

The shape used at allocation, or an empty vector if the buffer is not available.

size

uint64_t size() const

Returns the size of the buffer in bytes.

Returns

The buffer size in bytes, or 0 if the buffer is not available.

isAvailable

bool isAvailable() const

Checks whether the buffer is allocated and still usable.

A buffer is unavailable when allocation failed, when it was moved from, or when the owning Accelerator was destroyed and reclaimed the memory.

Returns

true if the buffer is available, false otherwise.