PinnedMemory¶
A buffer of NPU-accessible pinned (physically contiguous) memory.
A PinnedMemory wraps a host-mapped buffer that is pinned in device memory so the NPU can read from / write to it directly, avoiding extra copies during inference. Instances are created through Accelerator::allocatePinnedMemory and consumed by Model::inferPinnedMemory.
Note
This is an experimental API and is only supported on Regulus hardware.
Note
Destroying the last handle to a buffer (or destroying the owning Accelerator) frees the buffer and the relocation state derived from it. This must not happen while a Model::inferPinnedMemory call that uses the buffer is in flight on another thread; doing so is undefined behavior.
Warning
Model::inferPinnedMemory does not take ownership of the buffers passed to it. The caller must keep every PinnedMemory handle given as an input or output alive for the entire duration of the call. Do not destroy the last handle to such a buffer, and do not destroy the owning Accelerator, from another thread while the call is running — the NPU may still be reading from / writing to the buffer, so freeing it mid-flight is undefined behavior. This is the caller's responsibility; the runtime does not synchronize the buffer's lifetime against an in-flight inference.
The object has shared ownership of the underlying buffer: copying produces another handle to the same buffer (no data is duplicated), and the buffer is released only once the last handle is destroyed, or when the owning Accelerator is destroyed first (in which case the buffer is reclaimed and the handles become unavailable). Use isAvailable() to check whether the buffer is still valid before use.
Template parameters
T— The element type of the buffer (uint8_torfloat).
Member functions¶
| Member | Description |
|---|---|
PinnedMemory ×3 |
|
operator= ×2 |
|
data |
Returns a pointer to the host-mapped buffer. |
shape |
Returns the shape of the buffer. |
size |
Returns the size of the buffer in bytes. |
isAvailable |
Checks whether the buffer is allocated and still usable. |
PinnedMemory¶
operator=¶
data¶
Returns a pointer to the host-mapped buffer.
Returns
A pointer to the buffer, or nullptr if the buffer is not available.
shape¶
Returns the shape of the buffer.
Returns
The shape used at allocation, or an empty vector if the buffer is not available.
size¶
Returns the size of the buffer in bytes.
Returns
The buffer size in bytes, or 0 if the buffer is not available.
isAvailable¶
Checks whether the buffer is allocated and still usable.
A buffer is unavailable when allocation failed, when it was moved from, or when the owning Accelerator was destroyed and reclaimed the memory.
Returns
true if the buffer is available, false otherwise.