C++ API¶
C++ API provides core functionalities for the NPU.
Classes¶
| Name | Description |
|---|---|
Accelerator |
Represents an accelerator, i.e., an NPU, used for executing models. |
BatchParam |
Struct containing BatchLLM parameters. |
Buffer |
A simple byte-sized buffer. |
BufferInfo |
Struct representing input/output buffer information. |
CacheInfo |
Struct representing KV-cache information. |
CoreId |
Represents a unique identifier for an NPU core. |
Future |
Represents a future for retrieving the result of asynchronous inference. |
FutureImpl |
|
Model |
Represents an AI model loaded from an MXQ file. |
ModelConfig |
Configures a core mode and core allocation of a model for NPU inference. |
ModelPackage |
Represents an MXQ package (.mxqpkg) that packs multiple MXQ files into a single file. |
ModelVariantHandle |
Handle to a specific variant of a loaded model. |
NDArray |
A class representing an N-dimensional array (NDArray). |
internal::NDArrayData |
A class representing the data underlying an NDArray. |
NPUData |
Represents a model input or output tensor that can reside either on the host (CPU) or on the NPU. |
PinnedMemory |
A buffer of NPU-accessible pinned (physically contiguous) memory. |
Scale |
Struct for scale values. |
Enumerations¶
StatusCode¶
Enumerates status codes for the qbruntime.
This enumeration defines return codes used in the qbruntime to indicate success or specific error conditions. A value of StatusCode::OK (0) represents success, while any other value indicates a particular error type.
| Value | Description |
|---|---|
OK = 0 |
OK |
InternalError = 23 |
Should never be reached, but reached anyway |
NotImplemented = 18 |
Not implemented |
BadAlloc = 39 |
Bad allocation |
Acc_AmbiguousDevice = 72 |
Multiple devices found; select one explicitly |
Acc_CoreAlreadyInUse = 11 |
Core already in use. |
Acc_DeviceNotFound = 71 |
Requested device not found in the system |
Acc_NPUTimeout = 42 |
NPU timeout |
Acc_NoIMemInitFound = 43 |
No imem initialization found |
Acc_NoSuchModel = 12 |
No such model |
Acc_TaskQueueNotFound = 28 |
Task queue not found |
Acc_UnknownDevice = 73 |
Unknown or unsupported device name |
Driver_FailedToAllocateHostMemory = 13 |
Failed to allocate host memory |
Driver_FailedToAllocateModelMemory = 44 |
Failed to allocate model memory |
Driver_FailedToBuildCmaIoReq = 45 |
Failed to build CMA IO request |
Driver_FailedToClaimCores = 24 |
Failed to claim cores |
Driver_FailedToFreeHostMemory = 67 |
Failed to free host memory |
Driver_FailedToFreeModelMemory = 46 |
Failed to free model memory |
Driver_FailedToLockCore = 49 |
Failed to lock core |
Driver_FailedToPostInfer = 33 |
Failed to post infer |
Driver_FailedToReadMemoryBuffer = 15 |
Failed to read memory buffer |
Driver_FailedToSyncMappedMemory = 68 |
Failed to sync mmaped memory |
Driver_FailedToUnclaimCores = 25 |
Failed to unclaim cores |
Driver_FailedToUnlockCore = 50 |
Failed to unlock core |
Driver_FailedToWaitDone = 34 |
Failed to wait done |
Driver_FailedToWriteMemoryBuffer = 14 |
Failed to write memory buffer |
Driver_NotInitialized = 1 |
Not implemented |
Driver_WaitDoneTimeout = 35 |
Wait done timeout |
Driver_WrongBaseAddress = 47 |
Wrong base address |
MemoryPool_AllocatorNotSet = 30 |
Allocator not set |
MemoryPool_BufNotFound = 29 |
Buffer not found |
Model_AlreadyLaunched = 65 |
Model has been launched already |
Model_AsyncPipelineCheckFailed = 55 |
Asynchronous pipeline check failed |
Model_AsyncPipelineNotAlive = 56 |
Asynchronous pipeline not alive |
Model_AsyncPipelineTimeout = 57 |
Asynchronous pipeline Timeout |
Model_BrokenMXQ = 20 |
Broken mxq |
Model_BufferSizeMismatched = 63 |
Size of buffer is mismatched |
Model_CacheNotFound = 74 |
KV-cache with given name or layer hash not found |
Model_CacheOverflow = 53 |
KV-cache overflow |
Model_DtypeMismatched = 22 |
dtype mismatched |
Model_FailedToAllocMemory = 32 |
Failed to allocate memory |
Model_FailedToFindDirectory = 61 |
Failed to find specified directory |
Model_FailedToLoadMXQ = 3 |
Failed to load mxq |
Model_FailedToOpenCacheFile = 62 |
Failed to open cache file |
Model_FailedToOpenModelDescFile = 19 |
Failed to open model description file |
Model_FailedToOpenOutputScale = 6 |
Deprecated: Scale file doesn't exist any more. |
Model_FailedToOpenScaleFile = 4 |
Deprecated: Scale file doesn't exist any more. |
Model_FailedToOpenSectionBinary = 8 |
Failed to open section binary |
Model_FailedToSaveTensor = 37 |
Failed to save tensor |
Model_NumCacheMismatched = 60 |
Size of KV-cache mismatched |
Model_InvalidBatchParam = 66 |
Invalid BatchParam |
Model_InvalidNPUDtype = 51 |
Invalid NPU dtype |
Model_InvalidOutputScaleValue = 7 |
Deprecated: Scale file doesn't exist any more. |
Model_InvalidRmemType = 52 |
Invalid RMEM type |
Model_InvalidScaleValue = 5 |
Invalid scale value |
Model_InvalidSupplementary = 59 |
Invalid supplementary inference |
Model_InvalidVariantIdx = 64 |
Invalid model variant idx |
Model_IsNotPacked = 27 |
Deprecated: Packed logic doesn't exist any more. |
Model_IsNotSupportedHardware = 48 |
Model is not supported on this target device |
Model_IsPacked = 26 |
Deprecated: Packed logic doesn't exist any more. |
Model_MXQAndModelConfigNotMatch = 16 |
mxq and model config not match |
Model_NoCache = 54 |
Model does not support KV-cache |
Model_NoGlobalCoreWithGlobalMultiMode = 17 |
No global core with global multi mode |
Model_NoTargetCores = 2 |
No target cores |
Model_NotAlive = 10 |
Not alive |
Model_NotLaunched = 9 |
Not launched |
Model_NotSupported = 69 |
Not supported api |
Model_PredictError = 36 |
Predict error |
Model_ShapeMismatched = 21 |
Shape mismatched |
Model_TaskQueueClosed = 40 |
Task queue closed |
Model_TaskQueueTimeout = 41 |
Task queue timeout |
Model_UnexpectedMemoryFormat = 38 |
Deprecated: |
ModelPackage_EntryNotFound = 76 |
No such entry in the MXQ package |
ModelPackage_FailedToOpen = 75 |
Failed to open the MXQ package |
ModelPackage_FailedToReadEntry = 77 |
Failed to read an entry of the MXQ package |
PinnedMemory_NotAvailable = 70 |
PinnedMemory buffer is not available |
Future_NotValid = 58 |
Future is not valid |
Cluster¶
Enumerates clusters in the ARIES NPU.
Note
The ARIES NPU consists of two clusters, each containing one global core and four local cores, totaling eight local cores. REGULUS has only a single cluster (Cluster0) with one local core (Core0).
| Value | Description |
|---|---|
Cluster0 = 1 << 16 |
Cluster 0 |
Cluster1 = 2 << 16 |
Cluster 1 |
Error = 0x7FFF'0000 |
Represents an invalid or uninitialized state. |
Core¶
Enumerates cores within a cluster in the ARIES NPU.
Note
The ARIES NPU consists of two clusters, each containing one global core and four local cores, totaling eight local cores. REGULUS has only a single cluster (Cluster0) with one local core (Core0).
| Value | Description |
|---|---|
Core0 = 1 |
Local core 0 |
Core1 = 2 |
Local core 1 |
Core2 = 3 |
Local core 2 |
Core3 = 4 |
Local core 3 |
All = 0x0000'FFFC |
Deprecated |
GlobalCore = 0x0000'FFFE |
Global core |
Error = 0x0000'FFFF |
Represents an invalid or uninitialized state. |
CoreAllocationPolicy¶
Core allocation policy.
| Value | Description |
|---|---|
Auto |
Auto |
Manual |
Manual |
CoreMode¶
Defines the core mode for NPU execution.
Supported core modes include single-core, multi-core, global4-core, and global8-core. For detailed explanations of each mode, refer to the following functions:
ModelConfig::setAutoCoreModeModelConfig::setSingleCoreModeModelConfig::setMultiCoreModeModelConfig::setGlobal4CoreModeModelConfig::setGlobal8CoreMode
| Value | Description |
|---|---|
Single = 0 |
Single-core mode |
Multi = 1 |
Multi-core mode |
Global = 2 |
Deprecated |
Global4 = 3 |
Global4-core mode |
Global8 = 4 |
Global8-core mode |
Auto = 0xE |
Detects model's CoreMode automatically. |
Error = 0xF |
Represents an invalid or uninitialized state. |
LogLevel¶
| Value | Description |
|---|---|
Debug = 1 |
|
Info = 2 |
|
Warn = 3 |
|
Err = 4 |
|
Fatal = 5 |
|
Off = 6 |
CacheType¶
| Value | Description |
|---|---|
Default = 0 |
|
Batch |
|
FixedTailCache |
|
Error = 0x0F |
DataType¶
| Value | Description |
|---|---|
Float32 = 0 |
|
Float16 |
|
Int8 |
|
Uint8 |
|
BInt16 |
|
Error = 0x0F |
Functions¶
statusCodeToString¶
Convert StatusCode into string.
Parameters
status_code[in] — The StatusCode to convert.
Returns
String of each StatusCode if valid, else nullptr.
operator!¶
Checks whether the given StatusCode represents an error.
This operator enables StatusCode to be used in conditional statements for error checking. It returns false if sc is StatusCode::OK (0) and true otherwise.
StatusCode sc = someFunction();
if (!sc) { // `!sc` is true when `sc` is not `StatusCode::OK`.
// Handle error
}
Parameters
sc[in] — The StatusCode to evaluate.
Returns
True if sc is an error (nonzero), false if it is StatusCode::OK.
getQbRuntimeVersion¶
Retrieves the version of the qbruntime.
Returns
A string representing the runtime version.
getQbRuntimeGitVersion¶
Retrieves the Git commit hash of the qbruntime.
Returns
A string containing the Git hash.
getQbRuntimeVendor¶
Deprecated
This function returns nothing.
getQbRuntimeProduct¶
Deprecated
This function returns nothing.
setLogLevel¶
startTracingEvents¶
Starts event tracing and prepares to save the trace log to a specified file.
The trace log is recorded in "Chrome Tracing JSON format," which can be viewed at https://ui.perfetto.dev/.
The trace log is not written immediately; it is saved only when stopTracingEvents() is called.
Parameters
path[in] — The file path where the trace log should be stored.
Returns
True if tracing starts successfully, false otherwise.
stopTracingEvents¶
Stops event tracing and writes the recorded trace log.
This function finalizes tracing and saves the collected trace data to the file specified when startTracingEvents() was called.
getModelSummary¶
Generates a structured summary of the specified MXQ model.
Returns an overview of the model contained in the MXQ file, including:
- Target NPU device
- Supported core modes and their associated cores
- The total number of model variants
- For each variant:
- Input and output tensor shapes
- A list of layers with their types, output shapes, and input layer indices
The summary is returned as a human-readable string in a table and is useful for inspecting model compatibility, structure, and input/output shapes.
Parameters
mxq_path[in] — Path to the MXQ model file.
Returns
A formatted string containing the model summary.
Generates a structured summary of the specified MXQ entry in the MXQ package (.mxqpkg).
Same as the overload above, except that the model is read from the MXQ entry named entry_name in the package, without unpacking it.
Parameters
mxq_pkg_path[in] — The path to the MXQ package file.entry_name[in] — The name of the MXQ entry in the package (e.g. "block_0_expert_0.mxq").
Returns
A formatted string containing the model summary.
getAvailableDevices¶
Get the device names of the NPU devices detected on the system.
Returns
The detected device names (e.g. "aries-rb", "regulus-rb"). Empty if no NPU device is detected.
getAvailableDeviceNumbers¶
Get the number of available NPU devices.
Equivalent to getAvailableDeviceNumbers("auto").
Returns
The number of available NPU devices.
Get the numbers of available devices of the given device.
Parameters
device_name[in] — The device name ("auto"or a name such as"aries","aries-rb","regulus-rb", ...; case-insensitive).
Returns
The available device numbers. Empty if device_name cannot be resolved.