콘텐츠로 이동

C++ API

C++ API provides core functionalities for the NPU.

Classes

Name Description
Accelerator Represents an accelerator, i.e., an NPU, used for executing models.
BatchParam Struct containing BatchLLM parameters.
Buffer A simple byte-sized buffer.
BufferInfo Struct representing input/output buffer information.
CacheInfo Struct representing KV-cache information.
CoreId Represents a unique identifier for an NPU core.
Future Represents a future for retrieving the result of asynchronous inference.
FutureImpl
Model Represents an AI model loaded from an MXQ file.
ModelConfig Configures a core mode and core allocation of a model for NPU inference.
ModelVariantHandle Handle to a specific variant of a loaded model.
NDArray A class representing an N-dimensional array (NDArray).
internal::NDArrayData A class representing the data underlying an NDArray.
NPUData Represents a model input or output tensor that can reside either on the host (CPU) or on the NPU.
PinnedMemory A buffer of NPU-accessible pinned (physically contiguous) memory.
Scale Struct for scale values.

Enumerations

StatusCode

enum class StatusCode

Enumerates status codes for the qbruntime.

This enumeration defines return codes used in the qbruntime to indicate success or specific error conditions. A value of StatusCode::OK (0) represents success, while any other value indicates a particular error type.

Value Description
OK = 0 OK
InternalError = 23 Should never be reached, but reached anyway
NotImplemented = 18 Not implemented
BadAlloc = 39 Bad allocation
Acc_AmbiguousDevice = 72 Multiple devices found; select one explicitly
Acc_CoreAlreadyInUse = 11 Core already in use.
Acc_DeviceNotFound = 71 Requested device not found in the system
Acc_NPUTimeout = 42 NPU timeout
Acc_NoIMemInitFound = 43 No imem initialization found
Acc_NoSuchModel = 12 No such model
Acc_TaskQueueNotFound = 28 Task queue not found
Acc_UnknownDevice = 73 Unknown or unsupported device name
Driver_FailedToAllocateHostMemory = 13 Failed to allocate host memory
Driver_FailedToAllocateModelMemory = 44 Failed to allocate model memory
Driver_FailedToBuildCmaIoReq = 45 Failed to build CMA IO request
Driver_FailedToClaimCores = 24 Failed to claim cores
Driver_FailedToFreeHostMemory = 67 Failed to free host memory
Driver_FailedToFreeModelMemory = 46 Failed to free model memory
Driver_FailedToLockCore = 49 Failed to lock core
Driver_FailedToPostInfer = 33 Failed to post infer
Driver_FailedToReadMemoryBuffer = 15 Failed to read memory buffer
Driver_FailedToSyncMappedMemory = 68 Failed to sync mmaped memory
Driver_FailedToUnclaimCores = 25 Failed to unclaim cores
Driver_FailedToUnlockCore = 50 Failed to unlock core
Driver_FailedToWaitDone = 34 Failed to wait done
Driver_FailedToWriteMemoryBuffer = 14 Failed to write memory buffer
Driver_NotInitialized = 1 Not implemented
Driver_WaitDoneTimeout = 35 Wait done timeout
Driver_WrongBaseAddress = 47 Wrong base address
MemoryPool_AllocatorNotSet = 30 Allocator not set
MemoryPool_BufNotFound = 29 Buffer not found
Model_AlreadyLaunched = 65 Model has been launched already
Model_AsyncPipelineCheckFailed = 55 Asynchronous pipeline check failed
Model_AsyncPipelineNotAlive = 56 Asynchronous pipeline not alive
Model_AsyncPipelineTimeout = 57 Asynchronous pipeline Timeout
Model_BrokenMXQ = 20 Broken mxq
Model_BufferSizeMismatched = 63 Size of buffer is mismatched
Model_CacheNotFound = 74 KV-cache with given name or layer hash not found
Model_CacheOverflow = 53 KV-cache overflow
Model_DtypeMismatched = 22 dtype mismatched
Model_FailedToAllocMemory = 32 Failed to allocate memory
Model_FailedToFindDirectory = 61 Failed to find specified directory
Model_FailedToLoadMXQ = 3 Failed to load mxq
Model_FailedToOpenCacheFile = 62 Failed to open cache file
Model_FailedToOpenModelDescFile = 19 Failed to open model description file
Model_FailedToOpenOutputScale = 6 Deprecated: Scale file doesn't exist any more.
Model_FailedToOpenScaleFile = 4 Deprecated: Scale file doesn't exist any more.
Model_FailedToOpenSectionBinary = 8 Failed to open section binary
Model_FailedToSaveTensor = 37 Failed to save tensor
Model_NumCacheMismatched = 60 Size of KV-cache mismatched
Model_InvalidBatchParam = 66 Invalid BatchParam
Model_InvalidNPUDtype = 51 Invalid NPU dtype
Model_InvalidOutputScaleValue = 7 Deprecated: Scale file doesn't exist any more.
Model_InvalidRmemType = 52 Invalid RMEM type
Model_InvalidScaleValue = 5 Invalid scale value
Model_InvalidSupplementary = 59 Invalid supplementary inference
Model_InvalidVariantIdx = 64 Invalid model variant idx
Model_IsNotPacked = 27 Deprecated: Packed logic doesn't exist any more.
Model_IsNotSupportedHardware = 48 Model is not supported on this target device
Model_IsPacked = 26 Deprecated: Packed logic doesn't exist any more.
Model_MXQAndModelConfigNotMatch = 16 mxq and model config not match
Model_NoCache = 54 Model does not support KV-cache
Model_NoGlobalCoreWithGlobalMultiMode = 17 No global core with global multi mode
Model_NoTargetCores = 2 No target cores
Model_NotAlive = 10 Not alive
Model_NotLaunched = 9 Not launched
Model_NotSupported = 69 Not supported api
Model_PredictError = 36 Predict error
Model_ShapeMismatched = 21 Shape mismatched
Model_TaskQueueClosed = 40 Task queue closed
Model_TaskQueueTimeout = 41 Task queue timeout
Model_UnexpectedMemoryFormat = 38 Deprecated:
PinnedMemory_NotAvailable = 70 PinnedMemory buffer is not available
Future_NotValid = 58 Future is not valid

Cluster

enum class Cluster

Enumerates clusters in the ARIES NPU.

Note

The ARIES NPU consists of two clusters, each containing one global core and four local cores, totaling eight local cores. REGULUS has only a single cluster (Cluster0) with one local core (Core0).

Value Description
Cluster0 = 1 << 16 Cluster 0
Cluster1 = 2 << 16 Cluster 1
Error = 0x7FFF'0000 Represents an invalid or uninitialized state.

Core

enum class Core

Enumerates cores within a cluster in the ARIES NPU.

Note

The ARIES NPU consists of two clusters, each containing one global core and four local cores, totaling eight local cores. REGULUS has only a single cluster (Cluster0) with one local core (Core0).

Value Description
Core0 = 1 Local core 0
Core1 = 2 Local core 1
Core2 = 3 Local core 2
Core3 = 4 Local core 3
All = 0x0000'FFFC Deprecated
GlobalCore = 0x0000'FFFE Global core
Error = 0x0000'FFFF Represents an invalid or uninitialized state.

CoreAllocationPolicy

enum class CoreAllocationPolicy

Core allocation policy.

Value Description
Auto Auto
Manual Manual

CoreMode

enum class CoreMode

Defines the core mode for NPU execution.

Supported core modes include single-core, multi-core, global4-core, and global8-core. For detailed explanations of each mode, refer to the following functions:

Value Description
Single = 0 Single-core mode
Multi = 1 Multi-core mode
Global = 2 Deprecated
Global4 = 3 Global4-core mode
Global8 = 4 Global8-core mode
Auto = 0xE Detects model's CoreMode automatically.
Error = 0xF Represents an invalid or uninitialized state.

LogLevel

enum class LogLevel

LogLevel.

Value Description
Debug = 1
Info = 2
Warn = 3
Err = 4
Fatal = 5
Off = 6

CacheType

enum class CacheType

CacheType.

Value Description
Default = 0
Batch
FixedTailCache
Error = 0x0F

DataType

enum class DataType

DataType.

Value Description
Float32 = 0
Float16
Int8
Uint8
BInt16
Error = 0x0F

Functions

statusCodeToString

const std::string statusCodeToString(const StatusCode status_code)

Convert StatusCode into string.

Parameters

Returns

String of each StatusCode if valid, else nullptr.

operator!

bool operator!(StatusCode sc)

Checks whether the given StatusCode represents an error.

This operator enables StatusCode to be used in conditional statements for error checking. It returns false if sc is StatusCode::OK (0) and true otherwise.

StatusCode sc = someFunction();
if (!sc) { // `!sc` is true when `sc` is not `StatusCode::OK`.
    // Handle error
}

Parameters

Returns

True if sc is an error (nonzero), false if it is StatusCode::OK.

getQbRuntimeVersion

std::string getQbRuntimeVersion()

Retrieves the version of the qbruntime.

Returns

A string representing the runtime version.

getQbRuntimeGitVersion

std::string getQbRuntimeGitVersion()

Retrieves the Git commit hash of the qbruntime.

Returns

A string containing the Git hash.

getQbRuntimeVendor

std::string getQbRuntimeVendor()

Deprecated

This function returns nothing.

getQbRuntimeProduct

std::string getQbRuntimeProduct()

Deprecated

This function returns nothing.

setLogLevel

void setLogLevel(LogLevel level)

startTracingEvents

bool startTracingEvents(const char *path)

Starts event tracing and prepares to save the trace log to a specified file.

The trace log is recorded in "Chrome Tracing JSON format," which can be viewed at https://ui.perfetto.dev/.

The trace log is not written immediately; it is saved only when stopTracingEvents() is called.

Parameters

  • path [in] — The file path where the trace log should be stored.

Returns

True if tracing starts successfully, false otherwise.

stopTracingEvents

void stopTracingEvents()

Stops event tracing and writes the recorded trace log.

This function finalizes tracing and saves the collected trace data to the file specified when startTracingEvents() was called.

getModelSummary

std::string getModelSummary(const std::string &mxq_path)

Generates a structured summary of the specified MXQ model.

Returns an overview of the model contained in the MXQ file, including:

  • Target NPU device
  • Supported core modes and their associated cores
  • The total number of model variants
  • For each variant:
    • Input and output tensor shapes
    • A list of layers with their types, output shapes, and input layer indices

The summary is returned as a human-readable string in a table and is useful for inspecting model compatibility, structure, and input/output shapes.

Parameters

  • mxq_path [in] — Path to the MXQ model file.

Returns

A formatted string containing the model summary.

getAvailableDevices

std::vector<std::string> getAvailableDevices()

Get the device names of the NPU devices detected on the system.

Returns

The detected device names (e.g. "aries-rb", "regulus-rb"). Empty if no NPU device is detected.

getAvailableDeviceNumbers

std::vector<int> getAvailableDeviceNumbers()

Get the number of available NPU devices.

Equivalent to getAvailableDeviceNumbers("auto").

Returns

The number of available NPU devices.

std::vector<int> getAvailableDeviceNumbers(const std::string &device_name)

Get the numbers of available devices of the given device.

Parameters

  • device_name [in] — The device name ("auto" or a name such as "aries", "aries-rb", "regulus-rb", ...; case-insensitive).

Returns

The available device numbers. Empty if device_name cannot be resolved.

internal::numel

int64_t internal::numel(const std::vector<int64_t> &shape)