# Model

```cpp
#include <qbruntime/model.h>

class mobilint::Model
```

Represents an AI model loaded from an MXQ file.

This class loads an AI model from an MXQ file and provides functions to launch it on the NPU and perform inference.

> [!WARNING]
>
> The MXQ file, or the MXQ package file, is read again whenever the model is launched. Moving or deleting the file before the launch makes the launch fail, and modifying it may make the model produce wrong results.

## NHWC float-to-float inference

Performs inference with input and output elements of type float in NHWC (batch N, height H, width W, channels C) or HWC format.

Two input-output type pairs are supported:

1. [`std::vector<NDArray<float>>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) for both input and output

   - Recommended approach, as [`NDArray`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) allows the qbruntime to avoid unnecessary data copies internally.
2. `std::vector<float*>` for input and `std::vector<std::vector<float>>` for output

   - Provided for user convenience, but results in unavoidable extra copies within the qbruntime.

| Member | Description |
| --- | --- |
| [`infer`](<#infer>) ×14 | Performs inference. |

### infer

```cpp
StatusCode infer(const std::vector<NDArray<float>> &input, std::vector<NDArray<float>> &output)
```

Performs inference.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NHWC or HWC format.
- `output` *\[out\]* — A reference to a vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) that will store the inference results.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<NDArray<float>> infer(const std::vector<NDArray<float>> &input, StatusCode &sc)
```

This overload differs from the above function in that it directly returns the inference results instead of modifying an output parameter.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NHWC or HWC format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) containing the inference results.

```cpp
StatusCode infer(const std::vector<float *> &input, std::vector<std::vector<float>> &output)
```

This overload is provided for convenience but may result in additional data copies within the qbruntime.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in HWC format.
- `output` *\[out\]* — A reference to a vector of float vectors that will store the inference results.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<std::vector<float>> infer(const std::vector<float *> &input, StatusCode &sc)
```

This overload is provided for convenience but may result in additional data copies within the qbruntime.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in HWC format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of float vectors containing the inference results.

```cpp
StatusCode infer(const std::vector<float *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape)
```

This overload is provided for convenience but may result in additional data copies within the qbruntime.

Unlike other overloads, this version allows explicitly specifying the shape of each input data, which can be in NHWC or HWC format.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NHWC or HWC format.
- `output` *\[out\]* — A reference to a vector of float vectors that will store the inference results.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<std::vector<float>> infer(const std::vector<float *> &input, const std::vector<std::vector<int64_t>> &shape, StatusCode &sc)
```

This overload is provided for convenience but may result in additional data copies within the qbruntime.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NHWC or HWC format.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of float vectors containing the inference results.

```cpp
StatusCode infer(const std::vector<NDArray<float>> &input, std::vector<NDArray<float>> &output, uint32_t cache_size)
```

This overload supports inference with KV cache.

> [!NOTE]
>
> This function is relevant for LLM models that use KV cache.

**Parameters**

- `input` *\[in\]* — A vector of NDArrays, where each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) represents input data in NHWC or HWC format.
- `output` *\[out\]* — A reference to a vector of NDArrays that will store the inference results.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
StatusCode infer(const std::vector<NDArray<float>> &input, std::vector<NDArray<float>> &output, const std::vector<BatchParam> &params)
```

This overload is supports inference with [BatchParam](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>) for BatchLLM.

**Parameters**

- `input` *\[in\]* — A vector of NDArrays, where each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) represents input data in NHWC or HWC format.
- `output` *\[out\]* — A reference to a vector of NDArrays that will store the inference results.
- `params` *\[in\]* — A vector of [`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>), specifying each Batch's information for BatchLLM inference.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<NDArray<float>> infer(const std::vector<NDArray<float>> &input, uint32_t cache_size, StatusCode &sc)
```

This overload supports inference with KV cache.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

> [!NOTE]
>
> This function is relevant for LLM models that use KV cache.

**Parameters**

- `input` *\[in\]* — A vector of NDArrays, where each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) represents input data in NHWC or HWC format.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of NDArrays containing the inference results.

```cpp
std::vector<NDArray<float>> infer(const std::vector<NDArray<float>> &input, const std::vector<BatchParam> &params, StatusCode &sc)
```

This overload is supports inference with [BatchParam](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>) for BatchLLM.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

**Parameters**

- `input` *\[in\]* — A vector of NDArrays, where each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) represents input data in NHWC or HWC format.
- `params` *\[in\]* — A vector of [`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>), specifying each Batch's information for BatchLLM inference.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of NDArrays containing the inference results.

```cpp
StatusCode infer(const std::vector<float *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size)
```

This overload supports inference with KV cache.

> [!NOTE]
>
> This function is relevant for LLM models that use KV cache.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NHWC or HWC format.
- `output` *\[out\]* — A reference to a vector of float vectors that will store the inference results.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<std::vector<float>> infer(const std::vector<float *> &input, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size, StatusCode &sc)
```

This overload supports inference with KV cache.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

> [!NOTE]
>
> This function is relevant for LLM models that use KV cache.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NHWC or HWC format.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of float vectors containing the inference results.

```cpp
StatusCode infer(const std::vector<float *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape, const std::vector<BatchParam> &params)
```

This overload is supports inference with [BatchParam](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>) for BatchLLM.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NHWC or HWC format.
- `output` *\[out\]* — A reference to a vector of float vectors that will store the inference results.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.
- `params` *\[in\]* — A vector of [`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>), specifying each Batch's information for BatchLLM inference.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<std::vector<float>> infer(const std::vector<float *> &input, const std::vector<std::vector<int64_t>> &shape, const std::vector<BatchParam> &params, StatusCode &sc)
```

This overload is supports inference with [BatchParam](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>) for BatchLLM.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NHWC or HWC format.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.
- `params` *\[in\]* — A vector of [`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>), specifying each Batch's information for BatchLLM inference.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of float vectors containing the inference results.

## NCHW float-to-float inference

Performs inference with input and output elements of type float in NCHW (batch N, channels C, height H, width W) or CHW format.

Two input-output type pairs are supported:

1. [`std::vector<NDArray<float>>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) for both input and output

   - Recommended approach, as [`NDArray`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) allows the qbruntime to avoid unnecessary data copies internally.
2. `std::vector<float*>` for input and `std::vector<std::vector<float>>` for output

   - Provided for user convenience, but results in unavoidable extra copies within the qbruntime.

> [!NOTE]
>
> CHW is not the recommended format, as the NPU natively operates on HWC-ordered data. When input is provided in CHW format, it will be transposed internally, introducing additional overhead.

> [!NOTE]
>
> If your data is in HWC format, use [`Model::infer`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#infer>) instead of [`Model::inferCHW`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#inferchw>), as it avoids unnecessary format conversion.

| Member | Description |
| --- | --- |
| [`inferCHW`](<#inferchw>) ×10 | Performs inference. |

### inferCHW

```cpp
StatusCode inferCHW(const std::vector<NDArray<float>> &input, std::vector<NDArray<float>> &output)
```

Performs inference.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NCHW or CHW format.
- `output` *\[out\]* — A reference to a vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) that will store the inference results.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<NDArray<float>> inferCHW(const std::vector<NDArray<float>> &input, StatusCode &sc)
```

This overload differs from the above function in that it directly returns the inference results instead of modifying an output parameter.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NCHW or CHW format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) containing the inference results.

```cpp
StatusCode inferCHW(const std::vector<float *> &input, std::vector<std::vector<float>> &output)
```

This overload is provided for convenience but may result in additional data copies within the qbruntime.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in CHW format.
- `output` *\[out\]* — A reference to a vector of float vectors that will store the inference results.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<std::vector<float>> inferCHW(const std::vector<float *> &input, StatusCode &sc)
```

This overload is provided for convenience but may result in additional data copies within the qbruntime.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in CHW format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of float vectors containing the inference results.

```cpp
StatusCode inferCHW(const std::vector<float *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape)
```

This overload is provided for convenience but may result in additional data copies within the qbruntime.

Unlike other overloads, this version allows explicitly specifying the shape of each input data, which can be in NCHW or CHW format.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NCHW or CHW format.
- `output` *\[out\]* — A reference to a vector of float vectors that will store the inference results.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<std::vector<float>> inferCHW(const std::vector<float *> &input, const std::vector<std::vector<int64_t>> &shape, StatusCode &sc)
```

This overload is provided for convenience but may result in additional data copies within the qbruntime.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NCHW or CHW format.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of float vectors containing the inference results.

```cpp
StatusCode inferCHW(const std::vector<NDArray<float>> &input, std::vector<NDArray<float>> &output, uint32_t cache_size)
```

This overload supports inference with KV cache.

> [!NOTE]
>
> This function is relevant for LLM models that use KV cache.

**Parameters**

- `input` *\[in\]* — A vector of NDArrays, where each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) represents input data in NCHW or CHW format.
- `output` *\[out\]* — A reference to a vector of NDArrays that will store the inference results.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<NDArray<float>> inferCHW(const std::vector<NDArray<float>> &input, uint32_t cache_size, StatusCode &sc)
```

This overload supports inference with KV cache.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

> [!NOTE]
>
> This function is relevant for LLM models that use KV cache.

**Parameters**

- `input` *\[in\]* — A vector of NDArrays, where each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) represents input data in NCHW or CHW format.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of NDArrays containing the inference results.

```cpp
StatusCode inferCHW(const std::vector<float *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size)
```

This overload supports inference with KV cache.

> [!NOTE]
>
> This function is relevant for LLM models that use KV cache.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NCHW or CHW format.
- `output` *\[out\]* — A reference to a vector of float vectors that will store the inference results.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<std::vector<float>> inferCHW(const std::vector<float *> &input, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size, StatusCode &sc)
```

This overload supports inference with KV cache.

Unlike the above overload, this function returns the inference results directly instead of modifying an output parameter.

> [!NOTE]
>
> This function is relevant for LLM models that use KV cache.

**Parameters**

- `input` *\[in\]* — A vector of float pointers, where each pointer represents input data in NCHW or CHW format.
- `shape` *\[in\]* — A vector of vectors, where each inner vector specifies the shape of the corresponding input data.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference operation was successful or encountered an error.

**Returns**

A vector of float vectors containing the inference results.

## NHWC uint8\_t-to-float inference

Performs inference with input and output elements of type `uint8_t` in NHWC (batch N, height H, width W, channels C) or HWC format.

> [!NOTE]
>
> The `params` ([`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>)) argument is not supported yet in these overloads. It is silently ignored, and inference is performed as if no [`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>) had been given. For BatchLLM inference, use the float input overloads instead.

| Member | Description |
| --- | --- |
| [`infer`](<#infer-2>) ×14 |  |

### infer

```cpp
StatusCode infer(const std::vector<NDArray<uint8_t>> &input, std::vector<NDArray<float>> &output)
```

```cpp
std::vector<NDArray<float>> infer(const std::vector<NDArray<uint8_t>> &input, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<uint8_t *> &input, std::vector<std::vector<float>> &output)
```

```cpp
std::vector<std::vector<float>> infer(const std::vector<uint8_t *> &input, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<uint8_t *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape)
```

```cpp
std::vector<std::vector<float>> infer(const std::vector<uint8_t *> &input, const std::vector<std::vector<int64_t>> &shape, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<NDArray<uint8_t>> &input, std::vector<NDArray<float>> &output, uint32_t cache_size)
```

```cpp
std::vector<NDArray<float>> infer(const std::vector<NDArray<uint8_t>> &input, uint32_t cache_size, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<uint8_t *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size)
```

```cpp
std::vector<std::vector<float>> infer(const std::vector<uint8_t *> &input, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<NDArray<uint8_t>> &input, std::vector<NDArray<float>> &output, const std::vector<BatchParam> &params)
```

```cpp
std::vector<NDArray<float>> infer(const std::vector<NDArray<uint8_t>> &input, const std::vector<BatchParam> &params, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<uint8_t *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape, const std::vector<BatchParam> &params)
```

```cpp
std::vector<std::vector<float>> infer(const std::vector<uint8_t *> &input, const std::vector<std::vector<int64_t>> &shape, const std::vector<BatchParam> &params, StatusCode &sc)
```

## NCHW uint8\_t-to-float inference

Performs inference with input and output elements of type `uint8_t` in NCHW (batch N, channels C, height H, width W) or CHW format.

| Member | Description |
| --- | --- |
| [`inferCHW`](<#inferchw-2>) ×10 |  |

### inferCHW

```cpp
StatusCode inferCHW(const std::vector<NDArray<uint8_t>> &input, std::vector<NDArray<float>> &output)
```

```cpp
std::vector<NDArray<float>> inferCHW(const std::vector<NDArray<uint8_t>> &input, StatusCode &sc)
```

```cpp
StatusCode inferCHW(const std::vector<uint8_t *> &input, std::vector<std::vector<float>> &output)
```

```cpp
std::vector<std::vector<float>> inferCHW(const std::vector<uint8_t *> &input, StatusCode &sc)
```

```cpp
StatusCode inferCHW(const std::vector<uint8_t *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape)
```

```cpp
std::vector<std::vector<float>> inferCHW(const std::vector<uint8_t *> &input, const std::vector<std::vector<int64_t>> &shape, StatusCode &sc)
```

```cpp
StatusCode inferCHW(const std::vector<NDArray<uint8_t>> &input, std::vector<NDArray<float>> &output, uint32_t cache_size)
```

```cpp
std::vector<NDArray<float>> inferCHW(const std::vector<NDArray<uint8_t>> &input, uint32_t cache_size, StatusCode &sc)
```

```cpp
StatusCode inferCHW(const std::vector<uint8_t *> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size)
```

```cpp
std::vector<std::vector<float>> inferCHW(const std::vector<uint8_t *> &input, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size, StatusCode &sc)
```

## NHWC int8\_t-to-int8\_t inference

Performs inference with input and output elements of type `int8_t` in NHWC (batch N, height H, width W, channels C) or HWC format.

Using these inference APIs requires manual scaling (quantization) of float values to `int8_t` for input and `int8_t` to float for output.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

> [!NOTE]
>
> The `params` ([`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>)) argument is not supported yet in these overloads. It is silently ignored, and inference is performed as if no [`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/batchparam/index.md>) had been given. For BatchLLM inference, use the float input overloads instead.

| Member | Description |
| --- | --- |
| [`infer`](<#infer-3>) ×14 |  |

### infer

```cpp
StatusCode infer(const std::vector<NDArray<int8_t>> &input, std::vector<NDArray<int8_t>> &output)
```

```cpp
std::vector<NDArray<int8_t>> infer(const std::vector<NDArray<int8_t>> &input, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<int8_t *> &input, std::vector<std::vector<int8_t>> &output)
```

```cpp
std::vector<std::vector<int8_t>> infer(const std::vector<int8_t *> &input, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<int8_t *> &input, std::vector<std::vector<int8_t>> &output, const std::vector<std::vector<int64_t>> &shape)
```

```cpp
std::vector<std::vector<int8_t>> infer(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<NDArray<int8_t>> &input, std::vector<NDArray<int8_t>> &output, uint32_t cache_size)
```

```cpp
std::vector<NDArray<int8_t>> infer(const std::vector<NDArray<int8_t>> &input, uint32_t cache_size, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<int8_t *> &input, std::vector<std::vector<int8_t>> &output, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size)
```

```cpp
std::vector<std::vector<int8_t>> infer(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<NDArray<int8_t>> &input, std::vector<NDArray<int8_t>> &output, const std::vector<BatchParam> &params)
```

```cpp
std::vector<NDArray<int8_t>> infer(const std::vector<NDArray<int8_t>> &input, const std::vector<BatchParam> &params, StatusCode &sc)
```

```cpp
StatusCode infer(const std::vector<int8_t *> &input, std::vector<std::vector<int8_t>> &output, const std::vector<std::vector<int64_t>> &shape, const std::vector<BatchParam> &params)
```

```cpp
std::vector<std::vector<int8_t>> infer(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, const std::vector<BatchParam> &params, StatusCode &sc)
```

## NCHW int8\_t-to-int8\_t inference

Performs inference with input and output elements of type `int8_t` in NCHW (batch N, channels C, height H, width W) or CHW format.

Using these inference APIs requires manual scaling (quantization) of float values to `int8_t` for input and `int8_t` to float for output.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

| Member | Description |
| --- | --- |
| [`inferCHW`](<#inferchw-3>) ×10 |  |

### inferCHW

```cpp
StatusCode inferCHW(const std::vector<NDArray<int8_t>> &input, std::vector<NDArray<int8_t>> &output)
```

```cpp
std::vector<NDArray<int8_t>> inferCHW(const std::vector<NDArray<int8_t>> &input, StatusCode &sc)
```

```cpp
StatusCode inferCHW(const std::vector<int8_t *> &input, std::vector<std::vector<int8_t>> &output)
```

```cpp
std::vector<std::vector<int8_t>> inferCHW(const std::vector<int8_t *> &input, StatusCode &sc)
```

```cpp
StatusCode inferCHW(const std::vector<int8_t *> &input, std::vector<std::vector<int8_t>> &output, const std::vector<std::vector<int64_t>> &shape)
```

```cpp
std::vector<std::vector<int8_t>> inferCHW(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, StatusCode &sc)
```

```cpp
StatusCode inferCHW(const std::vector<NDArray<int8_t>> &input, std::vector<NDArray<int8_t>> &output, uint32_t cache_size)
```

```cpp
std::vector<NDArray<int8_t>> inferCHW(const std::vector<NDArray<int8_t>> &input, uint32_t cache_size, StatusCode &sc)
```

```cpp
StatusCode inferCHW(const std::vector<int8_t *> &input, std::vector<std::vector<int8_t>> &output, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size)
```

```cpp
std::vector<std::vector<int8_t>> inferCHW(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size, StatusCode &sc)
```

## NHWC int8\_t-to-float inference

Performs inference with input and output elements of type `int8_t` in NHWC (batch N, height H, width W, channels C) or HWC format.

Using these inference APIs requires manual scaling (quantization) of float values to `int8_t` for input.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

| Member | Description |
| --- | --- |
| [`inferToFloat`](<#infertofloat>) ×5 |  |

### inferToFloat

```cpp
std::vector<NDArray<float>> inferToFloat(const std::vector<NDArray<int8_t>> &input, StatusCode &sc)
```

```cpp
std::vector<std::vector<float>> inferToFloat(const std::vector<int8_t *> &input, StatusCode &sc)
```

```cpp
std::vector<std::vector<float>> inferToFloat(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, StatusCode &sc)
```

```cpp
std::vector<NDArray<float>> inferToFloat(const std::vector<NDArray<int8_t>> &input, uint32_t cache_size, StatusCode &sc)
```

```cpp
std::vector<std::vector<float>> inferToFloat(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size, StatusCode &sc)
```

## NCHW int8\_t-to-float inference

Performs inference with input and output elements of type `int8_t` in NCHW (batch N, channels C, height H, width W) or CHW format.

Using these inference APIs requires manual scaling (quantization) of float values to `int8_t` for input.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

| Member | Description |
| --- | --- |
| [`inferCHWToFloat`](<#inferchwtofloat>) ×5 |  |

### inferCHWToFloat

```cpp
std::vector<NDArray<float>> inferCHWToFloat(const std::vector<NDArray<int8_t>> &input, StatusCode &sc)
```

```cpp
std::vector<std::vector<float>> inferCHWToFloat(const std::vector<int8_t *> &input, StatusCode &sc)
```

```cpp
std::vector<std::vector<float>> inferCHWToFloat(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, StatusCode &sc)
```

```cpp
std::vector<NDArray<float>> inferCHWToFloat(const std::vector<NDArray<int8_t>> &input, uint32_t cache_size, StatusCode &sc)
```

```cpp
std::vector<std::vector<float>> inferCHWToFloat(const std::vector<int8_t *> &input, const std::vector<std::vector<int64_t>> &shape, uint32_t cache_size, StatusCode &sc)
```

## NHWC Buffer-to-Buffer inference

Performs inference using input and output elements in the NPU’s internal data type. The inference operates on buffers allocated via the following APIs:

- `Model::acquireInputBuffer`
- `Model::acquireOutputBuffer`
- `Model::acquireInputBuffers`
- `Model::acquireOutputBuffers`
- `ModelVariantHandle::acquireInputBuffer`
- `ModelVariantHandle::acquireOutputBuffer`
- `ModelVariantHandle::acquireInputBuffers`
- `ModelVariantHandle::acquireOutputBuffers`

Additionally, `Model::repositionInputs`, `Model::repositionOutputs`, `ModelVariantHandle::repositionInputs` and `ModelVariantHandle::repositionOutputs` must be used properly.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

| Member | Description |
| --- | --- |
| [`inferBuffer`](<#inferbuffer>) ×2 |  |

### inferBuffer

```cpp
StatusCode inferBuffer(const std::vector<Buffer> &input, std::vector<Buffer> &output, const std::vector<std::vector<int64_t>> &shape={}, uint32_t cache_size=0)
```

```cpp
StatusCode inferBuffer(const std::vector<std::vector<Buffer>> &input, std::vector<std::vector<Buffer>> &output, const std::vector<std::vector<int64_t>> &shape={}, uint32_t cache_size=0)
```

## NHWC Buffer-to-float inference

Performs inference using input and output elements in the NPU’s internal data type. The inference operates on buffers allocated via the following APIs:

- `Model::acquireInputBuffer`
- `Model::acquireInputBuffers`
- `ModelVariantHandle::acquireInputBuffer`
- `ModelVariantHandle::acquireInputBuffers`

Additionally, `Model::repositionInputs` and `ModelVariantHandle::repositionInputs` must be used properly.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

| Member | Description |
| --- | --- |
| [`inferBufferToFloat`](<#inferbuffertofloat>) ×4 |  |

### inferBufferToFloat

```cpp
StatusCode inferBufferToFloat(const std::vector<Buffer> &input, std::vector<NDArray<float>> &output, const std::vector<std::vector<int64_t>> &shape={}, uint32_t cache_size=0)
```

```cpp
StatusCode inferBufferToFloat(const std::vector<std::vector<Buffer>> &input, std::vector<NDArray<float>> &output, const std::vector<std::vector<int64_t>> &shape={}, uint32_t cache_size=0)
```

```cpp
StatusCode inferBufferToFloat(const std::vector<Buffer> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape={}, uint32_t cache_size=0)
```

```cpp
StatusCode inferBufferToFloat(const std::vector<std::vector<Buffer>> &input, std::vector<std::vector<float>> &output, const std::vector<std::vector<int64_t>> &shape={}, uint32_t cache_size=0)
```

## Asynchronous Inference

Performs inference asynchronously.

To use asynchronous inference, the model must be created using a [`ModelConfig`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/modelconfig/index.md>) object with the async pipeline configured to be enabled. This is done by calling [ModelConfig::setAsyncPipelineEnabled(true)](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/modelconfig/#setasyncpipelineenabled>) before passing the configuration to [`Model::create`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#create>).

Example:

```cpp
using namespace mobilint;

ModelConfig mc;

// Enables support for `inferAsync` and `inferAsyncCHW`
mc.setAsyncPipelineEnabled(true);

StatusCode sc;
std::unique_ptr<Model> model = Model::create("resnet50.mxq", mc, sc);
if (!sc) {
   // Handle error appropriately
}

// Now `inferAsync` can be called.
Future future = model->inferAsync(input, sc);
```

> [!NOTE]
>
> Functions in the `inferAsync` family (`inferAsync`, `inferAsyncCHW`, `inferAsyncToFloat`, `inferAsyncCHWToFloat`) typically return immediately. However, they may block if the input queue in the qbruntime is full.

> [!NOTE]
>
> For all functions in the `inferAsync` family (`inferAsync`, `inferAsyncCHW`, `inferAsyncToFloat`, `inferAsyncCHWToFloat`), the data provided through the `input` parameter must remain unmodified until the asynchronous inference has completed. Modifying this data during execution may result in invalid results.

> [!NOTE]
>
> Currently, only CNN-based models are supported, as asynchronous execution is particularly effective for this type of workload.

> [!NOTE]
>
> Limitations:
>
> - RNN/LSTM and LLM models are not supported yet.
> - Models requiring CPU offloading are not supported yet.
> - Currently, only single-batch inference is supported (i.e., N = 1).
> - Currently, [Buffer](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/buffer/index.md>) inference is not supported. The following types are supported in the synchronous API for advanced use cases, but are not yet available for asynchronous inference:
>
>   - [Buffer](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/buffer/index.md>) to [Buffer](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/buffer/index.md>)
>   - [Buffer](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/buffer/index.md>) to float

| Member | Description |
| --- | --- |
| [`inferAsync`](<#inferasync>) ×3 | Initiates asynchronous inference with input in NHWC (batch N, height H, width W, channels C) or HWC format. |
| [`inferAsyncCHW`](<#inferasyncchw>) ×3 | Initiates asynchronous inference with input in NCHW (batch N, channels C, height H, width W) or CHW format. |
| [`inferAsyncToFloat`](<#inferasynctofloat>) | This overload supports int8\_t-to-float asynchronous inference. |
| [`inferAsyncCHWToFloat`](<#inferasyncchwtofloat>) | This overload supports int8\_t-to-float asynchronous inference. |

### inferAsync

```cpp
Future<float> inferAsync(const std::vector<NDArray<float>> &input, StatusCode &sc)
```

Initiates asynchronous inference with input in NHWC (batch N, height H, width W, channels C) or HWC format.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NHWC or HWC format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the asynchronous inference request was successfully initiated or encountered an error.

**Returns**

A future that can be used to retrieve the inference result.

```cpp
Future<int8_t> inferAsync(const std::vector<NDArray<int8_t>> &input, StatusCode &sc)
```

This overload supports int8\_t-to-int8\_t asynchronous inference.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<int8_t>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NHWC or HWC format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the asynchronous inference request was successfully initiated or encountered an error.

**Returns**

A future that can be used to retrieve the inference result.

```cpp
Future<float> inferAsync(const std::vector<NDArray<uint8_t>> &input, StatusCode &sc)
```

This overload supports uint8\_t-to-float asynchronous inference.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<uint8_t>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NHWC or HWC format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the asynchronous inference request was successfully initiated or encountered an error.

**Returns**

A future that can be used to retrieve the inference result.

### inferAsyncCHW

```cpp
Future<float> inferAsyncCHW(const std::vector<NDArray<float>> &input, StatusCode &sc)
```

Initiates asynchronous inference with input in NCHW (batch N, channels C, height H, width W) or CHW format.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<float>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NCHW or CHW format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the asynchronous inference request was successfully initiated or encountered an error.

**Returns**

A future that can be used to retrieve the inference result.

```cpp
Future<int8_t> inferAsyncCHW(const std::vector<NDArray<int8_t>> &input, StatusCode &sc)
```

This overload supports int8\_t-to-int8\_t asynchronous inference.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<int8_t>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NCHW or CHW format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the asynchronous inference request was successfully initiated or encountered an error.

**Returns**

A future that can be used to retrieve the inference result.

```cpp
Future<float> inferAsyncCHW(const std::vector<NDArray<uint8_t>> &input, StatusCode &sc)
```

This overload supports uint8\_t-to-float asynchronous inference.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<uint8_t>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NCHW or CHW format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the asynchronous inference request was successfully initiated or encountered an error.

**Returns**

A future that can be used to retrieve the inference result.

### inferAsyncToFloat

```cpp
Future<float> inferAsyncToFloat(const std::vector<NDArray<int8_t>> &input, StatusCode &sc)
```

This overload supports int8\_t-to-float asynchronous inference.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<int8_t>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NHWC or HWC format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the asynchronous inference request was successfully initiated or encountered an error.

**Returns**

A future that can be used to retrieve the inference result.

### inferAsyncCHWToFloat

```cpp
Future<float> inferAsyncCHWToFloat(const std::vector<NDArray<int8_t>> &input, StatusCode &sc)
```

This overload supports int8\_t-to-float asynchronous inference.

**Parameters**

- `input` *\[in\]* — A vector of [`NDArray<int8_t>`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>). Each [NDArray](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/ndarray/index.md>) must be in NCHW or CHW format.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the asynchronous inference request was successfully initiated or encountered an error.

**Returns**

A future that can be used to retrieve the inference result.

## Buffer Management APIs

These APIs are required when calling `Model::inferBuffer` or `Model::inferBufferToFloat`.

Buffers are acquired using:

- `acquireInputBuffer`
- `acquireOutputBuffer`

Any acquired buffer must be released using:

- `releaseBuffer`
- `releaseBuffers`

Repositioning is handled by:

- `repositionInputs`
- `repositionOutputs`

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

| Member | Description |
| --- | --- |
| [`acquireInputBuffer`](<#acquireinputbuffer>) |  |
| [`acquireOutputBuffer`](<#acquireoutputbuffer>) |  |
| [`acquireInputBuffers`](<#acquireinputbuffers>) |  |
| [`acquireOutputBuffers`](<#acquireoutputbuffers>) |  |
| [`releaseBuffer`](<#releasebuffer>) |  |
| [`releaseBuffers`](<#releasebuffers>) |  |
| [`repositionInputs`](<#repositioninputs>) ×4 |  |
| [`repositionOutputs`](<#repositionoutputs>) ×4 |  |

### acquireInputBuffer

```cpp
std::vector<Buffer> acquireInputBuffer(const std::vector<std::vector<int>> &seqlens={}) const
```

### acquireOutputBuffer

```cpp
std::vector<Buffer> acquireOutputBuffer(const std::vector<std::vector<int>> &seqlens={}) const
```

### acquireInputBuffers

```cpp
std::vector<std::vector<Buffer>> acquireInputBuffers(const int batch_size, const std::vector<std::vector<int>> &seqlens={}) const
```

### acquireOutputBuffers

```cpp
std::vector<std::vector<Buffer>> acquireOutputBuffers(const int batch_size, const std::vector<std::vector<int>> &seqlens={}) const
```

### releaseBuffer

```cpp
StatusCode releaseBuffer(std::vector<Buffer> &buffer) const
```

### releaseBuffers

```cpp
StatusCode releaseBuffers(std::vector<std::vector<Buffer>> &buffers) const
```

### repositionInputs

```cpp
StatusCode repositionInputs(const std::vector<float *> &input, std::vector<Buffer> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
```

```cpp
StatusCode repositionInputs(const std::vector<uint8_t *> &input, std::vector<Buffer> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
```

```cpp
StatusCode repositionInputs(const std::vector<float *> &input, std::vector<std::vector<Buffer>> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
```

```cpp
StatusCode repositionInputs(const std::vector<uint8_t *> &input, std::vector<std::vector<Buffer>> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
```

### repositionOutputs

```cpp
StatusCode repositionOutputs(const std::vector<Buffer> &output_buf, std::vector<float *> &output, const std::vector<std::vector<int>> &seqlens={}) const
```

```cpp
StatusCode repositionOutputs(const std::vector<Buffer> &output_buf, std::vector<std::vector<float>> &output, const std::vector<std::vector<int>> &seqlens={}) const
```

```cpp
StatusCode repositionOutputs(const std::vector<std::vector<Buffer>> &output_buf, std::vector<float *> &output, const std::vector<std::vector<int>> &seqlens={}) const
```

```cpp
StatusCode repositionOutputs(const std::vector<std::vector<Buffer>> &output_buf, std::vector<std::vector<float>> &output, const std::vector<std::vector<int>> &seqlens={}) const
```

## NPUData Management APIs

These APIs hand out [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) objects, which represent a single model input or output tensor and own the storage backing it on the host and/or the NPU.

> [!WARNING]
>
> This API is in beta: it may still contain bugs, and its behavior may change in a future release.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

| Member | Description |
| --- | --- |
| [`acquireInputNPUData`](<#acquireinputnpudata>) ×2 | Acquires an [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) for the model input at the given index. |
| [`acquireOutputNPUData`](<#acquireoutputnpudata>) ×2 | Acquires an [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) for the model output at the given index. |
| [`infer`](<#infer-4>) ×2 | Performs inference using [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) for both inputs and outputs. |
| [`inferCHW`](<#inferchw-4>) ×2 |  |

### acquireInputNPUData

```cpp
NPUData acquireInputNPUData(const std::vector<int64_t> &shape, int idx, bool upload, StatusCode &sc)
```

Acquires an [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) for the model input at the given index.

**Parameters**

- `shape` *\[in\]* — The shape of the input tensor, in NHWC/HWC or NCHW/CHW layout.
- `idx` *\[in\]* — The index of the model input.
- `upload` *\[in\]* — If true, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the operation was successful or encountered an error.

**Returns**

The acquired [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>). Its contents are valid only when `sc` is OK.

```cpp
NPUData acquireInputNPUData(const std::vector<int64_t> &shape, int idx, bool upload, DataType dtype, StatusCode &sc)
```

Acquires an [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) with an explicit element type.

Same as the overload above, except that the element type is chosen by the caller instead of defaulting to the model's user-facing input type. Use this to hand the model data that is already quantized to the NPU-native type (typically `DataType::Int8`), which skips the host-side quantization the default (`DataType::Float32` / `DataType::Uint8`) types perform.

**Parameters**

- `shape` *\[in\]* — The shape of the input tensor, in NHWC/HWC or NCHW/CHW layout.
- `idx` *\[in\]* — The index of the model input.
- `upload` *\[in\]* — If true, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).
- `dtype` *\[in\]* — The element type of the data. Must be one of the types the model accepts for this input; otherwise `sc` is set to `Model_DtypeMismatched`.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the operation was successful or encountered an error.

**Returns**

The acquired [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>). Its contents are valid only when `sc` is OK.

### acquireOutputNPUData

```cpp
NPUData acquireOutputNPUData(const std::vector<int64_t> &shape, int idx, bool upload, StatusCode &sc)
```

Acquires an [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) for the model output at the given index.

**Parameters**

- `shape` *\[in\]* — The shape of the output tensor, in NHWC/HWC or NCHW/CHW layout.
- `idx` *\[in\]* — The index of the model output.
- `upload` *\[in\]* — If true, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the operation was successful or encountered an error.

**Returns**

The acquired [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>). Its contents are valid only when `sc` is OK.

```cpp
NPUData acquireOutputNPUData(const std::vector<int64_t> &shape, int idx, bool upload, DataType dtype, StatusCode &sc)
```

Acquires an [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) with an explicit element type.

Same as the overload above, except that the element type is chosen by the caller instead of defaulting to the model's user-facing output type. Use this to receive the raw NPU-native output (typically `DataType::Int8`), which skips the host-side dequantization the default type performs.

**Parameters**

- `shape` *\[in\]* — The shape of the output tensor, in NHWC/HWC or NCHW/CHW layout.
- `idx` *\[in\]* — The index of the model output.
- `upload` *\[in\]* — If true, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).
- `dtype` *\[in\]* — The element type of the data. Must be one of the types the model produces for this output; otherwise `sc` is set to `Model_DtypeMismatched`.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the operation was successful or encountered an error.

**Returns**

The acquired [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>). Its contents are valid only when `sc` is OK.

### infer

```cpp
StatusCode infer(std::vector<NPUData> &inputs, std::vector<NPUData> &outputs, uint32_t cache_size=0)
```

Performs inference using [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) for both inputs and outputs.

Every input and output [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) must share the same residency: either all of them are on the host (acquired with `upload = false`) or all of them are already on the NPU (acquired with `upload = true`). A mixed set results in an error.

When all of them are on the NPU, no reposition or copy is performed; the model instructions are relocated for the given input/output physical-address combination (only the first time a given combination is seen) and the NPU runs directly. The results are left in the output NPU memory and can be retrieved afterwards.

When all of them are on the host, the inputs are additionally repositioned and uploaded to the NPU before execution, and the outputs are downloaded and repositioned back to the host afterwards.

> [!WARNING]
>
> This API is in beta: it may still contain bugs, and its behavior may change in a future release.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

> [!NOTE]
>
> Only single-NPU-op (non-CPU-offload), relocatable (MXQv7+) models are supported.

> [!NOTE]
>
> `infer` requires every [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) to have been acquired with an NHWC/HWC shape and `inferCHW` with an NCHW/CHW shape; a mismatch fails with `Model_ShapeMismatched`.

**Parameters**

- `inputs` *\[in,out\]* — A vector of input [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>).
- `outputs` *\[in,out\]* — A vector of output [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>).
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.

**Returns**

A status code indicating whether the inference operation completed successfully or encountered an error.

```cpp
std::vector<NPUData> infer(std::vector<NPUData> &inputs, StatusCode &sc, uint32_t cache_size=0)
```

Performs inference using [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) inputs and returns the outputs.

Unlike the above overload, the output [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) are acquired internally (sized from the matching model variant and given the same residency as the inputs) and returned directly instead of being supplied by the caller.

> [!WARNING]
>
> This API is in beta: it may still contain bugs, and its behavior may change in a future release.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

> [!NOTE]
>
> Only single-NPU-op (non-CPU-offload), relocatable (MXQv7+) models are supported.

> [!NOTE]
>
> `infer` requires every [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) to have been acquired with an NHWC/HWC shape and `inferCHW` with an NCHW/CHW shape; a mismatch fails with `Model_ShapeMismatched`.

**Parameters**

- `inputs` *\[in,out\]* — A vector of input [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>).
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate success or the specific error condition.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.

**Returns**

A vector of output [`NPUData`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/npudata/index.md>) containing the inference results. Its contents are valid only when `sc` is OK.

### inferCHW

```cpp
StatusCode inferCHW(std::vector<NPUData> &inputs, std::vector<NPUData> &outputs, uint32_t cache_size=0)
```

```cpp
std::vector<NPUData> inferCHW(std::vector<NPUData> &inputs, StatusCode &sc, uint32_t cache_size=0)
```

## KV Cache Management

> [!NOTE]
>
> These APIs are used for LLM models that utilize KV cache.

| Member | Description |
| --- | --- |
| [`dumpCacheMemory`](<#dumpcachememory>) ×4 | Dumps the KV cache memory into buffers. |
| [`loadCacheMemory`](<#loadcachememory>) ×2 | Loads the KV cache memory from buffers. |
| [`dumpCacheMemoryByName`](<#dumpcachememorybyname>) ×2 | Dumps a single KV cache memory into a buffer. |
| [`dumpCacheMemoryByLayerHash`](<#dumpcachememorybylayerhash>) ×2 | Dumps a single KV cache memory into a buffer. |
| [`loadCacheMemoryByName`](<#loadcachememorybyname>) | Loads a single KV cache memory from a buffer. |
| [`loadCacheMemoryByLayerHash`](<#loadcachememorybylayerhash>) | Loads a single KV cache memory from a buffer. |
| [`resetFixedTailCacheMemory`](<#resetfixedtailcachememory>) | Reset the fixed tail KV cache memory. |
| [`filterCacheTail`](<#filtercachetail>) | Filter the tail of the KV cache memory. |
| [`moveCacheTail`](<#movecachetail>) | Moves the tail of the KV cache memory to the end of the head. |
| [`inferPinnedMemory`](<#inferpinnedmemory>) ×2 | Performs inference directly on pinned memory buffers. |

### dumpCacheMemory

```cpp
StatusCode dumpCacheMemory(std::vector<std::vector<int8_t>> &bufs, int cache_id=0)
```

Dumps the KV cache memory into buffers.

Writes the current KV cache data into provided buffers.

**Parameters**

- `bufs` *\[out\]* — A reference to vectors of byte vectors that will store the KV cache data.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the dump operation was successful or if an error occurred.

```cpp
std::vector<std::vector<int8_t>> dumpCacheMemory(StatusCode &sc)
```

Dumps the KV cache memory into buffers.

Writes the KV cache data into buffers and returns them.

**Parameters**

- `cache_id` *\[in\]* — Index of target cache.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the dump operation was successful or if an error occurred.

**Returns**

A vector of byte vectors containing the KV cache data.

```cpp
std::vector<std::vector<int8_t>> dumpCacheMemory(int cache_id, StatusCode &sc)
```

```cpp
StatusCode dumpCacheMemory(const std::string &cache_dir, int cache_id=0)
```

Dumps KV cache memory to files in the specified directory.

Writes the KV cache data to binary files within the given directory. Each file is named using the format: `cache_<layer_hash>.bin`.

**Parameters**

- `cache_dir` *\[in\]* — Path to the directory where KV cache files will be saved.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the dump operation was successful or if an error occurred.

### loadCacheMemory

```cpp
StatusCode loadCacheMemory(const std::vector<std::vector<int8_t>> &bufs, int cache_id=0)
```

Loads the KV cache memory from buffers.

Restores the KV cache from the provided buffers.

**Parameters**

- `bufs` *\[in\]* — A reference to a vector of byte vectors containing the KV cache data.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the load operation was successful or if an error occurred.

```cpp
StatusCode loadCacheMemory(const std::string &cache_dir, int cache_id=0)
```

Loads the KV cache memory from files in the specified directory.

Reads KV cache data from files within the given directory and restores them. Each file is named using the format: `cache_<layer_hash>.bin`.

**Parameters**

- `cache_dir` *\[in\]* — Path to the directory where KV cache files are saved.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the load operation was successful or if an error occurred.

### dumpCacheMemoryByName

```cpp
StatusCode dumpCacheMemoryByName(std::vector<int8_t> &buf, const std::string &name, int cache_id=0)
```

Dumps a single KV cache memory into a buffer.

Writes the current data of the KV cache whose `CacheInfo::name` matches `name` into the provided buffer.

**Parameters**

- `buf` *\[out\]* — A reference to a byte vector that will store the KV cache data. If it is empty, it is resized to the size of the target KV cache.
- `name` *\[in\]* — Name of the target KV cache. See `CacheInfo::name`.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the dump operation was successful or if an error occurred.

```cpp
std::vector<int8_t> dumpCacheMemoryByName(const std::string &name, StatusCode &sc, int cache_id=0)
```

Dumps a single KV cache memory into a buffer.

Writes the data of the KV cache whose `CacheInfo::name` matches `name` into a buffer and returns it.

**Parameters**

- `name` *\[in\]* — Name of the target KV cache. See `CacheInfo::name`.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the dump operation was successful or if an error occurred.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A byte vector containing the KV cache data.

### dumpCacheMemoryByLayerHash

```cpp
StatusCode dumpCacheMemoryByLayerHash(std::vector<int8_t> &buf, const std::string &layer_hash, int cache_id=0)
```

Dumps a single KV cache memory into a buffer.

Writes the current data of the KV cache whose `CacheInfo::layer_hash` matches `layer_hash` into the provided buffer.

**Parameters**

- `buf` *\[out\]* — A reference to a byte vector that will store the KV cache data. If it is empty, it is resized to the size of the target KV cache.
- `layer_hash` *\[in\]* — Layer hash of the target KV cache. See `CacheInfo::layer_hash`.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the dump operation was successful or if an error occurred.

```cpp
std::vector<int8_t> dumpCacheMemoryByLayerHash(const std::string &layer_hash, StatusCode &sc, int cache_id=0)
```

Dumps a single KV cache memory into a buffer.

Writes the data of the KV cache whose `CacheInfo::layer_hash` matches `layer_hash` into a buffer and returns it.

**Parameters**

- `layer_hash` *\[in\]* — Layer hash of the target KV cache. See `CacheInfo::layer_hash`.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the dump operation was successful or if an error occurred.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A byte vector containing the KV cache data.

### loadCacheMemoryByName

```cpp
StatusCode loadCacheMemoryByName(const std::vector<int8_t> &buf, const std::string &name, int cache_id=0)
```

Loads a single KV cache memory from a buffer.

Restores only the KV cache whose `CacheInfo::name` matches `name` from the provided buffer. The other KV caches are left untouched.

**Parameters**

- `buf` *\[in\]* — A byte vector containing the KV cache data. Its size must be equal to the size of the target KV cache.
- `name` *\[in\]* — Name of the target KV cache. See `CacheInfo::name`.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the load operation was successful or if an error occurred.

### loadCacheMemoryByLayerHash

```cpp
StatusCode loadCacheMemoryByLayerHash(const std::vector<int8_t> &buf, const std::string &layer_hash, int cache_id=0)
```

Loads a single KV cache memory from a buffer.

Restores only the KV cache whose `CacheInfo::layer_hash` matches `layer_hash` from the provided buffer. The other KV caches are left untouched.

**Parameters**

- `buf` *\[in\]* — A byte vector containing the KV cache data. Its size must be equal to the size of the target KV cache.
- `layer_hash` *\[in\]* — Layer hash of the target KV cache. See `CacheInfo::layer_hash`.
- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the load operation was successful or if an error occurred.

### resetFixedTailCacheMemory

```cpp
StatusCode resetFixedTailCacheMemory(int cache_id=0)
```

Reset the fixed tail KV cache memory.

Fills the entire memory region of every KV cache whose [`CacheType`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/#cachetype>) is `CacheType::FixedTailCache` with zeros. The caches of the other types are left untouched.

> [!NOTE]
>
> This does not update the cache size tracked by the caller; the caller must reset its own `cache_size` to 0 after this call.

**Parameters**

- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A status code indicating whether the reset operation was successful or if an error occurred.

### filterCacheTail

```cpp
int filterCacheTail(int cache_size, int tail_size, const std::vector<bool> &mask, StatusCode &sc)
```

Filter the tail of the KV cache memory.

Retains the desired caches in the tail of the KV cache memory, excludes the others, and shifts the remaining caches forward.

**Parameters**

- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.
- `tail_size` *\[in\]* — The tail size of the KV cache to filter (\<=32).
- `mask` *\[in\]* — A mask indicating tokens to retain or exclude at the tail of the KV cache.
- `sc` *\[out\]* — A status code indicating the outcome of the tail filtering.

**Returns**

New cache size after tail filtering.

### moveCacheTail

```cpp
int moveCacheTail(int num_head, int num_tail, int cache_size, StatusCode &sc)
```

Moves the tail of the KV cache memory to the end of the head.

Slice the tail of the KV cache memory up to the specified size and moves it to the designated cache position.

**Parameters**

- `num_head` *\[in\]* — The size of the KV cache head where the tail is appended.
- `num_tail` *\[in\]* — The size of the KV cache tail to be moved.
- `cache_size` *\[in\]* — The total number of tokens accumulated in the KV cache so far.
- `sc` *\[out\]* — A status code indicating the result of the tail move.

**Returns**

The updated cache size after moving the tail.

### inferPinnedMemory

```cpp
template <typename T>
std::vector<PinnedMemory<float>> inferPinnedMemory(const std::vector<PinnedMemory<T>> &inputs, uint32_t cache_size, StatusCode &sc)
```

Performs inference directly on pinned memory buffers.

This overload differs from the one below in that it directly returns the output pinned memory buffers instead of modifying an output parameter. The returned buffers are newly allocated.

Runs inference using NPU-accessible pinned buffers as the input storage and freshly allocated pinned buffers as the output storage, avoiding host-to-device copies of the I/O tensors.

> [!NOTE]
>
> This is an experimental API and is only supported on REGULUS device. It is not supported for models that use CPU offload.

> [!NOTE]
>
> The `inputs` buffers must be pre-allocated via [`Accelerator::allocatePinnedMemory`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/accelerator/#allocatepinnedmemory>) and must be available (see [`PinnedMemory::isAvailable`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/pinnedmemory/#isavailable>)) before calling this function.

> [!NOTE]
>
> Pinned inference on a given model must be called serially. Concurrent pinned inference calls on the same model are not supported.

**Template parameters**

- `T` — The element type of the input buffers (`uint8_t` or `float`).

**Parameters**

- `inputs` *\[in\]* — A vector of pinned memory buffers containing the input data.
- `cache_size` *\[in\]* — The size of the cache to use for inference.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the inference was successful or if an error occurred.

**Returns**

A vector of newly allocated pinned memory buffers containing the inference results.

```cpp
template <typename T>
StatusCode inferPinnedMemory(const std::vector<PinnedMemory<T>> &inputs, std::vector<PinnedMemory<float>> &outputs, uint32_t cache_size)
```

Performs inference directly on pinned memory buffers.

Runs inference using NPU-accessible pinned buffers as the input and output storage, avoiding host-to-device copies of the I/O tensors.

> [!NOTE]
>
> This is an experimental API and is only supported on REGULUS device. It is not supported for models that use CPU offload.

> [!NOTE]
>
> The `inputs` and `outputs` buffers must be pre-allocated via [`Accelerator::allocatePinnedMemory`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/accelerator/#allocatepinnedmemory>) and must be available (see [`PinnedMemory::isAvailable`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/pinnedmemory/#isavailable>)) before calling this function.

> [!NOTE]
>
> If `outputs` is empty (i.e. its size is 0), the qbruntime allocates the required output pinned memory buffers internally and populates `outputs` with them. Otherwise, `outputs` must be pre-allocated buffers whose count and shapes match the model's output; a mismatch results in an error.

> [!NOTE]
>
> Pinned inference on a given model must be called serially. Concurrent pinned inference calls on the same model are not supported.

**Template parameters**

- `T` — The element type of the input buffers (`uint8_t` or `float`).

**Parameters**

- `inputs` *\[in\]* — A vector of pinned memory buffers containing the input data.
- `outputs` *\[out\]* — A vector of pinned memory buffers to store the output data. If empty, output buffers are allocated internally and returned through this parameter.
- `cache_size` *\[in\]* — The size of the cache to use for inference.

**Returns**

A status code indicating whether the inference was successful or if an error occurred.

## Deprecated APIs

> [!NOTE]
>
> These APIs are deprecated and should not be used.

| Member | Description |
| --- | --- |
| [`infer`](<#infer-5>) ×2 |  |

### infer

```cpp
StatusCode infer(const std::vector<float *> &input, std::vector<std::vector<float>> &output, int batch_size)
```

> [!WARNING]
>
> **Deprecated**
>
> Use `infer(input, output, shape)` instead.

```cpp
std::vector<std::vector<float>> infer(const std::vector<float *> &input, int batch_size, StatusCode &sc)
```

> [!WARNING]
>
> **Deprecated**
>
> Use `infer(input, shape, sc)` instead.

## Static functions

| Member | Description |
| --- | --- |
| [`create`](<#create>) ×4 | Creates a [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object from the specified MXQ model file. |

### create

```cpp
static std::unique_ptr<Model> create(const std::string &mxq_path, StatusCode &sc)
```

Creates a [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object from the specified MXQ model file.

Parses the MXQ file and constructs a [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object. The model is initialized in auto-core mode.

> [!NOTE]
>
> The created [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object must be launched before performing inference. See [Model::launch](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#launch>) for more details.

**Parameters**

- `mxq_path` *\[in\]* — The path to the MXQ model file.
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the model was successfully created or if an error occurred.

**Returns**

A unique pointer to the created [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object.

```cpp
static std::unique_ptr<Model> create(const std::string &mxq_path, const ModelConfig &config, StatusCode &sc)
```

Creates a [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object from the specified MXQ model file and configuration.

Parses the MXQ file and constructs a [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object using the provided configuration, initializing the model with the given settings.

> [!NOTE]
>
> The created [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object must be launched before performing inference. See [Model::launch](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#launch>) for more details.

> [!NOTE]
>
> [`Model::create(path, {}, sc)`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#create>) is ambiguous with the MXQ package overload create(mxq\_pkg\_path, entry\_name, sc); pass `ModelConfig()` explicitly.

**Parameters**

- `mxq_path` *\[in\]* — The path to the MXQ model file.
- `config` *\[in\]* — The configuration settings to initialize the [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>).
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the model was successfully created or if an error occurred.

**Returns**

A unique pointer to the created [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object.

```cpp
static std::unique_ptr<Model> create(const std::string &mxq_pkg_path, const std::string &entry_name, StatusCode &sc)
```

Creates a [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object from an MXQ entry in an MXQ package (.mxqpkg).

Loads the MXQ entry named `entry_name` directly from the package without unpacking it. The model is initialized in auto-core mode.

> [!NOTE]
>
> The created [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object must be launched before performing inference. See [Model::launch](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#launch>) for more details.

> [!NOTE]
>
> [`Model::create(path, {}, sc)`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#create>) is ambiguous with this overload. To load a plain MXQ file with the default configuration, use create(mxq\_path, sc).

**Parameters**

- `mxq_pkg_path` *\[in\]* — The path to the MXQ package file.
- `entry_name` *\[in\]* — The name of the MXQ entry in the package (e.g. "block\_0\_expert\_0.mxq").
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the model was successfully created or if an error occurred.

**Returns**

A unique pointer to the created [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object.

```cpp
static std::unique_ptr<Model> create(const std::string &mxq_pkg_path, const std::string &entry_name, const ModelConfig &config, StatusCode &sc)
```

Creates a [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object from an MXQ entry in an MXQ package (.mxqpkg) and configuration.

Loads the MXQ entry named `entry_name` directly from the package without unpacking it, and constructs a [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object using the provided configuration, initializing the model with the given settings.

> [!NOTE]
>
> The created [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object must be launched before performing inference. See [Model::launch](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#launch>) for more details.

**Parameters**

- `mxq_pkg_path` *\[in\]* — The path to the MXQ package file.
- `entry_name` *\[in\]* — The name of the MXQ entry in the package (e.g. "block\_0\_expert\_0.mxq").
- `config` *\[in\]* — The configuration settings to initialize the [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>).
- `sc` *\[out\]* — A reference to a status code that will be updated to indicate whether the model was successfully created or if an error occurred.

**Returns**

A unique pointer to the created [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) object.

## Member functions

| Member | Description |
| --- | --- |
| [`Model`](<#model>) ×2 |  |
| [`operator=`](<#operator-assign>) ×2 |  |
| [`launch`](<#launch>) | Launches the model on the specified [Accelerator](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/accelerator/index.md>), which represents the actual NPU. |
| [`dispose`](<#dispose>) | Disposes of the model loaded onto the NPU. |
| [`getCoreMode`](<#getcoremode>) | Retrieves the core mode of the model. |
| [`isTarget`](<#istarget>) | Checks if the NPU core specified by [CoreId](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/coreid/index.md>) is the target of the model. In other words, whether the model is configured to use the given NPU core. |
| [`getTargetCores`](<#gettargetcores>) | Returns the NPU cores the model is configured to use. |
| [`inferSpeedrun`](<#inferspeedrun>) | Development-only API for measuring pure NPU inference speed. |
| [`getNumModelVariants`](<#getnummodelvariants>) | Returns the total number of model variants available in this model. |
| [`getModelVariantHandle`](<#getmodelvarianthandle>) | Retrieves a handle to the specified model variant. |
| [`getModelInputShape`](<#getmodelinputshape>) | Returns the input shape of the model. |
| [`getModelOutputShape`](<#getmodeloutputshape>) | Returns the output shape of the model. |
| [`getInputBufferInfo`](<#getinputbufferinfo>) | Returns the input buffer information for the model. |
| [`getOutputBufferInfo`](<#getoutputbufferinfo>) | Returns the output buffer information of the model. |
| [`getInputScale`](<#getinputscale>) | Returns the input quantization scale(s) of the model. |
| [`getOutputScale`](<#getoutputscale>) | Returns the output quantization scale(s) of the model. |
| [`getModelInputDataType`](<#getmodelinputdatatype>) | Returns a data type for model inputs. |
| [`getModelOutputDataType`](<#getmodeloutputdatatype>) | Returns a data type for model outputs. |
| [`getIdentifier`](<#getidentifier>) | Returns the model's unique identifier. |
| [`getModelPath`](<#getmodelpath>) | Returns the path to the MXQ model file associated with the [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>). |
| [`getPackageEntryName`](<#getpackageentryname>) | Returns the name of the MXQ entry if the [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) was created from an MXQ package, or an empty string otherwise. |
| [`getDeviceNames`](<#getdevicenames>) | Returns the supported target device name(s) this model can run on. |
| [`getCacheInfos`](<#getcacheinfos>) | Returns informations of KV-cache of the model. |

### Model

```cpp
Model(const Model &other)=delete
```

```cpp
Model(Model &&other) noexcept
```

### operator=

```cpp
Model & operator=(const Model &rhs)=delete
```

```cpp
Model & operator=(Model &&rhs) noexcept
```

### launch

```cpp
StatusCode launch(Accelerator &acc)
```

Launches the model on the specified [Accelerator](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/accelerator/index.md>), which represents the actual NPU.

**Parameters**

- `acc` *\[in\]* — The accelerator on which to launch the model.

**Returns**

A status code indicating whether the model was successfully launched or if an error occurred.

### dispose

```cpp
StatusCode dispose()
```

Disposes of the model loaded onto the NPU.

Releases any resources associated with the model on the NPU.

**Returns**

A status code indicating whether the disposal was successful or if an error occurred.

### getCoreMode

```cpp
CoreMode getCoreMode() const
```

Retrieves the core mode of the model.

**Returns**

The [CoreMode](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/#coremode>) of the model.

### isTarget

```cpp
bool isTarget(CoreId core_id) const
```

Checks if the NPU core specified by [CoreId](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/coreid/index.md>) is the target of the model. In other words, whether the model is configured to use the given NPU core.

**Parameters**

- `core_id` *\[in\]* — The [CoreId](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/coreid/index.md>) to check.

**Returns**

True if the model is configured to use the specified [CoreId](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/coreid/index.md>), false otherwise.

### getTargetCores

```cpp
std::vector<CoreId> getTargetCores() const
```

Returns the NPU cores the model is configured to use.

**Returns**

A vector of CoreIds representing the target NPU cores.

### inferSpeedrun

```cpp
StatusCode inferSpeedrun(int variant_idx=0)
```

Development-only API for measuring pure NPU inference speed.

Runs NPU inference without uploading inputs and without retrieving outputs.

> [!NOTE]
>
> For models with variable length inputs, this runs based on a sequence length of 1.

> [!NOTE]
>
> For models using a kv-cache, this runs based on a cache size of 0.

**Parameters**

- `variant_idx` *\[in\]* — Index of model variant to run

**Returns**

A status code indicating the result.

### getNumModelVariants

```cpp
int getNumModelVariants() const
```

Returns the total number of model variants available in this model.

The `variant_idx` parameter passed to [`Model::getModelVariantHandle`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#getmodelvarianthandle>) must be in the range \[0, return value of this function).

**Returns**

The total number of model variants.

### getModelVariantHandle

```cpp
std::unique_ptr<ModelVariantHandle> getModelVariantHandle(int variant_idx, StatusCode &sc) const
```

Retrieves a handle to the specified model variant.

Use the returned [`ModelVariantHandle`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/modelvarianthandle/index.md>) to query details such as input and output shapes for the selected variant.

**Parameters**

- `variant_idx` *\[in\]* — Index of the model variant to retrieve. Must be in the range \[0, [getNumModelVariants()](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#getnummodelvariants>)).
- `sc` *\[out\]* — A reference to a [StatusCode](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/#statuscode>) variable that will be updated to indicate success or failure.

**Returns**

A unique pointer to the corresponding [`ModelVariantHandle`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/modelvarianthandle/index.md>) if successful; otherwise, nullptr.

### getModelInputShape

```cpp
const std::vector<std::vector<int64_t>> & getModelInputShape() const
```

Returns the input shape of the model.

**Returns**

A reference to the input shape of the model.

### getModelOutputShape

```cpp
const std::vector<std::vector<int64_t>> & getModelOutputShape() const
```

Returns the output shape of the model.

**Returns**

A reference to the output shape of the model.

### getInputBufferInfo

```cpp
const std::vector<BufferInfo> & getInputBufferInfo() const
```

Returns the input buffer information for the model.

**Returns**

A reference to a vector of input buffer information.

### getOutputBufferInfo

```cpp
const std::vector<BufferInfo> & getOutputBufferInfo() const
```

Returns the output buffer information of the model.

**Returns**

A reference to a vector of output buffer information.

### getInputScale

```cpp
std::vector<Scale> getInputScale() const
```

Returns the input quantization scale(s) of the model.

**Returns**

A vector of input scales.

### getOutputScale

```cpp
std::vector<Scale> getOutputScale() const
```

Returns the output quantization scale(s) of the model.

**Returns**

A vector of output scales.

### getModelInputDataType

```cpp
DataType getModelInputDataType() const
```

Returns a data type for model inputs.

**Returns**

An input data type.

### getModelOutputDataType

```cpp
DataType getModelOutputDataType() const
```

Returns a data type for model outputs.

**Returns**

An output data type.

### getIdentifier

```cpp
uint32_t getIdentifier() const
```

Returns the model's unique identifier.

This identifier distinguishes multiple models within a single user program. It is assigned incrementally, starting from 0 (e.g., 0, 1, 2, 3, ...).

**Returns**

The model identifier.

### getModelPath

```cpp
std::string getModelPath() const
```

Returns the path to the MXQ model file associated with the [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>).

If the [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) was created from an MXQ package, this is the path to the package file. Use [getPackageEntryName()](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/#getpackageentryname>) to get the name of the MXQ entry in it.

**Returns**

The MXQ file path, or the MXQ package file path.

### getPackageEntryName

```cpp
std::string getPackageEntryName() const
```

Returns the name of the MXQ entry if the [Model](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/model/index.md>) was created from an MXQ package, or an empty string otherwise.

**Returns**

The MXQ entry name, or an empty string.

### getDeviceNames

```cpp
std::vector<std::string> getDeviceNames() const
```

Returns the supported target device name(s) this model can run on.

Returns target device names from the model's target device in the MXQ file. A single model may map to multiple names (e.g. a REGULUS model returns both SoC and USB devices). Any of the returned names can be passed to [`Accelerator::create`](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/accelerator/#create>).

**Returns**

A vector of supported target device names.

### getCacheInfos

```cpp
std::vector<CacheInfo> getCacheInfos() const
```

Returns informations of KV-cache of the model.

**Returns**

A vector of [CacheInfo](<https://docs.mobilint.com/runtime/kr/v1.6/api/cpp/cacheinfo/index.md>) objects.
