# Model

```python
class Model
```

Represents an AI model loaded from an MXQ file.

This class loads an AI model from an MXQ file and provides functions to launch it on the NPU and perform inference.

## Attributes

| Name | Description |
| --- | --- |
| [`target_cores`](<#target-cores>) |  |

### target\_cores

```python
property target_cores: List[CoreId]
```

> [!WARNING]
>
> **Deprecated**

## Methods

| Name | Description |
| --- | --- |
| [`__init__`](<#init>) | Creates a [Model](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/index.md>) object from the specified MXQ model file and configuration. |
| [`launch`](<#launch>) | Launches the model on the specified [Accelerator](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Accelerator/index.md>), which represents the actual NPU. |
| [`dispose`](<#dispose>) | Disposes of the model loaded onto the NPU. |
| [`is_target`](<#is-target>) | Checks if the NPU core specified by [CoreId](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/CoreId/index.md>) is the target of the model. In other words, whether the model is configured to use the given NPU core. |
| [`get_core_mode`](<#get-core-mode>) | Retrieves the core mode of the model. |
| [`get_target_cores`](<#get-target-cores>) | Returns the NPU cores the model is configured to use. |
| [`infer`](<#infer>) | Performs inference. |
| [`infer_hwc`](<#infer-hwc>) |  |
| [`infer_chw`](<#infer-chw>) |  |
| [`infer_to_float`](<#infer-to-float>) | int8\_t-to-float inference Performs inference with input and output elements of type `int8_t` |
| [`infer_hwc_to_float`](<#infer-hwc-to-float>) |  |
| [`infer_chw_to_float`](<#infer-chw-to-float>) |  |
| [`infer_buffer`](<#infer-buffer>) | [Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>)-to-Buffer inference |
| [`infer_speedrun`](<#infer-speedrun>) | Development-only API for measuring pure NPU inference speed. |
| [`infer_pinned_memory`](<#infer-pinned-memory>) | Performs inference directly on pinned memory buffers (zero-copy). |
| [`infer_async`](<#infer-async>) | Asynchronous Inference |
| [`infer_async_to_float`](<#infer-async-to-float>) | This method supports int8\_t-to-float asynchronous inference. |
| [`reposition_inputs`](<#reposition-inputs>) | Reposition input |
| [`reposition_outputs`](<#reposition-outputs>) | Reposition output |
| [`get_num_model_variants`](<#get-num-model-variants>) | Returns the total number of model variants available in this model. |
| [`get_model_variant_handle`](<#get-model-variant-handle>) | Retrieves a handle to the specified model variant. |
| [`get_model_input_shape`](<#get-model-input-shape>) | Returns the input shape of the model. |
| [`get_model_output_shape`](<#get-model-output-shape>) | Returns the output shape of the model. |
| [`get_input_scale`](<#get-input-scale>) | Returns the input quantization scale(s) of the model. |
| [`get_output_scale`](<#get-output-scale>) | Returns the output quantization scale(s) of the model. |
| [`get_input_buffer_info`](<#get-input-buffer-info>) | Returns the input buffer information for the model. |
| [`get_output_buffer_info`](<#get-output-buffer-info>) | Returns the output buffer information of the model. |
| [`get_model_input_data_type`](<#get-model-input-data-type>) | Returns a data type for model inputs. |
| [`get_model_output_data_type`](<#get-model-output-data-type>) | Returns a data type for model outputs. |
| [`acquire_input_buffer`](<#acquire-input-buffer>) | [Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) Management API |
| [`acquire_output_buffer`](<#acquire-output-buffer>) | [Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) Management API |
| [`release_buffer`](<#release-buffer>) | [Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) Management API |
| [`get_identifier`](<#get-identifier>) | Returns the model's unique identifier. |
| [`get_model_path`](<#get-model-path>) | Returns the path to the MXQ model file associated with the [Model](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/index.md>). |
| [`get_cache_infos`](<#get-cache-infos>) | Returns informations of KV-cache of the model. |
| [`get_latency_consumed`](<#get-latency-consumed>) |  |
| [`get_latency_finished`](<#get-latency-finished>) |  |
| [`dump_cache_memory`](<#dump-cache-memory>) | Dumps the KV cache memory into buffers. |
| [`load_cache_memory`](<#load-cache-memory>) | Loads the KV cache memory from buffers. |
| [`dump_cache_memory_to`](<#dump-cache-memory-to>) | Dumps KV cache memory to files in the specified directory. |
| [`load_cache_memory_from`](<#load-cache-memory-from>) | Loads the KV cache memory from files in the specified directory. |
| [`filter_cache_tail`](<#filter-cache-tail>) | Filter the tail of the KV cache memory |
| [`move_cache_tail`](<#move-cache-tail>) | Moves the tail of the KV cache memory to the end of the head. |

### \_\_init\_\_

```python
def __init__(self, path: str, model_config: Optional[ModelConfig] = None)
```

Creates a [Model](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/index.md>) object from the specified MXQ model file and configuration.

Parses the MXQ file and constructs a [Model](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/index.md>) object using the provided configuration, initializing the model with the given settings.

> [!NOTE]
>
> The created [Model](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/index.md>) object must be launched before performing inference. See [Model.launch](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/#launch>) for more details.

**Parameters**

- `path` *\[in\]* — The path to the MXQ model file.
- `model_config` *\[in\]* — The configuration settings to initialize the [Model](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/index.md>).

### launch

```python
def launch(self, acc: Accelerator) -> None
```

Launches the model on the specified [Accelerator](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Accelerator/index.md>), which represents the actual NPU.

**Parameters**

- `acc` *\[in\]* — The accelerator on which to launch the model.

### dispose

```python
def dispose(self) -> None
```

Disposes of the model loaded onto the NPU.

Releases any resources associated with the model on the NPU.

### is\_target

```python
def is_target(self, core_id: CoreId) -> bool
```

Checks if the NPU core specified by [CoreId](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/CoreId/index.md>) is the target of the model. In other words, whether the model is configured to use the given NPU core.

**Parameters**

- `core_id` *\[in\]* — The [CoreId](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/CoreId/index.md>) to check.

**Returns**

True if the model is configured to use the specified [CoreId](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/CoreId/index.md>), false otherwise.

### get\_core\_mode

```python
def get_core_mode(self) -> CoreMode
```

Retrieves the core mode of the model.

**Returns**

The [CoreMode](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/CoreMode/index.md>) of the model.

### get\_target\_cores

```python
def get_target_cores(self) -> List[CoreId]
```

Returns the NPU cores the model is configured to use.

**Returns**

A list of CoreIds representing the target NPU cores.

### infer

```python
def infer(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
    outputs: Optional[List[np.ndarray]] = None,
    cache_size: int = 0,
    params: Optional[List[BatchParam]] = None,
) -> Optional[List[np.ndarray]]
```

Performs inference.

Fowllowing types of inference supported.

1. infer(in:List\[numpy\]) -\> List\[numpy\] (float / int)
2. infer(in:numpy) -\> List\[numpy\] (float / int)
3. infer(in:List\[numpy\], out:List\[numpy\]) (float / int)
4. infer(in:List\[numpy\], out:List\[\]) (float / int)
5. infer(in:numpy, out:List\[numpy\]) (float / int)
6. infer(in:numpy, out:List\[\]) (float / int)

**Parameters**

- `inputs` *\[in\]* — Input data as a single numpy.ndarray or a list of numpy.ndarray's.
- `outputs` *\[out\]* — Optional pre-allocated list of numpy.ndarray's to store inference results.
- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.
- `params` *\[in\]* — A List of [`BatchParam`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/BatchParam/index.md>), specifying each batch's information for BatchLLM inference. If `params` is specified, `cache_size` is ignored.

**Returns**

Inference results as a list of numpy.ndarray.

### infer\_hwc

```python
def infer_hwc(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
    outputs: Optional[List[np.ndarray]] = None,
    cache_size: int = 0,
    params: Optional[List[BatchParam]] = None,
) -> Optional[List[np.ndarray]]
```

### infer\_chw

```python
def infer_chw(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
    outputs: Optional[List[np.ndarray]] = None,
    cache_size: int = 0,
    params: Optional[List[BatchParam]] = None,
) -> Optional[List[np.ndarray]]
```

### infer\_to\_float

```python
def infer_to_float(
    self,
    inputs: Union[
        np.ndarray,
        List[np.ndarray],
    ],
    cache_size: int = 0,
) -> List[np.ndarray]
```

int8\_t-to-float inference Performs inference with input and output elements of type `int8_t`

Using these inference APIs requires manual scaling (quantization) of float values to `int8_t` for input.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

### infer\_hwc\_to\_float

```python
def infer_hwc_to_float(
    self,
    inputs: Union[
        np.ndarray,
        List[np.ndarray],
    ],
    cache_size: int = 0,
) -> List[np.ndarray]
```

### infer\_chw\_to\_float

```python
def infer_chw_to_float(
    self,
    inputs: Union[
        np.ndarray,
        List[np.ndarray],
    ],
    cache_size: int = 0,
) -> List[np.ndarray]
```

### infer\_buffer

```python
def infer_buffer(
    self,
    inputs: List[Buffer],
    outputs: List[Buffer],
    shape: List[List[int]] = [],
    cache_size: int = 0,
) -> None
```

[Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>)-to-Buffer inference

Performs inference using input and output elements in the NPU’s internal data type. The inference operates on buffers allocated via the following APIs:

- [`Model.acquire_input_buffer()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/#acquire-input-buffer>)
- [`Model.acquire_output_buffer()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/#acquire-output-buffer>)
- [`ModelVariantHandle.acquire_input_buffer()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/ModelVariantHandle/#acquire-input-buffer>)
- [`ModelVariantHandle.acquire_output_buffer()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/ModelVariantHandle/#acquire-output-buffer>)

Additionally, [`Model.reposition_inputs()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/#reposition-inputs>), [`Model.reposition_outputs()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/#reposition-outputs>), [`ModelVariantHandle.reposition_inputs()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/ModelVariantHandle/#reposition-inputs>), [`ModelVariantHandle.reposition_outputs()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/ModelVariantHandle/#reposition-outputs>) must be used properly.

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

### infer\_speedrun

```python
def infer_speedrun(self) -> None
```

Development-only API for measuring pure NPU inference speed.

Runs NPU inference without uploading inputs and without retrieving outputs.

### infer\_pinned\_memory

```python
def infer_pinned_memory(
    self,
    inputs: List[PinnedMemory],
    outputs: Optional[List[PinnedMemory]] = None,
    cache_size: int = 0,
) -> Optional[List[PinnedMemory]]
```

Performs inference directly on pinned memory buffers (zero-copy).

The NPU reads the inputs from and writes the outputs into the pinned buffers in place, avoiding host-to-device copies of the I/O tensors. Write the input data into each input buffer via indexing (e.g. `pm[...] = data`) before calling, and read the results from the output buffers afterwards.

> [!NOTE]
>
> This is an experimental API and is only supported on Regulus hardware. It is not supported for models that use CPU offload.

**Parameters**

- `inputs` — A list of [PinnedMemory](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/PinnedMemory/index.md>) buffers holding the input data. All inputs must share the same dtype (`numpy.float32` or `numpy.uint8`).
- `outputs` — A list of pre-allocated [PinnedMemory](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/PinnedMemory/index.md>) buffers (dtype `numpy.float32`) that will receive the output data. If `None` (the default), the output buffers are allocated internally and returned by this method.
- `cache_size` — The size of the cache to use for inference.

**Returns**

If `outputs` is `None`, a list of newly allocated [PinnedMemory](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/PinnedMemory/index.md>) buffers (dtype `numpy.float32`) holding the output data; otherwise `None` (the given `outputs` are filled in place).

### infer\_async

```python
def infer_async(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
) -> Future
```

Asynchronous Inference

Performs inference asynchronously.

To use asynchronous inference, the model must be created using a [`ModelConfig`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/ModelConfig/index.md>) object with the async pipeline configured to be enabled. This is done by calling [ModelConfig.set\_async\_pipeline\_enabled(True)](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/ModelConfig/#set-async-pipeline-enabled>) before passing the configuration to `Model()`.

Example:

```python
import qbruntime

mc = qbruntime.ModelConfig()
mc.set_async_pipeline_enabled(True)

model = qbruntime.Model(MXQ_PATH, mc)
acc = qbruntime.Accelerator()

model.launch(acc)

future = model.infer_async(inputs)

ret = future.get()
```

> [!NOTE]
>
> Currently, only CNN-based models are supported, as asynchronous execution is particularly effective for this type of workload.

> [!NOTE]
>
> Limitations:
>
> - RNN/LSTM and LLM models are not supported yet.
> - Models requiring CPU offloading are not supported yet.
> - Currently, only single-batch inference is supported (i.e., N = 1).
> - Currently, [Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) inference is not supported. The following types are supported in the synchronous API for advanced use cases, but are not yet available for asynchronous inference:
> - [Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) to [Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>)
> - [Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) to float

### infer\_async\_to\_float

```python
def infer_async_to_float(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
) -> Future
```

This method supports int8\_t-to-float asynchronous inference.

**Parameters**

- `inputs` *\[in\]* — Input data as a single numpy.ndarray or a list of numpy.ndarray's.

**Returns**

A future that can be used to retrieve the inference result.

### reposition\_inputs

```python
def reposition_inputs(
    self,
    inputs: List[np.ndarray],
    input_bufs: List[Buffer],
    seqlens: List[List[int]] = [],
) -> None
```

Reposition input

### reposition\_outputs

```python
def reposition_outputs(
    self,
    output_bufs: List[Buffer],
    outputs: List[np.ndarray],
    seqlens: List[List[int]] = [],
) -> None
```

Reposition output

### get\_num\_model\_variants

```python
def get_num_model_variants(self) -> int
```

Returns the total number of model variants available in this model.

The `variant_idx` parameter passed to [`Model.get_model_variant_handle()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/#get-model-variant-handle>) must be in the range \[0, return value of this function).

**Returns**

The total number of model variants.

### get\_model\_variant\_handle

```python
def get_model_variant_handle(self, variant_idx) -> ModelVariantHandle
```

Retrieves a handle to the specified model variant.

Use the returned [`ModelVariantHandle`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/ModelVariantHandle/index.md>) to query details such as input and output shapes for the selected variant.

**Parameters**

- `variant_idx` *\[in\]* — Index of the model variant to retrieve. Must be in the range \[0, getNumModelVariants()).

**Returns**

A [`ModelVariantHandle`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/ModelVariantHandle/index.md>) object if successful; otherwise, raise qbruntime.QbRuntimeError "Model\_InvalidVariantIdx".

### get\_model\_input\_shape

```python
def get_model_input_shape(self) -> List[_Shape]
```

Returns the input shape of the model.

**Returns**

A list of input shape of the model.

### get\_model\_output\_shape

```python
def get_model_output_shape(self) -> List[_Shape]
```

Returns the output shape of the model.

**Returns**

A list of output shape of the model.

### get\_input\_scale

```python
def get_input_scale(self) -> List[Scale]
```

Returns the input quantization scale(s) of the model.

**Returns**

A list of input scales.

### get\_output\_scale

```python
def get_output_scale(self) -> List[Scale]
```

Returns the output quantization scale(s) of the model.

**Returns**

A list of output scales.

### get\_input\_buffer\_info

```python
def get_input_buffer_info(self) -> List[BufferInfo]
```

Returns the input buffer information for the model.

**Returns**

A list of input buffer information.

### get\_output\_buffer\_info

```python
def get_output_buffer_info(self) -> List[BufferInfo]
```

Returns the output buffer information of the model.

**Returns**

A list of output buffer information.

### get\_model\_input\_data\_type

```python
def get_model_input_data_type(self) -> DataType
```

Returns a data type for model inputs.

**Returns**

An input data type.

### get\_model\_output\_data\_type

```python
def get_model_output_data_type(self) -> DataType
```

Returns a data type for model outputs.

**Returns**

An output data type.

### acquire\_input\_buffer

```python
def acquire_input_buffer(self, seqlens: List[List[int]] = []) -> List[Buffer]
```

[Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) Management API

Acquires list of [`Buffer`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) for input. These API is required when calling [`Model.infer_buffer()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/#infer-buffer>).

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

### acquire\_output\_buffer

```python
def acquire_output_buffer(self, seqlens: List[List[int]] = []) -> List[Buffer]
```

[Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) Management API

Acquires list of [`Buffer`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) for output. These API is required when calling [`Model.infer_buffer()`](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/#infer-buffer>).

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

### release\_buffer

```python
def release_buffer(self, buffer: List[Buffer]) -> None
```

[Buffer](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Buffer/index.md>) Management API

Deallocate acquired Input/Output buffer

> [!NOTE]
>
> These APIs are intended for advanced use rather than typical usage.

### get\_identifier

```python
def get_identifier(self) -> int
```

Returns the model's unique identifier.

This identifier distinguishes multiple models within a single user program. It is assigned incrementally, starting from 0 (e.g., 0, 1, 2, 3, ...).

**Returns**

The model identifier.

### get\_model\_path

```python
def get_model_path(self) -> str
```

Returns the path to the MXQ model file associated with the [Model](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/Model/index.md>).

**Returns**

The MXQ file path.

### get\_cache\_infos

```python
def get_cache_infos(self) -> List[CacheInfo]
```

Returns informations of KV-cache of the model.

**Returns**

A list of [CacheInfo](<https://docs.mobilint.com/runtime/kr/v1.3/api/python/CacheInfo/index.md>) objects.

### get\_latency\_consumed

```python
def get_latency_consumed(self) -> int
```

> [!WARNING]
>
> **Deprecated**

### get\_latency\_finished

```python
def get_latency_finished(self) -> int
```

> [!WARNING]
>
> **Deprecated**

### dump\_cache\_memory

```python
def dump_cache_memory(self, cache_id: int = 0) -> List[bytes]
```

Dumps the KV cache memory into buffers.

Writes the current KV cache data into provided buffers.

**Parameters**

- `cache_id` *\[in\]* — Index of target cache.

**Returns**

A list of bytes containing the KV cache data.

### load\_cache\_memory

```python
def load_cache_memory(self, bufs: List[bytes], cache_id: int = 0) -> None
```

Loads the KV cache memory from buffers.

Restores the KV cache from the provided buffers.

**Parameters**

- `bufs` *\[in\]* — A list of bytes containing the KV cache

### dump\_cache\_memory\_to

```python
def dump_cache_memory_to(self, cache_dir: str, cache_id: int = 0) -> None
```

Dumps KV cache memory to files in the specified directory.

Writes the KV cache data to binary files within the given directory. Each file is named using the format: `cache_<layer_hash>.bin`.

**Parameters**

- `cache_dir` *\[in\]* — Path to the directory where KV cache files will be saved.
- `cache_id` *\[in\]* — Index of target cache.

### load\_cache\_memory\_from

```python
def load_cache_memory_from(self, cache_dir: str, cache_id: int = 0) -> None
```

Loads the KV cache memory from files in the specified directory.

Reads KV cache data from files within the given directory and restores them. Each file is named using the format: `cache_<layer_hash>.bin`.

**Parameters**

- `cache_dir` *\[in\]* — Path to the directory where KV cache files are saved.

### filter\_cache\_tail

```python
def filter_cache_tail(
    self, cache_size: int, tail_size: int, mask: List[bool]
) -> int
```

Filter the tail of the KV cache memory

Retains the desired caches in the tail of the KV cache memory, excludes the others, and shifts the remaining caches forward.

**Parameters**

- `cache_size` *\[in\]* — The number of tokens accumulated in the KV cache so far.
- `tail_size` *\[in\]* — The tail size of the KV cache to filter (\<=32).
- `mask` *\[in\]* — A mask indicating tokens to retain or exclude at the tail of the KV cache.

**Returns**

New cache size after tail filtering.

### move\_cache\_tail

```python
def move_cache_tail(self, num_head: int, num_tail: int, cache_size: int) -> int
```

Moves the tail of the KV cache memory to the end of the head.

Slice the tail of the KV cache memory up to the specified size and moves it to the designated cache position.

**Parameters**

- `num_head` *\[in\]* — The size of the KV cache head where the tail is appended.
- `num_tail` *\[in\]* — The size of the KV cache tail to be moved.
- `cache_size` *\[in\]* — The total number of tokens accumulated in the KV cache so far.

**Returns**

The updated cache size after moving the tail.
