Model¶
Represents an AI model loaded from an MXQ file.
This class loads an AI model from an MXQ file and provides functions to launch it on the NPU and perform inference.
Attributes¶
| Name | Description |
|---|---|
target_cores |
target_cores¶
Deprecated
Methods¶
| Name | Description |
|---|---|
__init__ |
Creates a Model object from the specified MXQ model file and configuration. |
launch |
Launches the model on the specified Accelerator, which represents the actual NPU. |
dispose |
Disposes of the model loaded onto the NPU. |
is_target |
Checks if the NPU core specified by CoreId is the target of the model. In other words, whether the model is configured to use the given NPU core. |
get_core_mode |
Retrieves the core mode of the model. |
get_target_cores |
Returns the NPU cores the model is configured to use. |
infer |
Performs inference. |
infer_hwc |
|
infer_chw |
|
infer_to_float |
int8_t-to-float inference Performs inference with input and output elements of type int8_t |
infer_hwc_to_float |
|
infer_chw_to_float |
|
infer_buffer |
Buffer-to-Buffer inference |
infer_speedrun |
Development-only API for measuring pure NPU inference speed. |
infer_async |
Asynchronous Inference |
infer_async_to_float |
This method supports int8_t-to-float asynchronous inference. |
reposition_inputs |
Reposition input |
reposition_outputs |
Reposition output |
get_num_model_variants |
Returns the total number of model variants available in this model. |
get_model_variant_handle |
Retrieves a handle to the specified model variant. |
get_model_input_shape |
Returns the input shape of the model. |
get_model_output_shape |
Returns the output shape of the model. |
get_input_scale |
Returns the input quantization scale(s) of the model. |
get_output_scale |
Returns the output quantization scale(s) of the model. |
get_input_buffer_info |
Returns the input buffer information for the model. |
get_output_buffer_info |
Returns the output buffer information of the model. |
get_model_input_data_type |
Returns a data type for model inputs. |
get_model_output_data_type |
Returns a data type for model outputs. |
acquire_input_buffer |
Buffer Management API |
acquire_output_buffer |
Buffer Management API |
release_buffer |
Buffer Management API |
get_identifier |
Returns the model's unique identifier. |
get_model_path |
Returns the path to the MXQ model file associated with the Model. |
get_cache_infos |
Returns informations of KV-cache of the model. |
get_latency_consumed |
|
get_latency_finished |
|
dump_cache_memory |
Dumps the KV cache memory into buffers. |
load_cache_memory |
Loads the KV cache memory from buffers. |
dump_cache_memory_to |
Dumps KV cache memory to files in the specified directory. |
load_cache_memory_from |
Loads the KV cache memory from files in the specified directory. |
filter_cache_tail |
Filter the tail of the KV cache memory |
move_cache_tail |
Moves the tail of the KV cache memory to the end of the head. |
__init__¶
Creates a Model object from the specified MXQ model file and configuration.
Parses the MXQ file and constructs a Model object using the provided configuration, initializing the model with the given settings.
Note
The created Model object must be launched before performing inference. See Model.launch for more details.
Parameters
path[in] — The path to the MXQ model file.model_config[in] — The configuration settings to initialize the Model.
launch¶
Launches the model on the specified Accelerator, which represents the actual NPU.
Parameters
acc[in] — The accelerator on which to launch the model.
dispose¶
Disposes of the model loaded onto the NPU.
Releases any resources associated with the model on the NPU.
is_target¶
Checks if the NPU core specified by CoreId is the target of the model. In other words, whether the model is configured to use the given NPU core.
Parameters
core_id[in] — The CoreId to check.
Returns
True if the model is configured to use the specified CoreId, false otherwise.
get_core_mode¶
Retrieves the core mode of the model.
Returns
The CoreMode of the model.
get_target_cores¶
Returns the NPU cores the model is configured to use.
Returns
A list of CoreIds representing the target NPU cores.
infer¶
def infer(
self,
inputs: Union[np.ndarray, List[np.ndarray]],
outputs: Optional[List[np.ndarray]] = None,
cache_size: int = 0,
) -> Optional[List[np.ndarray]]
Performs inference.
Fowllowing types of inference supported.
- infer(in:List[numpy]) -> List[numpy] (float / int)
- infer(in:numpy) -> List[numpy] (float / int)
- infer(in:List[numpy], out:List[numpy]) (float / int)
- infer(in:List[numpy], out:List[]) (float / int)
- infer(in:numpy, out:List[numpy]) (float / int)
- infer(in:numpy, out:List[]) (float / int)
Parameters
inputs[in] — Input data as a single numpy.ndarray or a list of numpy.ndarray's.outputs[out] — Optional pre-allocated list of numpy.ndarray's to store inference results.
Returns
Inference results as a list of numpy.ndarray.
infer_hwc¶
def infer_hwc(
self,
inputs: Union[np.ndarray, List[np.ndarray]],
outputs: Optional[List[np.ndarray]] = None,
cache_size: int = 0,
) -> Optional[List[np.ndarray]]
infer_chw¶
def infer_chw(
self,
inputs: Union[np.ndarray, List[np.ndarray]],
outputs: Optional[List[np.ndarray]] = None,
cache_size: int = 0,
) -> Optional[List[np.ndarray]]
infer_to_float¶
def infer_to_float(
self,
inputs: Union[
np.ndarray,
List[np.ndarray],
],
cache_size: int = 0,
) -> List[np.ndarray]
int8_t-to-float inference Performs inference with input and output elements of type int8_t
Using these inference APIs requires manual scaling (quantization) of float values to int8_t for input.
Note
These APIs are intended for advanced use rather than typical usage.
infer_hwc_to_float¶
def infer_hwc_to_float(
self,
inputs: Union[
np.ndarray,
List[np.ndarray],
],
cache_size: int = 0,
) -> List[np.ndarray]
infer_chw_to_float¶
def infer_chw_to_float(
self,
inputs: Union[
np.ndarray,
List[np.ndarray],
],
cache_size: int = 0,
) -> List[np.ndarray]
infer_buffer¶
def infer_buffer(
self,
inputs: List[Buffer],
outputs: List[Buffer],
shape: List[List[int]] = [],
cache_size: int = 0,
) -> None
Buffer-to-Buffer inference
Performs inference using input and output elements in the NPU’s internal data type. The inference operates on buffers allocated via the following APIs:
Model.acquire_input_buffer()Model.acquire_output_buffer()ModelVariantHandle.acquire_input_buffer()ModelVariantHandle.acquire_output_buffer()
Additionally, Model.reposition_inputs(), Model.reposition_outputs(), ModelVariantHandle.reposition_inputs(), ModelVariantHandle.reposition_outputs() must be used properly.
Note
These APIs are intended for advanced use rather than typical usage.
infer_speedrun¶
Development-only API for measuring pure NPU inference speed.
Runs NPU inference without uploading inputs and without retrieving outputs.
infer_async¶
Asynchronous Inference
Performs inference asynchronously.
To use asynchronous inference, the model must be created using a ModelConfig object with the async pipeline configured to be enabled. This is done by calling ModelConfig.set_async_pipeline_enabled(True) before passing the configuration to Model().
Example:
import qbruntime
mc = qbruntime.ModelConfig()
mc.set_async_pipeline_enabled(True)
model = qbruntime.Model(MXQ_PATH, mc)
acc = qbruntime.Accelerator()
model.launch(acc)
future = model.infer_async(inputs)
ret = future.get()
Note
Currently, only CNN-based models are supported, as asynchronous execution is particularly effective for this type of workload.
Note
Limitations:
- RNN/LSTM and LLM models are not supported yet.
- Models requiring CPU offloading are not supported yet.
- Currently, only single-batch inference is supported (i.e., N = 1).
- Currently, Buffer inference is not supported. The following types are supported in the synchronous API for advanced use cases, but are not yet available for asynchronous inference:
- Buffer to Buffer
- Buffer to float
infer_async_to_float¶
This method supports int8_t-to-float asynchronous inference.
Parameters
inputs[in] — Input data as a single numpy.ndarray or a list of numpy.ndarray's.
Returns
A future that can be used to retrieve the inference result.
reposition_inputs¶
def reposition_inputs(
self,
inputs: List[np.ndarray],
input_bufs: List[Buffer],
seqlens: List[List[int]] = [],
) -> None
Reposition input
reposition_outputs¶
def reposition_outputs(
self,
output_bufs: List[Buffer],
outputs: List[np.ndarray],
seqlens: List[List[int]] = [],
) -> None
Reposition output
get_num_model_variants¶
Returns the total number of model variants available in this model.
The variant_idx parameter passed to Model.get_model_variant_handle() must be in the range [0, return value of this function).
Returns
The total number of model variants.
get_model_variant_handle¶
Retrieves a handle to the specified model variant.
Use the returned ModelVariantHandle to query details such as input and output shapes for the selected variant.
Parameters
variant_idx[in] — Index of the model variant to retrieve. Must be in the range [0, getNumModelVariants()).
Returns
A ModelVariantHandle object if successful; otherwise, raise qbruntime.QbRuntimeError "Model_InvalidVariantIdx".
get_model_input_shape¶
Returns the input shape of the model.
Returns
A list of input shape of the model.
get_model_output_shape¶
Returns the output shape of the model.
Returns
A list of output shape of the model.
get_input_scale¶
Returns the input quantization scale(s) of the model.
Returns
A list of input scales.
get_output_scale¶
Returns the output quantization scale(s) of the model.
Returns
A list of output scales.
get_input_buffer_info¶
Returns the input buffer information for the model.
Returns
A list of input buffer information.
get_output_buffer_info¶
Returns the output buffer information of the model.
Returns
A list of output buffer information.
get_model_input_data_type¶
Returns a data type for model inputs.
Returns
An input data type.
get_model_output_data_type¶
Returns a data type for model outputs.
Returns
An output data type.
acquire_input_buffer¶
Buffer Management API
Acquires list of Buffer for input. These API is required when calling Model.infer_buffer().
Note
These APIs are intended for advanced use rather than typical usage.
acquire_output_buffer¶
Buffer Management API
Acquires list of Buffer for output. These API is required when calling Model.infer_buffer().
Note
These APIs are intended for advanced use rather than typical usage.
release_buffer¶
Buffer Management API
Deallocate acquired Input/Output buffer
Note
These APIs are intended for advanced use rather than typical usage.
get_identifier¶
Returns the model's unique identifier.
This identifier distinguishes multiple models within a single user program. It is assigned incrementally, starting from 0 (e.g., 0, 1, 2, 3, ...).
Returns
The model identifier.
get_model_path¶
Returns the path to the MXQ model file associated with the Model.
Returns
The MXQ file path.
get_cache_infos¶
Returns informations of KV-cache of the model.
Returns
A list of CacheInfo objects.
get_latency_consumed¶
Deprecated
get_latency_finished¶
Deprecated
dump_cache_memory¶
Dumps the KV cache memory into buffers.
Writes the current KV cache data into provided buffers.
Returns
A list of bytes containing the KV cache data.
load_cache_memory¶
Loads the KV cache memory from buffers.
Restores the KV cache from the provided buffers.
Parameters
bufs[in] — A list of bytes containing the KV cache
dump_cache_memory_to¶
Dumps KV cache memory to files in the specified directory.
Writes the KV cache data to binary files within the given directory. Each file is named using the format: cache_<layer_hash>.bin.
Parameters
cache_dir[in] — Path to the directory where KV cache files will be saved.
load_cache_memory_from¶
Loads the KV cache memory from files in the specified directory.
Reads KV cache data from files within the given directory and restores them. Each file is named using the format: cache_<layer_hash>.bin.
Parameters
cache_dir[in] — Path to the directory where KV cache files are saved.
filter_cache_tail¶
Filter the tail of the KV cache memory
Retains the desired caches in the tail of the KV cache memory, excludes the others, and shifts the remaining caches forward.
Parameters
cache_size[in] — The number of tokens accumulated in the KV cache so far.tail_size[in] — The tail size of the KV cache to filter (<=32).mask[in] — A mask indicating tokens to retain or exclude at the tail of the KV cache.
Returns
New cache size after tail filtering.
move_cache_tail¶
Moves the tail of the KV cache memory to the end of the head.
Slice the tail of the KV cache memory up to the specified size and moves it to the designated cache position.
Parameters
num_head[in] — The size of the KV cache head where the tail is appended.num_tail[in] — The size of the KV cache tail to be moved.cache_size[in] — The total number of tokens accumulated in the KV cache so far.
Returns
The updated cache size after moving the tail.