Skip to content

Model

class Model

Represents an AI model loaded from an MXQ file.

This class loads an AI model from an MXQ file and provides functions to launch it on the NPU and perform inference.

Warning

The MXQ file, or the MXQ package file, is read again whenever the model is launched. Moving or deleting the file before the launch makes the launch fail, and modifying it may make the model produce wrong results.

Attributes

Name Description
target_cores

target_cores

property target_cores: List[CoreId]

Deprecated

Methods

Name Description
__init__ Creates a Model object from an MXQ model file, or from an MXQ entry in an MXQ package (.mxqpkg).
from_cpp
launch Launches the model on the specified Accelerator, which represents the actual NPU.
dispose Disposes of the model loaded onto the NPU.
is_target Checks if the NPU core specified by CoreId is the target of the model. In other words, whether the model is configured to use the given NPU core.
get_core_mode Retrieves the core mode of the model.
get_device_names Returns the supported target device name(s) this model can run on.
get_target_cores Returns the NPU cores the model is configured to use.
infer Performs inference.
infer_hwc
infer_chw
infer_to_float int8_t-to-float inference Performs inference with input and output elements of type int8_t
infer_hwc_to_float
infer_chw_to_float
infer_buffer Buffer-to-Buffer inference
infer_speedrun Development-only API for measuring pure NPU inference speed.
infer_pinned_memory Performs inference directly on pinned memory buffers (zero-copy).
acquire_input_npu_data Acquires an NPUData for the model input at the given index.
acquire_output_npu_data Acquires an NPUData for the model output at the given index.
infer_npu_data Performs inference using NPUData for both inputs and outputs.
infer_npu_data_chw Performs inference using NPUData in NCHW/CHW layout.
infer_async Asynchronous Inference
infer_async_to_float This method supports int8_t-to-float asynchronous inference.
reposition_inputs Reposition input
reposition_outputs Reposition output
get_num_model_variants Returns the total number of model variants available in this model.
get_model_variant_handle Retrieves a handle to the specified model variant.
get_model_input_shape Returns the input shape of the model.
get_model_output_shape Returns the output shape of the model.
get_input_scale Returns the input quantization scale(s) of the model.
get_output_scale Returns the output quantization scale(s) of the model.
get_input_buffer_info Returns the input buffer information for the model.
get_output_buffer_info Returns the output buffer information of the model.
get_model_input_data_type Returns a data type for model inputs.
get_model_output_data_type Returns a data type for model outputs.
acquire_input_buffer Buffer Management API
acquire_output_buffer Buffer Management API
release_buffer Buffer Management API
get_identifier Returns the model's unique identifier.
get_model_path Returns the path to the MXQ model file associated with the Model.
get_package_entry_name Returns the name of the MXQ entry if the Model was created from an MXQ package, or an empty string otherwise.
get_cache_infos Returns informations of KV-cache of the model.
dump_cache_memory Dumps the KV cache memory into buffers.
load_cache_memory Loads the KV cache memory from buffers.
dump_cache_memory_to Dumps KV cache memory to files in the specified directory.
load_cache_memory_from Loads the KV cache memory from files in the specified directory.
dump_cache_memory_by_name Dumps a single KV cache memory into a buffer.
dump_cache_memory_by_layer_hash Dumps a single KV cache memory into a buffer.
load_cache_memory_by_name Loads a single KV cache memory from a buffer.
load_cache_memory_by_layer_hash Loads a single KV cache memory from a buffer.
reset_fixed_tail_cache_memory Reset the fixed tail KV cache memory.
filter_cache_tail Filter the tail of the KV cache memory
move_cache_tail Moves the tail of the KV cache memory to the end of the head.

__init__

def __init__(self, path: str, model_config: Optional[ModelConfig] = None)
def __init__(
    self, path: str, entry_name: str, model_config: Optional[ModelConfig] = None
)

Creates a Model object from an MXQ model file, or from an MXQ entry in an MXQ package (.mxqpkg).

Model(path[, model_config]) loads a plain MXQ file. Model(path, entry_name[, model_config]) loads the MXQ entry named entry_name directly from the MXQ package at path without unpacking it.

Note

The created Model object must be launched before performing inference. See Model.launch for more details.

Parameters

  • path [in] — The path to the MXQ model file, or to the MXQ package file.
  • entry_name [in] — The name of the MXQ entry in the package (e.g. "block_0_expert_0.mxq"). Omit it to load a plain MXQ file.
  • model_config [in] — The configuration settings to initialize the Model.

from_cpp

def from_cpp(cls, _model: _cQbRuntime.Model):
    # ModelPackage.create_model 처럼 C++ 에서 이미 만든 Model 을 감쌀 때 쓴다.
    # __init__ 은 C++ Model 을 새로 만들므로 거치지 않는다.

launch

def launch(self, acc: Accelerator) -> None

Launches the model on the specified Accelerator, which represents the actual NPU.

Parameters

  • acc [in] — The accelerator on which to launch the model.

dispose

def dispose(self) -> None

Disposes of the model loaded onto the NPU.

Releases any resources associated with the model on the NPU.

is_target

def is_target(self, core_id: CoreId) -> bool

Checks if the NPU core specified by CoreId is the target of the model. In other words, whether the model is configured to use the given NPU core.

Parameters

  • core_id [in] — The CoreId to check.

Returns

True if the model is configured to use the specified CoreId, false otherwise.

get_core_mode

def get_core_mode(self) -> CoreMode

Retrieves the core mode of the model.

Returns

The CoreMode of the model.

get_device_names

def get_device_names(self) -> List[str]

Returns the supported target device name(s) this model can run on.

Returns device names from the model's target device in the MXQ file. A single model may map to multiple names (e.g. a REGULUS model returns both SoC and USB devices). Any of the returned names can be passed to Accelerator(device_name=).

Returns

A list of supported target device names.

get_target_cores

def get_target_cores(self) -> List[CoreId]

Returns the NPU cores the model is configured to use.

Returns

A list of CoreIds representing the target NPU cores.

infer

def infer(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
    outputs: Optional[List[np.ndarray]] = None,
    cache_size: int = 0,
    params: Optional[List[BatchParam]] = None,
) -> Optional[List[np.ndarray]]

Performs inference.

Fowllowing types of inference supported.

  1. infer(in:List[numpy]) -> List[numpy] (float / int)
  2. infer(in:numpy) -> List[numpy] (float / int)
  3. infer(in:List[numpy], out:List[numpy]) (float / int)
  4. infer(in:List[numpy], out:List[]) (float / int)
  5. infer(in:numpy, out:List[numpy]) (float / int)
  6. infer(in:numpy, out:List[]) (float / int)

Parameters

  • inputs [in] — Input data as a single numpy.ndarray or a list of numpy.ndarray's.
  • outputs [out] — Optional pre-allocated list of numpy.ndarray's to store inference results.
  • cache_size [in] — The number of tokens accumulated in the KV cache so far.
  • params [in] — A List of BatchParam, specifying each batch's information for BatchLLM inference. If params is specified, cache_size is ignored.

Returns

Inference results as a list of numpy.ndarray.

infer_hwc

def infer_hwc(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
    outputs: Optional[List[np.ndarray]] = None,
    cache_size: int = 0,
    params: Optional[List[BatchParam]] = None,
) -> Optional[List[np.ndarray]]

infer_chw

def infer_chw(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
    outputs: Optional[List[np.ndarray]] = None,
    cache_size: int = 0,
    params: Optional[List[BatchParam]] = None,
) -> Optional[List[np.ndarray]]

infer_to_float

def infer_to_float(
    self,
    inputs: Union[
        np.ndarray,
        List[np.ndarray],
    ],
    cache_size: int = 0,
) -> List[np.ndarray]

int8_t-to-float inference Performs inference with input and output elements of type int8_t

Using these inference APIs requires manual scaling (quantization) of float values to int8_t for input.

Note

These APIs are intended for advanced use rather than typical usage.

infer_hwc_to_float

def infer_hwc_to_float(
    self,
    inputs: Union[
        np.ndarray,
        List[np.ndarray],
    ],
    cache_size: int = 0,
) -> List[np.ndarray]

infer_chw_to_float

def infer_chw_to_float(
    self,
    inputs: Union[
        np.ndarray,
        List[np.ndarray],
    ],
    cache_size: int = 0,
) -> List[np.ndarray]

infer_buffer

def infer_buffer(
    self,
    inputs: List[Buffer],
    outputs: List[Buffer],
    shape: List[List[int]] = [],
    cache_size: int = 0,
) -> None

Buffer-to-Buffer inference

Performs inference using input and output elements in the NPU’s internal data type. The inference operates on buffers allocated via the following APIs:

Additionally, Model.reposition_inputs(), Model.reposition_outputs(), ModelVariantHandle.reposition_inputs(), ModelVariantHandle.reposition_outputs() must be used properly.

Note

These APIs are intended for advanced use rather than typical usage.

infer_speedrun

def infer_speedrun(self) -> None

Development-only API for measuring pure NPU inference speed.

Runs NPU inference without uploading inputs and without retrieving outputs.

infer_pinned_memory

def infer_pinned_memory(
    self,
    inputs: List[PinnedMemory],
    outputs: Optional[List[PinnedMemory]] = None,
    cache_size: int = 0,
) -> Optional[List[PinnedMemory]]

Performs inference directly on pinned memory buffers (zero-copy).

The NPU reads the inputs from and writes the outputs into the pinned buffers in place, avoiding host-to-device copies of the I/O tensors. Write the input data into each input buffer via indexing (e.g. pm[...] = data) before calling, and read the results from the output buffers afterwards.

Note

This is an experimental API and is only supported on REGULUS device. It is not supported for models that use CPU offload.

Parameters

  • inputs — A list of PinnedMemory buffers holding the input data. All inputs must share the same dtype (numpy.float32 or numpy.uint8).
  • outputs — A list of pre-allocated PinnedMemory buffers (dtype numpy.float32) that will receive the output data. If None (the default), the output buffers are allocated internally and returned by this method.
  • cache_size — The size of the cache to use for inference.

Returns

If outputs is None, a list of newly allocated PinnedMemory buffers (dtype numpy.float32) holding the output data; otherwise None (the given outputs are filled in place).

acquire_input_npu_data

def acquire_input_npu_data(
    self,
    shape: List[int],
    idx: int = 0,
    upload: bool = False,
    dtype: Optional[DataType] = None,
) -> NPUData

Acquires an NPUData for the model input at the given index.

Note

This is an advanced API rather than a typical usage. (Running inference on the acquired NPUData via infer_npu_data additionally requires a single-NPU-op, non-CPU-offload, relocatable (MXQv7+) model.)

Warning

This API is in beta: it may still contain bugs, and its behavior may change in a future release.

Parameters

  • shape — The shape of the input tensor, in NHWC/HWC or NCHW/CHW layout.
  • idx — The index of the model input.
  • upload — If True, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).
  • dtype — The element type of the data. None (the default) uses the model's user-facing input type. Pass DataType.Int8 to hand the model data that is already quantized to the NPU-native type, which skips the host-side quantization.

Returns

The acquired NPUData.

acquire_output_npu_data

def acquire_output_npu_data(
    self,
    shape: List[int],
    idx: int = 0,
    upload: bool = False,
    dtype: Optional[DataType] = None,
) -> NPUData

Acquires an NPUData for the model output at the given index.

Note

This is an advanced API rather than a typical usage. (Running inference on the acquired NPUData via infer_npu_data additionally requires a single-NPU-op, non-CPU-offload, relocatable (MXQv7+) model.)

Warning

This API is in beta: it may still contain bugs, and its behavior may change in a future release.

Parameters

  • shape — The shape of the output tensor, in NHWC/HWC or NCHW/CHW layout.
  • idx — The index of the model output.
  • upload — If True, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).
  • dtype — The element type of the data. None (the default) uses the model's user-facing output type. Pass DataType.Int8 to receive the raw NPU-native output, which skips the host-side dequantization.

Returns

The acquired NPUData.

infer_npu_data

def infer_npu_data(
    self,
    inputs: List[NPUData],
    outputs: Optional[List[NPUData]] = None,
    cache_size: int = 0,
) -> Optional[List[NPUData]]

Performs inference using NPUData for both inputs and outputs.

Every input and output NPUData must share the same residency: either all of them are on the host (acquired with upload=False) or all of them are already on the NPU (acquired with upload=True). A mixed set results in an error.

Note

This is an advanced API rather than a typical usage. Only single-NPU-op (non-CPU-offload), relocatable (MXQv7+) models are supported.

Note

Every NPUData must have been acquired with an NHWC/HWC shape; use infer_npu_data_chw for NCHW/CHW-shaped NPUData. A mismatch raises Model_ShapeMismatched.

Warning

This API is in beta: it may still contain bugs, and its behavior may change in a future release.

Parameters

  • inputs — A list of input NPUData.
  • outputs — An optional list of pre-allocated output NPUData. If None (the default), the output NPUData are acquired internally and returned.
  • cache_size — The number of tokens accumulated in the KV cache so far.

Returns

If outputs is None, a list of output NPUData holding the inference results; otherwise None (the given outputs are filled in place).

infer_npu_data_chw

def infer_npu_data_chw(
    self,
    inputs: List[NPUData],
    outputs: Optional[List[NPUData]] = None,
    cache_size: int = 0,
) -> Optional[List[NPUData]]

Performs inference using NPUData in NCHW/CHW layout.

Same as infer_npu_data except that every input and output NPUData must have been acquired with a NCHW/CHW shape; a mismatch raises Model_ShapeMismatched.

Note

This is an advanced API rather than a typical usage. Only single-NPU-op (non-CPU-offload), relocatable (MXQv7+) models are supported.

Warning

This API is in beta: it may still contain bugs, and its behavior may change in a future release.

Parameters

  • inputs — A list of input NPUData.
  • outputs — An optional list of pre-allocated output NPUData. If None (the default), the output NPUData are acquired internally and returned.
  • cache_size — The number of tokens accumulated in the KV cache so far.

Returns

If outputs is None, a list of output NPUData holding the inference results; otherwise None (the given outputs are filled in place).

infer_async

def infer_async(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
) -> Future

Asynchronous Inference

Performs inference asynchronously.

To use asynchronous inference, the model must be created using a ModelConfig object with the async pipeline configured to be enabled. This is done by calling ModelConfig.set_async_pipeline_enabled(True) before passing the configuration to Model().

Example:

import qbruntime

mc = qbruntime.ModelConfig()
mc.set_async_pipeline_enabled(True)

model = qbruntime.Model(MXQ_PATH, mc)
acc = qbruntime.Accelerator()

model.launch(acc)

future = model.infer_async(inputs)

ret = future.get()

Note

Currently, only CNN-based models are supported, as asynchronous execution is particularly effective for this type of workload.

Note

Limitations:

  • RNN/LSTM and LLM models are not supported yet.
  • Models requiring CPU offloading are not supported yet.
  • Currently, only single-batch inference is supported (i.e., N = 1).
  • Currently, Buffer inference is not supported. The following types are supported in the synchronous API for advanced use cases, but are not yet available for asynchronous inference:
  • Buffer to Buffer
  • Buffer to float

infer_async_to_float

def infer_async_to_float(
    self,
    inputs: Union[np.ndarray, List[np.ndarray]],
) -> Future

This method supports int8_t-to-float asynchronous inference.

Parameters

  • inputs [in] — Input data as a single numpy.ndarray or a list of numpy.ndarray's.

Returns

A future that can be used to retrieve the inference result.

reposition_inputs

def reposition_inputs(
    self,
    inputs: List[np.ndarray],
    input_bufs: List[Buffer],
    seqlens: List[List[int]] = [],
) -> None

Reposition input

reposition_outputs

def reposition_outputs(
    self,
    output_bufs: List[Buffer],
    outputs: List[np.ndarray],
    seqlens: List[List[int]] = [],
) -> None

Reposition output

get_num_model_variants

def get_num_model_variants(self) -> int

Returns the total number of model variants available in this model.

The variant_idx parameter passed to Model.get_model_variant_handle() must be in the range [0, return value of this function).

Returns

The total number of model variants.

get_model_variant_handle

def get_model_variant_handle(self, variant_idx) -> ModelVariantHandle

Retrieves a handle to the specified model variant.

Use the returned ModelVariantHandle to query details such as input and output shapes for the selected variant.

Parameters

  • variant_idx [in] — Index of the model variant to retrieve. Must be in the range [0, getNumModelVariants()).

Returns

A ModelVariantHandle object if successful; otherwise, raise qbruntime.QbRuntimeError "Model_InvalidVariantIdx".

get_model_input_shape

def get_model_input_shape(self) -> List[_Shape]

Returns the input shape of the model.

Returns

A list of input shape of the model.

get_model_output_shape

def get_model_output_shape(self) -> List[_Shape]

Returns the output shape of the model.

Returns

A list of output shape of the model.

get_input_scale

def get_input_scale(self) -> List[Scale]

Returns the input quantization scale(s) of the model.

Returns

A list of input scales.

get_output_scale

def get_output_scale(self) -> List[Scale]

Returns the output quantization scale(s) of the model.

Returns

A list of output scales.

get_input_buffer_info

def get_input_buffer_info(self) -> List[BufferInfo]

Returns the input buffer information for the model.

Returns

A list of input buffer information.

get_output_buffer_info

def get_output_buffer_info(self) -> List[BufferInfo]

Returns the output buffer information of the model.

Returns

A list of output buffer information.

get_model_input_data_type

def get_model_input_data_type(self) -> DataType

Returns a data type for model inputs.

Returns

An input data type.

get_model_output_data_type

def get_model_output_data_type(self) -> DataType

Returns a data type for model outputs.

Returns

An output data type.

acquire_input_buffer

def acquire_input_buffer(self, seqlens: List[List[int]] = []) -> List[Buffer]

Buffer Management API

Acquires list of Buffer for input. These API is required when calling Model.infer_buffer().

Note

These APIs are intended for advanced use rather than typical usage.

acquire_output_buffer

def acquire_output_buffer(self, seqlens: List[List[int]] = []) -> List[Buffer]

Buffer Management API

Acquires list of Buffer for output. These API is required when calling Model.infer_buffer().

Note

These APIs are intended for advanced use rather than typical usage.

release_buffer

def release_buffer(self, buffer: List[Buffer]) -> None

Buffer Management API

Deallocate acquired Input/Output buffer

Note

These APIs are intended for advanced use rather than typical usage.

get_identifier

def get_identifier(self) -> int

Returns the model's unique identifier.

This identifier distinguishes multiple models within a single user program. It is assigned incrementally, starting from 0 (e.g., 0, 1, 2, 3, ...).

Returns

The model identifier.

get_model_path

def get_model_path(self) -> str

Returns the path to the MXQ model file associated with the Model.

If the Model was created from an MXQ package, this is the path to the package file. Use get_package_entry_name() to get the name of the MXQ entry in it.

Returns

The MXQ file path, or the MXQ package file path.

get_package_entry_name

def get_package_entry_name(self) -> str

Returns the name of the MXQ entry if the Model was created from an MXQ package, or an empty string otherwise.

Returns

The MXQ entry name, or an empty string.

get_cache_infos

def get_cache_infos(self) -> List[CacheInfo]

Returns informations of KV-cache of the model.

Returns

A list of CacheInfo objects.

dump_cache_memory

def dump_cache_memory(self, cache_id: int = 0) -> List[bytes]

Dumps the KV cache memory into buffers.

Writes the current KV cache data into provided buffers.

Parameters

  • cache_id [in] — Index of target cache.

Returns

A list of bytes containing the KV cache data.

load_cache_memory

def load_cache_memory(self, bufs: List[bytes], cache_id: int = 0) -> None

Loads the KV cache memory from buffers.

Restores the KV cache from the provided buffers.

Parameters

  • bufs [in] — A list of bytes containing the KV cache

dump_cache_memory_to

def dump_cache_memory_to(self, cache_dir: str, cache_id: int = 0) -> None

Dumps KV cache memory to files in the specified directory.

Writes the KV cache data to binary files within the given directory. Each file is named using the format: cache_<layer_hash>.bin.

Parameters

  • cache_dir [in] — Path to the directory where KV cache files will be saved.
  • cache_id [in] — Index of target cache.

load_cache_memory_from

def load_cache_memory_from(self, cache_dir: str, cache_id: int = 0) -> None

Loads the KV cache memory from files in the specified directory.

Reads KV cache data from files within the given directory and restores them. Each file is named using the format: cache_<layer_hash>.bin.

Parameters

  • cache_dir [in] — Path to the directory where KV cache files are saved.

dump_cache_memory_by_name

def dump_cache_memory_by_name(self, name: str, cache_id: int = 0) -> bytes

Dumps a single KV cache memory into a buffer.

Writes the current data of the KV cache whose CacheInfo.name matches name into a buffer and returns it.

Parameters

  • name [in] — Name of the target KV cache. See CacheInfo.name.
  • cache_id [in] — Index of target cache.

Returns

Bytes containing the KV cache data.

dump_cache_memory_by_layer_hash

def dump_cache_memory_by_layer_hash(
    self, layer_hash: str, cache_id: int = 0
) -> bytes

Dumps a single KV cache memory into a buffer.

Writes the current data of the KV cache whose CacheInfo.layer_hash matches layer_hash into a buffer and returns it.

Parameters

  • layer_hash [in] — Layer hash of the target KV cache. See CacheInfo.layer_hash.
  • cache_id [in] — Index of target cache.

Returns

Bytes containing the KV cache data.

load_cache_memory_by_name

def load_cache_memory_by_name(
    self, buf: bytes, name: str, cache_id: int = 0
) -> None

Loads a single KV cache memory from a buffer.

Restores only the KV cache whose CacheInfo.name matches name from the given buffer. The other KV caches are left untouched.

Parameters

  • buf [in] — Bytes containing the KV cache data. Its size must be equal to the size of the target KV cache.
  • name [in] — Name of the target KV cache. See CacheInfo.name.
  • cache_id [in] — Index of target cache.

load_cache_memory_by_layer_hash

def load_cache_memory_by_layer_hash(
    self, buf: bytes, layer_hash: str, cache_id: int = 0
) -> None

Loads a single KV cache memory from a buffer.

Restores only the KV cache whose CacheInfo.layer_hash matches layer_hash from the given buffer. The other KV caches are left untouched.

Parameters

  • buf [in] — Bytes containing the KV cache data. Its size must be equal to the size of the target KV cache.
  • layer_hash [in] — Layer hash of the target KV cache. See CacheInfo.layer_hash.
  • cache_id [in] — Index of target cache.

reset_fixed_tail_cache_memory

def reset_fixed_tail_cache_memory(self, cache_id: int = 0) -> None

Reset the fixed tail KV cache memory.

Fills the entire memory region of every KV cache whose CacheType is CacheType.FixedTailCache with zeros. The caches of the other types are left untouched.

Note

This does not update the cache size tracked by the caller; the caller must reset its own cache_size to 0 after this call.

Parameters

  • cache_id [in] — Index of target cache.

filter_cache_tail

def filter_cache_tail(
    self, cache_size: int, tail_size: int, mask: List[bool]
) -> int

Filter the tail of the KV cache memory

Retains the desired caches in the tail of the KV cache memory, excludes the others, and shifts the remaining caches forward.

Parameters

  • cache_size [in] — The number of tokens accumulated in the KV cache so far.
  • tail_size [in] — The tail size of the KV cache to filter (<=32).
  • mask [in] — A mask indicating tokens to retain or exclude at the tail of the KV cache.

Returns

New cache size after tail filtering.

move_cache_tail

def move_cache_tail(self, num_head: int, num_tail: int, cache_size: int) -> int

Moves the tail of the KV cache memory to the end of the head.

Slice the tail of the KV cache memory up to the specified size and moves it to the designated cache position.

Parameters

  • num_head [in] — The size of the KV cache head where the tail is appended.
  • num_tail [in] — The size of the KV cache tail to be moved.
  • cache_size [in] — The total number of tokens accumulated in the KV cache so far.

Returns

The updated cache size after moving the tail.