Skip to content

ModelConfig

#include <qbruntime/type.h>

class mobilint::ModelConfig

Configures a core mode and core allocation of a model for NPU inference.

The ModelConfig class provides methods for setting a core mode and allocating cores for NPU inference. Supported core modes are single-core, multi-core, global4-core, and global8-core. Users can also specify which cores to allocate for the model. Additionally, the configuration offers an option to enforce the use of a specific NPU bundle.

Note

Deprecated functions are included for backward compatibility, but it is recommended to use the newer core mode configuration methods.

Attributes

Member Description
early_latencies
finish_latencies

early_latencies

std::vector<uint64_t> early_latencies

Deprecated

This setting has no effect.

finish_latencies

std::vector<uint64_t> finish_latencies

Deprecated

This setting has no effect.

Member functions

Member Description
ModelConfig ×2 Default constructor. This default-constructed object is initially set to auto-core mode.
setAutoCoreMode Sets the model to detect CoreMode automatically.
setSingleCoreMode ×2 Sets the model to use single-core mode for inference with a specified number of local cores.
setMultiCoreMode Sets the model to use multi-core mode for batch inference.
setGlobal4CoreMode Sets the model to use global4-core mode for inference with a specified set of NPU clusters.
setGlobal8CoreMode Sets the model to use global8-core mode for inference.
getCoreMode Gets the core mode to be applied to the model.
getCoreAllocationPolicy Gets the core allocation policy to be applied to the model.
getNumCores Gets the number of cores to be allocated for the model.
forceSingleNPUBundle Forces the use of a specific NPU bundle.
getForcedNPUBundleIndex Retrieves the index of the forced NPU bundle.
getCoreIds Returns the list of NPU CoreIds to be used for model inference.
getClusters
setAsyncPipelineEnabled Enables or disables the asynchronous pipeline required for asynchronous inference.
getAsyncPipelineEnabled Returns whether the asynchronous pipeline is enabled in this configuration.
setActivationSlots Sets activation buffer slots for multi-activation supported model.
getActivationSlots Returns activation buffer slot count.
setGlobalCoreMode

ModelConfig

ModelConfig()

Default constructor. This default-constructed object is initially set to auto-core mode.

ModelConfig(int num_cores)

deprecated

setAutoCoreMode

bool setAutoCoreMode()

Sets the model to detect CoreMode automatically.

In auto-core mode, the model automatically detects a supported CoreMode while using all available NPU cores.

Note

If the model has more than one CoreMode, CoreMode::Auto is not supported.

Note

activation buffer slots will be reset after setAutoCoreMode is called.

Returns

true if the mode was successfully set, false otherwise.

setSingleCoreMode

bool setSingleCoreMode(int num_cores)

Sets the model to use single-core mode for inference with a specified number of local cores.

In single-core mode, each local core executes model inference independently. The number of cores used is specified by the num_cores parameter, and the core allocation policy is set to CoreAllocationPolicy::Auto, meaning the model will be automatically allocated to available local cores when the model is launched to the NPU, specifically when the Model::launch function is called.

Parameters

  • num_cores [in] — The number of local cores to use for inference.

Returns

true if the mode was successfully set, false otherwise.

bool setSingleCoreMode(std::vector<CoreId> core_ids)

Sets the model to use single-core mode for inference with a specific set of NPU local cores.

In single-core mode, each local core executes model inference independently. The user can specify a vector of CoreIds to determine which cores to use for inference.

Parameters

  • core_ids [in] — A vector of CoreIds to be used for model inference.

Returns

true if the mode was successfully set, false otherwise.

setMultiCoreMode

bool setMultiCoreMode(std::vector<Cluster> clusters={Cluster::Cluster0, Cluster::Cluster1})

Sets the model to use multi-core mode for batch inference.

In multi-core mode, on Aries NPU, the four local cores within a cluster work together to process batch inference tasks efficiently. This mode is optimized for batch processing.

Note

By default, the configuration is set to use all clusters.

Parameters

  • clusters [in] — A vector of clusters to be used for multi-core batch inference.

Returns

true if the mode was successfully set, false otherwise.

setGlobal4CoreMode

bool setGlobal4CoreMode(std::vector<Cluster> clusters={Cluster::Cluster0, Cluster::Cluster1})

Sets the model to use global4-core mode for inference with a specified set of NPU clusters.

For Aries NPU, there are two clusters, each consisting of four local cores. In global4-core mode, four local cores within the same cluster work together to execute the model inference.

Note

By default, the configuration is set to use all clusters.

Parameters

  • clusters [in] — A vector of clusters to be used for model inference.

Returns

true if the mode was successfully set, false otherwise.

setGlobal8CoreMode

bool setGlobal8CoreMode()

Sets the model to use global8-core mode for inference.

For Aries NPU, there are two clusters, each consisting of four local cores. In global8-core mode, all eight local cores across the two clusters work together to execute the model inference.

Returns

true if the mode was successfully set, false otherwise.

getCoreMode

CoreMode getCoreMode() const

Gets the core mode to be applied to the model.

This reflects the core mode that will be used when the model is created.

Returns

The CoreMode to be applied to the model.

getCoreAllocationPolicy

CoreAllocationPolicy getCoreAllocationPolicy() const

Gets the core allocation policy to be applied to the model.

This reflects the core allocation policy that will be used when the model is created.

Returns

The CoreAllocationPolicy to be applied to the model.

getNumCores

int getNumCores() const

Gets the number of cores to be allocated for the model.

This represents the number of cores that will be allocated for inference when the model is launched to the NPU.

Returns

The number of cores to be allocated for the model.

forceSingleNPUBundle

bool forceSingleNPUBundle(int npu_bundle_index)

Forces the use of a specific NPU bundle.

This function forces the selection of a specific NPU bundle. If a non-negative index is provided, the corresponding NPU bundle is selected and runs without CPU offloading. If -1 is provided, all NPU bundles are used with CPU offloading enabled.

Parameters

  • npu_bundle_index [in] — The index of the NPU bundle to force. A non-negative integer selects a specific NPU bundle (runs without CPU offloading), or -1 to enable all NPU bundles with CPU offloading.

Returns

true if the index is valid and the NPU bundle is successfully set, false if the index is invalid (less than -1).

getForcedNPUBundleIndex

int getForcedNPUBundleIndex() const

Retrieves the index of the forced NPU bundle.

This function returns the index of the NPU bundle that has been forced using the forceSingleNPUBundle function. If no NPU bundle is forced, the returned value will be -1.

Returns

The index of the forced NPU bundle, or -1 if no bundle is forced.

getCoreIds

const std::vector<CoreId> & getCoreIds() const

Returns the list of NPU CoreIds to be used for model inference.

This function returns a reference to the vector of NPU CoreIds that the model will use for inference. When setSingleCoreMode(int num_cores) is called and the core allocation policy is set to CoreAllocationPolicy::Auto, it will return an empty vector.

Returns

A constant reference to the vector of NPU CoreIds.

getClusters

const std::vector<Cluster> & getClusters() const

setAsyncPipelineEnabled

void setAsyncPipelineEnabled(bool enable)

Enables or disables the asynchronous pipeline required for asynchronous inference.

Call this function with enable set to true if you intend to use Model::inferAsync or Model::inferAsyncCHW, as the asynchronous pipeline is necessary for their operation.

If you are only using synchronous inference, such as Model::infer or Model::inferCHW, it is recommended to keep the asynchronous pipeline disabled to avoid unnecessary overhead.

Parameters

  • enable [in] — Set to true to enable the asynchronous pipeline; set to false to disable it.

getAsyncPipelineEnabled

bool getAsyncPipelineEnabled() const

Returns whether the asynchronous pipeline is enabled in this configuration.

Returns

true if the asynchronous pipeline is enabled; false otherwise.

setActivationSlots

void setActivationSlots(int count)

Sets activation buffer slots for multi-activation supported model.

Call this function if you want to set the number of activation buffer slots manually.

If you do not call this function, the default number of activation buffer slots is set differently depending on the CoreMode.

Note

This function has no effect on MXQ file in version earlier than MXQv7.

Note

Currently, LLM model's activation slot is fixed to 1 and ignoring count.

Parameters

  • count [in] — Multi activation counts. Must be >= 1.

getActivationSlots

int getActivationSlots() const

Returns activation buffer slot count.

Note

This function has no meaning on MXQ file in version earlier than MXQv7.

Returns

Activation buffer slot count.

setGlobalCoreMode

bool setGlobalCoreMode(std::vector<Cluster> clusters)

deprecated