Core Mode

Core Mode#

Core mode defines how the host software maps inference work onto the ARIES accelerator’s cores. A model’s core mode is fixed when it is compiled into an MXQ and is applied at runtime through ModelConfig, as shown in the examples below.

Note

To select the cores and core mode for a specific model in your inference code, see ModelConfig Configuration.

Overview#

ARIES supports four core collaboration modes (hereafter referred to as core mode). In user manual and tutorials, the Single mode - the most intuitive core mode - is primarily used to help understanding. However, depending on the deep learning task you intend to perform, a different core mode may be more suitable. Utilizing more appropriate core modes can provide significant performance improvements.

Currently, ARIES supports following core modes.

Important

Core modes are determined at the compilation stage. If you have already compiled MXQ, you cannot use core mode other than specified in your MXQ. You can figure out designated core mode in MXQ using mobilint-cli mxqtool.

To use your desired core mode, you must either

  • Compile the model directly with the desired mode, or

  • use an MXQ file that was compiled with desired mode.

Before getting started, it’s important to understand the overall internal architecture of ARIES. Fundamentally, each core in the ARIES (referred to as a Local Core) can be regarded as the smallest unit capable of independently executing a single model. These Local Cores are combined to form a Cluster. Within each Cluster, there is an additional Global Core, which is responsible for coordinating tasks under the Multi/Global4,8 Modes. The Global Core manages the additional operations and communications required by these modes. The structure can be illustrated as follows:

NPU_Internal

On ARIES, the eight Local Cores are organized into two Clusters (Cluster0 and Cluster1), each containing four Local Cores (Core0–Core3). The runtime identifies a specific core by its CoreId — the combination of its cluster and core index, for example CoreId(Cluster.Cluster0, Core.Core0). This is the numbering that core mode configuration and the runtime API use to place work on the device. Because each Local Core can execute a model independently, assigning different cores to different models lets several workloads share a single ARIES device.

Single Mode#

Single Mode is the most intuitive mode, where each Local Core operates independently. In this mode, the user sends inference directly to individual cores and collects the results one by one to complete the desired task. Since independent tasks can be executed across 8 cores, assigning work to all 8 cores continuously enables maximum performance (throughput).

NPU_Single

  • Note: While this mode is the most simple and intuitive for functional verification, it requires a mechanism that can send requests in non-blocking manner. Without a proper design - such as using multithreading or async I/O to send inference requests to multiple cores simultaneously - the system will fall back to fully sequential processing, resulting in wasting core utilization (utilizing only one core). Since multithreading and async I/O are common design patterns in software engineering, please refer to related resources when designing them.

## Compilation stage
# Generate MXQ Model with Single-core Inference Scheme
from qbcompiler import mxq_compile

mxq_compile(
        ...
        inference_scheme="single" # Technically, this line can be omitted.
        ...
    )

Multi Mode#

This mode actually behaves as a batch processing mode, where multiple inputs are fed into the NPU at once, and multiple results are retrived from NPU together. It operates at Cluster level, with each cluster consisting of 4 Local Cores. Through coordination among the 4 Local Cores within a Cluster-managed by Global Core-the system performs operations optimized for 4-batch processing.

NPU_Multi

  • Use Cases

    • To struct the NPU inference logic in the same way as conventional batch processing

    • To achieve high performance(throughput) in 4-batch units without implementing multithreading or async I/O

## Compilation stage
# Generate MXQ Model with Multi-core Inference Scheme
from qbcompiler import mxq_compile

mxq_compile(
        ...
        inference_scheme="multi"
        ...
    )

Global Modes#

This is an inference method in which multiple cores work collaboratively to process a single piece of data. For example, in the case of large and computationally heavy model, splitting the single input data across cores can potentially provide performance benefit in terms of latency. Global Mode operates by utilizing Local Cores within a cluster, and is offered under the names Global4 Mode and Global8 Mode.

NPU_Global

  • Use Cases

    • To reduce latency in large model.

Global4 Mode#

4 Local Cores (one Cluster) work together to process a single piece of data. You can freely select which Cluster to execute in qbruntime. Compilation and execution can be performed as shown in examples below:

## Compilation stage
# Generate MXQ Model with Global4 Inference Scheme
from qbcompiler import mxq_compile

mxq_compile(
        ...
        inference_scheme="global4"
        ...
    )

Global8 Mode#

8 Local Cores (2 Clusters) work together to process a single piece of data. Compilation and execution can be performed as shown in the examples below:

## Compilation stage
# Generate MXQ Model with Global8 Inference Scheme
from qbcompiler import mxq_compile

mxq_compile(
        ...
        inference_scheme="global8"
        ...
    )