# Core Mode

`Core mode` defines how the host software maps inference work onto the ARIES
accelerator's cores. A model's core mode is fixed when it is compiled into an
MXQ and is applied at runtime through `ModelConfig`, as shown in the examples below.

```{note}
To select the cores and core mode for a specific model in your inference code, see [ModelConfig Configuration](modelconfig.md).
```

## Overview

ARIES supports four core collaboration modes (hereafter referred to as `core mode`).
In user manual and tutorials, the `Single` mode - the most intuitive core mode - is primarily used to help understanding.
However, depending on the deep learning task you intend to perform, a different `core mode` may be more suitable.
Utilizing more appropriate core modes can provide significant performance improvements.

Currently, ARIES supports following core modes.
- [Single](#single-mode)
- [Multi](#multi-mode)
- [Global4](#global4-mode)
- [Global8](#global8-mode)

```{important}
Core modes are determined at the compilation stage. If you have already compiled MXQ, you cannot use core mode other than specified in your MXQ. You can figure out designated core mode in MXQ using `mobilint-cli mxqtool`.

To use your desired core mode, you must either
- Compile the model directly with the desired mode, or
- use an MXQ file that was compiled with desired mode.
```

Before getting started, it's important to understand the overall internal architecture of ARIES.
Fundamentally, each core in the ARIES (referred to as a `Local Core`) can be regarded as the smallest unit capable of independently executing a single model.
These `Local Cores` are combined to form a `Cluster`.
Within each `Cluster`, there is an additional `Global Core`, which is responsible for coordinating tasks under the `Multi/Global4,8 Modes`. The `Global Core` manages the additional operations and communications required by these modes.
The structure can be illustrated as follows:

![NPU_Internal](../res/image/NPU_Internal.png "NPU Internal Architecture")

On ARIES, the eight `Local Cores` are organized into two `Clusters` (`Cluster0`
and `Cluster1`), each containing four `Local Cores` (`Core0`–`Core3`). The
runtime identifies a specific core by its `CoreId` — the combination of its
cluster and core index, for example `CoreId(Cluster.Cluster0, Core.Core0)`. This
is the numbering that `core mode` configuration and the runtime API use to place
work on the device. Because each `Local Core` can execute a model independently,
assigning different cores to different models lets several workloads share a
single ARIES device.

## Single Mode

`Single Mode` is the most intuitive mode, where each `Local Core` operates independently.
In this mode, the user sends inference directly to individual cores and collects the results one by one to complete the desired task.
Since independent tasks can be executed across 8 cores, assigning work to all 8 cores continuously enables maximum performance (throughput).

![NPU_Single](../res/image/NPU_Single.png "NPU Single Mode")

- Note: While this mode is the most simple and intuitive for functional verification, it requires a mechanism that can send requests in non-blocking manner.
Without a proper design - such as using multithreading or async I/O to send inference requests to multiple cores simultaneously - the system will fall back to fully sequential processing, resulting in wasting core utilization (utilizing only one core).
Since multithreading and async I/O are common design patterns in software engineering, please refer to related resources when designing them.

```python
## Compilation stage
# Generate MXQ Model with Single-core Inference Scheme
from qbcompiler import mxq_compile

mxq_compile(
        ...
        inference_scheme="single" # Technically, this line can be omitted.
        ...
    )
```

## Multi Mode

This mode actually behaves as a batch processing mode, where multiple inputs are fed into the NPU at once, and multiple results are retrived from NPU together.
It operates at `Cluster` level, with each cluster consisting of 4 `Local Cores`.
Through coordination among the 4 `Local Cores` within a `Cluster`-managed by `Global Core`-the system performs operations optimized for 4-batch processing.

![NPU_Multi](../res/image/NPU_Multi.png "NPU Multi Mode")

- Use Cases
    - To struct the NPU inference logic in the same way as conventional batch processing
    - To achieve high performance(throughput) in 4-batch units without implementing multithreading or async I/O

```python
## Compilation stage
# Generate MXQ Model with Multi-core Inference Scheme
from qbcompiler import mxq_compile

mxq_compile(
        ...
        inference_scheme="multi"
        ...
    )
```

## Global Modes

This is an inference method in which multiple cores work collaboratively to process a single piece of data.
For example, in the case of large and computationally heavy model, splitting the single input data across cores can potentially provide performance benefit in terms of latency.
`Global Mode` operates by utilizing `Local Cores` within a cluster, and is offered under the names `Global4 Mode` and `Global8 Mode`.

![NPU_Global](../res/image/NPU_Global.png "NPU Global Mode")

- Use Cases
    - To reduce latency in large model.

### Global4 Mode

4 `Local Cores` (one `Cluster`) work together to process a single piece of data. You can freely select which `Cluster` to execute in `qbruntime`. Compilation and execution can be performed as shown in examples below:

```python
## Compilation stage
# Generate MXQ Model with Global4 Inference Scheme
from qbcompiler import mxq_compile

mxq_compile(
        ...
        inference_scheme="global4"
        ...
    )
```

### Global8 Mode

8 `Local Cores` (2 `Clusters`) work together to process a single piece of data. Compilation and execution can be performed as shown in the examples below:

```python
## Compilation stage
# Generate MXQ Model with Global8 Inference Scheme
from qbcompiler import mxq_compile

mxq_compile(
        ...
        inference_scheme="global8"
        ...
    )
```