# CPU Offloading

## CPU Offloading Overview

CPU Offloading lets qb Compiler keep on the CPU those parts of a model that are unsupported or unsuitable for the NPU, while compiling the rest of the graph for the Mobilint NPU. The output is still an MXQ package, but the runtime must support CPU-offloaded MXQ execution.

Use CPU Offloading when the model has small preprocessing, postprocessing, shape, or indexing operations that are not good NPU candidates. If the unsupported region is large or appears in the middle of the performance-critical body, it is usually better to change the original model, export a cleaner graph, or compile only the NPU body subgraph.

Enable CPU Offloading explicitly in the compile configuration or Python API:

```python
from qbcompiler import mxq_compile

mxq_compile(
    model="model.onnx",
    backend="onnx",
    target_device="regulus-rb",
    calib_data_path="./calib",
    save_path="./model_cpu_offload.mxq",
    cpu_offload=True,
)
```

The same setting can be specified as `cpuOffload: true` in a compile config file, or as `--cpu-offload` if the installed CLI supports that flag.

Confirm that the runtime version on the device supports MXQ files compiled with CPU Offloading. CPU Offloading is not a replacement for arbitrary Python, PyTorch, TensorFlow, or custom application execution — it only covers the graph regions that the CPU offload path supports.

## CPU Offload Operator Scope

The following MXQ layer types are executable by the CPU Offload path in qb Runtime 1.3.0. The installed runtime determines CPU execution support independently of the qb Compiler release. The model must also satisfy operator attributes, input shapes and dtypes, and graph partitioning requirements.

### Inputs, constants, and dtypes

- `Bridge`
- `Cast`
- `ConstantOfShape`
- `Input`
- `InputConstant`
- `Range`

### Arithmetic and math

- `Abs`
- `Adding`
- `AddingConstant`
- `Div`
- `DivConstant`
- `Exp`
- `Log`
- `MatMul`
- `Mod`
- `Multiply`
- `MultiplyConstant`
- `Pow`
- `Sqrt`
- `Sub`
- `SubConstant`

### Activations and normalization

- `Clip`
- `GroupNormalization`
- `InstanceNormalization`
- `LayerNormalization`
- `Relu`
- `Sigmoid`
- `Softmax`
- `Softplus`
- `Tanh`

### Comparisons and logic

- `And`
- `Equal`
- `Greater`
- `Less`
- `Not`
- `Or`
- `Where`
- `Xor`

### Shapes and axes

- `Concatenate`
- `Expand`
- `Flatten`
- `Flip`
- `HeaderView`
- `Reshape`
- `Reverse`
- `Shape`
- `Split`
- `Squeeze`
- `Tile`
- `Transpose`
- `Unsqueeze`

### Indexing, selection, and reduction

- `ArgMax`
- `Gather`
- `GatherConstant`
- `GatherElements`
- `GatherND`
- `NonZero`
- `ReduceMax`
- `ReduceMean`
- `ReduceMin`
- `ReduceProd`
- `ReduceSum`
- `ScatterND`
- `Slice`
- `TopKIndices`
- `TopKValues`

### Spatial and specialized operations

- `GridSample`
- `MeloTTSPiecewiseRationalQuadraticTransform`
- `Pooling`
- `Resize`

An operator absent from this table is not covered by this runtime's CPU Offload path. The names are MXQ layer types, which can differ from source framework operator names.

## Unsupported Operator Handling

During `parse`, qb Compiler converts the original model to MBLT and marks which layers are supported by the selected target device. During MXQ generation, CPU Offloading can split the graph into three logical regions:

- `CPU head`: operators before the NPU-executable region.
- `NPU body`: the contiguous body subgraph compiled for the NPU.
- `CPU tail`: operators after the NPU body, often task-specific postprocessing.

This is most useful for models such as YOLO, where the convolutional backbone and neck are NPU-friendly but the final decoding or filtering logic may contain unsupported operators. Netron can be used to inspect the MBLT and confirm the supported layer region before compiling.

The presence of unsupported operators is not automatically harmless. CPU Offloading is intended for graph boundary regions and simple CPU-executable operators. If an unsupported operator splits the NPU body into many small islands, CPU/NPU transfer overhead can dominate latency, and the compiler may fail to produce a practical MXQ.

## Calibration Shape Differences

Calibration samples must match the input contract of the graph being compiled: shape, layout, dtype, and any preprocessing defined by the compile configuration. Enabling CPU Offloading alone does not change that contract. A difference arises when a CPU head is included or removed from the compiled graph.

- **CPU head included:** Calibrate with the input seen before the CPU head, usually the original exported model input. For example, use `NCHW` if that is the model-visible layout, or `NHWC`/`uint8` if the compiled input expects raw image data. If the compile configuration adds preprocessing before the model, follow the compiler-visible input format instead.
- **No CPU head, with or without a CPU tail:** Calibrate with the NPU body input. Offloading only postprocessing to the CPU does not change the required input shape.
- **CPU head excluded and only the NPU body compiled:** Calibrate with the extracted NPU body's input, which corresponds to the CPU head's output when that head is run separately.

Calibration samples must also follow the same preprocessing and numeric distribution as inference inputs. If accuracy differs from the original model, check the input contract, preprocessing order, channel order, normalization, and output interpretation before tuning quantization.
