CPU Offloading (Beta Version)#
Remark To proceed inference with CPU offloading, it requires a runtime library that supports the MXQ file that is compiled for CPU offloading.
From qubee v0.7, we provide a Beta version of CPU offloading for MXQ compilation. CPU offloading makes it easier for users to compile their models by automatically offloading the computation that Mobilint NPU does not support to the CPU. For example, if a pre-processing or post-processing included in the model involves operations that the NPU does not support, the user would have to implement them manually after compiling. However, CPU offloading covers most of these operations and eliminates the need for additional work.
When CPU offloading is employed, the procedures for preparing the calibration dataset and compiling the model vary slightly as follows:
The pre-processed input shape should match the original model’s input shape, whereas the pre-processed input shape should be in the format (H, W, C) to compile the model without CPU offloading.
Set the argument cpu_offload of function mxq_compile True to enable CPU offloading.
