Advanced Usage#
This section covers some of the advanced features of the runtime library that allow developers to optimize performance and customize resource allocation on the NPU.
These features are not required for basic operation, but can significantly enhance throughput and flexibility in multi-stream, multi-model, or other various environments where fine-grained control or scalability is needed.
Tip
ARIES contains eight (8) independent NPU processing cores, and REGULUS has one (1) independent NPU core.
(Important) Multi-threading#
Note
The following instruction is applicable to ARIES-powered products only.
By default, inference using the runtime library (via mobilint::Model::infer() method) operates
in a blocking or synchronous I/O manner. This approach has been chosen as the
default because it is the simplest structurally and the easiest to understand when
working with an unfamiliar runtime library.
However, blocking I/O has the following limitation:
When inference is performed using a single thread, the CPU sends a task to NPU and waits for result each time. As a result, CPU cannot immediately sends next inputs, so multiple NPU cores(or clusters) cannot be utilized simultaneously. In other words, CPU must wait until core/cluster finishes its task before sending next input, which prevents taking advantage of multiple cores or clusters.
To address this issue, you can either:
Implement multi-threaded program directly, or
Use
inferAsync()method provided by Mobilint runtime.
For users familiar with multi-threaded programming, performance can be improved by implementing a custom multi-threaded program, as shown in simple example below.
void work(Model* model) {
StatusCode sc;
NDArray<float> input(model->getModelInputShape()[0], sc);
std::vector<NDArray<float>> output;
for (int i = 0; i < 10; i++) {
sc = model->infer({input}, output);
}
}
int main() {
StatusCode sc;
auto acc = Accelerator::create(sc);
auto model = Model::create(MXQ_PATH, sc);
model->launch(*acc);
std::vector<std::thread> threads;
for (int i = 0; i < NUM_THREAD; i++) {
threads.emplace_back(work, model.get());
}
for (int i = 0; i < NUM_THREAD; i++) {
threads[i].join();
}
return 0;
}
Alternatively, if implementing a multi-threaded program is challenging, you can use inferAsync() method provided by Mobilint runtime library, as show in example below.
Caution
The inferAsync() method currently has the following limitations:
RNN/LSTM/LLM models are not yet supported.
Models requiring CPU offloading are not yet supported.
Only single-batch inference supported.
For much details, please refer to API Reference under the Model class, section “Asynchronous Inference”.
## Python example
import qbruntime
from collections import deque
mc = qbruntime.ModelConfig()
mc.set_async_pipeline_enabled(True)
acc = qbruntime.Accelerator()
model = qbruntime.Model(MXQ_PATH, mc)
model.launch(acc)
## Method 1: Simple usage example
future_results = []
results = []
for i in range(NUM_INFERENCE):
rand_input = np.random.rand(224, 224, 3).astype(np.float32)
future_result = model.infer_async(rand_input)
future_results.append(future_result)
for future in future_results:
res = future.get()
results.append(res)
## Method 2: Focus on real-time data processing
future_results = deque()
for i in range(NUM_INFERENCE):
rand_input = np.random.rand(224, 224, 3).astype(np.float32)
future_result = model.infer_async(rand_input)
future_results.append(future_result)
while future_results and (future_results[0].wait_for(0) or i == NUM_INFERENCE - 1):
future = future_results.popleft()
res = future.get()
## RESULT PROCESSING ...
// C++ example
#include "qbruntime/qbruntime.h"
#include <queue>
#include <vector>
int main() {
mobilint::StatusCode sc;
mobilint::ModelConfig mc;
auto acc = mobilint::Accelerator::create(sc);
if (!sc) exit(1);
mc.setAsyncPipelineEnabled(true);
auto model = mobilint::Model::create(MXQ_PATH, mc, sc);
if (!sc) exit(1);
sc = model->launch(*acc);
if (!sc) exit(1);
// Note: For simplicity, validating `sc` value are omitted below.
// Method 1: Simple usage example
std::vector<mobilint::Future<float>> future_results;
std::vector<std::vector<mobilint::NDArray<float>>> results;
for (int i = 0; i < NUM_INFERENCE; i++) {
auto rnd_inputs = mobilint::NDArray<float>({224, 224, 3}, sc);
mobilint::Future<float> future = model->inferAsync({rnd_inputs}, sc);
future_results.push_back(std::move(future));
}
for (auto& future_result: future_results) {
std::vector<mobilint::NDArray<float>> res = future_result.get(sc);
results.push_back(std::move(res));
}
// Method 2: Focus on real-time data processing
std::queue<mobilint::Future<float>> future_results;
for (int i = 0; i < NUM_INFERENCE; i++) {
auto rnd_inputs = mobilint::NDArray<float>({224, 224, 3}, sc);
mobilint::Future<float> future = model->inferAsync({rnd_inputs}, sc);
future_results.push(std::move(future));
while (!future_results.empty() && (future_results.front().waitFor(0) ||
i == NUM_INFERENCE - 1)) {
auto future_result = std::move(future_results.front());
future_results.pop();
std::vector<mobilint::NDArray<float>> res = future_result.get(sc);
// RESULT PROCESSING ...
}
}
}
Tracing NPU Usage#
Runtime library offers a tracing feature. You can start tracing with startTracingEvents().
User can specify the trace log’s path in path argument. The log should result in a .json file format, which can be viewed in Perfetto UI. This trace log would be recorded until stopTracingEvents() is called.
You can enable tracing by using the following methods:
// c++ example
#include "qbruntime/qbruntime.h"
mobilint::startTracingEvents("path/to/trace.json");
// Tracing target NPU events
mobilint::stopTracingEvents();
## python example
import qbruntime
qbruntime.start_tracing_events("path/to/trace.json")
## Tracing target NPU events
qbruntime.stop_tracing_events()