Reference¶
This chapter is a reference for the current qbcompiler command line, Python API, configuration schema, supported inputs, and related terms. The source of truth is the current qbcompiler implementation.
CLI Reference¶
Run the CLI as:
The current command set is compile, parse, quantize, dump-config, info, check, presets, validate, and extract-body. The table below covers the five this manual builds its examples on; presets is described under Compile Configuration.
| Command | Purpose | Main options |
|---|---|---|
compile |
End-to-end model to .mxq: parse to temporary .mblt, then quantize. |
--model, --output, --backend, --device, --target-device, --cpu-offload / --no-cpu-offload, --config-preset, --compile-config, --calib-data-path, --use-random-calib / --no-use-random-calib, --config-save-path |
parse |
Convert an input model to .mblt intermediate IR. |
--model, --output, --backend, --device, --target-device, --cpu-offload / --no-cpu-offload, --config-preset, --compile-config |
quantize |
Convert a runnable .mblt file to .mxq. |
--mblt, --output, --target-device, --calib-data-path, --use-random-calib / --no-use-random-calib, --config-preset, --compile-config, --device, --config-save-path |
dump-config |
Write a default or preset-derived CompileConfig template. |
--output, --preset |
info |
Print the provenance recorded in a .mblt. |
--mblt |
--backend defaults to onnx. --calib-data-path is repeatable. --compile-config accepts JSON, YAML, or YML and is the extension point for settings that do not have dedicated CLI flags. dump-config chooses JSON or YAML from the output file suffix.
For subprocess integrations, the CLI has an opt-in JSONL protocol. Set QBCOMPILER_JSONL=1, true, yes, or on to emit structured status, progress, result, log, and error messages on stdout. Without that environment variable, normal logs are written to stderr.
Python API Reference¶
Public functions are available from qbcompiler.
| API | Purpose |
|---|---|
mxq_compile(...) |
Main Python entry point. Compiles a model or runnable .mblt to .mxq, delegating to one of the two functions below based on the input. |
mxq_compile_from_source(...) |
Parses a source model (ONNX / PyTorch / TensorFlow / TF Lite / TorchScript), then compiles it to .mxq. Takes the same arguments as mxq_compile. |
mxq_compile_from_mblt(...) |
Compiles an existing .mblt, or a list of .mblt files that are one model at different sizes, to .mxq. Its first parameter is mblt, not model. The graph is already parsed, so the parser-only arguments (save_subgraph_type, output_subgraph_path, feed_dict, dynamic_axes, multi_shape, yolo_decode_include, exclude_first_subgraph) are absent; if you pass any of them, a warning names them and they are ignored. model_part and model_part_options also apply only to parsing: pass them to mblt_compile() when producing the .mblt. mxq_compile() given a .mblt together with either of them raises ValueError. |
mxq_compile_with_callback(...) |
mxq_compile with a progress callback. |
mblt_compile_with_callback(...) |
mblt_compile with a progress callback. Takes its output path before its target device. |
read_provenance(mblt_path) |
Returns the provenance record of a .mblt. None when the file carries no record — written before provenance existed, or with recording off. Both container formats carry one otherwise. |
mblt_compile(...) |
Exports a model to .mblt without producing .mxq. |
list_presets() |
Returns preset metadata dictionaries. |
dump_default_config(output_path, preset=None) |
Writes a CompileConfig template to JSON/YAML. |
get_body_subgraph(mblt_path, output_path) |
Writes a .mblt containing the largest NPU-runnable subgraph, together with any subgraphs it calls. A subgraph that is only ever called is not chosen as the body. |
set_log_level(level) |
Sets the log level of every qbcompiler logger at once, as a logging level or its name such as "debug" or "warning". New in qbcompiler 1.4. |
The main mxq_compile parameters are:
from qbcompiler import mxq_compile
mxq_compile(
model="model.onnx",
target_device="aries-rb",
calib_data_path="calib",
save_path="model.mxq",
backend="onnx",
device="gpu",
config_preset="classification",
compile_config=None,
)
Important parameters include model, target_device, calib_data_path, save_path, backend, feed_dict, dynamic_axes, multi_shape, yolo_decode_include, exclude_first_subgraph, device, inference_scheme, use_random_calib, cpu_offload, optimize_option, bias_correction, buffer_mode, force_npu_input_reposition, force_npu_output_reposition, image_channels, split_blocks, split_parts, config_preset, compile_config, config_save_path, model_part, model_part_options, and sub-config objects such as calibration_config, bit_config, llm_config, hessian_quant_config, bias_correction_config, mod_config, equivalent_transformation_config, search_weight_scale_config, uint8_input_config, preprocessing_config, save_sample_config, resource_management_config, and the keyword-only extra_output_config.
multi_shape and extra_output_config are new in qbcompiler 1.4:
-
multi_shapecompiles one model at several sizes of one input dimension in a single call. It is keyed by input name likefeed_dict, and each entry names the axis and the sizes it takes, for example{"x": {"axis": 3, "values": [100, 200, 300]}}or{"x": {"axis": (2, 3), "values": [(224, 224), (448, 448)]}}for axes that move together. The input axis named bymulti_shape.axismust already be marked as a dynamic axis when the model is exported to ONNX.widths = (100, 200, 300) mxq_compile( model="model.onnx", backend="onnx", target_device="aries-rb", feed_dict={"x": example}, multi_shape={"x": {"axis": 3, "values": widths}}, calib_data_path=[f"calib/w{w}" for w in widths], save_path="model.mxq", )multi_shaperequiresfeed_dict. It resizes the named axis of its tensors to create an example tensor for each input size, and parses the model once per size. Several inputs are walked together by index, so theirvalueslists must be the same length.multi_shapecannot be combined with thedynamic_axesargument.mblt_compile()writes one.mbltper size: pass a list ofmblt_save_pathvalues, or one path to get<stem>_<i>.mblt. When an original model is passed withmulti_shape,mxq_compile()creates one.mbltper input size in a temporary directory, runs quantization once, and packs the results into one.mxq. Alternatively, pass the paths of existing.mbltfiles asmodel. Because those files are already parsed, parser arguments such asfeed_dictandmulti_shapedo not apply and produce a warning if passed. Before quantization, the compiler instead compares the source-model information and graph structure recorded in the files, rejecting a set that does not represent the same model and reporting the difference. In both multi-shape cases,calib_data_pathtakes one calibration directory per size, paired by index withvalues. -
extra_output_configadds named intermediate layers to the MXQ's outputs. See Compile Configuration — Exporting intermediate activations.
qbcompiler 1.4 also removes the legacy Model_Dict parser and the arguments only it read. qbcompiler.Model_Dict, qbcompiler.compiler.LegacyCompiler and qbcompiler.compiler.compiler_legacy no longer exist; use mxq_compile() or mblt_compile(). Passing in_dformats or input_shape_dict raises ValueError. Input layouts are inferred from the model, and multi_shape replaces input_shape_dict. layer_bias_correction and layer_bias_correction_config are renamed to bias_correction and bias_correction_config.
config_save_path, model_part and model_part_options are new in qbcompiler 1.3:
config_save_pathwrites the fully resolvedCompileConfig— the normalized configuration after every layer of the precedence order below has been applied — before compilation starts. A.yamlor.ymlsuffix writes YAML, any other suffix writes JSON, and parent directories are created. Because it is written up front, the record survives a compile that fails after it starts, and the file can be passed straight back ascompile_config=to reproduce the compile. It is written by the quantize phase, so acompilethat fails while parsing produces nothing. The CLI exposes it as--config-save-pathonquantizeandcompile.model_partnames which part of a multi-part torch model to parse ("vision","language","encoder", and so on), andmodel_part_optionspasses extra arguments that part needs, such as{"mel_frames": 100}. A model declaring exactly one part resolves it fromNone; a model declaring several requires a name. List them withavailable_parts(model)fromqbcompiler.model_dict.parser.patcher.parts. These replace the removedhf_configargument, which now raisesValueError.
Configuration resolution priority (highest wins): explicit arguments → sub-config objects → compile_config file or config_preset, whichever is given → defaults. Passing both uses the preset and ignores the file. For details and usage examples, see Compile Configuration — Config Resolution Priority.
CompileConfig Schema¶
CompileConfig is a Pydantic model with extra="forbid" and aliases enabled. Use alias names in JSON/YAML files. A full template can be generated with:
python -m qbcompiler dump-config --output compile_config.yaml
python -m qbcompiler dump-config --preset yolo_640 --output yolo_640.yaml
Top-level fields:
| Alias | Python field | Default | Meaning |
|---|---|---|---|
modelPaths |
model_paths |
[] |
Model path list. Usually supplied by API/CLI instead. |
calibDataPaths |
calib_data_path |
[] |
Calibration dataset path list. |
savePaths |
save_paths |
["./tmp.mxq"] |
MXQ output path list. |
useRandomCalib |
use_random_calib |
false |
Generate random calibration data. |
inferenceScheme |
inference_scheme |
single |
Compile-time NPU core-assignment scheme used by the generated MXQ. See inferenceScheme. |
cpuOffload |
cpu_offload |
false |
Enable CPU offloading for unsupported groups. |
forceNpuInputReposition |
force_npu_input_reposition |
false |
Force input reposition operations onto NPU. |
forceNpuOutputReposition |
force_npu_output_reposition |
false |
Force output reposition operations onto NPU. |
optimizeOption |
optimize_option |
1 |
Back-end optimization selector. Accepted values are 0, 1, 2, 3, 4 and 99. From qbcompiler 1.4 any other value is rejected during back-end compilation instead of being ignored. |
bufferMode |
buffer_mode |
1 |
Buffer serialization mode. |
device |
device |
gpu |
Compilation compute device: gpu or cpu. |
dtype |
dtype |
float |
Computation dtype metadata. |
debug |
debug |
false |
Enable debug mode. |
trace |
trace |
false |
Enable trace mode. |
imageChannels |
image_channels |
0 |
Number of image channels; 0 means auto-detect. |
configVersion |
config_version |
1.0.0 |
Config schema version. |
splitBlocks |
split_blocks |
[] |
LLM multi-MXQ split points by block index. |
splitParts |
split_parts |
0 |
Split LLM transformer blocks into N parts. |
Nested sections:
| Alias | Type | Purpose |
|---|---|---|
uint8Input |
Uint8InputConfig |
Treat model inputs as uint8, optionally by input name. |
preprocessing |
PreprocessingConfig |
Input preprocessing pipeline such as resize, letterbox, crop, normalize, and format conversion. |
resourceManagement |
ResourceManagementConfig |
Weight dtype, the GPU memory budget for weight quantization (gpuMemoryBudgetMB, in MiB; -1 automatic by default, 0 unlimited), and weight-memory method. useGPUOnlyForCalibration was removed in 1.4. |
calibration |
CalibrationConfig |
Quantization method, output quantization, calibration mode, clipping/statistics, and layer overrides. |
bit |
BitConfig |
Quantization precision settings. |
hessianQuant |
HessianQuantConfig |
HessianQuant optimization settings, including hessianDtype, solver, rescomp and attributes.alpha. accumulationDevice was removed in 1.4. See Model Quantization. |
biasCorrection |
BiasCorrectionConfig |
Corrects the per-channel bias that quantization introduces. On by default. Replaces layerBiasCorrection from 1.4. |
mod |
ModConfig |
MOD training/optimization settings. |
llm |
LlmConfig |
LLM sequence/cache/runtime/debug settings. If runtime.batchSize is greater than 1, inferenceScheme must be single, global4 or global8 — multi and all are rejected — and the Batch LLM runtime API uses BatchParam metadata; see qb Runtime Release Notes. |
moe |
MoeConfig |
Sparse MoE calibration expert selection. |
equivalentTransformation |
EquivalentTransformationConfig |
SmoothQuant-like, rotation, and FFN equivalent transformations. |
searchWeightScale |
SearchWeightScaleConfig |
Weight-scale search. |
loadScale |
LoadScaleConfig |
Load externally supplied scale entries. |
runtimeOptions |
RuntimeOptions |
Internal runtime metadata. |
saveSample |
SaveSampleConfig |
Save sample data for inspection/debugging. |
extraOutput |
ExtraOutputConfig |
Add named intermediate layers to the MXQ's outputs (apply, layers). New in 1.4. |
CompileConfig.from_file() accepts .json, .yaml, and .yml. For compatibility, it also flattens grouped keys such as quantization.calibration, quantization.bit, advancedQuantization.hessianQuant, advancedQuantization.mod, advancedQuantization.EquivalentTransformation, advancedQuantization.searchWeightScale, advancedQuantization.loadScale, and advancedQuantization.biasCorrection, which replaces 1.3's advancedQuantization.layerBiasCorrection from 1.4. That grouped spelling is also the one the tool writes — dump-config and --config-save-path both emit it. Flattening overwrites rather than merges, so a top-level calibration block added beside a quantization.calibration one is discarded with no message: do not mix the two spellings in one file.
inferenceScheme¶
inferenceScheme selects how NPU core work is assigned when qb Runtime uses the MXQ for inference. Because the compiled MXQ is prepared for the selected core-assignment scheme, an MXQ built only for one mode cannot later be switched arbitrarily to another mode at runtime. One built with all carries every mode the model and target support, but the caller must name one: qb Runtime's default CoreMode::Auto accepts only an MXQ with exactly one mode, so an all build needs an explicit setSingleCoreMode() or setGlobal4CoreMode() — see ARIES ModelConfig Configuration.
Available choices are single, multi, global4, global8, and all. single uses independent Local Cores, multi is the cluster-level 4-batch mode, and global4/global8 use 4 or 8 Local Cores together for one input. all prepares the MXQ for every core mode supported by the selected model and target. Support can vary by model, target, and compiler version. REGULUS targets have a single NPU core, so MXQ files compiled for REGULUS can use only single mode. From qbcompiler 1.3 this is enforced rather than left to fail later: multi, global4 and global8 are rejected outright on a single-core target, and all narrows to single there. The check runs before quantization regardless of whether the scheme or the target device was set first. global is a legacy/compatibility value; use global4 or global8 for new settings.
If llm.attributes.runtime.batchSize is greater than 1, LLM/KV-cache transformer compilation accepts single, global4 and global8, and rejects multi and all rather than narrowing them — each segment compiles under exactly one scenario, and guessing which one was meant would build for a core set nobody asked for. Vision batch inference and LLM batch inference do not use the same core-mode rule.
For what each mode means, see ARIES Core Mode. For runtime core/cluster selection, see ARIES ModelConfig Configuration.
RuntimeOptions¶
RuntimeOptions currently contains only:
| Field | Default | Meaning |
|---|---|---|
version |
0.0.0 |
Compiler/runtime version metadata. |
This section is runtime metadata rather than a normal user tuning surface. Prefer the compile, calibration, preprocessing, LLM, and optimization sections for user-controlled behavior.
Preset List¶
| Preset | Extends | Behavior |
|---|---|---|
classification |
none | Image classification defaults: calibration.mode=1, calibration.output=0. |
detection |
none | Object detection defaults: calibration.mode=1, calibration.output=1. |
classification_torchvision |
classification |
Enables uint8 input and standard Torchvision preprocessing: resize shortest side to 256 with the Pillow backend (backend: "pil"), center crop 224x224, normalize ImageNet mean/std, imageChannels=3. |
yolo_640 |
detection |
Enables uint8 input and 640x640 letterbox preprocessing with pad value 114 and the OpenCV backend (backend: "opencv"), imageChannels=3. |
yolo_1280 |
detection |
Enables uint8 input and 1280x1280 letterbox preprocessing with pad value 114 and the OpenCV backend (backend: "opencv"), imageChannels=3. |
From qbcompiler 1.4 the three image presets use the Pillow and OpenCV backends to match their reference evaluators, so their calibration tensors and quantization results can differ from 1.3.
| llm | none | Enables LLM config with maxSequenceLength=4096, maxCacheLength=4096, calibration.mode=0, calibration.output=0, full-sequence-length LLM calibration, and the QK, UD, VO, SpinR1, SpinR2 and OptimizeFFN equivalent transformations. |
| llm_fast | llm | Turns off those six equivalent transformations and full-sequence-length calibration, trading accuracy for compile time. |
| vision_transformer | none | Transformer-oriented calibration/bit settings. |
| multimodal | none | Enables LLM handling and uses calibration.method=3. |
Supported Frameworks¶
The public backend strings are below. Backend names are case-insensitive. From qbcompiler 1.4 every backend uses the current parser; tf, tflite and torchscript no longer go through the legacy parser.
| Backend | Input |
|---|---|
onnx |
ONNX model path. This is the primary path and is used by validate_model. |
tf |
TensorFlow SavedModel directory, Keras .keras / .h5 file, or frozen GraphDef .pb. tensorflow and keras are accepted as other names for this backend. |
tflite |
TensorFlow Lite .tflite model path. |
torchscript |
Path of a torch.jit.save archive. It is exported to ONNX first, so feed_dict is required. |
torch |
PyTorch model or HuggingFace/transformers model path/object flows. HuggingFace LLMs use this backend path, not a separate hf backend. |
Supported Target Device¶
Use these target device strings in CLI and Python API calls:
| target device | Meaning |
|---|---|
regulus-ra |
Target device for regulus-ra. |
aries-rb |
Target device for aries-rb. |
regulus-rb |
Target device for regulus-rb. |
regulus-rb-usb |
USB-connected target device with the same NPU and CPU Offload operator scope as regulus-rb. |
The validator accepts the public strings above. Internal enum names use underscores, but user-facing values use hyphens.
Supported Operators¶
NPU operator support lists supported and constrained framework operators and Mobilint IR operations. Operations outside those tables may still compile after graph-level transformation. Whether a given operation runs on the NPU depends on the model, backend, shapes, and target device; qb Compiler determines this for each compile. For CPU execution, see CPU Offloading.
Available reference paths:
- For non-ONNX backends, parse/compile the model and inspect unsupported groups with
.mbltand CPU offloading workflows. - If a model contains unsupported groups,
cpuOffload/cpu_offloadcan partition those groups for CPU execution where the runtime flow supports it.
Glossary¶
| Term | Meaning |
|---|---|
MXQ |
Mobilint executable package produced by quantization/compilation and run on Mobilint NPUs. |
MBLT |
Mobilint intermediate model format used between parsing and MXQ generation. |
parse |
Convert a source model into .mblt. |
quantize |
Convert a runnable .mblt into .mxq using calibration/config settings. |
compile |
End-to-end parse plus quantize flow. |
backend |
Source model framework identifier such as onnx, torch, or tflite. |
target_device |
Mobilint NPU target device string such as aries-rb. |
| NPU Chip | Mobilint NPU product name such as ARIES or REGULUS. |
| target device | Compile target such as aries-rb, regulus-ra, regulus-rb, or regulus-rb-usb; used as the target device string. |
calibration data |
Representative input tensors used to derive quantization scales/statistics. |
CPU offloading |
Partitioning unsupported graph groups for CPU execution while supported body subgraphs run on NPU. |
body subgraph |
The largest NPU-runnable supported subgraph extracted from a partitioned .mblt. |
preset |
Built-in partial CompileConfig for common model families. |
License¶
See License for the open-source license notices included with this manual.