qbcompiler.frontend¶
Attributes¶
| Name | Description |
|---|---|
logger |
logger¶
Model_Dict¶
Wrapper around the Mobilint compiler to support compilation and inference workflows.
Capabilities include:
- Compilation of models into MXQ artifacts runnable on Mobilint NPUs.
- Inference using the full-precision high-level compiled model on CPU or GPU.
- Inference with the quantized model on CPU or GPU.
Methods¶
| Name | Description |
|---|---|
__init__ |
Initialize the Mobilint compiler wrapper. |
compile |
Compile the wrapped model into an MXQ artifact. |
__init__¶
def __init__(
self,
model,
backend="onnx",
device="cpu",
feed_dict=None,
dynamic_axes=None,
in_dformats=None,
yolo_decode_include=False,
exclude_first_subgraph=False,
**kwargs,
)
Initialize the Mobilint compiler wrapper.
Parameters
model— string or model instance. Model path or in-memory model to compile.backend— string. Framework identifier for the model (for example "onnx"). Defaults to "onnx".device— string. Target device for inference ("cpu" or "gpu"). Defaults to "cpu".feed_dict— dict. Example inputs for shape resolution.dynamic_axes— dict. Marks model axes as dynamic.in_dformats— dict. Describes input data formats.yolo_decode_include— bool. Runs YOLO decode on NPU whenTrue.exclude_first_subgraph— bool. Applies only when CPU offloading is enabled: exclude the first subgraph from the final graph if it is unsupported.kwargs— dict. Additional compiler arguments.
compile¶
def compile(
self,
save_subgraph_type: int = 0,
output_subgraph_path="",
compile_config: CompileConfig = None,
**kwargs,
)
Compile the wrapped model into an MXQ artifact.
Parameters
save_subgraph_type— int. Controls optional MBLT exports: 0 disables; 1 saves graph structure; 2 saves structure and weights; 3 splits structure into multiple subgraphs; 4 splits structure and weights.output_subgraph_path— string. Destination for exported .mblt whensave_subgraph_typeis 1–4.compile_config— CompileConfig. Compile configuration object.kwargs— dict. Additional arguments forwarded to the compiler.
Functions¶
| Name | Description |
|---|---|
mxq_compile |
Compile a model into a Mobilint eXeCUtable (MXQ) package for execution on Mobilint NPUs. |
mxq_compile_from_source |
Compile a raw framework model (ONNX / PyTorch / TensorFlow / TF-Lite) into an MXQ package. |
mxq_compile_from_mblt |
Compile an existing Mobilint IR (.mblt) file into an MXQ package. |
mblt_compile |
Export a model to the Mobilint .mblt format without producing an MXQ package. |
mblt_compile_with_callback |
Compile-to-mblt entry point with progress callbacks. |
mxq_compile_with_callback |
Compile/quantize entry point with progress callbacks. |
mxq_compile¶
def mxq_compile(
model,
target_device: str,
calib_data_path: Union[str, List[str]] = UNSET,
save_subgraph_type: int = 0,
output_subgraph_path="",
save_path: Union[str, List[str]] = UNSET,
backend="onnx",
# -- ModelDict params --
feed_dict=None,
dynamic_axes=None,
in_dformats=None,
yolo_decode_include=False,
exclude_first_subgraph=False,
# -- CompileConfig-overridable (UNSET = defer to config) --
device=UNSET,
inference_scheme=UNSET,
use_random_calib=UNSET,
cpu_offload=UNSET,
optimize_option=UNSET,
buffer_mode=UNSET,
input_shape_dict=UNSET,
force_npu_input_reposition=UNSET,
force_npu_output_reposition=UNSET,
image_channels=UNSET,
split_blocks=UNSET,
split_parts=UNSET,
# -- Config objects --
config_preset: Optional[str] = None,
compile_config: Optional[CompileConfig] = None,
resource_management_config: Optional[ResourceManagementConfig] = None,
calibration_config: Optional[CalibrationConfig] = None,
bit_config: Optional[BitConfig] = None,
llm_config: Optional[LlmConfig] = None,
hessian_quant_config: Optional[HessianQuantConfig] = None,
mod_config: Optional[ModConfig] = None,
equivalent_transformation_config: Optional[EquivalentTransformationConfig] = None,
search_weight_scale_config: Optional[SearchWeightScaleConfig] = None,
save_sample_config: Optional[SaveSampleConfig] = None,
uint8_input_config: Optional[Uint8InputConfig] = None,
preprocessing_config: Optional[PreprocessingConfig] = None,
layer_bias_correction=UNSET,
layer_bias_correction_config: Optional[LayerBiasCorrectionConfig] = None,
model_part: Optional[str] = None,
model_part_options: Optional[dict] = None,
config_save_path: Optional[str] = None,
**kwargs,
)
Compile a model into a Mobilint eXeCUtable (MXQ) package for execution on Mobilint NPUs.
When no explicit value is provided for a parameter marked with UNSET, the default from CompileConfig is used. This allows a compile_config file or object to supply the value without being overridden by function-level defaults.
Configuration is resolved in priority order (highest to lowest):
- Explicitly passed function arguments
- Individual sub-config objects (calibration_config, llm_config, etc.)
- kwargs partial overrides (quantization_method, weight_dtype, etc.)
- compile_config (CompileConfig object or JSON/YAML file) or config_preset
- CompileConfig field defaults
Parameters
model— string or model instance. Model path. When usingbackend="onnx", this should be the path to an ONNX model file. Whenbackendis "torchscript", provide a TorchScript module; when "torch", provide a standard PyTorch model. Forbackend="tf", pass the directory that contains the TensorFlow SavedModel graph and assets. Forbackend="tflite", pass the TF Lite model path.calib_data_path— string or list of strings. Path(s) to the calibration dataset. Accepts either a text/json file that lists NumPy files or a directory that contains the pre-processed NumPy files.save_subgraph_type— int. Controls optional MBLT subgraph exports: 0 disables exports; 1 saves only the graph structure; 2 saves graph structure plus weights; 3 saves the graph structure split into multiple subgraphs; 4 saves both structure and weights split into multiple subgraphs. Defaults to 0.output_subgraph_path— string. Destination path for the exported .mblt file whensave_subgraph_typeis 1–4. The resulting file can be used for visualization. Defaults to "".save_path— string or list of strings. Output MXQ filename(s). When omitted, defaults to "{model_name}.mxq" derived from the model path basename.backend— string. Framework used to generate the Mobilint IR. Case-insensitive, so "ONNX" and "onnx" are the same backend. "onnx" and "torch" are parsed by the current parser; "tf" (also spelled "tensorflow"), "tflite" and "torchscript" go through the legacy parser. Any other value raisesValueError. Defaults to "onnx". Whenmodelis a path to an already-compiled .mblt file (the output ofmblt_compile()), it is compiled directly with MXQ regardless of thebackendvalue, and the file is validated beforehand; it must be a runnable artifact produced bymblt_compile()rather than asave_subgraph_typepreview export.target_device— string. Target NPU device for parser/compiler configuration (for example "aries-rb"). Required by execution paths that parse or quantize a model.device— string. Compilation and inference device: "cpu" or "gpu". When omitted, uses CompileConfig default.feed_dict— dict. Example input tensors for shape inference and inference validation.dynamic_axes— dict. Marks model axes as dynamic. Keys are input names and values map dimension indices to aliases (for example {"input": {2: "seq_len"}}).in_dformats— dict. Describes input tensor data formats.inference_scheme— string. NPU inference scheme. One of "single", "multi", "global", "global4", or "global8". When omitted, uses CompileConfig default.yolo_decode_include— bool. Determines whether YOLO decode runs on NPU. Defaults toFalse.use_random_calib— bool. Generates random calibration data to validate model compilability. When omitted, uses CompileConfig default.cpu_offload— bool. Enables CPU offloading during NPU inference. When omitted, uses CompileConfig default.optimize_option— int. Compiler optimization strategy selector. When omitted, uses CompileConfig default.buffer_mode— int. Buffer serialization mode: 0 uses a naive buffer, 1 uses an mmap-backed buffer. When omitted, uses CompileConfig default.input_shape_dict— dict. Multi-shape compilation specification for STT/TTS models only (e.g., Conformer-CTC, MeloTTS). The dynamic axis is defined with respect to each input tensor's shape. During compilation, the same axis is varied across all model inputs using the provided values. Currently, only a single entry key ("multi_shape0") is supported. Example: {"multi_shape0": {"axis": 2, "values": [100, 200, 300]}}force_npu_input_reposition— bool. Force input reposition operations to run on NPU instead of CPU. When omitted, uses CompileConfig default.force_npu_output_reposition— bool. Force output reposition operations to run on NPU instead of CPU. When omitted, uses CompileConfig default.image_channels— int. Number of image channels (0 for auto-detect). When omitted, uses CompileConfig default.split_blocks— list of int. Multi-MXQ split points by transformer block index. Only supported for LLM models. When omitted, uses CompileConfig default.split_parts— int. Evenly split transformer blocks into N MXQ parts. Only supported for LLM models. When omitted, uses CompileConfig default.layer_bias_correction— bool. Enables calibration-derived layer bias correction. When omitted, useslayer_bias_correction_configor the CompileConfig value.exclude_first_subgraph— bool. Applies only when CPU offloading is enabled: exclude the first subgraph from the final graph if it is unsupported. Defaults toFalse.config_preset— string. Name of a built-in configuration preset. When provided, loads the preset via CompileConfig.from_preset(). Defaults to None (no preset). Available presets: - "classification": Image classification models (ResNet, EfficientNet, ViT, etc.) - "detection": Object detection models (YOLO, SSD, DETR, etc.) - "classification_torchvision": Torchvision classification models with standard preprocessing - "yolo_640": YOLO detection models with 640x640 letterbox preprocessing - "yolo_1280": YOLO detection models with 1280x1280 letterbox preprocessing - "llm": Large Language Models (LLaMA, Qwen, Gemma, etc.) - "llm_fast": LLM with faster compilation (less accuracy optimization) - "vision_transformer": Vision Transformer models (ViT, DeiT, Swin, etc.) - "multimodal": Multimodal models (CLIP, BLIP, LLaVA, etc.)compile_config— CompileConfig or string. CompileConfig object or path to JSON/YAML configuration file containing all compilation settings (resourceManagement, calibration, bit, hessianQuant, mod, llm, etc.).config_save_path— string. When provided, the fully resolved configuration - the normalized CompileConfig after every layer of the precedence order above has been applied, including all sub-configurations - is written to this path before compilation starts. The format follows the path suffix: ".yaml"/".yml" produce YAML, anything else produces JSON. Parent directories are created as needed. The saved file can be fed back ascompile_configto reproduce the same compilation. Defaults to None (no config file is written).resource_management_config— ResourceManagementConfig. Resource management configuration object.calibration_config— CalibrationConfig. Calibration configuration object.bit_config— BitConfig. Bit configuration object for quantization precision settings.llm_config— LlmConfig. LLM configuration object.hessian_quant_config— HessianQuantConfig. HessianQuant (Hessian-based Quantization) configuration object.layer_bias_correction_config— LayerBiasCorrectionConfig. Calibration-derived layer bias correction settings.model_part— string. Selects a specific part of a torch model to parse ("vision", "language", "encoder", ...). Typically used to support models with complex architectures (e.g. Qwen3-VL) by compiling one part at a time. Parts a model declares: parser.patcher.parts.available_parts(model). A model declaring exactly one part resolves it from None; one declaring several requires a name.model_part_options— dict. Extra arguments for the named part (e.g. {"mel_frames": 100} for Qwen3-ASR audio).mod_config— ModConfig. MOD (Metric-based Optimization and Distillation) configuration object.equivalent_transformation_config— EquivalentTransformationConfig. Configuration for equivalent transformations like SmoothQuant.search_weight_scale_config— SearchWeightScaleConfig. Configuration for weight scale search.save_sample_config— SaveSampleConfig. Configuration for sample data generation and saving.uint8_input_config— Uint8InputConfig. Configuration for uint8 input handling.preprocessing_config— PreprocessingConfig. Preprocessing pipeline configuration.kwargs— dict. Additional compiler arguments. Supports partial config overrides such asquantization_method,quantization_mode,percentile,weight_dtype,ram_usage,max_sequence_length, etc. Also accepts deprecated parameters (quantization_config,advanced_quantization_config,input_process_config,save_sample,sample_dtype) which will emit DeprecationWarning.
Returns
None.
Using configuration
There are three ways to configure quantization settings:
-
Load all settings from a JSON/YAML config file:
-
Pass individual sub-config objects:
from qbcompiler import mxq_compile from qbcompiler.configs import ( ResourceManagementConfig, CalibrationConfig, BitConfig, HessianQuantConfig, ModConfig, LlmConfig, ) resource_mgmt = ResourceManagementConfig(weight_dtype="float32") calib_cfg = CalibrationConfig(method=1, mode=1) bit_cfg = BitConfig(...) hessian_quant_cfg = HessianQuantConfig(apply=True) mod_cfg = ModConfig(apply=False) llm_cfg = LlmConfig(apply=True) mxq_compile( model="path/to/model.onnx", target_device="aries-rb", calib_data_path="path/to/calib", resource_management_config=resource_mgmt, calibration_config=calib_cfg, bit_config=bit_cfg, hessian_quant_config=hessian_quant_cfg, mod_config=mod_cfg, llm_config=llm_cfg, device="gpu", ) -
Automatically applied partial configuration overrides:
from qbcompiler import mxq_compile mxq_compile( model="path/to/model.onnx", target_device="aries-rb", calib_data_path="path/to/calib", quantization_method=1, # per channel quantization quantization_mode=1, # max percentile quantization percentile=0.999, # percentile value for max percentile quantization quantization_output=0, # per layer quantization for the output layer device="gpu", )
Please refer to the mxq_compile function and the quantization configuration section for the meaning of the quantization-related numeric values.
Compiling models with custom inputs (ONNX/Torch/TensorFlow)
Provide NumPy inputs whenever the model omits shape information so that qbcompiler can infer unknown dimensions and data formats.
from qbcompiler import mxq_compile
import numpy as np
example_input = {
"input_node_name_1": np.random.randn(1, 3, 224, 224).astype(np.float32),
"input_node_name_2": np.random.randn(1, 8, 224, 224).astype(np.float32),
"input_node_name_3": np.random.randn(1, 4, 56, 56).astype(np.float32),
}
in_dformats = {
"input_node_name_1": "NCHW",
"input_node_name_2": "NCHW",
"input_node_name_3": "NCHW",
}
onnx_model_path = "path/to/your/model.onnx"
mxq_compile(
model=onnx_model_path,
target_device="aries-rb",
feed_dict=example_input,
in_dformats=in_dformats,
backend="onnx",
compile_config="path/to/config.json",
)
mxq_compile_from_source¶
def mxq_compile_from_source(
model,
target_device: str,
calib_data_path: Union[str, List[str]] = UNSET,
save_subgraph_type: int = 0,
output_subgraph_path="",
save_path: Union[str, List[str]] = UNSET,
backend="onnx",
# -- ModelDict params --
feed_dict=None,
dynamic_axes=None,
in_dformats=None,
yolo_decode_include=False,
exclude_first_subgraph=False,
# -- CompileConfig-overridable (UNSET = defer to config) --
device=UNSET,
inference_scheme=UNSET,
use_random_calib=UNSET,
cpu_offload=UNSET,
optimize_option=UNSET,
buffer_mode=UNSET,
input_shape_dict=UNSET,
force_npu_input_reposition=UNSET,
force_npu_output_reposition=UNSET,
image_channels=UNSET,
split_blocks=UNSET,
split_parts=UNSET,
# -- Config objects --
config_preset: Optional[str] = None,
compile_config: Optional[CompileConfig] = None,
resource_management_config: Optional[ResourceManagementConfig] = None,
calibration_config: Optional[CalibrationConfig] = None,
bit_config: Optional[BitConfig] = None,
llm_config: Optional[LlmConfig] = None,
hessian_quant_config: Optional[HessianQuantConfig] = None,
mod_config: Optional[ModConfig] = None,
equivalent_transformation_config: Optional[EquivalentTransformationConfig] = None,
search_weight_scale_config: Optional[SearchWeightScaleConfig] = None,
save_sample_config: Optional[SaveSampleConfig] = None,
uint8_input_config: Optional[Uint8InputConfig] = None,
preprocessing_config: Optional[PreprocessingConfig] = None,
layer_bias_correction=UNSET,
layer_bias_correction_config: Optional[LayerBiasCorrectionConfig] = None,
model_part: Optional[str] = None,
model_part_options: Optional[dict] = None,
config_save_path: Optional[str] = None,
**kwargs,
)
Compile a raw framework model (ONNX / PyTorch / TensorFlow / TF-Lite) into an MXQ package.
This is the raw-model half of mxq_compile(): it parses the model into Mobilint IR and then quantizes and compiles it in one pass. The parameters, their defaults, and the configuration precedence rules are identical to mxq_compile() — see that function for the full reference and usage examples.
Passing the path of an existing .mblt file here raises ValueError; use mxq_compile_from_mblt() for that input instead. mxq_compile() routes between the two automatically.
Parameters
model— string or model instance. Raw model or path to one. Must not be an existing.mbltfile.target_device— string. Target NPU device (for example "aries-rb").
Returns
None.
mxq_compile_from_mblt¶
def mxq_compile_from_mblt(
mblt: str,
target_device: str,
calib_data_path: Union[str, List[str]] = UNSET,
save_path: Union[str, List[str]] = UNSET,
backend="onnx",
device=UNSET,
inference_scheme=UNSET,
use_random_calib=UNSET,
cpu_offload=UNSET,
optimize_option=UNSET,
buffer_mode=UNSET,
input_shape_dict=UNSET,
force_npu_input_reposition=UNSET,
force_npu_output_reposition=UNSET,
image_channels=UNSET,
split_blocks=UNSET,
split_parts=UNSET,
config_preset: Optional[str] = None,
compile_config: Optional[CompileConfig] = None,
resource_management_config: Optional[ResourceManagementConfig] = None,
calibration_config: Optional[CalibrationConfig] = None,
bit_config: Optional[BitConfig] = None,
llm_config: Optional[LlmConfig] = None,
hessian_quant_config: Optional[HessianQuantConfig] = None,
mod_config: Optional[ModConfig] = None,
equivalent_transformation_config: Optional[EquivalentTransformationConfig] = None,
search_weight_scale_config: Optional[SearchWeightScaleConfig] = None,
save_sample_config: Optional[SaveSampleConfig] = None,
uint8_input_config: Optional[Uint8InputConfig] = None,
preprocessing_config: Optional[PreprocessingConfig] = None,
layer_bias_correction=UNSET,
layer_bias_correction_config: Optional[LayerBiasCorrectionConfig] = None,
config_save_path: Optional[str] = None,
**kwargs,
)
Compile an existing Mobilint IR (.mblt) file into an MXQ package.
This is the pre-parsed half of mxq_compile(): parsing already happened (via mblt_compile()), so this function only quantizes and compiles. Quantization parameters, their defaults, and the configuration precedence rules are identical to mxq_compile() — see that function for the full reference and usage examples.
Parser-only arguments (save_subgraph_type, output_subgraph_path, feed_dict, dynamic_axes, in_dformats, yolo_decode_include, exclude_first_subgraph) are absent because the graph is already parsed. mxq_compile() accepts them for backward compatibility and drops them when routing here.
Parameters
mblt— string. Path to a runnable.mbltproduced bymblt_compile(). The file is validated up front; asave_subgraph_typepreview export is rejected.target_device— string. Target NPU device (for example "aries-rb").backend— string. Retained for backward compatibility and ignored: the graph is already parsed, so no parser backend is selected.
Returns
None.
Throws
ValueError—mbltis not an existing.mbltfile, or it is not a runnable artifact for the requestedcpu_offloadsetting.
mblt_compile¶
def mblt_compile(
model: str | Any,
mblt_save_path: str,
target_device: str,
backend="onnx",
device="cpu",
feed_dict=None,
dynamic_axes=None,
in_dformats=None,
yolo_decode_include=False,
cpu_offload=False,
exclude_first_subgraph=False,
model_part: Optional[str] = None,
model_part_options: Optional[dict] = None,
**kwargs,
)
Export a model to the Mobilint .mblt format without producing an MXQ package.
Parameters
model— string or model instance. Source model or path to compile.mblt_save_path— string. Output path for the .mblt artifact.backend— string. Framework identifier, case-insensitive. "onnx" and "torch" are parsed by the current parser; "tf" (also spelled "tensorflow"), "tflite" and "torchscript" go through the legacy parser. Any other value raisesValueError. Defaults to "onnx".device— string. Compilation device ("cpu" or "gpu"). Defaults to "cpu".feed_dict— dict. Example inputs used for shape inference.dynamic_axes— dict. Declares dynamic axes per input name.in_dformats— dict. Input dataformat metadata.yolo_decode_include— bool. Runs YOLO decode on NPU when True.cpu_offload— bool. Enables CPU offloading for unsupported groups.exclude_first_subgraph— bool. Applies only when CPU offloading is enabled: exclude the first subgraph from the final graph if it is unsupported.model_part— string. Selects a specific part of a torch model to parse ("vision", "language", "encoder", ...). Typically used to support models with complex architectures (e.g. Qwen3-VL) by compiling one part at a time. Parts a model declares: parser.patcher.parts.available_parts(model). A model declaring exactly one part resolves it from None; one declaring several requires a name.model_part_options— dict. Extra arguments for the named part (e.g. {"mel_frames": 100} for Qwen3-ASR audio).kwargs— dict. Additional arguments forwarded to the compiler.
Returns
None.
mblt_compile_with_callback¶
def mblt_compile_with_callback(
model: str,
mblt_save_path: str,
target_device: str,
backend: str = "onnx",
device: Optional[str] = None,
cpu_offload: Optional[bool] = None,
config_preset: Optional[str] = None,
compile_config: Optional[CompileConfig | str] = None,
compilation_mode: Optional[str] = None,
*,
progress_callback: Callable[[int, str], None],
) -> None
Compile-to-mblt entry point with progress callbacks.
Only forwards explicitly-set values so the UNSET defaults (and the merged [CompileConfig](qbcompiler.configs.models.md#compileconfig) behind them) apply when the caller omits a flag -- which is why this goes through _mblt_compile_dispatch rather than :func:mblt_compile, whose own device / cpu_offload defaults would override the config. target_device is required and has no config fallback — callers (e.g. the CLI) validate it via validate_target_device before invoking (QC-228).
mxq_compile_with_callback¶
def mxq_compile_with_callback(
model: str,
target_device: str,
save_path: str,
backend: str = "onnx",
device: Optional[str] = None,
calib_data_path: Optional[Union[str, List[str]]] = None,
use_random_calib: Optional[bool] = None,
config_preset: Optional[str] = None,
compile_config: Optional[CompileConfig | str] = None,
config_save_path: Optional[str] = None,
*,
progress_callback: Callable[[int, str], None],
) -> None
Compile/quantize entry point with progress callbacks.
target_device is required and has no config fallback — callers (e.g. the CLI) validate it via validate_target_device before invoking, and the wrapper forwards it unconditionally (QC-228).
Like :func:mblt_compile_with_callback, only explicitly-set values are forwarded, so :func:mxq_compile's UNSET defaults (and the merged [CompileConfig](qbcompiler.configs.models.md#compileconfig) behind them) still apply to whatever the caller omits. config_save_path passes straight through, and the resolved config is written there before compilation starts; this is what backs the CLI's --config-save-path on quantize / compile.