Skip to content

qbcompiler.frontend

Attributes

Name Description
logger

logger

logger = get_logger(__name__)

Model_Dict

class Model_Dict(LegacyCompiler)

Wrapper around the Mobilint compiler to support compilation and inference workflows.

Capabilities include:

  • Compilation of models into MXQ artifacts runnable on Mobilint NPUs.
  • Inference using the full-precision high-level compiled model on CPU or GPU.
  • Inference with the quantized model on CPU or GPU.

Methods

Name Description
__init__ Initialize the Mobilint compiler wrapper.
compile Compile the wrapped model into an MXQ artifact.

__init__

def __init__(
    self,
    model,
    backend="onnx",
    device="cpu",
    feed_dict=None,
    dynamic_axes=None,
    in_dformats=None,
    yolo_decode_include=False,
    exclude_first_subgraph=False,
    **kwargs,
)

Initialize the Mobilint compiler wrapper.

Parameters

  • model — string or model instance. Model path or in-memory model to compile.
  • backend — string. Framework identifier for the model (for example "onnx"). Defaults to "onnx".
  • device — string. Target device for inference ("cpu" or "gpu"). Defaults to "cpu".
  • feed_dict — dict. Example inputs for shape resolution.
  • dynamic_axes — dict. Marks model axes as dynamic.
  • in_dformats — dict. Describes input data formats.
  • yolo_decode_include — bool. Runs YOLO decode on NPU when True.
  • exclude_first_subgraph — bool. Applies only when CPU offloading is enabled: exclude the first subgraph from the final graph if it is unsupported.
  • kwargs — dict. Additional compiler arguments.

compile

def compile(
    self,
    save_subgraph_type: int = 0,
    output_subgraph_path="",
    compile_config: CompileConfig = None,
    **kwargs,
)

Compile the wrapped model into an MXQ artifact.

Parameters

  • save_subgraph_type — int. Controls optional MBLT exports: 0 disables; 1 saves graph structure; 2 saves structure and weights; 3 splits structure into multiple subgraphs; 4 splits structure and weights.
  • output_subgraph_path — string. Destination for exported .mblt when save_subgraph_type is 1–4.
  • compile_config — CompileConfig. Compile configuration object.
  • kwargs — dict. Additional arguments forwarded to the compiler.

Functions

Name Description
mxq_compile Compile a model into a Mobilint eXeCUtable (MXQ) package for execution on Mobilint NPUs.
mxq_compile_from_source Compile a raw framework model (ONNX / PyTorch / TensorFlow / TF-Lite) into an MXQ package.
mxq_compile_from_mblt Compile an existing Mobilint IR (.mblt) file into an MXQ package.
mblt_compile Export a model to the Mobilint .mblt format without producing an MXQ package.
mblt_compile_with_callback Compile-to-mblt entry point with progress callbacks.
mxq_compile_with_callback Compile/quantize entry point with progress callbacks.

mxq_compile

def mxq_compile(
    model,
    target_device: str,
    calib_data_path: Union[str, List[str]] = UNSET,
    save_subgraph_type: int = 0,
    output_subgraph_path="",
    save_path: Union[str, List[str]] = UNSET,
    backend="onnx",
    # -- ModelDict params --
    feed_dict=None,
    dynamic_axes=None,
    in_dformats=None,
    yolo_decode_include=False,
    exclude_first_subgraph=False,
    # -- CompileConfig-overridable (UNSET = defer to config) --
    device=UNSET,
    inference_scheme=UNSET,
    use_random_calib=UNSET,
    cpu_offload=UNSET,
    optimize_option=UNSET,
    buffer_mode=UNSET,
    input_shape_dict=UNSET,
    force_npu_input_reposition=UNSET,
    force_npu_output_reposition=UNSET,
    image_channels=UNSET,
    split_blocks=UNSET,
    split_parts=UNSET,
    # -- Config objects --
    config_preset: Optional[str] = None,
    compile_config: Optional[CompileConfig] = None,
    resource_management_config: Optional[ResourceManagementConfig] = None,
    calibration_config: Optional[CalibrationConfig] = None,
    bit_config: Optional[BitConfig] = None,
    llm_config: Optional[LlmConfig] = None,
    hessian_quant_config: Optional[HessianQuantConfig] = None,
    mod_config: Optional[ModConfig] = None,
    equivalent_transformation_config: Optional[EquivalentTransformationConfig] = None,
    search_weight_scale_config: Optional[SearchWeightScaleConfig] = None,
    save_sample_config: Optional[SaveSampleConfig] = None,
    uint8_input_config: Optional[Uint8InputConfig] = None,
    preprocessing_config: Optional[PreprocessingConfig] = None,
    layer_bias_correction=UNSET,
    layer_bias_correction_config: Optional[LayerBiasCorrectionConfig] = None,
    model_part: Optional[str] = None,
    model_part_options: Optional[dict] = None,
    config_save_path: Optional[str] = None,
    **kwargs,
)

Compile a model into a Mobilint eXeCUtable (MXQ) package for execution on Mobilint NPUs.

When no explicit value is provided for a parameter marked with UNSET, the default from CompileConfig is used. This allows a compile_config file or object to supply the value without being overridden by function-level defaults.

Configuration is resolved in priority order (highest to lowest):

  1. Explicitly passed function arguments
  2. Individual sub-config objects (calibration_config, llm_config, etc.)
  3. kwargs partial overrides (quantization_method, weight_dtype, etc.)
  4. compile_config (CompileConfig object or JSON/YAML file) or config_preset
  5. CompileConfig field defaults

Parameters

  • model — string or model instance. Model path. When using backend="onnx", this should be the path to an ONNX model file. When backend is "torchscript", provide a TorchScript module; when "torch", provide a standard PyTorch model. For backend="tf", pass the directory that contains the TensorFlow SavedModel graph and assets. For backend="tflite", pass the TF Lite model path.
  • calib_data_path — string or list of strings. Path(s) to the calibration dataset. Accepts either a text/json file that lists NumPy files or a directory that contains the pre-processed NumPy files.
  • save_subgraph_type — int. Controls optional MBLT subgraph exports: 0 disables exports; 1 saves only the graph structure; 2 saves graph structure plus weights; 3 saves the graph structure split into multiple subgraphs; 4 saves both structure and weights split into multiple subgraphs. Defaults to 0.
  • output_subgraph_path — string. Destination path for the exported .mblt file when save_subgraph_type is 1–4. The resulting file can be used for visualization. Defaults to "".
  • save_path — string or list of strings. Output MXQ filename(s). When omitted, defaults to "{model_name}.mxq" derived from the model path basename.
  • backend — string. Framework used to generate the Mobilint IR. Case-insensitive, so "ONNX" and "onnx" are the same backend. "onnx" and "torch" are parsed by the current parser; "tf" (also spelled "tensorflow"), "tflite" and "torchscript" go through the legacy parser. Any other value raises ValueError. Defaults to "onnx". When model is a path to an already-compiled .mblt file (the output of mblt_compile()), it is compiled directly with MXQ regardless of the backend value, and the file is validated beforehand; it must be a runnable artifact produced by mblt_compile() rather than a save_subgraph_type preview export.
  • target_device — string. Target NPU device for parser/compiler configuration (for example "aries-rb"). Required by execution paths that parse or quantize a model.
  • device — string. Compilation and inference device: "cpu" or "gpu". When omitted, uses CompileConfig default.
  • feed_dict — dict. Example input tensors for shape inference and inference validation.
  • dynamic_axes — dict. Marks model axes as dynamic. Keys are input names and values map dimension indices to aliases (for example {"input": {2: "seq_len"}}).
  • in_dformats — dict. Describes input tensor data formats.
  • inference_scheme — string. NPU inference scheme. One of "single", "multi", "global", "global4", or "global8". When omitted, uses CompileConfig default.
  • yolo_decode_include — bool. Determines whether YOLO decode runs on NPU. Defaults to False.
  • use_random_calib — bool. Generates random calibration data to validate model compilability. When omitted, uses CompileConfig default.
  • cpu_offload — bool. Enables CPU offloading during NPU inference. When omitted, uses CompileConfig default.
  • optimize_option — int. Compiler optimization strategy selector. When omitted, uses CompileConfig default.
  • buffer_mode — int. Buffer serialization mode: 0 uses a naive buffer, 1 uses an mmap-backed buffer. When omitted, uses CompileConfig default.
  • input_shape_dict — dict. Multi-shape compilation specification for STT/TTS models only (e.g., Conformer-CTC, MeloTTS). The dynamic axis is defined with respect to each input tensor's shape. During compilation, the same axis is varied across all model inputs using the provided values. Currently, only a single entry key ("multi_shape0") is supported. Example: {"multi_shape0": {"axis": 2, "values": [100, 200, 300]}}
  • force_npu_input_reposition — bool. Force input reposition operations to run on NPU instead of CPU. When omitted, uses CompileConfig default.
  • force_npu_output_reposition — bool. Force output reposition operations to run on NPU instead of CPU. When omitted, uses CompileConfig default.
  • image_channels — int. Number of image channels (0 for auto-detect). When omitted, uses CompileConfig default.
  • split_blocks — list of int. Multi-MXQ split points by transformer block index. Only supported for LLM models. When omitted, uses CompileConfig default.
  • split_parts — int. Evenly split transformer blocks into N MXQ parts. Only supported for LLM models. When omitted, uses CompileConfig default.
  • layer_bias_correction — bool. Enables calibration-derived layer bias correction. When omitted, uses layer_bias_correction_config or the CompileConfig value.
  • exclude_first_subgraph — bool. Applies only when CPU offloading is enabled: exclude the first subgraph from the final graph if it is unsupported. Defaults to False.
  • config_preset — string. Name of a built-in configuration preset. When provided, loads the preset via CompileConfig.from_preset(). Defaults to None (no preset). Available presets: - "classification": Image classification models (ResNet, EfficientNet, ViT, etc.) - "detection": Object detection models (YOLO, SSD, DETR, etc.) - "classification_torchvision": Torchvision classification models with standard preprocessing - "yolo_640": YOLO detection models with 640x640 letterbox preprocessing - "yolo_1280": YOLO detection models with 1280x1280 letterbox preprocessing - "llm": Large Language Models (LLaMA, Qwen, Gemma, etc.) - "llm_fast": LLM with faster compilation (less accuracy optimization) - "vision_transformer": Vision Transformer models (ViT, DeiT, Swin, etc.) - "multimodal": Multimodal models (CLIP, BLIP, LLaVA, etc.)
  • compile_config — CompileConfig or string. CompileConfig object or path to JSON/YAML configuration file containing all compilation settings (resourceManagement, calibration, bit, hessianQuant, mod, llm, etc.).
  • config_save_path — string. When provided, the fully resolved configuration - the normalized CompileConfig after every layer of the precedence order above has been applied, including all sub-configurations - is written to this path before compilation starts. The format follows the path suffix: ".yaml"/".yml" produce YAML, anything else produces JSON. Parent directories are created as needed. The saved file can be fed back as compile_config to reproduce the same compilation. Defaults to None (no config file is written).
  • resource_management_config — ResourceManagementConfig. Resource management configuration object.
  • calibration_config — CalibrationConfig. Calibration configuration object.
  • bit_config — BitConfig. Bit configuration object for quantization precision settings.
  • llm_config — LlmConfig. LLM configuration object.
  • hessian_quant_config — HessianQuantConfig. HessianQuant (Hessian-based Quantization) configuration object.
  • layer_bias_correction_config — LayerBiasCorrectionConfig. Calibration-derived layer bias correction settings.
  • model_part — string. Selects a specific part of a torch model to parse ("vision", "language", "encoder", ...). Typically used to support models with complex architectures (e.g. Qwen3-VL) by compiling one part at a time. Parts a model declares: parser.patcher.parts.available_parts(model). A model declaring exactly one part resolves it from None; one declaring several requires a name.
  • model_part_options — dict. Extra arguments for the named part (e.g. {"mel_frames": 100} for Qwen3-ASR audio).
  • mod_config — ModConfig. MOD (Metric-based Optimization and Distillation) configuration object.
  • equivalent_transformation_config — EquivalentTransformationConfig. Configuration for equivalent transformations like SmoothQuant.
  • search_weight_scale_config — SearchWeightScaleConfig. Configuration for weight scale search.
  • save_sample_config — SaveSampleConfig. Configuration for sample data generation and saving.
  • uint8_input_config — Uint8InputConfig. Configuration for uint8 input handling.
  • preprocessing_config — PreprocessingConfig. Preprocessing pipeline configuration.
  • kwargs — dict. Additional compiler arguments. Supports partial config overrides such as quantization_method, quantization_mode, percentile, weight_dtype, ram_usage, max_sequence_length, etc. Also accepts deprecated parameters (quantization_config, advanced_quantization_config, input_process_config, save_sample, sample_dtype) which will emit DeprecationWarning.

Returns

None.

Using configuration

There are three ways to configure quantization settings:

  1. Load all settings from a JSON/YAML config file:

    from qbcompiler import mxq_compile
    
    mxq_compile(
        model="path/to/model.onnx",
        target_device="aries-rb",
        calib_data_path="path/to/calib",
        compile_config="path/to/config.json",  # or config.yaml
        device="gpu",
    )
    
  2. Pass individual sub-config objects:

    from qbcompiler import mxq_compile
    from qbcompiler.configs import (
        ResourceManagementConfig,
        CalibrationConfig,
        BitConfig,
        HessianQuantConfig,
        ModConfig,
        LlmConfig,
    )
    
    resource_mgmt = ResourceManagementConfig(weight_dtype="float32")
    calib_cfg = CalibrationConfig(method=1, mode=1)
    bit_cfg = BitConfig(...)
    hessian_quant_cfg = HessianQuantConfig(apply=True)
    mod_cfg = ModConfig(apply=False)
    llm_cfg = LlmConfig(apply=True)
    
    mxq_compile(
        model="path/to/model.onnx",
        target_device="aries-rb",
        calib_data_path="path/to/calib",
        resource_management_config=resource_mgmt,
        calibration_config=calib_cfg,
        bit_config=bit_cfg,
        hessian_quant_config=hessian_quant_cfg,
        mod_config=mod_cfg,
        llm_config=llm_cfg,
        device="gpu",
    )
    
  3. Automatically applied partial configuration overrides:

    from qbcompiler import mxq_compile
    
    mxq_compile(
        model="path/to/model.onnx",
        target_device="aries-rb",
        calib_data_path="path/to/calib",
        quantization_method=1, # per channel quantization
        quantization_mode=1, # max percentile quantization
        percentile=0.999, # percentile value for max percentile quantization
        quantization_output=0, # per layer quantization for the output layer
        device="gpu",
    )
    

Please refer to the mxq_compile function and the quantization configuration section for the meaning of the quantization-related numeric values.

Compiling models with custom inputs (ONNX/Torch/TensorFlow)

Provide NumPy inputs whenever the model omits shape information so that qbcompiler can infer unknown dimensions and data formats.

from qbcompiler import mxq_compile
import numpy as np

example_input = {
    "input_node_name_1": np.random.randn(1, 3, 224, 224).astype(np.float32),
    "input_node_name_2": np.random.randn(1, 8, 224, 224).astype(np.float32),
    "input_node_name_3": np.random.randn(1, 4, 56, 56).astype(np.float32),
}

in_dformats = {
    "input_node_name_1": "NCHW",
    "input_node_name_2": "NCHW",
    "input_node_name_3": "NCHW",
}

onnx_model_path = "path/to/your/model.onnx"
mxq_compile(
    model=onnx_model_path,
    target_device="aries-rb",
    feed_dict=example_input,
    in_dformats=in_dformats,
    backend="onnx",
    compile_config="path/to/config.json",
)

mxq_compile_from_source

def mxq_compile_from_source(
    model,
    target_device: str,
    calib_data_path: Union[str, List[str]] = UNSET,
    save_subgraph_type: int = 0,
    output_subgraph_path="",
    save_path: Union[str, List[str]] = UNSET,
    backend="onnx",
    # -- ModelDict params --
    feed_dict=None,
    dynamic_axes=None,
    in_dformats=None,
    yolo_decode_include=False,
    exclude_first_subgraph=False,
    # -- CompileConfig-overridable (UNSET = defer to config) --
    device=UNSET,
    inference_scheme=UNSET,
    use_random_calib=UNSET,
    cpu_offload=UNSET,
    optimize_option=UNSET,
    buffer_mode=UNSET,
    input_shape_dict=UNSET,
    force_npu_input_reposition=UNSET,
    force_npu_output_reposition=UNSET,
    image_channels=UNSET,
    split_blocks=UNSET,
    split_parts=UNSET,
    # -- Config objects --
    config_preset: Optional[str] = None,
    compile_config: Optional[CompileConfig] = None,
    resource_management_config: Optional[ResourceManagementConfig] = None,
    calibration_config: Optional[CalibrationConfig] = None,
    bit_config: Optional[BitConfig] = None,
    llm_config: Optional[LlmConfig] = None,
    hessian_quant_config: Optional[HessianQuantConfig] = None,
    mod_config: Optional[ModConfig] = None,
    equivalent_transformation_config: Optional[EquivalentTransformationConfig] = None,
    search_weight_scale_config: Optional[SearchWeightScaleConfig] = None,
    save_sample_config: Optional[SaveSampleConfig] = None,
    uint8_input_config: Optional[Uint8InputConfig] = None,
    preprocessing_config: Optional[PreprocessingConfig] = None,
    layer_bias_correction=UNSET,
    layer_bias_correction_config: Optional[LayerBiasCorrectionConfig] = None,
    model_part: Optional[str] = None,
    model_part_options: Optional[dict] = None,
    config_save_path: Optional[str] = None,
    **kwargs,
)

Compile a raw framework model (ONNX / PyTorch / TensorFlow / TF-Lite) into an MXQ package.

This is the raw-model half of mxq_compile(): it parses the model into Mobilint IR and then quantizes and compiles it in one pass. The parameters, their defaults, and the configuration precedence rules are identical to mxq_compile() — see that function for the full reference and usage examples.

Passing the path of an existing .mblt file here raises ValueError; use mxq_compile_from_mblt() for that input instead. mxq_compile() routes between the two automatically.

Parameters

  • model — string or model instance. Raw model or path to one. Must not be an existing .mblt file.
  • target_device — string. Target NPU device (for example "aries-rb").

Returns

None.

mxq_compile_from_mblt

def mxq_compile_from_mblt(
    mblt: str,
    target_device: str,
    calib_data_path: Union[str, List[str]] = UNSET,
    save_path: Union[str, List[str]] = UNSET,
    backend="onnx",
    device=UNSET,
    inference_scheme=UNSET,
    use_random_calib=UNSET,
    cpu_offload=UNSET,
    optimize_option=UNSET,
    buffer_mode=UNSET,
    input_shape_dict=UNSET,
    force_npu_input_reposition=UNSET,
    force_npu_output_reposition=UNSET,
    image_channels=UNSET,
    split_blocks=UNSET,
    split_parts=UNSET,
    config_preset: Optional[str] = None,
    compile_config: Optional[CompileConfig] = None,
    resource_management_config: Optional[ResourceManagementConfig] = None,
    calibration_config: Optional[CalibrationConfig] = None,
    bit_config: Optional[BitConfig] = None,
    llm_config: Optional[LlmConfig] = None,
    hessian_quant_config: Optional[HessianQuantConfig] = None,
    mod_config: Optional[ModConfig] = None,
    equivalent_transformation_config: Optional[EquivalentTransformationConfig] = None,
    search_weight_scale_config: Optional[SearchWeightScaleConfig] = None,
    save_sample_config: Optional[SaveSampleConfig] = None,
    uint8_input_config: Optional[Uint8InputConfig] = None,
    preprocessing_config: Optional[PreprocessingConfig] = None,
    layer_bias_correction=UNSET,
    layer_bias_correction_config: Optional[LayerBiasCorrectionConfig] = None,
    config_save_path: Optional[str] = None,
    **kwargs,
)

Compile an existing Mobilint IR (.mblt) file into an MXQ package.

This is the pre-parsed half of mxq_compile(): parsing already happened (via mblt_compile()), so this function only quantizes and compiles. Quantization parameters, their defaults, and the configuration precedence rules are identical to mxq_compile() — see that function for the full reference and usage examples.

Parser-only arguments (save_subgraph_type, output_subgraph_path, feed_dict, dynamic_axes, in_dformats, yolo_decode_include, exclude_first_subgraph) are absent because the graph is already parsed. mxq_compile() accepts them for backward compatibility and drops them when routing here.

Parameters

  • mblt — string. Path to a runnable .mblt produced by mblt_compile(). The file is validated up front; a save_subgraph_type preview export is rejected.
  • target_device — string. Target NPU device (for example "aries-rb").
  • backend — string. Retained for backward compatibility and ignored: the graph is already parsed, so no parser backend is selected.

Returns

None.

Throws

  • ValueError — mblt is not an existing .mblt file, or it is not a runnable artifact for the requested cpu_offload setting.

mblt_compile

def mblt_compile(
    model: str | Any,
    mblt_save_path: str,
    target_device: str,
    backend="onnx",
    device="cpu",
    feed_dict=None,
    dynamic_axes=None,
    in_dformats=None,
    yolo_decode_include=False,
    cpu_offload=False,
    exclude_first_subgraph=False,
    model_part: Optional[str] = None,
    model_part_options: Optional[dict] = None,
    **kwargs,
)

Export a model to the Mobilint .mblt format without producing an MXQ package.

Parameters

  • model — string or model instance. Source model or path to compile.
  • mblt_save_path — string. Output path for the .mblt artifact.
  • backend — string. Framework identifier, case-insensitive. "onnx" and "torch" are parsed by the current parser; "tf" (also spelled "tensorflow"), "tflite" and "torchscript" go through the legacy parser. Any other value raises ValueError. Defaults to "onnx".
  • device — string. Compilation device ("cpu" or "gpu"). Defaults to "cpu".
  • feed_dict — dict. Example inputs used for shape inference.
  • dynamic_axes — dict. Declares dynamic axes per input name.
  • in_dformats — dict. Input dataformat metadata.
  • yolo_decode_include — bool. Runs YOLO decode on NPU when True.
  • cpu_offload — bool. Enables CPU offloading for unsupported groups.
  • exclude_first_subgraph — bool. Applies only when CPU offloading is enabled: exclude the first subgraph from the final graph if it is unsupported.
  • model_part — string. Selects a specific part of a torch model to parse ("vision", "language", "encoder", ...). Typically used to support models with complex architectures (e.g. Qwen3-VL) by compiling one part at a time. Parts a model declares: parser.patcher.parts.available_parts(model). A model declaring exactly one part resolves it from None; one declaring several requires a name.
  • model_part_options — dict. Extra arguments for the named part (e.g. {"mel_frames": 100} for Qwen3-ASR audio).
  • kwargs — dict. Additional arguments forwarded to the compiler.

Returns

None.

mblt_compile_with_callback

def mblt_compile_with_callback(
    model: str,
    mblt_save_path: str,
    target_device: str,
    backend: str = "onnx",
    device: Optional[str] = None,
    cpu_offload: Optional[bool] = None,
    config_preset: Optional[str] = None,
    compile_config: Optional[CompileConfig | str] = None,
    compilation_mode: Optional[str] = None,
    *,
    progress_callback: Callable[[int, str], None],
) -> None

Compile-to-mblt entry point with progress callbacks.

Only forwards explicitly-set values so the UNSET defaults (and the merged [CompileConfig](qbcompiler.configs.models.md#compileconfig) behind them) apply when the caller omits a flag -- which is why this goes through _mblt_compile_dispatch rather than :func:mblt_compile, whose own device / cpu_offload defaults would override the config. target_device is required and has no config fallback — callers (e.g. the CLI) validate it via validate_target_device before invoking (QC-228).

mxq_compile_with_callback

def mxq_compile_with_callback(
    model: str,
    target_device: str,
    save_path: str,
    backend: str = "onnx",
    device: Optional[str] = None,
    calib_data_path: Optional[Union[str, List[str]]] = None,
    use_random_calib: Optional[bool] = None,
    config_preset: Optional[str] = None,
    compile_config: Optional[CompileConfig | str] = None,
    config_save_path: Optional[str] = None,
    *,
    progress_callback: Callable[[int, str], None],
) -> None

Compile/quantize entry point with progress callbacks.

target_device is required and has no config fallback — callers (e.g. the CLI) validate it via validate_target_device before invoking, and the wrapper forwards it unconditionally (QC-228).

Like :func:mblt_compile_with_callback, only explicitly-set values are forwarded, so :func:mxq_compile's UNSET defaults (and the merged [CompileConfig](qbcompiler.configs.models.md#compileconfig) behind them) still apply to whatever the caller omits. config_save_path passes straight through, and the resolved config is written there before compilation starts; this is what backs the CLI's --config-save-path on quantize / compile.