qbcompiler.configs.models¶
Auto-generated Pydantic models from config_schema.yaml.
Attributes¶
| Name | Description |
|---|---|
SCHEMA_VERSION |
SCHEMA_VERSION¶
Uint8InputConfig¶
Configuration for uint8 input handling
Defines whether inputs should be treated as uint8 and which specific inputs to apply this to.
Parameters
apply— bool. If true, treat specified inputs as uint8inputs— List[str]. List of input names to treat as uint8. If empty and apply is true, applies to all inputsdivision_factor— float. Division factor for uint8 to float conversion (e.g., 255.0 for [0,1], 127.5 for [0,2])
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
inputs |
|
division_factor |
model_config¶
apply¶
inputs¶
inputs: List[str] = Field(
default=[],
description="List of input names to treat as uint8. If empty and apply is true, applies to all inputs",
)
division_factor¶
division_factor: float = Field(
default=255.0,
alias="divisionFactor",
description="Division factor for uint8 to float conversion (e.g., 255.0 for [0,1], 127.5 for [0,2])",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
PreprocessingConfig¶
Configuration for input preprocessing pipeline
Defines preprocessing operations to be applied to model inputs, including operations like resize, normalize, color conversion, etc.
Parameters
apply— bool. If true, apply preprocessing pipelineauto_convert_format— bool. If true, automatically convert input formatpipeline— List[Dict[str, Any]]. List of preprocessing operations to apply globallyinput_configs— Dict[str, Any]. Per-input preprocessing configurations. Keys are input names
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
auto_convert_format |
|
pipeline |
|
input_configs |
model_config¶
apply¶
auto_convert_format¶
auto_convert_format: bool = Field(
default=False,
alias="autoConvertFormat",
description="If true, automatically convert input format",
)
pipeline¶
pipeline: Any = Field(
default=[], description="List of preprocessing operations to apply globally"
)
input_configs¶
input_configs: Any = Field(
default={},
alias="inputConfigs",
description="Per-input preprocessing configurations. Keys are input names",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
ResourceManagementConfig¶
Configuration for resource management during model compilation
Controls GPU and memory management settings during the compilation process.
Parameters
weight_dtype— str. Weight data type for calibration (e.g., 'float32', 'float16')use_gpu_only_for_calibration— bool. If True, use GPU only during the calibration phaseweight_memory— WeightMemory. Weight memory management configuration
Attributes¶
| Name | Description |
|---|---|
model_config |
|
weight_dtype |
|
use_gpu_only_for_calibration |
|
weight_memory |
model_config¶
weight_dtype¶
weight_dtype: str = Field(
default="float32",
alias="weightDtype",
description="Weight data type for calibration (e.g., 'float32', 'float16')",
)
use_gpu_only_for_calibration¶
use_gpu_only_for_calibration: bool = Field(
default=True,
alias="useGPUOnlyForCalibration",
description="If True, use GPU only during the calibration phase",
)
weight_memory¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
ResourceManagementConfig.WeightMemory¶
Weight memory management configuration
Parameters
method— int. Weight memory management method index:
0: DeleteFloat - Delete float weights after quantization.
1: SaveFloat - Save float weights to disk.
2: MoveFloat - Move float weights to CPU.
3: KeepFloat - Keep float weights in memory.
4: KeepAll - Keep all weights in memory.
Attributes¶
| Name | Description |
|---|---|
method_list |
|
method |
method_list¶
method_list: List[str] = Field(
default=["DeleteFloat", "SaveFloat", "MoveFloat", "KeepFloat", "KeepAll"],
alias="methodList",
)
method¶
CalibrationConfig¶
Configuration for calibration during quantization
Defines calibration and quantization parameterization used to derive activation/weight scales and related statistics during quantized compilation.
Parameters
method— int. Calibration method index:
0: WChALayer - Weight per-channel, Activation per-layer.
1: WChAMulti - Weight per-channel, Activation multi-layer.
2: WChALayerZeropoint - Weight per-channel, Activation per-layer with zeropoint.
3: WChAMultiZeropoint - Weight per-channel, Activation multi-layer with zeropoint.output— int. Output quantization type index:
0: Layer - Per-layer quantization.
1: Ch - Per-channel quantization.
2: Sigmoid - Sigmoid-based quantization.mode— int. Quantization mode index:
0: Max - Maximum value calibration.
1: MaxPercentile - Maximum percentile calibration.
2: Histogram - Histogram-based calibration.act_scale_min— float. Minimum allowed activation scale (lower bound clamp)act16_scale_min— float. Minimum 16-bit activation scale (actScaleMin / 256)weight_scale_min— float. Minimum allowed weight scale (lower bound clamp)weight16_scale_min— float. Minimum 16-bit weight scale (weightScaleMin / 256)min_clip_ratio— float. Minimum clip ratio constraint applied during calibrationmax_calib_data_size— int. Maximum number of calibration samples kept after loading or generationmax_sample_size_for_quant_scheme— int. Maximum number of calibration samples used per quant scheme stagemax_percentile— MaxPercentile. MaxPercentile mode configurationfast_dist— FastDist. Fast distribution calibration configurationhistogram— Histogram. Histogram-based calibration configurationlayer_overrides— LayerOverrides. Layer-specific override settings for calibrationstatistics— Statistics. Statistics save/load configuration with percentile selectiongroup_lut— GroupLut. LUT grouping algorithm configurationoptimize_lut— OptimizeLut. LUT optimization configuration
Attributes¶
model_config¶
method_list¶
method_list: List[str] = Field(
default=["WChALayer", "WChAMulti", "WChALayerZeropoint", "WChAMultiZeropoint"],
alias="methodList",
)
method¶
output_list¶
output¶
mode_list¶
mode¶
act_scale_min¶
act_scale_min: float = Field(
default=0.0005,
alias="actScaleMin",
description="Minimum allowed activation scale (lower bound clamp)",
ge=0,
le=1,
)
act16_scale_min¶
act16_scale_min: float = Field(
default=1.953125e-06,
alias="act16ScaleMin",
description="Minimum 16-bit activation scale (actScaleMin / 256)",
)
weight_scale_min¶
weight_scale_min: float = Field(
default=1e-06,
alias="weightScaleMin",
description="Minimum allowed weight scale (lower bound clamp)",
ge=0,
le=1,
)
weight16_scale_min¶
weight16_scale_min: float = Field(
default=3.90625e-09,
alias="weight16ScaleMin",
description="Minimum 16-bit weight scale (weightScaleMin / 256)",
)
min_clip_ratio¶
min_clip_ratio: float = Field(
default=-1,
alias="minClipRatio",
description="Minimum clip ratio constraint applied during calibration",
ge=-1,
le=1,
)
max_calib_data_size¶
max_calib_data_size: int = Field(
default=-1,
alias="maxCalibDataSize",
description="Maximum number of calibration samples kept after loading or generation",
ge=-1,
)
max_sample_size_for_quant_scheme¶
max_sample_size_for_quant_scheme: int = Field(
default=16,
alias="maxSampleSizeForQuantScheme",
description="Maximum number of calibration samples used per quant scheme stage",
ge=1,
)
max_percentile¶
fast_dist¶
histogram¶
layer_overrides¶
statistics¶
group_lut¶
optimize_lut¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
CalibrationConfig.MaxPercentile¶
Attributes¶
| Name | Description |
|---|---|
model_config |
MaxPercentile mode configuration |
percentile |
|
topk_ratio |
|
max_each |
|
max_total |
|
per_ch_divisor |
model_config¶
MaxPercentile mode configuration
Parameters
percentile— float. Percentile value for maxPercentile modetopk_ratio— float. Top-k ratio used in maxPercentile modemax_each— int. Maximum number of samples processed per iterationmax_total— int. Total maximum number of samplesper_ch_divisor— int. Divisor for per-channel buffer capacity (bufferCap = max(maxTotal / perChDivisor, maxEach))
percentile¶
topk_ratio¶
topk_ratio: float = Field(
default=0.01,
alias="topKRatio",
description="Top-k ratio used in maxPercentile mode",
)
max_each¶
max_each: int = Field(
default=128,
alias="maxEach",
description="Maximum number of samples processed per iteration",
)
max_total¶
max_total: int = Field(
default=65536,
alias="maxTotal",
description="Total maximum number of samples",
)
per_ch_divisor¶
per_ch_divisor: int = Field(
default=16,
alias="perChDivisor",
description="Divisor for per-channel buffer capacity (bufferCap = max(maxTotal / perChDivisor, maxEach))",
ge=1,
)
CalibrationConfig.FastDist¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Fast distribution calibration configuration |
size_cali |
|
kernel_size |
|
stack_size |
model_config¶
Fast distribution calibration configuration
Parameters
size_cali— int.kernel_size— int.stack_size— int.
size_cali¶
kernel_size¶
stack_size¶
CalibrationConfig.Histogram¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Histogram-based calibration configuration |
search_type_list |
|
search_type |
|
percentile |
|
use_gpu |
|
num_bins |
|
num_samples |
|
buffer_size |
|
min_bin_width |
|
search_percentile_min |
|
search_percentile_max |
|
num_search |
model_config¶
Histogram-based calibration configuration
Parameters
search_type— int. Search type for histogram calibration:
0: Percentile.
1: MSE.
2: KL.percentile— float. Percentile value for histogram calibrationuse_gpu— bool. Use GPU for histogram computationnum_bins— int. Number of bins for histogramnum_samples— int. Number of samples for histogram calibrationbuffer_size— int. Buffer size for histogram computation (-1 for auto)min_bin_width— float. Minimum bin width for histogramsearch_percentile_min— float. Minimum search percentilesearch_percentile_max— float. Maximum search percentilenum_search— int. Number of search iterations
search_type_list¶
search_type¶
percentile¶
percentile: float = Field(
default=0.9999, description="Percentile value for histogram calibration"
)
use_gpu¶
use_gpu: bool = Field(
default=True,
alias="useGPU",
description="Use GPU for histogram computation",
)
num_bins¶
num_samples¶
num_samples: int = Field(
default=128,
alias="numSamples",
description="Number of samples for histogram calibration",
)
buffer_size¶
buffer_size: int = Field(
default=-1,
alias="bufferSize",
description="Buffer size for histogram computation (-1 for auto)",
)
min_bin_width¶
min_bin_width: float = Field(
default=1e-06,
alias="minBinWidth",
description="Minimum bin width for histogram",
)
search_percentile_min¶
search_percentile_min: float = Field(
default=0.9999,
alias="searchPercentileMin",
description="Minimum search percentile",
)
search_percentile_max¶
search_percentile_max: float = Field(
default=1.0,
alias="searchPercentileMax",
description="Maximum search percentile",
)
num_search¶
num_search: int = Field(
default=128, alias="numSearch", description="Number of search iterations"
)
CalibrationConfig.LayerOverrides¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Layer-specific override settings for calibration |
act_scale_min |
|
percentile |
|
method |
model_config¶
Layer-specific override settings for calibration
Parameters
act_scale_min— dict. Per-layer activation scale minimum overrides. Keys are actScaleMin values (e.g. '0.0005'), values are lists of layer names to apply that override. (e.g. {'0.0005' : ['layer1', 'layer2']})percentile— dict. Per-layer maxPercentile percentile overrides. Keys are layer names, values are percentile floats. (e.g. {'/model/layer0/conv': 0.999, '/model/layer5/attn': 0.99})method— dict. Per-layer calibration method overrides. Keys are layer names, values are method ints (0=WChALayer, 1=WChAMulti, 2=WChALayerZeropoint, 3=WChAMultiZeropoint). (e.g. {'/model/layer0/conv': 1, '/model/layer5/attn': 3})
act_scale_min¶
act_scale_min: Any = Field(
default={},
alias="actScaleMin",
description="Per-layer activation scale minimum overrides. Keys are actScaleMin values (e.g. '0.0005'), values are lists of layer names to apply that override. (e.g. {'0.0005' : ['layer1', 'layer2']})",
)
percentile¶
percentile: Any = Field(
default={},
description="Per-layer maxPercentile percentile overrides. Keys are layer names, values are percentile floats. (e.g. {'/model/layer0/conv': 0.999, '/model/layer5/attn': 0.99})",
)
method¶
method: Any = Field(
default={},
description="Per-layer calibration method overrides. Keys are layer names, values are method ints (0=WChALayer, 1=WChAMulti, 2=WChALayerZeropoint, 3=WChAMultiZeropoint). (e.g. {'/model/layer0/conv': 1, '/model/layer5/attn': 3})",
)
CalibrationConfig.Statistics¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Statistics save/load configuration with percentile selection |
apply |
|
save_path |
|
load_path |
|
percentiles |
|
percentile_index |
model_config¶
Statistics save/load configuration with percentile selection
Parameters
apply— bool. Enable statistics save/loadsave_path— str. Path to save statistics. If empty, not savedload_path— str. Path to load statistics. If empty, not loadedpercentiles— List[float]. List of percentile candidatespercentile_index— int. Index into percentiles list to select active percentile
apply¶
save_path¶
save_path: str = Field(
default="",
alias="savePath",
description="Path to save statistics. If empty, not saved",
)
load_path¶
load_path: str = Field(
default="",
alias="loadPath",
description="Path to load statistics. If empty, not loaded",
)
percentiles¶
percentiles: List[float] = Field(
default=[0.9999, 0.999, 0.99, 0.9],
description="List of percentile candidates",
)
percentile_index¶
percentile_index: int = Field(
default=0,
alias="percentileIndex",
description="Index into percentiles list to select active percentile",
)
CalibrationConfig.GroupLut¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LUT grouping algorithm configuration |
irls_iter |
|
dro_eps |
|
cover_floor |
model_config¶
LUT grouping algorithm configuration
Parameters
irls_iter— int. Number of IRLS iterations for optimal scale computation (1 = single WLS)dro_eps— float. DRO worst-point coefficient; larger = more conservative clipping (0 = disable)cover_floor— float. LUTAM cluster scale lower bound as fraction of coverage
irls_iter¶
irls_iter: int = Field(
default=3,
alias="irlsIter",
description="Number of IRLS iterations for optimal scale computation (1 = single WLS)",
ge=1,
)
dro_eps¶
dro_eps: float = Field(
default=0,
alias="droEps",
description="DRO worst-point coefficient; larger = more conservative clipping (0 = disable)",
ge=0,
)
cover_floor¶
cover_floor: float = Field(
default=0.85,
alias="coverFloor",
description="LUTAM cluster scale lower bound as fraction of coverage",
ge=0,
le=1,
)
CalibrationConfig.OptimizeLut¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LUT optimization configuration |
optimization_level |
model_config¶
LUT optimization configuration
Parameters
optimization_level— int. Optimizer search depth (0 = surrogate-only fast, 1 = surrogate + true-objective refinement)
optimization_level¶
optimization_level: int = Field(
default=1,
alias="optimizationLevel",
description="Optimizer search depth (0 = surrogate-only fast, 1 = surrogate + true-objective refinement)",
ge=0,
le=1,
)
BitConfig¶
Configuration for bit precision
Defines bit-width parameterization for activations and weights used in mixed-precision quantization (e.g., attention and FFN components).
Parameters
transformer— Transformer. Transformer-specific bit-width configurationsave_info— SaveInfo. Bit allocation save/load configurationlayer_overrides— LayerOverrides. Layer-specific bit-width override settings
Attributes¶
| Name | Description |
|---|---|
model_config |
|
transformer |
|
save_info |
|
layer_overrides |
model_config¶
transformer¶
save_info¶
layer_overrides¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
BitConfig.Transformer¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Transformer-specific bit-width configuration |
activation |
|
weight |
|
mixed_precision |
model_config¶
Transformer-specific bit-width configuration
Parameters
activation— Activation. Activation bit-widths for transformer componentsweight— Weight. Weight bit-widths for transformer componentsmixed_precision— MixedPrecision. Mixed precision configuration
activation¶
weight¶
mixed_precision¶
BitConfig.Transformer.Activation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Activation bit-widths for transformer components |
query |
|
key |
|
value |
|
output |
|
head |
|
router |
|
ffn |
model_config¶
Activation bit-widths for transformer components
Parameters
query— int. Query activation bit-widthkey— int. Key activation bit-widthvalue— int. Value activation bit-widthoutput— int. Output activation bit-widthhead— int. Head activation bit-widthrouter— int. MoE router gate activation bit-widthffn— Ffn. FFN activation bit-widths (int shorthand sets all sublayers)
query¶
key¶
value¶
output¶
head¶
router¶
ffn¶
BitConfig.Transformer.Activation.Ffn¶
FFN activation bit-widths (int shorthand sets all sublayers)
Parameters
up— int. FFN up-projection activation bit-widthgate— int. FFN gate (SwiGLU) activation bit-widthdown— int. FFN down-projection activation bit-width
Attributes¶
| Name | Description |
|---|---|
up |
|
gate |
|
down |
up¶
gate¶
down¶
Methods¶
| Name | Description |
|---|---|
expand_int_shorthand |
expand_int_shorthand¶
BitConfig.Transformer.Weight¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Weight bit-widths for transformer components |
query |
|
key |
|
value |
|
output |
|
head |
|
router |
|
ffn |
model_config¶
Weight bit-widths for transformer components
Parameters
query— int. Query weight bit-widthkey— int. Key weight bit-widthvalue— int. Value weight bit-widthoutput— int. Output weight bit-widthhead— int. Head weight bit-widthrouter— int. MoE router gate weight bit-widthffn— Ffn. FFN weight bit-widths (int shorthand sets all sublayers)
query¶
key¶
value¶
output¶
head¶
router¶
ffn¶
BitConfig.Transformer.Weight.Ffn¶
FFN weight bit-widths (int shorthand sets all sublayers)
Parameters
up— int. FFN up-projection weight bit-widthgate— int. FFN gate (SwiGLU) weight bit-widthdown— int. FFN down-projection weight bit-width
Attributes¶
| Name | Description |
|---|---|
up |
|
gate |
|
down |
up¶
gate¶
down¶
Methods¶
| Name | Description |
|---|---|
expand_int_shorthand |
expand_int_shorthand¶
BitConfig.Transformer.MixedPrecision¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Mixed precision configuration |
weight |
|
activation |
model_config¶
Mixed precision configuration
Parameters
weight— Weight. Mixed precision configuration for weightsactivation— Activation. Per-layer activation mixed precision configuration
weight¶
activation¶
BitConfig.Transformer.MixedPrecision.Weight¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Mixed precision configuration for weights |
apply |
|
type_wise |
|
prune |
|
bit_2 |
|
bit_4 |
|
bit_8 |
|
importance_threshold_low |
|
importance_threshold_high |
model_config¶
Mixed precision configuration for weights
Parameters
apply— bool. If true, apply mixed-precision according to the specified bit-widthstype_wise— bool. Apply type-wise mixed precisionprune— float. Pruning ratiobit_2— float. Ratio of 2-bit quantizationbit_4— float. Ratio of 4-bit quantizationbit_8— float. Ratio of 8-bit quantizationimportance_threshold_low— float. Low importance thresholdimportance_threshold_high— float. High importance threshold
apply¶
apply: bool = Field(
default=False,
description="If true, apply mixed-precision according to the specified bit-widths",
)
type_wise¶
type_wise: bool = Field(
default=True,
alias="typeWise",
description="Apply type-wise mixed precision",
)
prune¶
bit_2¶
bit_4¶
bit_8¶
importance_threshold_low¶
importance_threshold_low: float = Field(
default=-1,
alias="importanceThreshold_low",
description="Low importance threshold",
)
importance_threshold_high¶
importance_threshold_high: float = Field(
default=-1,
alias="importanceThreshold_high",
description="High importance threshold",
)
BitConfig.Transformer.MixedPrecision.Activation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Per-layer activation mixed precision configuration |
apply |
|
ratio_16bit |
|
importance_threshold |
|
search_range |
model_config¶
Per-layer activation mixed precision configuration
Parameters
apply— bool. If true, apply per-layer activation mixed precisionratio_16bit— float. Ratio of layers assigned 16-bit (used when importanceThreshold < 0)importance_threshold— float. Normalized importance threshold for 16-bit assignment (negative = use ratio_16bit)search_range— int. Target layer range: -1=all layers, N=first N layers
apply¶
apply: bool = Field(
default=False,
description="If true, apply per-layer activation mixed precision",
)
ratio_16bit¶
ratio_16bit: float = Field(
default=0.45,
alias="ratio16Bit",
description="Ratio of layers assigned 16-bit (used when importanceThreshold < 0)",
)
importance_threshold¶
importance_threshold: float = Field(
default=-1,
alias="importanceThreshold",
description="Normalized importance threshold for 16-bit assignment (negative = use ratio_16bit)",
)
search_range¶
search_range: int = Field(
default=-1,
alias="searchRange",
description="Target layer range: -1=all layers, N=first N layers",
)
BitConfig.SaveInfo¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Bit allocation save/load configuration |
save_path |
|
load_path |
model_config¶
Bit allocation save/load configuration
Parameters
save_path— str. Path to save the bit allocation. If empty, not savedload_path— str. Path to load the bit allocation. If empty, not loaded
save_path¶
save_path: str = Field(
default="",
alias="savePath",
description="Path to save the bit allocation. If empty, not saved",
)
load_path¶
load_path: str = Field(
default="",
alias="loadPath",
description="Path to load the bit allocation. If empty, not loaded",
)
BitConfig.LayerOverrides¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Layer-specific bit-width override settings |
activation_16bits |
|
weight_16bits |
|
weight_8bits |
model_config¶
Layer-specific bit-width override settings
Parameters
activation_16bits— list[string]. Layer names to force 16-bit activationsweight_16bits— list[string]. Layer names to force 16-bit weightsweight_8bits— list[string]. Layer names to force 8-bit weights
activation_16bits¶
activation_16bits: List[str] = Field(
default=[],
alias="activation16Bits",
description="Layer names to force 16-bit activations",
)
weight_16bits¶
weight_16bits: List[str] = Field(
default=[],
alias="weight16Bits",
description="Layer names to force 16-bit weights",
)
weight_8bits¶
weight_8bits: List[str] = Field(
default=[],
alias="weight8Bits",
description="Layer names to force 8-bit weights",
)
HessianQuantConfig¶
Configuration for HessianQuant algorithm
Defines parameters controlling whether and how HessianQuant is applied during quantization, including layer-level inclusion/exclusion lists.
Parameters
apply— bool. If true, apply HessianQuanthessian_dtype— str. Storage dtype for the accumulated HessianQuant Hessian. bf16 halves its memory footprint (host RAM when the Hessian lives on CPU, VRAM when on GPU) — critical for large models such as MoE with thousands of expert FFN Hessians. Compute stays float32 regardless: the per-batch matmul and accumulation run in float32 and the solve upcasts back to float32; only the persistent accumulator is bfloat16.:
0: fp32 - Store the Hessian in float32.
1: bf16 - Store the Hessian in bfloat16 (half the memory).accumulation_device— str. Device the HessianQuant Hessian is accumulated on during calibration. gpu accumulates on the calibration device, which is 3-5x faster because the whole d x d accumulator is otherwise copied off the device once per layer per batch; the Hessian is parked on the host as soon as the last batch is in, so this does not raise peak VRAM for the rest of the compile. cpu accumulates on the host instead, bounding VRAM during calibration itself for models whose summed Hessian does not fit alongside the activations. auto follows resourceManagement.useGPUOnlyForCalibration, which is the knob that already says whether this compile is VRAM-bound: false picks gpu, true picks cpu. Resolved once at config load and logged; a CPU compile always accumulates on the host regardless.:
0: auto - Follow useGPUOnlyForCalibration: false picks gpu, true picks cpu.
1: cpu - Accumulate on the host, bounding VRAM during calibration itself.
2: gpu - Accumulate on the calibration device, then park on the host.attributes— Attributes. HessianQuant algorithm attributes
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
hessian_dtype |
|
accumulation_device |
|
attributes |
model_config¶
apply¶
hessian_dtype¶
hessian_dtype: str = Field(
default="fp32",
alias="hessianDtype",
description="Storage dtype for the accumulated HessianQuant Hessian. bf16 halves its memory footprint (host RAM when the Hessian lives on CPU, VRAM when on GPU) — critical for large models such as MoE with thousands of expert FFN Hessians. Compute stays float32 regardless: the per-batch matmul and accumulation run in float32 and the solve upcasts back to float32; only the persistent accumulator is bfloat16.",
)
accumulation_device¶
accumulation_device: str = Field(
default="auto",
alias="accumulationDevice",
description="Device the HessianQuant Hessian is accumulated on during calibration. gpu accumulates on the calibration device, which is 3-5x faster because the whole d x d accumulator is otherwise copied off the device once per layer per batch; the Hessian is parked on the host as soon as the last batch is in, so this does not raise peak VRAM for the rest of the compile. cpu accumulates on the host instead, bounding VRAM during calibration itself for models whose summed Hessian does not fit alongside the activations. auto follows resourceManagement.useGPUOnlyForCalibration, which is the knob that already says whether this compile is VRAM-bound: false picks gpu, true picks cpu. Resolved once at config load and logged; a CPU compile always accumulates on the host regardless.",
)
attributes¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
HessianQuantConfig.Attributes¶
Attributes¶
| Name | Description |
|---|---|
model_config |
HessianQuant algorithm attributes |
act_order |
|
block_size |
|
perc_damp |
|
apply_layers |
|
exclude_layers |
model_config¶
HessianQuant algorithm attributes
Parameters
act_order— bool. If true, use activation orderblock_size— int. Block size used for HessianQuantperc_damp— float. Percentage dampening factorapply_layers— List[str]. Layer names to apply HessianQuant. If empty, applies to all eligible layersexclude_layers— List[str]. Layer names to exclude from HessianQuant
act_order¶
act_order: bool = Field(
default=True, alias="actOrder", description="If true, use activation order"
)
block_size¶
block_size: int = Field(
default=256,
alias="blockSize",
description="Block size used for HessianQuant",
)
perc_damp¶
perc_damp: float = Field(
default=0.01, alias="percDamp", description="Percentage dampening factor"
)
apply_layers¶
apply_layers: List[str] = Field(
default=[],
alias="applyLayers",
description="Layer names to apply HessianQuant. If empty, applies to all eligible layers",
)
exclude_layers¶
exclude_layers: List[str] = Field(
default=[],
alias="excludeLayers",
description="Layer names to exclude from HessianQuant",
)
LayerBiasCorrectionConfig¶
Configuration for calibration-derived layer bias correction
Measures systematic per-channel float-to-integer activation error throughout the quantized network and folds damped corrections into integer convolution biases. Corrections are recomputed between iterations to account for upstream changes. This is calibration-only bias correction and does not use labels or Minimum Output Difference optimization.
Parameters
apply— bool. If true, correct systematic per-layer quantization biasattributes— Attributes. Layer bias correction attributes
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
attributes |
model_config¶
apply¶
apply: bool = Field(
default=False,
description="If true, correct systematic per-layer quantization bias",
)
attributes¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
LayerBiasCorrectionConfig.Attributes¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Layer bias correction attributes |
num_samples |
|
iterations |
|
correction_rate |
model_config¶
Layer bias correction attributes
Parameters
num_samples— int. Maximum number of calibration samples used per iterationiterations— int. Number of damped correction iterationscorrection_rate— float. Fraction of the measured bias error applied per iteration
num_samples¶
num_samples: int = Field(
default=256,
alias="numSamples",
description="Maximum number of calibration samples used per iteration",
ge=1,
)
iterations¶
correction_rate¶
correction_rate: float = Field(
default=0.05,
alias="correctionRate",
description="Fraction of the measured bias error applied per iteration",
ge=1e-06,
le=1.0,
)
ModConfig¶
Configuration for Minimum Output Difference algorithm
Defines parameters controlling whether and how MOD is applied during quantization, including layer-level inclusion/exclusion lists.
Parameters
apply— bool. If true, apply MODattributes— Attributes. MOD algorithm attributes
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
attributes |
model_config¶
apply¶
attributes¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
ModConfig.Attributes¶
Attributes¶
| Name | Description |
|---|---|
model_config |
MOD algorithm attributes |
epochs |
|
warmup_epochs |
|
lr_min_ratio |
|
save_dir |
|
seed |
|
apply_layers |
|
exclude_layers |
|
mod_after_layer_name |
|
anchors |
|
use_xyxy |
|
learning_rates |
|
training |
|
loss |
|
post_processing |
model_config¶
MOD algorithm attributes
Parameters
epochs— int. Number of training epochswarmup_epochs— int. Number of warmup epochslr_min_ratio— float. Minimum learning rate ratiosave_dir— str. Directory to save MOD resultsseed— int. Random seed for MODapply_layers— List[str]. Layer names to apply MOD. If empty, applies to all eligible layersexclude_layers— List[str]. Layer names to exclude from MODmod_after_layer_name— str. Apply MOD after this layeranchors— List. Anchor configurations for detection models. Nested list of anchor box sizes for detection models. Structure: List[List[List[int]]] where outer list is per detection head (e.g. small/medium/large), middle list is anchors per head, inner list is [width, height]. Example: [[[12,16],[19,36],[40,28]], [[36,75],[76,55],[72,146]], [[142,110],[192,243],[459,401]]]use_xyxy— bool. Use XYXY format for bounding boxeslearning_rates— LearningRates. Learning rate configuration for MODtraining— Training. MOD training configurationloss— Loss. MOD loss configurationpost_processing— PostProcessing. Post-processing configuration for detection models
epochs¶
warmup_epochs¶
warmup_epochs: int = Field(
default=1, alias="warmupEpochs", description="Number of warmup epochs"
)
lr_min_ratio¶
lr_min_ratio: float = Field(
default=0.0001,
alias="lrMinRatio",
description="Minimum learning rate ratio",
)
save_dir¶
seed¶
apply_layers¶
apply_layers: List[str] = Field(
default=[],
alias="applyLayers",
description="Layer names to apply MOD. If empty, applies to all eligible layers",
)
exclude_layers¶
exclude_layers: List[str] = Field(
default=[],
alias="excludeLayers",
description="Layer names to exclude from MOD",
)
mod_after_layer_name¶
mod_after_layer_name: str = Field(
default="",
alias="modAfterLayerName",
description="Apply MOD after this layer",
)
anchors¶
anchors: Any = Field(
default=[],
description="Anchor configurations for detection models. Nested list of anchor box sizes for detection models. Structure: List[List[List[int]]] where outer list is per detection head (e.g. small/medium/large), middle list is anchors per head, inner list is [width, height]. Example: [[[12,16],[19,36],[40,28]], [[36,75],[76,55],[72,146]], [[142,110],[192,243],[459,401]]]",
)
use_xyxy¶
use_xyxy: bool = Field(
default=False,
alias="useXYXY",
description="Use XYXY format for bounding boxes",
)
learning_rates¶
training¶
loss¶
post_processing¶
ModConfig.Attributes.LearningRates¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Learning rate configuration for MOD |
act_scale |
|
zeropoint |
|
weight_scale |
|
weight |
|
bias |
model_config¶
Learning rate configuration for MOD
Parameters
act_scale— float. Learning rate for activation scalezeropoint— float. Learning rate for zeropointweight_scale— float. Learning rate for weight scaleweight— float. Learning rate for weightbias— float. Learning rate for bias
act_scale¶
act_scale: float = Field(
default=0.0,
alias="actScale",
description="Learning rate for activation scale",
)
zeropoint¶
weight_scale¶
weight_scale: float = Field(
default=0.0,
alias="weightScale",
description="Learning rate for weight scale",
)
weight¶
bias¶
ModConfig.Attributes.Training¶
Attributes¶
| Name | Description |
|---|---|
model_config |
MOD training configuration |
batch_size |
|
q_drop |
|
quantize_weight |
|
weight_scale_init |
|
downresol_mode |
|
scheduler_type |
model_config¶
MOD training configuration
Parameters
batch_size— int. Batch size for MOD trainingq_drop— float. Quantization drop probabilityquantize_weight— bool. Whether to quantize weightsweight_scale_init— str. Weight scale initialization methoddownresol_mode— str. Downresolution modescheduler_type— str. LR scheduler type
batch_size¶
q_drop¶
quantize_weight¶
quantize_weight: bool = Field(
default=True,
alias="quantizeWeight",
description="Whether to quantize weights",
)
weight_scale_init¶
weight_scale_init: str = Field(
default="MinMax",
alias="weightScaleInit",
description="Weight scale initialization method",
)
downresol_mode¶
downresol_mode: str = Field(
default="STE", alias="downresolMode", description="Downresolution mode"
)
scheduler_type¶
scheduler_type: str = Field(
default="Cosine", alias="schedulerType", description="LR scheduler type"
)
ModConfig.Attributes.Loss¶
Attributes¶
| Name | Description |
|---|---|
model_config |
MOD loss configuration |
type |
|
use_outputs |
|
kl_temperature |
|
recon_prob |
|
recon_coeff |
|
lambda_0 |
|
lambda_1 |
|
lambda_2 |
|
lambda_3 |
|
custom_loss_jit_path |
model_config¶
MOD loss configuration
Parameters
type— str. Loss type (MSE, KL, etc.)use_outputs— bool. Use model outputs for loss computationkl_temperature— float. KL divergence temperaturerecon_prob— float. Reconstruction probabilityrecon_coeff— float. Reconstruction coefficientlambda_0— float. Loss weight lambda_0lambda_1— float. Loss weight lambda_1lambda_2— float. Loss weight lambda_2lambda_3— float. Loss weight lambda_3custom_loss_jit_path— str. Path to custom JIT-compiled loss function. Refer to /workspace/quantizer/pyutils/mel.pt
type¶
use_outputs¶
use_outputs: bool = Field(
default=False,
alias="useOutputs",
description="Use model outputs for loss computation",
)
kl_temperature¶
kl_temperature: float = Field(
default=1.0,
alias="KLTemperature",
description="KL divergence temperature",
)
recon_prob¶
recon_prob: float = Field(
default=1.0, alias="reconProb", description="Reconstruction probability"
)
recon_coeff¶
recon_coeff: float = Field(
default=1.0,
alias="reconCoeff",
description="Reconstruction coefficient",
)
lambda_0¶
lambda_1¶
lambda_2¶
lambda_3¶
custom_loss_jit_path¶
custom_loss_jit_path: str = Field(
default="",
alias="customLossJITPath",
description="Path to custom JIT-compiled loss function. Refer to /workspace/quantizer/pyutils/mel.pt",
)
ModConfig.Attributes.PostProcessing¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Post-processing configuration for detection models |
post |
|
box_conf_thres |
|
box_iou_thres |
model_config¶
Post-processing configuration for detection models
Parameters
post— str. Post-processing typebox_conf_thres— float. Box confidence thresholdbox_iou_thres— float. Box IoU threshold
post¶
box_conf_thres¶
box_conf_thres: float = Field(
default=0, alias="boxConfThres", description="Box confidence threshold"
)
box_iou_thres¶
LlmConfig¶
Configuration for Large Language Model (LLM) compilation
Defines LLM-specific settings including sequence lengths, cache configurations, and runtime parameters for efficient LLM inference.
Parameters
apply— bool. If True, apply LLM-specific configurationsnpu_parallel_degree— int. Number of NPU partitions for FFN tensor parallelism (1 = disabled). Applied before OptimizeFFN if both active.attributes— Attributes. LLM attributes configuration
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
npu_parallel_degree |
|
attributes |
model_config¶
apply¶
npu_parallel_degree¶
npu_parallel_degree: int = Field(
default=1,
alias="npuParallelDegree",
description="Number of NPU partitions for FFN tensor parallelism (1 = disabled). Applied before OptimizeFFN if both active.",
)
attributes¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
LlmConfig.Attributes¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LLM attributes configuration |
max_data_length |
|
max_sequence_length |
|
max_cache_length |
|
max_core_data_length |
|
calibration |
|
runtime |
|
debug |
model_config¶
LLM attributes configuration
Parameters
max_data_length— int. Maximum data lengthmax_sequence_length— int. Maximum sequence lengthmax_cache_length— int. Maximum cache lengthmax_core_data_length— int. Maximum core data lengthcalibration— Calibration. LLM calibration settingsruntime— Runtime. LLM runtime settingsdebug— Debug. LLM debug settings
max_data_length¶
max_data_length: int = Field(
default=4096, alias="maxDataLength", description="Maximum data length"
)
max_sequence_length¶
max_sequence_length: int = Field(
default=4096,
alias="maxSequenceLength",
description="Maximum sequence length",
)
max_cache_length¶
max_cache_length: int = Field(
default=4096, alias="maxCacheLength", description="Maximum cache length"
)
max_core_data_length¶
max_core_data_length: int = Field(
default=128,
alias="maxCoreDataLength",
description="Maximum core data length",
)
calibration¶
runtime¶
debug¶
LlmConfig.Attributes.Calibration¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LLM calibration settings |
random_seq_length |
|
use_full_seq_length |
model_config¶
LLM calibration settings
Parameters
random_seq_length— int. Random sequence length used for calibrationuse_full_seq_length— bool. If True, use the full sequence length for calibration
random_seq_length¶
random_seq_length: int = Field(
default=80,
alias="randomSeqLength",
description="Random sequence length used for calibration",
)
use_full_seq_length¶
use_full_seq_length: bool = Field(
default=False,
alias="useFullSeqLength",
description="If True, use the full sequence length for calibration",
)
LlmConfig.Attributes.Runtime¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LLM runtime settings |
use_global_core |
|
batch_size |
|
npu_core_ids |
|
dynamic_rope |
|
dynamic_mask |
model_config¶
LLM runtime settings
Parameters
use_global_core— bool. If True, use a global corebatch_size— int. Batch sizenpu_core_ids— List[int]. List of NPU core IDsdynamic_rope— bool. If True, enable dynamic RoPE (rotary position embedding)dynamic_mask— bool. If True, enable dynamic mask (attention mask as runtime input)
use_global_core¶
use_global_core: bool = Field(
default=False,
alias="useGlobalCore",
description="If True, use a global core",
)
batch_size¶
npu_core_ids¶
npu_core_ids: List[int] = Field(
default=[0], alias="npuCoreIds", description="List of NPU core IDs"
)
dynamic_rope¶
dynamic_rope: bool = Field(
default=False,
alias="dynamicRope",
description="If True, enable dynamic RoPE (rotary position embedding)",
)
dynamic_mask¶
dynamic_mask: bool = Field(
default=False,
alias="dynamicMask",
description="If True, enable dynamic mask (attention mask as runtime input)",
)
LlmConfig.Attributes.Debug¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LLM debug settings |
apply |
|
batch_debug_bundle_size |
model_config¶
LLM debug settings
Parameters
apply— bool. Enable LLM debug modebatch_debug_bundle_size— int. Batch debug bundle size
apply¶
batch_debug_bundle_size¶
batch_debug_bundle_size: int = Field(
default=0,
alias="batchDebugBundleSize",
description="Batch debug bundle size",
)
MoeConfig¶
Sparse MoE expert-selection configuration (calibration only)
Controls which experts are calibrated inside SparseMoe modules. selectionMode is a calibration-only knob: it picks which experts collect statistics (and, for TopK, on which tokens). It does NOT change inference routing — the forward path always routes the router's top-K experts regardless of this setting. scoreThreshold is only used when selectionMode is Threshold.
Parameters
selection_mode— int. Expert selection mode index (calibration only; inference is always top-K):
0: TopK - Calibrate only the router's top-K experts per token (matches inference routing).
1: All - Calibrate every expert on the full sequence.
2: Threshold - Calibrate all experts whose routing score exceeds scoreThreshold.score_threshold— float. Routing score threshold used when selectionMode is Threshold (calibration only)
Attributes¶
| Name | Description |
|---|---|
model_config |
|
selection_mode_list |
|
selection_mode |
|
score_threshold |
model_config¶
selection_mode_list¶
selection_mode_list: List[str] = Field(
default=["TopK", "All", "Threshold"], alias="selectionModeList"
)
selection_mode¶
score_threshold¶
score_threshold: float = Field(
default=0.0,
alias="scoreThreshold",
description="Routing score threshold used when selectionMode is Threshold (calibration only)",
ge=0,
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
EquivalentTransformationConfig¶
Configuration for equivalent transformation techniques
Defines parameters for various equivalent transformation methods including NormConv, QK smoothing, and rotation matrices for improved quantization.
Parameters
seed— int. Random seed for transformationapply_hadamard_rotation_matrix— bool. Apply Hadamard rotation matrixnorm_conv— NormConv. NormConv equivalent transformationqk— Qk. QK smoothing transformationud— Ud. UD transformationvo— Vo. VO transformationfeed_forward_multi_lut— FeedForwardMultiLut. Feed-forward multi-LUT transformationspin_r1— SpinR1. SpinR1 rotation transformationhead_out_ch_rotation— HeadOutChRotation. Head output channel rotation transformationin_rotation— InRotation. Input rotation transformationspin_r2— SpinR2. SpinR2 rotation transformationqk_rotation— QkRotation. QK rotation transformationflatten_quant— FlattenQuant. Flatten quantization transformationoptimize_ffn— OptimizeFfn. FFN optimization
Attributes¶
| Name | Description |
|---|---|
model_config |
|
seed |
|
apply_hadamard_rotation_matrix |
|
norm_conv |
|
qk |
|
ud |
|
vo |
|
feed_forward_multi_lut |
|
spin_r1 |
|
head_out_ch_rotation |
|
in_rotation |
|
spin_r2 |
|
qk_rotation |
|
flatten_quant |
|
optimize_ffn |
model_config¶
seed¶
apply_hadamard_rotation_matrix¶
apply_hadamard_rotation_matrix: bool = Field(
default=True,
alias="applyHadamardRotationMatrix",
description="Apply Hadamard rotation matrix",
)
norm_conv¶
qk¶
ud¶
vo¶
feed_forward_multi_lut¶
feed_forward_multi_lut: FeedForwardMultiLut = Field(
default_factory=FeedForwardMultiLut, alias="FeedForwardMultiLUT"
)
spin_r1¶
head_out_ch_rotation¶
head_out_ch_rotation: HeadOutChRotation = Field(
default_factory=HeadOutChRotation, alias="HeadOutChRotation"
)
in_rotation¶
spin_r2¶
qk_rotation¶
flatten_quant¶
optimize_ffn¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
EquivalentTransformationConfig.NormConv¶
Attributes¶
| Name | Description |
|---|---|
model_config |
NormConv equivalent transformation |
apply |
|
learn |
|
smoothing_factor |
|
min_gamma |
|
max_gamma |
model_config¶
NormConv equivalent transformation
Parameters
apply— bool. Apply NormConv transformationlearn— bool. Learn transformation parameterssmoothing_factor— float. Smoothing factormin_gamma— float. Minimum gamma valuemax_gamma— float. Maximum gamma value
apply¶
learn¶
smoothing_factor¶
smoothing_factor: float = Field(
default=0.5, alias="smoothingFactor", description="Smoothing factor"
)
min_gamma¶
max_gamma¶
EquivalentTransformationConfig.Qk¶
Attributes¶
| Name | Description |
|---|---|
model_config |
QK smoothing transformation |
apply |
|
smoothing_factor |
|
min_gamma |
|
max_gamma |
model_config¶
QK smoothing transformation
Parameters
apply— bool. Apply QK transformationsmoothing_factor— float. Smoothing factormin_gamma— float. Minimum gamma valuemax_gamma— float. Maximum gamma value
apply¶
smoothing_factor¶
smoothing_factor: float = Field(
default=0.5, alias="smoothingFactor", description="Smoothing factor"
)
min_gamma¶
max_gamma¶
EquivalentTransformationConfig.Ud¶
Attributes¶
| Name | Description |
|---|---|
model_config |
UD transformation |
apply |
|
learn |
|
smoothing_factor |
|
min_gamma |
|
max_gamma |
model_config¶
UD transformation
Parameters
apply— bool. Apply UD transformationlearn— bool. Learn transformation parameterssmoothing_factor— float. Smoothing factormin_gamma— float. Minimum gamma valuemax_gamma— float. Maximum gamma value
apply¶
learn¶
smoothing_factor¶
smoothing_factor: float = Field(
default=0.5, alias="smoothingFactor", description="Smoothing factor"
)
min_gamma¶
max_gamma¶
EquivalentTransformationConfig.Vo¶
Attributes¶
| Name | Description |
|---|---|
model_config |
VO transformation |
apply |
|
smoothing_factor |
|
min_gamma |
|
max_gamma |
model_config¶
VO transformation
Parameters
apply— bool. Apply VO transformationsmoothing_factor— float. Smoothing factormin_gamma— float. Minimum gamma valuemax_gamma— float. Maximum gamma value
apply¶
smoothing_factor¶
smoothing_factor: float = Field(
default=0.5, alias="smoothingFactor", description="Smoothing factor"
)
min_gamma¶
max_gamma¶
EquivalentTransformationConfig.FeedForwardMultiLut¶
Feed-forward multi-LUT transformation
Parameters
apply— bool. Apply feed-forward multi-LUT transformationbreakpoints— List[float]. Breakpoints for multi-LUT
Attributes¶
| Name | Description |
|---|---|
apply |
|
breakpoints |
apply¶
breakpoints¶
breakpoints: List[float] = Field(
default=[-8.0, -4.0, 0], description="Breakpoints for multi-LUT"
)
EquivalentTransformationConfig.SpinR1¶
Attributes¶
| Name | Description |
|---|---|
model_config |
SpinR1 rotation transformation |
apply |
|
matrix_path |
model_config¶
SpinR1 rotation transformation
Parameters
apply— bool. Apply SpinR1 transformationmatrix_path— str. Path to rotation matrix file
apply¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
EquivalentTransformationConfig.HeadOutChRotation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Head output channel rotation transformation |
apply |
|
matrix_path |
model_config¶
Head output channel rotation transformation
Parameters
apply— bool. Apply head output channel rotationmatrix_path— str. Path to rotation matrix file
apply¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
EquivalentTransformationConfig.InRotation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Input rotation transformation |
apply |
|
matrix_path |
|
input_names |
model_config¶
Input rotation transformation
Parameters
apply— bool. Apply input rotationmatrix_path— str. Path to rotation matrix fileinput_names— List[str]. Names of the input layers to rotate
apply¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
input_names¶
input_names: List[str] = Field(
default=[],
alias="inputNames",
description="Names of the input layers to rotate",
)
EquivalentTransformationConfig.SpinR2¶
Attributes¶
| Name | Description |
|---|---|
model_config |
SpinR2 rotation transformation |
apply |
|
learn |
|
matrix_path |
model_config¶
SpinR2 rotation transformation
Parameters
apply— bool. Apply SpinR2 transformationlearn— bool. Learn rotation matrixmatrix_path— str. Path to rotation matrix file
apply¶
learn¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
EquivalentTransformationConfig.QkRotation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
QK rotation transformation |
apply |
|
matrix_path |
model_config¶
QK rotation transformation
Parameters
apply— bool. Apply QK rotation transformationmatrix_path— str. Path to rotation matrix file
apply¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
EquivalentTransformationConfig.FlattenQuant¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Flatten quantization transformation |
apply |
|
learn |
|
apply_threshold |
|
max_overhead |
model_config¶
Flatten quantization transformation
Parameters
apply— bool. Apply flatten quantizationlearn— bool. Learn flattening parametersapply_threshold— float. Threshold for applying flatten quantizationmax_overhead— float. Maximum overhead allowed for flattening
apply¶
learn¶
apply_threshold¶
apply_threshold: float = Field(
default=0.33,
alias="applyThreshold",
description="Threshold for applying flatten quantization",
)
max_overhead¶
max_overhead: float = Field(
default=0.02,
alias="maxOverhead",
description="Maximum overhead allowed for flattening",
)
EquivalentTransformationConfig.OptimizeFfn¶
Attributes¶
| Name | Description |
|---|---|
model_config |
FFN optimization |
apply |
|
ch_per_ffn |
model_config¶
FFN optimization
Parameters
apply— bool. Apply FFN optimizationch_per_ffn— int. Optimize FFN split (-1 for auto)
apply¶
ch_per_ffn¶
ch_per_ffn: int = Field(
default=-1, alias="chPerFFN", description="Optimize FFN split (-1 for auto)"
)
SearchWeightScaleConfig¶
Configuration for weight scale search
Defines which transformer components should have their weight scales searched for optimal quantization.
Parameters
apply— bool. If true, apply weight scale searchtransformer— Transformer. Transformer components for weight scale search
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
transformer |
model_config¶
apply¶
transformer¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
SearchWeightScaleConfig.Transformer¶
Transformer components for weight scale search
Parameters
query— bool. Search weight scale for querykey— bool. Search weight scale for keyvalue— bool. Search weight scale for valueout— bool. Search weight scale for outputffn— bool. Search weight scale for FFN
Attributes¶
| Name | Description |
|---|---|
query |
|
key |
|
value |
|
out |
|
ffn |
query¶
key¶
value¶
out¶
ffn¶
LoadScaleConfig¶
QAT activation scale loading configuration
Loads pre-trained activation scales from safetensors files and applies them to specified layers before scale/zeropoint computation. NOTE: entries is stored as raw JSON (type: list) because the generator does not support list[CustomStruct]. The schema records the field shape; parsing is done manually in applyQATLoadScaleQuantType / applyQATLoadScales.
Parameters
apply— bool. If true, load and apply QAT scales from safetensors filesentries— List. List of file entries. Each entry is a dict: { path: str, scales: [ { key: str, layers: [str] } ] } path: safetensors file path; key: tensor name in the file; layers: layer names whose activation scale will be overridden.
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
entries |
model_config¶
apply¶
apply: bool = Field(
default=False,
description="If true, load and apply QAT scales from safetensors files",
)
entries¶
entries: Any = Field(
default=[],
description="List of file entries. Each entry is a dict: { path: str, scales: [ { key: str, layers: [str] } ] } path: safetensors file path; key: tensor name in the file; layers: layer names whose activation scale will be overridden.",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
RuntimeOptions¶
Runtime options for compilation
Contains runtime-specific settings like version info and cache options.
Parameters
version— str. Compiler version string (e.g., 0.0.0)
Attributes¶
| Name | Description |
|---|---|
model_config |
|
version |
model_config¶
version¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
SaveSampleConfig¶
Sample data generation and saving configuration
Parameters
apply— bool. Enable sample data savingmode— str. Inference mode: infer (standard) or inferWithCache (LLM cache models)batch_size— int. Number of inference batches to generatebatch_seq_lens— List. Per-batch step-wise sequence lengths for inferWithCache mode. e.g. [[80, 1], [240, 10]]save_folder— str. Output folder for sample datadtype— str. Data type for saved samples: float or int8
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
mode |
|
batch_size |
|
batch_seq_lens |
|
save_folder |
|
dtype |
model_config¶
apply¶
mode¶
mode: str = Field(
default="infer",
description="Inference mode: infer (standard) or inferWithCache (LLM cache models)",
)
batch_size¶
batch_size: int = Field(
default=1,
alias="batchSize",
description="Number of inference batches to generate",
)
batch_seq_lens¶
batch_seq_lens: Any = Field(
default=[],
alias="batchSeqLens",
description="Per-batch step-wise sequence lengths for inferWithCache mode. e.g. [[80, 1], [240, 10]]",
)
save_folder¶
save_folder: str = Field(
default="sampleInout",
alias="saveFolder",
description="Output folder for sample data",
)
dtype¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
GroupWiseConfig¶
Group-wise streaming quantization configuration for large LLMs
Configuration for the group-wise (streaming) quantization pipeline used for LLMs that do not fit fully on GPU. Partitions the model into groups of transformer blocks and quantizes each group independently.
Parameters
apply— bool. Enable group-wise streaming quantization pipelinegroup_size— int. Group size in number of transformer blocks (0 = auto)gpu_budget_gb— float. GPU memory budget in GiB for group-wise execution (0 = auto-detect)gpu_safety_margin_gb— float. Safety margin in GiB subtracted from detected GPU budgetcache_dir— str. Directory for group-wise activation/state cache (empty = system temp)keep_cache— bool. Retain group-wise cache after run completes (for debugging)partition_policy— str. Partitioning policy (e.g. transformer_block, moe_expert_subgroup)expert_groups— List[List[int]]. MoE fallback partitioner: list of expert-index lists (List[List[int]]).retain_topology_weights— bool. Keep inflated FP weights across groups (false = release after each group for tight memory budgets).checkpoint— bool. Save per-group checkpoint so quantization can resume from the last completed group after a crash.
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
group_size |
|
gpu_budget_gb |
|
gpu_safety_margin_gb |
|
cache_dir |
|
keep_cache |
|
partition_policy |
|
expert_groups |
|
retain_topology_weights |
|
checkpoint |
model_config¶
apply¶
apply: bool = Field(
default=False, description="Enable group-wise streaming quantization pipeline"
)
group_size¶
group_size: int = Field(
default=0,
alias="groupSize",
description="Group size in number of transformer blocks (0 = auto)",
)
gpu_budget_gb¶
gpu_budget_gb: float = Field(
default=0,
alias="gpuBudgetGb",
description="GPU memory budget in GiB for group-wise execution (0 = auto-detect)",
)
gpu_safety_margin_gb¶
gpu_safety_margin_gb: float = Field(
default=4,
alias="gpuSafetyMarginGb",
description="Safety margin in GiB subtracted from detected GPU budget",
)
cache_dir¶
cache_dir: str = Field(
default="",
alias="cacheDir",
description="Directory for group-wise activation/state cache (empty = system temp)",
)
keep_cache¶
keep_cache: bool = Field(
default=False,
alias="keepCache",
description="Retain group-wise cache after run completes (for debugging)",
)
partition_policy¶
partition_policy: str = Field(
default="transformer_block",
alias="partitionPolicy",
description="Partitioning policy (e.g. transformer_block, moe_expert_subgroup)",
)
expert_groups¶
expert_groups: Any = Field(
default=[],
alias="expertGroups",
description="MoE fallback partitioner: list of expert-index lists (List[List[int]]).",
)
retain_topology_weights¶
retain_topology_weights: bool = Field(
default=True,
alias="retainTopologyWeights",
description="Keep inflated FP weights across groups (false = release after each group for tight memory budgets).",
)
checkpoint¶
checkpoint: bool = Field(
default=False,
description="Save per-group checkpoint so quantization can resume from the last completed group after a crash.",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
CompileConfig¶
Unified compilation configuration for Mobilint MXQ compilation.
Attributes¶
model_config¶
model_paths¶
model_paths: List[str] = Field(
default=[], alias="modelPaths", description="Paths to model files"
)
calib_data_path¶
calib_data_path: List[str] = Field(
default=[], alias="calibDataPaths", description="Paths to calibration datasets"
)
save_paths¶
save_paths: List[str] = Field(
default=["./tmp.mxq"],
alias="savePaths",
description="Output MXQ filename/paths",
)
use_random_calib¶
use_random_calib: bool = Field(
default=False, alias="useRandomCalib", description="Use random calibration"
)
inference_scheme¶
inference_scheme: str = Field(
default="single", alias="inferenceScheme", description="NPU inference scheme"
)
cpu_offload¶
cpu_offload: bool = Field(
default=False,
alias="cpuOffload",
description="Enable CPU offload for unsupported operators",
)
force_npu_input_reposition¶
force_npu_input_reposition: bool = Field(
default=False,
alias="forceNpuInputReposition",
description="Force input reposition operations to run on NPU instead of CPU",
)
force_npu_output_reposition¶
force_npu_output_reposition: bool = Field(
default=False,
alias="forceNpuOutputReposition",
description="Force output reposition operations to run on NPU instead of CPU",
)
optimize_option¶
optimize_option: int = Field(
default=1,
alias="optimizeOption",
description="Compiler optimization selector",
ge=0,
)
buffer_mode¶
input_shape_dict¶
input_shape_dict: Any = Field(
default={}, alias="inputShapeDict", description="Dictionary of input shapes"
)
device¶
dtype¶
debug¶
trace¶
image_channels¶
image_channels: int = Field(
default=0,
alias="imageChannels",
description="Number of image channels (0 for auto-detect)",
)
config_version¶
config_version: str = Field(
default="1.0.0", alias="configVersion", description="Config schema version"
)
split_blocks¶
split_blocks: List[int] = Field(
default=[],
alias="splitBlocks",
description="Multi-MXQ split points by transformer block index",
)
split_parts¶
split_parts: int = Field(
default=0,
alias="splitParts",
description="Evenly split transformer blocks into N MXQ parts",
)
uint8_input¶
preprocessing¶
resource_management¶
resource_management: ResourceManagementConfig = Field(
default_factory=ResourceManagementConfig, alias="resourceManagement"
)
calibration¶
bit¶
hessian_quant¶
hessian_quant: HessianQuantConfig = Field(
default_factory=HessianQuantConfig, alias="hessianQuant"
)
layer_bias_correction¶
layer_bias_correction: LayerBiasCorrectionConfig = Field(
default_factory=LayerBiasCorrectionConfig, alias="layerBiasCorrection"
)
mod¶
llm¶
moe¶
equivalent_transformation¶
equivalent_transformation: EquivalentTransformationConfig = Field(
default_factory=EquivalentTransformationConfig, alias="equivalentTransformation"
)
search_weight_scale¶
search_weight_scale: SearchWeightScaleConfig = Field(
default_factory=SearchWeightScaleConfig, alias="searchWeightScale"
)
load_scale¶
runtime_options¶
save_sample¶
group_wise¶
Methods¶
| Name | Description |
|---|---|
with_uint8_input |
Return a copy with uint8_input settings enabled. |
with_preprocessing |
Return a copy with preprocessing settings enabled. |
with_hessian_quant |
Return a copy with hessian_quant settings enabled. |
with_layer_bias_correction |
Return a copy with layer_bias_correction settings enabled. |
with_mod |
Return a copy with mod settings enabled. |
with_llm |
Return a copy with llm settings enabled. |
with_search_weight_scale |
Return a copy with search_weight_scale settings enabled. |
with_load_scale |
Return a copy with load_scale settings enabled. |
with_save_sample |
Return a copy with save_sample settings enabled. |
with_group_wise |
Return a copy with group_wise settings enabled. |
from_file |
Load config from YAML or JSON file. |
from_preset |
Load config from a preset. |
to_file |
Save config to YAML or JSON file. |
with_uint8_input¶
Return a copy with uint8_input settings enabled.
with_preprocessing¶
Return a copy with preprocessing settings enabled.
with_hessian_quant¶
Return a copy with hessian_quant settings enabled.
with_layer_bias_correction¶
Return a copy with layer_bias_correction settings enabled.
with_mod¶
Return a copy with mod settings enabled.
with_llm¶
Return a copy with llm settings enabled.
with_search_weight_scale¶
Return a copy with search_weight_scale settings enabled.
with_load_scale¶
Return a copy with load_scale settings enabled.
with_save_sample¶
Return a copy with save_sample settings enabled.
with_group_wise¶
Return a copy with group_wise settings enabled.
from_file¶
Load config from YAML or JSON file.
from_preset¶
Load config from a preset.
to_file¶
Save config to YAML or JSON file.