qbcompiler.configs.models¶
Auto-generated Pydantic models from config_schema.yaml.
Attributes¶
| Name | Description |
|---|---|
SCHEMA_VERSION |
SCHEMA_VERSION¶
Uint8InputConfig¶
Configuration for uint8 input handling
Defines whether inputs should be treated as uint8 and which specific inputs to apply this to.
Parameters
apply— bool. If true, treat specified inputs as uint8inputs— List[str]. List of input names to treat as uint8. If empty and apply is true, applies to all inputsdivision_factor— float. Division factor for uint8 to float conversion (e.g., 255.0 for [0,1], 127.5 for [0,2])
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
inputs |
|
division_factor |
model_config¶
apply¶
inputs¶
inputs: List[str] = Field(
default=[],
description="List of input names to treat as uint8. If empty and apply is true, applies to all inputs",
)
division_factor¶
division_factor: float = Field(
default=255.0,
alias="divisionFactor",
description="Division factor for uint8 to float conversion (e.g., 255.0 for [0,1], 127.5 for [0,2])",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
PreprocessingConfig¶
Configuration for input preprocessing pipeline
Defines preprocessing operations to be applied to model inputs, including operations like resize, normalize, color conversion, etc. Each pipeline entry is {"op": <name>, ...op keys}; unknown keys are rejected. resize/letterbox take "backend": "torch" (default, float interpolate), "pil" (Pillow, bit-exact, bilinear/bicubic) or "opencv" (cv2 INTER_LINEAR, bit-exact), so calibration sees the same pixels as an evaluator built on that library. letterbox "alignType" is 0 (default: center, padding split around the image) or 1 (top-left: padding only at the bottom and right, as YOLOX and DAMO-YOLO use). letterbox "roundType" is 0 (default: each backend's rounding of the resized size) or 1 (floor: upstream YOLOX and DAMO-YOLO's int(w * r)); a fuseIntoFirstLayer letterbox ignores it, since its exact odd decimation has no fraction to round.
Parameters
apply— bool. If true, apply preprocessing pipelineauto_convert_format— bool. If true, automatically convert input formatpipeline— List[Dict[str, Any]]. List of preprocessing operations to apply globallyinput_configs— Dict[str, Any]. Per-input preprocessing configurations. Keys are input names
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
auto_convert_format |
|
pipeline |
|
input_configs |
model_config¶
apply¶
auto_convert_format¶
auto_convert_format: bool = Field(
default=False,
alias="autoConvertFormat",
description="If true, automatically convert input format",
)
pipeline¶
pipeline: Any = Field(
default=[], description="List of preprocessing operations to apply globally"
)
input_configs¶
input_configs: Any = Field(
default={},
alias="inputConfigs",
description="Per-input preprocessing configurations. Keys are input names",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
ResourceManagementConfig¶
Configuration for resource management during model compilation
Controls GPU and memory management settings during the compilation process.
Parameters
weight_dtype— str. Weight data type for calibration (e.g., 'float32', 'float16')gpu_memory_budget_mb— int. Device memory budget for weight quantization in MiB; 0 = unlimited, -1 = autoweight_memory— WeightMemory. Weight memory management configuration
Attributes¶
| Name | Description |
|---|---|
model_config |
|
weight_dtype |
|
gpu_memory_budget_mb |
|
weight_memory |
model_config¶
weight_dtype¶
weight_dtype: str = Field(
default="float32",
alias="weightDtype",
description="Weight data type for calibration (e.g., 'float32', 'float16')",
)
gpu_memory_budget_mb¶
gpu_memory_budget_mb: int = Field(
default=-1,
alias="gpuMemoryBudgetMB",
description="Device memory budget for weight quantization in MiB; 0 = unlimited, -1 = auto",
)
weight_memory¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
ResourceManagementConfig.WeightMemory¶
Weight memory management configuration
Parameters
method— int. Weight memory management method index:
0: DeleteFloat - Delete float weights after quantization.
1: SaveFloat - Save float weights to disk.
2: MoveFloat - Move float weights to CPU.
3: KeepFloat - Keep float weights in memory.
4: KeepAll - Keep all weights in memory.
Attributes¶
| Name | Description |
|---|---|
method_list |
|
method |
method_list¶
method_list: List[str] = Field(
default=["DeleteFloat", "SaveFloat", "MoveFloat", "KeepFloat", "KeepAll"],
alias="methodList",
)
method¶
CalibrationConfig¶
Configuration for calibration during quantization
Defines calibration and quantization parameterization used to derive activation/weight scales and related statistics during quantized compilation.
Parameters
method— int. Calibration method index:
0: WChALayer - Weight per-channel, Activation per-layer.
1: WChAMulti - Weight per-channel, Activation multi-layer.
2: WChALayerZeropoint - Weight per-channel, Activation per-layer with zeropoint.
3: WChAMultiZeropoint - Weight per-channel, Activation multi-layer with zeropoint.output— int. Output quantization type index:
0: Layer - Per-layer quantization.
1: Ch - Per-channel quantization.
2: Sigmoid - Sigmoid-based quantization.mode— int. Quantization mode index:
0: Max - Maximum value calibration.
1: MaxPercentile - Maximum percentile calibration.
2: Histogram - Histogram-based calibration.
3: GeometricSketch - Geometric-bin relative-error sketch (DDSketch family).act_scale_min— float. Minimum allowed activation scale (lower bound clamp)act16_scale_min— float. Minimum 16-bit activation scale (actScaleMin / 256)weight_scale_min— float. Minimum allowed weight scale (lower bound clamp)weight16_scale_min— float. Minimum 16-bit weight scale (weightScaleMin / 256)min_clip_ratio— float. Minimum clip ratio constraint applied during calibrationmax_calib_data_size— int. Maximum number of calibration samples kept after loading or generationmax_sample_size_for_quant_scheme— int. Maximum number of calibration samples used per quant scheme stagemax_percentile— MaxPercentile. MaxPercentile mode configurationfast_dist— FastDist. Fast distribution calibration configurationhistogram— Histogram. Histogram-based calibration configurationgeometric_sketch— GeometricSketch. GeometricSketch mode configurationlayer_overrides— LayerOverrides. Layer-specific override settings for calibrationstatistics— Statistics. Statistics save/load configuration with percentile selectiongroup_lut— GroupLut. LUT grouping algorithm configuration
Attributes¶
model_config¶
method_list¶
method_list: List[str] = Field(
default=["WChALayer", "WChAMulti", "WChALayerZeropoint", "WChAMultiZeropoint"],
alias="methodList",
)
method¶
output_list¶
output¶
mode_list¶
mode_list: List[str] = Field(
default=["Max", "MaxPercentile", "Histogram", "GeometricSketch"],
alias="modeList",
)
mode¶
act_scale_min¶
act_scale_min: float = Field(
default=0.0005,
alias="actScaleMin",
description="Minimum allowed activation scale (lower bound clamp)",
ge=0,
le=1,
)
act16_scale_min¶
act16_scale_min: float = Field(
default=1.953125e-06,
alias="act16ScaleMin",
description="Minimum 16-bit activation scale (actScaleMin / 256)",
)
weight_scale_min¶
weight_scale_min: float = Field(
default=1e-06,
alias="weightScaleMin",
description="Minimum allowed weight scale (lower bound clamp)",
ge=0,
le=1,
)
weight16_scale_min¶
weight16_scale_min: float = Field(
default=3.90625e-09,
alias="weight16ScaleMin",
description="Minimum 16-bit weight scale (weightScaleMin / 256)",
)
min_clip_ratio¶
min_clip_ratio: float = Field(
default=-1,
alias="minClipRatio",
description="Minimum clip ratio constraint applied during calibration",
ge=-1,
le=1,
)
max_calib_data_size¶
max_calib_data_size: int = Field(
default=-1,
alias="maxCalibDataSize",
description="Maximum number of calibration samples kept after loading or generation",
ge=-1,
)
max_sample_size_for_quant_scheme¶
max_sample_size_for_quant_scheme: int = Field(
default=16,
alias="maxSampleSizeForQuantScheme",
description="Maximum number of calibration samples used per quant scheme stage",
ge=1,
)
max_percentile¶
fast_dist¶
histogram¶
geometric_sketch¶
geometric_sketch: GeometricSketch = Field(
default_factory=GeometricSketch, alias="geometricSketch"
)
layer_overrides¶
statistics¶
group_lut¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
CalibrationConfig.MaxPercentile¶
Attributes¶
| Name | Description |
|---|---|
model_config |
MaxPercentile mode configuration |
percentile |
|
topk_ratio |
|
max_each |
|
max_total |
|
per_ch_divisor |
model_config¶
MaxPercentile mode configuration
Parameters
percentile— float. Percentile value for maxPercentile modetopk_ratio— float. Top-k ratio used in maxPercentile modemax_each— int. Maximum number of samples processed per iterationmax_total— int. Total maximum number of samplesper_ch_divisor— int. Divisor for per-channel buffer capacity (bufferCap = max(maxTotal / perChDivisor, maxEach))
percentile¶
topk_ratio¶
topk_ratio: float = Field(
default=0.01,
alias="topKRatio",
description="Top-k ratio used in maxPercentile mode",
)
max_each¶
max_each: int = Field(
default=128,
alias="maxEach",
description="Maximum number of samples processed per iteration",
)
max_total¶
max_total: int = Field(
default=65536,
alias="maxTotal",
description="Total maximum number of samples",
)
per_ch_divisor¶
per_ch_divisor: int = Field(
default=16,
alias="perChDivisor",
description="Divisor for per-channel buffer capacity (bufferCap = max(maxTotal / perChDivisor, maxEach))",
ge=1,
)
CalibrationConfig.FastDist¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Fast distribution calibration configuration |
size_cali |
|
kernel_size |
|
stack_size |
model_config¶
Fast distribution calibration configuration
Parameters
size_cali— int.kernel_size— int.stack_size— int.
size_cali¶
kernel_size¶
stack_size¶
CalibrationConfig.Histogram¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Histogram-based calibration configuration |
search_type_list |
|
search_type |
|
percentile |
|
use_gpu |
|
num_bins |
|
num_samples |
|
buffer_size |
|
min_bin_width |
|
search_percentile_min |
|
search_percentile_max |
|
num_search |
model_config¶
Histogram-based calibration configuration
Parameters
search_type— int. Search type for histogram calibration:
0: Percentile.
1: MSE.
2: KL.percentile— float. Percentile value for histogram calibrationuse_gpu— bool. Use GPU for histogram computationnum_bins— int. Number of bins for histogramnum_samples— int. Number of samples for histogram calibrationbuffer_size— int. Buffer size for histogram computation (-1 for auto)min_bin_width— float. Minimum bin width for histogramsearch_percentile_min— float. Minimum search percentilesearch_percentile_max— float. Maximum search percentilenum_search— int. Number of search iterations
search_type_list¶
search_type¶
percentile¶
percentile: float = Field(
default=0.9999, description="Percentile value for histogram calibration"
)
use_gpu¶
use_gpu: bool = Field(
default=True,
alias="useGPU",
description="Use GPU for histogram computation",
)
num_bins¶
num_samples¶
num_samples: int = Field(
default=128,
alias="numSamples",
description="Number of samples for histogram calibration",
)
buffer_size¶
buffer_size: int = Field(
default=-1,
alias="bufferSize",
description="Buffer size for histogram computation (-1 for auto)",
)
min_bin_width¶
min_bin_width: float = Field(
default=1e-06,
alias="minBinWidth",
description="Minimum bin width for histogram",
)
search_percentile_min¶
search_percentile_min: float = Field(
default=0.9999,
alias="searchPercentileMin",
description="Minimum search percentile",
)
search_percentile_max¶
search_percentile_max: float = Field(
default=1.0,
alias="searchPercentileMax",
description="Maximum search percentile",
)
num_search¶
num_search: int = Field(
default=128, alias="numSearch", description="Number of search iterations"
)
CalibrationConfig.GeometricSketch¶
Attributes¶
| Name | Description |
|---|---|
model_config |
GeometricSketch mode configuration |
clip_ratio8 |
|
clip_ratio16 |
|
relative_error |
|
exp_min |
|
exp_max |
|
bin_budget |
model_config¶
GeometricSketch mode configuration
Parameters
clip_ratio8— float. Fraction clipped from each tail for INT8; must be >= 0 and < 0.5clip_ratio16— float. Fraction clipped from each tail for INT16; must be >= 0 and < 0.5relative_error— float. Geometric bin relative error; bin k holds magnitudes in ((1+r)^(k-1), (1+r)^k]exp_min— int. Smallest power-of-two exponent the magnitude grid covers; values below clamp to the end binexp_max— int. Largest power-of-two exponent the magnitude grid covers; values above clamp to the end binbin_budget— int. Stored bins per sign ladder and channel; magnitudes more than that many bins below the channel's largest fall into one floor bin
clip_ratio8¶
clip_ratio8: float = Field(
default=5e-06,
alias="clipRatio8",
description="Fraction clipped from each tail for INT8; must be >= 0 and < 0.5",
ge=0,
lt=0.5,
)
clip_ratio16¶
clip_ratio16: float = Field(
default=1e-06,
alias="clipRatio16",
description="Fraction clipped from each tail for INT16; must be >= 0 and < 0.5",
ge=0,
lt=0.5,
)
relative_error¶
relative_error: float = Field(
default=0.05,
alias="relativeError",
description="Geometric bin relative error; bin k holds magnitudes in ((1+r)^(k-1), (1+r)^k]",
gt=0,
lt=1,
)
exp_min¶
exp_min: int = Field(
default=-14,
alias="expMin",
description="Smallest power-of-two exponent the magnitude grid covers; values below clamp to the end bin",
)
exp_max¶
exp_max: int = Field(
default=24,
alias="expMax",
description="Largest power-of-two exponent the magnitude grid covers; values above clamp to the end bin",
)
bin_budget¶
bin_budget: int = Field(
default=256,
alias="binBudget",
description="Stored bins per sign ladder and channel; magnitudes more than that many bins below the channel's largest fall into one floor bin",
ge=8,
)
CalibrationConfig.LayerOverrides¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Layer-specific override settings for calibration |
act_scale_min |
|
percentile |
|
method |
model_config¶
Layer-specific override settings for calibration
Parameters
act_scale_min— dict. Per-layer activation scale minimum overrides. Keys are actScaleMin values (e.g. '0.0005'), values are lists of layer names to apply that override. (e.g. {'0.0005' : ['layer1', 'layer2']})percentile— dict. Per-layer maxPercentile percentile overrides. Keys are layer names, values are percentile floats. (e.g. {'/model/layer0/conv': 0.999, '/model/layer5/attn': 0.99})method— dict. Per-layer calibration method overrides. Keys are layer names, values are method ints (0=WChALayer, 1=WChAMulti, 2=WChALayerZeropoint, 3=WChAMultiZeropoint). (e.g. {'/model/layer0/conv': 1, '/model/layer5/attn': 3})
act_scale_min¶
act_scale_min: Any = Field(
default={},
alias="actScaleMin",
description="Per-layer activation scale minimum overrides. Keys are actScaleMin values (e.g. '0.0005'), values are lists of layer names to apply that override. (e.g. {'0.0005' : ['layer1', 'layer2']})",
)
percentile¶
percentile: Any = Field(
default={},
description="Per-layer maxPercentile percentile overrides. Keys are layer names, values are percentile floats. (e.g. {'/model/layer0/conv': 0.999, '/model/layer5/attn': 0.99})",
)
method¶
method: Any = Field(
default={},
description="Per-layer calibration method overrides. Keys are layer names, values are method ints (0=WChALayer, 1=WChAMulti, 2=WChALayerZeropoint, 3=WChAMultiZeropoint). (e.g. {'/model/layer0/conv': 1, '/model/layer5/attn': 3})",
)
CalibrationConfig.Statistics¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Statistics save/load configuration with percentile selection |
apply |
|
save_path |
|
load_path |
|
percentiles |
|
percentile_index |
model_config¶
Statistics save/load configuration with percentile selection
Parameters
apply— bool. Enable statistics save/loadsave_path— str. Path to save statistics. If empty, not savedload_path— str. Path to load statistics. If empty, not loadedpercentiles— List[float]. List of percentile candidatespercentile_index— int. Index into percentiles list to select active percentile
apply¶
save_path¶
save_path: str = Field(
default="",
alias="savePath",
description="Path to save statistics. If empty, not saved",
)
load_path¶
load_path: str = Field(
default="",
alias="loadPath",
description="Path to load statistics. If empty, not loaded",
)
percentiles¶
percentiles: List[float] = Field(
default=[0.9999, 0.999, 0.99, 0.9],
description="List of percentile candidates",
)
percentile_index¶
percentile_index: int = Field(
default=0,
alias="percentileIndex",
description="Index into percentiles list to select active percentile",
)
CalibrationConfig.GroupLut¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LUT grouping algorithm configuration |
irls_iter |
|
dro_eps |
|
cover_floor |
model_config¶
LUT grouping algorithm configuration
Parameters
irls_iter— int. Number of IRLS iterations for optimal scale computation (1 = single WLS)dro_eps— float. DRO worst-point coefficient; larger = more conservative clipping (0 = disable)cover_floor— float. LUTAM cluster scale lower bound as fraction of coverage
irls_iter¶
irls_iter: int = Field(
default=3,
alias="irlsIter",
description="Number of IRLS iterations for optimal scale computation (1 = single WLS)",
ge=1,
)
dro_eps¶
dro_eps: float = Field(
default=0,
alias="droEps",
description="DRO worst-point coefficient; larger = more conservative clipping (0 = disable)",
ge=0,
)
cover_floor¶
cover_floor: float = Field(
default=0.85,
alias="coverFloor",
description="LUTAM cluster scale lower bound as fraction of coverage",
ge=0,
le=1,
)
BitConfig¶
Configuration for bit precision
Defines bit-width parameterization for activations and weights used in mixed-precision quantization (e.g., attention and FFN components).
Parameters
transformer— Transformer. Transformer-specific bit-width configurationsave_info— SaveInfo. Bit allocation save/load configurationlayer_overrides— LayerOverrides. Layer-specific bit-width override settings
Attributes¶
| Name | Description |
|---|---|
model_config |
|
transformer |
|
save_info |
|
layer_overrides |
model_config¶
transformer¶
save_info¶
layer_overrides¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
BitConfig.Transformer¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Transformer-specific bit-width configuration |
activation |
|
weight |
|
mixed_precision |
model_config¶
Transformer-specific bit-width configuration
Parameters
activation— Activation. Activation bit-widths for transformer componentsweight— Weight. Weight bit-widths for transformer componentsmixed_precision— MixedPrecision. Mixed precision configuration
activation¶
weight¶
mixed_precision¶
BitConfig.Transformer.Activation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Activation bit-widths for transformer components |
query |
|
key |
|
value |
|
output |
|
head |
|
router |
|
ffn |
model_config¶
Activation bit-widths for transformer components
Parameters
query— int. Query activation bit-widthkey— int. Key activation bit-widthvalue— int. Value activation bit-widthoutput— int. Output activation bit-widthhead— int. Head activation bit-widthrouter— int. MoE router gate activation bit-widthffn— Ffn. FFN activation bit-widths (int shorthand sets all sublayers)
query¶
key¶
value¶
output¶
head¶
router¶
ffn¶
BitConfig.Transformer.Activation.Ffn¶
FFN activation bit-widths (int shorthand sets all sublayers)
Parameters
up— int. FFN up-projection activation bit-widthgate— int. FFN gate (SwiGLU) activation bit-widthdown— int. FFN down-projection activation bit-width
Attributes¶
| Name | Description |
|---|---|
up |
|
gate |
|
down |
up¶
gate¶
down¶
Methods¶
| Name | Description |
|---|---|
expand_int_shorthand |
expand_int_shorthand¶
BitConfig.Transformer.Weight¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Weight bit-widths for transformer components |
query |
|
key |
|
value |
|
output |
|
head |
|
router |
|
ffn |
model_config¶
Weight bit-widths for transformer components
Parameters
query— int. Query weight bit-widthkey— int. Key weight bit-widthvalue— int. Value weight bit-widthoutput— int. Output weight bit-widthhead— int. Head weight bit-widthrouter— int. MoE router gate weight bit-widthffn— Ffn. FFN weight bit-widths (int shorthand sets all sublayers)
query¶
key¶
value¶
output¶
head¶
router¶
ffn¶
BitConfig.Transformer.Weight.Ffn¶
FFN weight bit-widths (int shorthand sets all sublayers)
Parameters
up— int. FFN up-projection weight bit-widthgate— int. FFN gate (SwiGLU) weight bit-widthdown— int. FFN down-projection weight bit-width
Attributes¶
| Name | Description |
|---|---|
up |
|
gate |
|
down |
up¶
gate¶
down¶
Methods¶
| Name | Description |
|---|---|
expand_int_shorthand |
expand_int_shorthand¶
BitConfig.Transformer.MixedPrecision¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Mixed precision configuration |
weight |
|
activation |
model_config¶
Mixed precision configuration
Parameters
weight— Weight. Mixed precision configuration for weightsactivation— Activation. Per-layer activation mixed precision configuration
weight¶
activation¶
BitConfig.Transformer.MixedPrecision.Weight¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Mixed precision configuration for weights |
apply |
|
type_wise |
|
prune |
|
bit_2 |
|
bit_4 |
|
bit_8 |
|
importance_threshold_low |
|
importance_threshold_high |
model_config¶
Mixed precision configuration for weights
Parameters
apply— bool. If true, apply mixed-precision according to the specified bit-widthstype_wise— bool. Apply type-wise mixed precisionprune— float. Pruning ratiobit_2— float. Ratio of 2-bit quantizationbit_4— float. Ratio of 4-bit quantizationbit_8— float. Ratio of 8-bit quantizationimportance_threshold_low— float. Low importance thresholdimportance_threshold_high— float. High importance threshold
apply¶
apply: bool = Field(
default=False,
description="If true, apply mixed-precision according to the specified bit-widths",
)
type_wise¶
type_wise: bool = Field(
default=True,
alias="typeWise",
description="Apply type-wise mixed precision",
)
prune¶
bit_2¶
bit_2: float = Field(
default=0,
validation_alias=AliasChoices("bit_2", "bit2"),
description="Ratio of 2-bit quantization",
)
bit_4¶
bit_4: float = Field(
default=0,
validation_alias=AliasChoices("bit_4", "bit4"),
description="Ratio of 4-bit quantization",
)
bit_8¶
bit_8: float = Field(
default=1,
validation_alias=AliasChoices("bit_8", "bit8"),
description="Ratio of 8-bit quantization",
)
importance_threshold_low¶
importance_threshold_low: float = Field(
default=-1,
alias="importanceThreshold_low",
description="Low importance threshold",
)
importance_threshold_high¶
importance_threshold_high: float = Field(
default=-1,
alias="importanceThreshold_high",
description="High importance threshold",
)
BitConfig.Transformer.MixedPrecision.Activation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Per-layer activation mixed precision configuration |
apply |
|
ratio_16bit |
|
importance_threshold |
|
search_range |
model_config¶
Per-layer activation mixed precision configuration
Parameters
apply— bool. If true, apply per-layer activation mixed precisionratio_16bit— float. Ratio of layers assigned 16-bit (used when importanceThreshold < 0)importance_threshold— float. Normalized importance threshold for 16-bit assignment (negative = use ratio_16bit)search_range— int. Target layer range: -1=all layers, N=first N layers
apply¶
apply: bool = Field(
default=False,
description="If true, apply per-layer activation mixed precision",
)
ratio_16bit¶
ratio_16bit: float = Field(
default=0.45,
alias="ratio16Bit",
description="Ratio of layers assigned 16-bit (used when importanceThreshold < 0)",
)
importance_threshold¶
importance_threshold: float = Field(
default=-1,
alias="importanceThreshold",
description="Normalized importance threshold for 16-bit assignment (negative = use ratio_16bit)",
)
search_range¶
search_range: int = Field(
default=-1,
alias="searchRange",
description="Target layer range: -1=all layers, N=first N layers",
)
BitConfig.SaveInfo¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Bit allocation save/load configuration |
save_path |
|
load_path |
model_config¶
Bit allocation save/load configuration
Parameters
save_path— str. Path to save the bit allocation. If empty, not savedload_path— str. Path to load the bit allocation. If empty, not loaded
save_path¶
save_path: str = Field(
default="",
alias="savePath",
description="Path to save the bit allocation. If empty, not saved",
)
load_path¶
load_path: str = Field(
default="",
alias="loadPath",
description="Path to load the bit allocation. If empty, not loaded",
)
BitConfig.LayerOverrides¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Layer-specific bit-width override settings |
activation_16bits |
|
weight_16bits |
|
weight_8bits |
model_config¶
Layer-specific bit-width override settings
Parameters
activation_16bits— list[string]. Layer names to force 16-bit activationsweight_16bits— list[string]. Layer names to force 16-bit weightsweight_8bits— list[string]. Layer names to force 8-bit weights
activation_16bits¶
activation_16bits: List[str] = Field(
default=[],
alias="activation16Bits",
description="Layer names to force 16-bit activations",
)
weight_16bits¶
weight_16bits: List[str] = Field(
default=[],
alias="weight16Bits",
description="Layer names to force 16-bit weights",
)
weight_8bits¶
weight_8bits: List[str] = Field(
default=[],
alias="weight8Bits",
description="Layer names to force 8-bit weights",
)
HessianQuantConfig¶
Configuration for HessianQuant algorithm
Defines parameters controlling whether and how HessianQuant is applied during quantization, including layer-level inclusion/exclusion lists.
Parameters
apply— bool. If true, apply HessianQuanthessian_dtype— str. Storage dtype for the accumulated HessianQuant Hessian. bf16 halves its memory footprint (host RAM when the Hessian lives on CPU, VRAM when on GPU) — critical for large models such as MoE with thousands of expert FFN Hessians. Compute stays float32 regardless: the per-batch matmul and accumulation run in float32 and the solve upcasts back to float32; only the persistent accumulator is bfloat16. Accepted values:
"fp32" - Store the Hessian in float32. The default.
"bf16" - Store the Hessian in bfloat16 (half the memory).solver— str. HessianQuant solve algorithm used to compute per-layer integer weights. symmetric targets the output on the quantized input; the asymmetric solvers target the original-weight float output and differ in how they absorb the input mismatch. Accepted values:
"symmetric" - The default. min ||(W - Q) X_q||^2: sequential greedy rounding via a Cholesky-based Hessian solve.
"asymmetric_causal" - min ||W X_fp - Q X_q||^2: each column's input-mismatch residual is fed only to the columns after it, scaled by alpha.
"asymmetric_refit" - min ||W X_fp - Q X_q||^2: least-squares refit W^T = H^-1 G W^T of every column, then a symmetric solve around W.rescomp— bool. Enable residual-compensation correction. When true, each solver additionally applies an R correction term that compensates for accumulated weight drift from inter-block error propagation. Requires crossGram collection even for solver=symmetric.attributes— Attributes. HessianQuant algorithm attributes
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
hessian_dtype |
|
solver |
|
rescomp |
|
attributes |
model_config¶
apply¶
hessian_dtype¶
hessian_dtype: str = Field(
default="fp32",
alias="hessianDtype",
description="Storage dtype for the accumulated HessianQuant Hessian. bf16 halves its memory footprint (host RAM when the Hessian lives on CPU, VRAM when on GPU) — critical for large models such as MoE with thousands of expert FFN Hessians. Compute stays float32 regardless: the per-batch matmul and accumulation run in float32 and the solve upcasts back to float32; only the persistent accumulator is bfloat16.",
)
solver¶
solver: str = Field(
default="symmetric",
description="HessianQuant solve algorithm used to compute per-layer integer weights. symmetric targets the output on the quantized input; the asymmetric solvers target the original-weight float output and differ in how they absorb the input mismatch.",
)
rescomp¶
rescomp: bool = Field(
default=False,
description="Enable residual-compensation correction. When true, each solver additionally applies an R correction term that compensates for accumulated weight drift from inter-block error propagation. Requires crossGram collection even for solver=symmetric.",
)
attributes¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
HessianQuantConfig.Attributes¶
Attributes¶
| Name | Description |
|---|---|
model_config |
HessianQuant algorithm attributes |
act_order |
|
block_size |
|
perc_damp |
|
apply_layers |
|
exclude_layers |
|
alpha |
model_config¶
HessianQuant algorithm attributes
Parameters
act_order— bool. If true, use activation orderblock_size— int. Block size used for HessianQuantperc_damp— float. Percentage dampening factorapply_layers— List[str]. Layer names to apply HessianQuant. If empty, applies to all eligible layersexclude_layers— List[str]. Layer names to exclude from HessianQuantalpha— float. Scale of the secondary correction terms: the asymmetric_causal P term and the rescomp R term. symmetric and asymmetric_refit use it only through rescomp.
act_order¶
act_order: bool = Field(
default=True, alias="actOrder", description="If true, use activation order"
)
block_size¶
block_size: int = Field(
default=256,
alias="blockSize",
description="Block size used for HessianQuant",
)
perc_damp¶
perc_damp: float = Field(
default=0.01, alias="percDamp", description="Percentage dampening factor"
)
apply_layers¶
apply_layers: List[str] = Field(
default=[],
alias="applyLayers",
description="Layer names to apply HessianQuant. If empty, applies to all eligible layers",
)
exclude_layers¶
exclude_layers: List[str] = Field(
default=[],
alias="excludeLayers",
description="Layer names to exclude from HessianQuant",
)
alpha¶
alpha: float = Field(
default=0.25,
description="Scale of the secondary correction terms: the asymmetric_causal P term and the rescomp R term. symmetric and asymmetric_refit use it only through rescomp.",
)
BiasCorrectionConfig¶
Configuration for in-schedule integer bias correction
During weight quantization, measures the per-channel activation error of each quantized convolution (plain, depthwise or transposed; not grouped) against the original-weight float path over every calibration sample, folds the correction into the integer bias within its range, skipping a channel whose slope is too flat to invert, and patches the already-produced integer activations so downstream layers quantize against corrected outputs. Corrections propagate layer by layer in one pass; there is no damping rate and no iteration count. Under group-wise quantization both streams start each group from the float boundary batch, so the correction covers the error the group itself adds. A run the engine cannot drive leaf by leaf (expert aggregation, an external input narrower than its per-channel qparam) keeps the calibrated bias. A layer whose LUT window is re-chosen is measured on its accumulator, ahead of the window; any other layer on its requantized codes.
Parameters
apply— bool. If true, correct systematic per-layer quantization biasattributes— Attributes. Bias correction attributes
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
attributes |
model_config¶
apply¶
apply: bool = Field(
default=True,
description="If true, correct systematic per-layer quantization bias",
)
attributes¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
BiasCorrectionConfig.Attributes¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Bias correction attributes |
apply_layers |
|
exclude_layers |
model_config¶
Bias correction attributes
Parameters
apply_layers— List[str]. Layer names to apply bias correction. If empty, applies to all eligible layersexclude_layers— List[str]. Layer names to exclude from bias correction
apply_layers¶
apply_layers: List[str] = Field(
default=[],
alias="applyLayers",
description="Layer names to apply bias correction. If empty, applies to all eligible layers",
)
exclude_layers¶
exclude_layers: List[str] = Field(
default=[],
alias="excludeLayers",
description="Layer names to exclude from bias correction",
)
ModConfig¶
Configuration for Minimum Output Difference algorithm
Defines parameters controlling whether and how MOD is applied during quantization, including layer-level inclusion/exclusion lists.
Parameters
apply— bool. If true, apply MODattributes— Attributes. MOD algorithm attributes
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
attributes |
model_config¶
apply¶
attributes¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
ModConfig.Attributes¶
Attributes¶
| Name | Description |
|---|---|
model_config |
MOD algorithm attributes |
epochs |
|
warmup_epochs |
|
lr_min_ratio |
|
save_dir |
|
seed |
|
apply_layers |
|
exclude_layers |
|
mod_after_layer_name |
|
anchors |
|
use_xyxy |
|
learning_rates |
|
training |
|
loss |
|
post_processing |
model_config¶
MOD algorithm attributes
Parameters
epochs— int. Number of training epochswarmup_epochs— int. Number of warmup epochslr_min_ratio— float. Minimum learning rate ratiosave_dir— str. Directory to save MOD resultsseed— int. Random seed for MODapply_layers— List[str]. Layer names to apply MOD. If empty, applies to all eligible layersexclude_layers— List[str]. Layer names to exclude from MODmod_after_layer_name— str. Apply MOD after this layeranchors— List. Anchor configurations for detection models. Nested list of anchor box sizes for detection models. Structure: List[List[List[int]]] where outer list is per detection head (e.g. small/medium/large), middle list is anchors per head, inner list is [width, height]. Example: [[[12,16],[19,36],[40,28]], [[36,75],[76,55],[72,146]], [[142,110],[192,243],[459,401]]]use_xyxy— bool. Use XYXY format for bounding boxeslearning_rates— LearningRates. Learning rate configuration for MODtraining— Training. MOD training configurationloss— Loss. MOD loss configurationpost_processing— PostProcessing. Post-processing configuration for detection models
epochs¶
warmup_epochs¶
warmup_epochs: int = Field(
default=1, alias="warmupEpochs", description="Number of warmup epochs"
)
lr_min_ratio¶
lr_min_ratio: float = Field(
default=0.0001,
alias="lrMinRatio",
description="Minimum learning rate ratio",
)
save_dir¶
seed¶
apply_layers¶
apply_layers: List[str] = Field(
default=[],
alias="applyLayers",
description="Layer names to apply MOD. If empty, applies to all eligible layers",
)
exclude_layers¶
exclude_layers: List[str] = Field(
default=[],
alias="excludeLayers",
description="Layer names to exclude from MOD",
)
mod_after_layer_name¶
mod_after_layer_name: str = Field(
default="",
alias="modAfterLayerName",
description="Apply MOD after this layer",
)
anchors¶
anchors: Any = Field(
default=[],
description="Anchor configurations for detection models. Nested list of anchor box sizes for detection models. Structure: List[List[List[int]]] where outer list is per detection head (e.g. small/medium/large), middle list is anchors per head, inner list is [width, height]. Example: [[[12,16],[19,36],[40,28]], [[36,75],[76,55],[72,146]], [[142,110],[192,243],[459,401]]]",
)
use_xyxy¶
use_xyxy: bool = Field(
default=False,
alias="useXYXY",
description="Use XYXY format for bounding boxes",
)
learning_rates¶
training¶
loss¶
post_processing¶
ModConfig.Attributes.LearningRates¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Learning rate configuration for MOD |
act_scale |
|
zeropoint |
|
weight_scale |
|
weight |
|
bias |
model_config¶
Learning rate configuration for MOD
Parameters
act_scale— float. Learning rate for activation scalezeropoint— float. Learning rate for zeropointweight_scale— float. Learning rate for weight scaleweight— float. Learning rate for weightbias— float. Learning rate for bias
act_scale¶
act_scale: float = Field(
default=0.0,
alias="actScale",
description="Learning rate for activation scale",
)
zeropoint¶
weight_scale¶
weight_scale: float = Field(
default=0.0,
alias="weightScale",
description="Learning rate for weight scale",
)
weight¶
bias¶
ModConfig.Attributes.Training¶
Attributes¶
| Name | Description |
|---|---|
model_config |
MOD training configuration |
batch_size |
|
q_drop |
|
quantize_weight |
|
weight_scale_init |
|
downresol_mode |
|
scheduler_type |
model_config¶
MOD training configuration
Parameters
batch_size— int. Batch size for MOD trainingq_drop— float. Quantization drop probabilityquantize_weight— bool. Whether to quantize weightsweight_scale_init— str. Weight scale initialization methoddownresol_mode— str. Downresolution modescheduler_type— str. LR scheduler type
batch_size¶
q_drop¶
quantize_weight¶
quantize_weight: bool = Field(
default=True,
alias="quantizeWeight",
description="Whether to quantize weights",
)
weight_scale_init¶
weight_scale_init: str = Field(
default="MinMax",
alias="weightScaleInit",
description="Weight scale initialization method",
)
downresol_mode¶
downresol_mode: str = Field(
default="STE", alias="downresolMode", description="Downresolution mode"
)
scheduler_type¶
scheduler_type: str = Field(
default="Cosine", alias="schedulerType", description="LR scheduler type"
)
ModConfig.Attributes.Loss¶
Attributes¶
| Name | Description |
|---|---|
model_config |
MOD loss configuration |
type |
|
use_outputs |
|
kl_temperature |
|
recon_prob |
|
recon_coeff |
|
lambda_0 |
|
lambda_1 |
|
lambda_2 |
|
lambda_3 |
|
custom_loss_jit_path |
model_config¶
MOD loss configuration
Parameters
type— str. Loss type (MSE, KL, etc.)use_outputs— bool. Use model outputs for loss computationkl_temperature— float. KL divergence temperaturerecon_prob— float. Reconstruction probabilityrecon_coeff— float. Reconstruction coefficientlambda_0— float. Loss weight lambda_0lambda_1— float. Loss weight lambda_1lambda_2— float. Loss weight lambda_2lambda_3— float. Loss weight lambda_3custom_loss_jit_path— str. Path to custom JIT-compiled loss function. Refer to /workspace/quantizer/pyutils/mel.pt
type¶
use_outputs¶
use_outputs: bool = Field(
default=False,
alias="useOutputs",
description="Use model outputs for loss computation",
)
kl_temperature¶
kl_temperature: float = Field(
default=1.0,
alias="KLTemperature",
description="KL divergence temperature",
)
recon_prob¶
recon_prob: float = Field(
default=1.0, alias="reconProb", description="Reconstruction probability"
)
recon_coeff¶
recon_coeff: float = Field(
default=1.0,
alias="reconCoeff",
description="Reconstruction coefficient",
)
lambda_0¶
lambda_0: float = Field(
default=1.0,
validation_alias=AliasChoices("lambda_0", "lambda0"),
description="Loss weight lambda_0",
)
lambda_1¶
lambda_1: float = Field(
default=1.0,
validation_alias=AliasChoices("lambda_1", "lambda1"),
description="Loss weight lambda_1",
)
lambda_2¶
lambda_2: float = Field(
default=1.0,
validation_alias=AliasChoices("lambda_2", "lambda2"),
description="Loss weight lambda_2",
)
lambda_3¶
lambda_3: float = Field(
default=1.0,
validation_alias=AliasChoices("lambda_3", "lambda3"),
description="Loss weight lambda_3",
)
custom_loss_jit_path¶
custom_loss_jit_path: str = Field(
default="",
alias="customLossJITPath",
description="Path to custom JIT-compiled loss function. Refer to /workspace/quantizer/pyutils/mel.pt",
)
ModConfig.Attributes.PostProcessing¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Post-processing configuration for detection models |
post |
|
box_conf_thres |
|
box_iou_thres |
model_config¶
Post-processing configuration for detection models
Parameters
post— str. Post-processing typebox_conf_thres— float. Box confidence thresholdbox_iou_thres— float. Box IoU threshold
post¶
box_conf_thres¶
box_conf_thres: float = Field(
default=0, alias="boxConfThres", description="Box confidence threshold"
)
box_iou_thres¶
LlmConfig¶
Configuration for Large Language Model (LLM) compilation
Defines LLM-specific settings including sequence lengths, cache configurations, and runtime parameters for efficient LLM inference.
Parameters
apply— bool. If True, apply LLM-specific configurationsnpu_parallel_degree— int. Number of NPU partitions for FFN tensor parallelism (1 = disabled). Applied before OptimizeFFN if both active.attributes— Attributes. LLM attributes configuration
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
npu_parallel_degree |
|
attributes |
model_config¶
apply¶
npu_parallel_degree¶
npu_parallel_degree: int = Field(
default=1,
alias="npuParallelDegree",
description="Number of NPU partitions for FFN tensor parallelism (1 = disabled). Applied before OptimizeFFN if both active.",
)
attributes¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
LlmConfig.Attributes¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LLM attributes configuration |
max_data_length |
|
max_sequence_length |
|
max_cache_length |
|
max_core_data_length |
|
calibration |
|
runtime |
|
debug |
model_config¶
LLM attributes configuration
Parameters
max_data_length— int. Maximum data lengthmax_sequence_length— int. Maximum sequence lengthmax_cache_length— int. Maximum cache lengthmax_core_data_length— int. Maximum core data lengthcalibration— Calibration. LLM calibration settingsruntime— Runtime. LLM runtime settingsdebug— Debug. LLM debug settings
max_data_length¶
max_data_length: int = Field(
default=4096, alias="maxDataLength", description="Maximum data length"
)
max_sequence_length¶
max_sequence_length: int = Field(
default=4096,
alias="maxSequenceLength",
description="Maximum sequence length",
)
max_cache_length¶
max_cache_length: int = Field(
default=4096, alias="maxCacheLength", description="Maximum cache length"
)
max_core_data_length¶
max_core_data_length: int = Field(
default=128,
alias="maxCoreDataLength",
description="Maximum core data length",
)
calibration¶
runtime¶
debug¶
LlmConfig.Attributes.Calibration¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LLM calibration settings |
random_seq_length |
|
use_full_seq_length |
model_config¶
LLM calibration settings
Parameters
random_seq_length— int. Random sequence length used for calibrationuse_full_seq_length— bool. If True, use the full sequence length for calibration
random_seq_length¶
random_seq_length: int = Field(
default=80,
alias="randomSeqLength",
description="Random sequence length used for calibration",
)
use_full_seq_length¶
use_full_seq_length: bool = Field(
default=False,
alias="useFullSeqLength",
description="If True, use the full sequence length for calibration",
)
LlmConfig.Attributes.Runtime¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LLM runtime settings |
use_global_core |
|
batch_size |
|
npu_core_ids |
|
dynamic_rope |
|
dynamic_mask |
model_config¶
LLM runtime settings
Parameters
use_global_core— bool. If True, use a global corebatch_size— int. Batch sizenpu_core_ids— List[int]. List of NPU core IDsdynamic_rope— bool. If True, enable dynamic RoPE (rotary position embedding)dynamic_mask— bool. If True, enable dynamic mask (attention mask as runtime input)
use_global_core¶
use_global_core: bool = Field(
default=False,
alias="useGlobalCore",
description="If True, use a global core",
)
batch_size¶
npu_core_ids¶
npu_core_ids: List[int] = Field(
default=[0], alias="npuCoreIds", description="List of NPU core IDs"
)
dynamic_rope¶
dynamic_rope: bool = Field(
default=False,
alias="dynamicRope",
description="If True, enable dynamic RoPE (rotary position embedding)",
)
dynamic_mask¶
dynamic_mask: bool = Field(
default=False,
alias="dynamicMask",
description="If True, enable dynamic mask (attention mask as runtime input)",
)
LlmConfig.Attributes.Debug¶
Attributes¶
| Name | Description |
|---|---|
model_config |
LLM debug settings |
apply |
|
batch_debug_bundle_size |
model_config¶
LLM debug settings
Parameters
apply— bool. Enable LLM debug modebatch_debug_bundle_size— int. Batch debug bundle size
apply¶
batch_debug_bundle_size¶
batch_debug_bundle_size: int = Field(
default=0,
alias="batchDebugBundleSize",
description="Batch debug bundle size",
)
MoeConfig¶
Sparse MoE expert-selection configuration (calibration only)
Controls which experts are calibrated inside SparseMoe modules. selectionMode is a calibration-only knob: it picks which experts collect statistics (and, for TopK, on which tokens). It does NOT change inference routing — the forward path always routes the router's top-K experts regardless of this setting. scoreThreshold is only used when selectionMode is Threshold.
Parameters
selection_mode— int. Expert selection mode index (calibration only; inference is always top-K):
0: TopK - Calibrate only the router's top-K experts per token (matches inference routing).
1: All - Calibrate every expert on the full sequence.
2: Threshold - Calibrate all experts whose routing score exceeds scoreThreshold.score_threshold— float. Routing score threshold used when selectionMode is Threshold (calibration only)
Attributes¶
| Name | Description |
|---|---|
model_config |
|
selection_mode_list |
|
selection_mode |
|
score_threshold |
model_config¶
selection_mode_list¶
selection_mode_list: List[str] = Field(
default=["TopK", "All", "Threshold"], alias="selectionModeList"
)
selection_mode¶
score_threshold¶
score_threshold: float = Field(
default=0.0,
alias="scoreThreshold",
description="Routing score threshold used when selectionMode is Threshold (calibration only)",
ge=0,
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
EquivalentTransformationConfig¶
Configuration for equivalent transformation techniques
Defines parameters for various equivalent transformation methods including NormConv, QK smoothing, and rotation matrices for improved quantization.
Parameters
seed— int. Random seed for transformationapply_hadamard_rotation_matrix— bool. Apply Hadamard rotation matrixnorm_conv— NormConv. NormConv equivalent transformationqk— Qk. QK smoothing transformationud— Ud. UD transformationvo— Vo. VO transformationfeed_forward_multi_lut— FeedForwardMultiLut. Feed-forward multi-LUT transformationspin_r1— SpinR1. SpinR1 rotation transformationhead_out_ch_rotation— HeadOutChRotation. Head output channel rotation transformationin_rotation— InRotation. Input rotation transformationspin_r2— SpinR2. SpinR2 rotation transformationqk_rotation— QkRotation. QK rotation transformationflatten_quant— FlattenQuant. Flatten quantization transformationoptimize_ffn— OptimizeFfn. FFN optimization
Attributes¶
| Name | Description |
|---|---|
model_config |
|
seed |
|
apply_hadamard_rotation_matrix |
|
norm_conv |
|
qk |
|
ud |
|
vo |
|
feed_forward_multi_lut |
|
spin_r1 |
|
head_out_ch_rotation |
|
in_rotation |
|
spin_r2 |
|
qk_rotation |
|
flatten_quant |
|
optimize_ffn |
model_config¶
seed¶
apply_hadamard_rotation_matrix¶
apply_hadamard_rotation_matrix: bool = Field(
default=True,
alias="applyHadamardRotationMatrix",
description="Apply Hadamard rotation matrix",
)
norm_conv¶
qk¶
ud¶
vo¶
feed_forward_multi_lut¶
feed_forward_multi_lut: FeedForwardMultiLut = Field(
default_factory=FeedForwardMultiLut, alias="FeedForwardMultiLUT"
)
spin_r1¶
head_out_ch_rotation¶
head_out_ch_rotation: HeadOutChRotation = Field(
default_factory=HeadOutChRotation, alias="HeadOutChRotation"
)
in_rotation¶
spin_r2¶
qk_rotation¶
flatten_quant¶
optimize_ffn¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
EquivalentTransformationConfig.NormConv¶
Attributes¶
| Name | Description |
|---|---|
model_config |
NormConv equivalent transformation |
apply |
|
learn |
|
smoothing_factor |
|
min_gamma |
|
max_gamma |
model_config¶
NormConv equivalent transformation
Parameters
apply— bool. Apply NormConv transformationlearn— bool. Learn transformation parameterssmoothing_factor— float. Smoothing factormin_gamma— float. Minimum gamma valuemax_gamma— float. Maximum gamma value
apply¶
learn¶
smoothing_factor¶
smoothing_factor: float = Field(
default=0.5, alias="smoothingFactor", description="Smoothing factor"
)
min_gamma¶
max_gamma¶
EquivalentTransformationConfig.Qk¶
Attributes¶
| Name | Description |
|---|---|
model_config |
QK smoothing transformation |
apply |
|
smoothing_factor |
|
min_gamma |
|
max_gamma |
model_config¶
QK smoothing transformation
Parameters
apply— bool. Apply QK transformationsmoothing_factor— float. Smoothing factormin_gamma— float. Minimum gamma valuemax_gamma— float. Maximum gamma value
apply¶
smoothing_factor¶
smoothing_factor: float = Field(
default=0.5, alias="smoothingFactor", description="Smoothing factor"
)
min_gamma¶
max_gamma¶
EquivalentTransformationConfig.Ud¶
Attributes¶
| Name | Description |
|---|---|
model_config |
UD transformation |
apply |
|
learn |
|
smoothing_factor |
|
min_gamma |
|
max_gamma |
model_config¶
UD transformation
Parameters
apply— bool. Apply UD transformationlearn— bool. Learn transformation parameterssmoothing_factor— float. Smoothing factormin_gamma— float. Minimum gamma valuemax_gamma— float. Maximum gamma value
apply¶
learn¶
smoothing_factor¶
smoothing_factor: float = Field(
default=0.5, alias="smoothingFactor", description="Smoothing factor"
)
min_gamma¶
max_gamma¶
EquivalentTransformationConfig.Vo¶
Attributes¶
| Name | Description |
|---|---|
model_config |
VO transformation |
apply |
|
smoothing_factor |
|
min_gamma |
|
max_gamma |
model_config¶
VO transformation
Parameters
apply— bool. Apply VO transformationsmoothing_factor— float. Smoothing factormin_gamma— float. Minimum gamma valuemax_gamma— float. Maximum gamma value
apply¶
smoothing_factor¶
smoothing_factor: float = Field(
default=0.5, alias="smoothingFactor", description="Smoothing factor"
)
min_gamma¶
max_gamma¶
EquivalentTransformationConfig.FeedForwardMultiLut¶
Feed-forward multi-LUT transformation
Parameters
apply— bool. Apply feed-forward multi-LUT transformationbreakpoints— List[float]. Breakpoints for multi-LUT
Attributes¶
| Name | Description |
|---|---|
apply |
|
breakpoints |
apply¶
breakpoints¶
breakpoints: List[float] = Field(
default=[-8.0, -4.0, 0], description="Breakpoints for multi-LUT"
)
EquivalentTransformationConfig.SpinR1¶
Attributes¶
| Name | Description |
|---|---|
model_config |
SpinR1 rotation transformation |
apply |
|
matrix_path |
|
insert_output_conv |
model_config¶
SpinR1 rotation transformation
Parameters
apply— bool. Apply SpinR1 transformationmatrix_path— str. Path to rotation matrix fileinsert_output_conv— bool. Insert an identity output conv when the residual stream ends without a head conv
apply¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
insert_output_conv¶
insert_output_conv: bool = Field(
default=False,
alias="insertOutputConv",
description="Insert an identity output conv when the residual stream ends without a head conv",
)
EquivalentTransformationConfig.HeadOutChRotation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Head output channel rotation transformation |
apply |
|
matrix_path |
model_config¶
Head output channel rotation transformation
Parameters
apply— bool. Apply head output channel rotationmatrix_path— str. Path to rotation matrix file
apply¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
EquivalentTransformationConfig.InRotation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Input rotation transformation |
apply |
|
matrix_path |
|
input_names |
model_config¶
Input rotation transformation
Parameters
apply— bool. Apply input rotationmatrix_path— str. Path to rotation matrix fileinput_names— List[str]. Names of the input layers to rotate
apply¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
input_names¶
input_names: List[str] = Field(
default=[],
alias="inputNames",
description="Names of the input layers to rotate",
)
EquivalentTransformationConfig.SpinR2¶
Attributes¶
| Name | Description |
|---|---|
model_config |
SpinR2 rotation transformation |
apply |
|
learn |
|
matrix_path |
model_config¶
SpinR2 rotation transformation
Parameters
apply— bool. Apply SpinR2 transformationlearn— bool. Learn rotation matrixmatrix_path— str. Path to rotation matrix file
apply¶
learn¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
EquivalentTransformationConfig.QkRotation¶
Attributes¶
| Name | Description |
|---|---|
model_config |
QK rotation transformation |
apply |
|
matrix_path |
model_config¶
QK rotation transformation
Parameters
apply— bool. Apply QK rotation transformationmatrix_path— str. Path to rotation matrix file
apply¶
matrix_path¶
matrix_path: str = Field(
default="", alias="matrixPath", description="Path to rotation matrix file"
)
EquivalentTransformationConfig.FlattenQuant¶
Attributes¶
| Name | Description |
|---|---|
model_config |
Flatten quantization transformation |
apply |
|
learn |
|
apply_threshold |
|
max_overhead |
model_config¶
Flatten quantization transformation
Parameters
apply— bool. Apply flatten quantizationlearn— bool. Learn flattening parametersapply_threshold— float. Threshold for applying flatten quantizationmax_overhead— float. Maximum overhead allowed for flattening
apply¶
learn¶
apply_threshold¶
apply_threshold: float = Field(
default=0.33,
alias="applyThreshold",
description="Threshold for applying flatten quantization",
)
max_overhead¶
max_overhead: float = Field(
default=0.02,
alias="maxOverhead",
description="Maximum overhead allowed for flattening",
)
EquivalentTransformationConfig.OptimizeFfn¶
Attributes¶
| Name | Description |
|---|---|
model_config |
FFN optimization |
apply |
|
ch_per_ffn |
model_config¶
FFN optimization
Parameters
apply— bool. Apply FFN optimizationch_per_ffn— int. Optimize FFN split (-1 for auto)
apply¶
ch_per_ffn¶
ch_per_ffn: int = Field(
default=-1, alias="chPerFFN", description="Optimize FFN split (-1 for auto)"
)
SearchWeightScaleConfig¶
Configuration for weight scale search
Defines which transformer components should have their weight scales searched for optimal quantization.
Parameters
apply— bool. If true, apply weight scale searchtransformer— Transformer. Transformer components for weight scale search
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
transformer |
model_config¶
apply¶
transformer¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
SearchWeightScaleConfig.Transformer¶
Transformer components for weight scale search
Parameters
query— bool. Search weight scale for querykey— bool. Search weight scale for keyvalue— bool. Search weight scale for valueout— bool. Search weight scale for outputffn— bool. Search weight scale for FFN
Attributes¶
| Name | Description |
|---|---|
query |
|
key |
|
value |
|
out |
|
ffn |
query¶
key¶
value¶
out¶
ffn¶
LoadScaleConfig¶
QAT activation scale loading configuration
Loads activation scales from .safetensors, torch.save tensor dictionaries, or libtorch OutputArchive .pt/.pth files. Scalars select per-layer quantization; channel vectors select per-channel quantization. Invalid imports fail compilation. Sigmoid/Compare imports must match the hardware-fixed scale and zero-point (within 1e-6 relative scale tolerance). actScaleMin and enabled minClipRatio clipping take precedence over imported values; applied scales and clipping adjustments are logged, with quantization ranges updated to match. NOTE: entries is stored as raw JSON (type: list) because the generator does not support list[CustomStruct]. The schema records the field shape; parsing is done manually in activation_scale_loader::read.
Parameters
apply— bool. If true, load and apply QAT activation scales; invalid imports are errorsentries— List. List of file entries. Each entry is a dict: { path: str, scales: [ { key: str, layers: [str], zeropointKey?: str } ] } path: .safetensors or .pt/.pth file; key: floating scalar or channel vector. zeropointKey: optional I32/I64 tensor; .actScale keys automatically use a sibling .zeropoint if present, otherwise zero. Scalar zeropoints broadcast. layers: compiler layer names whose independent activation scales are overridden.
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
entries |
model_config¶
apply¶
apply: bool = Field(
default=False,
description="If true, load and apply QAT activation scales; invalid imports are errors",
)
entries¶
entries: Any = Field(
default=[],
description="List of file entries. Each entry is a dict: { path: str, scales: [ { key: str, layers: [str], zeropointKey?: str } ] } path: .safetensors or .pt/.pth file; key: floating scalar or channel vector. zeropointKey: optional I32/I64 tensor; .actScale keys automatically use a sibling .zeropoint if present, otherwise zero. Scalar zeropoints broadcast. layers: compiler layer names whose independent activation scales are overridden.",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
RuntimeOptions¶
Runtime options for compilation
Contains runtime-specific settings like version info and cache options.
Parameters
version— str. Compiler version string (e.g., 0.0.0)deterministic_algorithms— bool. Run LUT grouping, the only compile stage found to vary between runs on one GPU, with PyTorch deterministic algorithms; false gives a faster grouping whose result can differ between runs
Attributes¶
| Name | Description |
|---|---|
model_config |
|
version |
|
deterministic_algorithms |
model_config¶
version¶
deterministic_algorithms¶
deterministic_algorithms: bool = Field(
default=True,
alias="deterministicAlgorithms",
description="Run LUT grouping, the only compile stage found to vary between runs on one GPU, with PyTorch deterministic algorithms; false gives a faster grouping whose result can differ between runs",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
SaveSampleConfig¶
Sample data generation and saving configuration
Parameters
apply— bool. Enable sample data savingmode— str. Inference mode: infer (standard) or inferWithCache (LLM cache models)batch_size— int. Number of inference batches to generatebatch_seq_lens— List. Per-batch step-wise sequence lengths for inferWithCache mode. e.g. [[80, 1], [240, 10]]save_folder— str. Output folder for sample datadtype— str. Data type for saved samples: float or int8
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
mode |
|
batch_size |
|
batch_seq_lens |
|
save_folder |
|
dtype |
model_config¶
apply¶
mode¶
mode: str = Field(
default="infer",
description="Inference mode: infer (standard) or inferWithCache (LLM cache models)",
)
batch_size¶
batch_size: int = Field(
default=1,
alias="batchSize",
description="Number of inference batches to generate",
)
batch_seq_lens¶
batch_seq_lens: Any = Field(
default=[],
alias="batchSeqLens",
description="Per-batch step-wise sequence lengths for inferWithCache mode. e.g. [[80, 1], [240, 10]]",
)
save_folder¶
save_folder: str = Field(
default="sampleInout",
alias="saveFolder",
description="Output folder for sample data",
)
dtype¶
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
ExtraOutputConfig¶
Promote named intermediate layers to additional model outputs
Applied after quantization, so the rest of the bundle is unchanged. Names are matched against the post-fusion graph, and an unknown one is an error.
Parameters
apply— bool. Promote the named layers. Off by default: promotion is an explicit opt-in.layers— List[str]. Layer names to promote; one that is already an output is left alone
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
layers |
model_config¶
apply¶
apply: bool = Field(
default=False,
description="Promote the named layers. Off by default: promotion is an explicit opt-in.",
)
layers¶
layers: List[str] = Field(
default=[],
description="Layer names to promote; one that is already an output is left alone",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
GroupWiseConfig¶
Group-wise streaming quantization configuration for large LLMs
Configuration for the group-wise (streaming) quantization pipeline used for LLMs that do not fit fully on GPU. Partitions the model into groups of transformer blocks and quantizes each group independently.
Parameters
apply— bool. Enable group-wise streaming quantization pipelinegroup_size— int. Group size in number of transformer blocks (0 = auto)gpu_budget_gb— float. GPU memory budget in GiB for group-wise execution (0 = auto-detect)gpu_safety_margin_gb— float. Safety margin in GiB subtracted from detected GPU budgetcache_dir— str. Directory for group-wise activation/state cache (empty = system temp)keep_cache— bool. Retain group-wise cache after run completes (for debugging)partition_policy— str. Partitioning policy (e.g. transformer_block, moe_expert_subgroup)expert_groups— List[List[int]]. MoE fallback partitioner: list of expert-index lists (List[List[int]]).retain_topology_weights— bool. Keep inflated FP weights across groups (false = release after each group for tight memory budgets).checkpoint— bool. Save per-group checkpoint so quantization can resume from the last completed group after a crash.
Attributes¶
| Name | Description |
|---|---|
model_config |
|
apply |
|
group_size |
|
gpu_budget_gb |
|
gpu_safety_margin_gb |
|
cache_dir |
|
keep_cache |
|
partition_policy |
|
expert_groups |
|
retain_topology_weights |
|
checkpoint |
model_config¶
apply¶
apply: bool = Field(
default=False, description="Enable group-wise streaming quantization pipeline"
)
group_size¶
group_size: int = Field(
default=0,
alias="groupSize",
description="Group size in number of transformer blocks (0 = auto)",
)
gpu_budget_gb¶
gpu_budget_gb: float = Field(
default=0,
alias="gpuBudgetGb",
description="GPU memory budget in GiB for group-wise execution (0 = auto-detect)",
)
gpu_safety_margin_gb¶
gpu_safety_margin_gb: float = Field(
default=4,
alias="gpuSafetyMarginGb",
description="Safety margin in GiB subtracted from detected GPU budget",
)
cache_dir¶
cache_dir: str = Field(
default="",
alias="cacheDir",
description="Directory for group-wise activation/state cache (empty = system temp)",
)
keep_cache¶
keep_cache: bool = Field(
default=False,
alias="keepCache",
description="Retain group-wise cache after run completes (for debugging)",
)
partition_policy¶
partition_policy: str = Field(
default="transformer_block",
alias="partitionPolicy",
description="Partitioning policy (e.g. transformer_block, moe_expert_subgroup)",
)
expert_groups¶
expert_groups: Any = Field(
default=[],
alias="expertGroups",
description="MoE fallback partitioner: list of expert-index lists (List[List[int]]).",
)
retain_topology_weights¶
retain_topology_weights: bool = Field(
default=True,
alias="retainTopologyWeights",
description="Keep inflated FP weights across groups (false = release after each group for tight memory budgets).",
)
checkpoint¶
checkpoint: bool = Field(
default=False,
description="Save per-group checkpoint so quantization can resume from the last completed group after a crash.",
)
Methods¶
| Name | Description |
|---|---|
with_updates |
Return a copy with updated fields. |
with_updates¶
Return a copy with updated fields.
CompileConfig¶
Unified compilation configuration for Mobilint MXQ compilation.
Attributes¶
model_config¶
model_paths¶
model_paths: List[str] = Field(
default=[], alias="modelPaths", description="Paths to model files"
)
calib_data_path¶
calib_data_path: List[str] = Field(
default=[], alias="calibDataPaths", description="Paths to calibration datasets"
)
save_paths¶
save_paths: List[str] = Field(
default=["./tmp.mxq"],
alias="savePaths",
description="Output MXQ filename/paths",
)
use_random_calib¶
use_random_calib: bool = Field(
default=False, alias="useRandomCalib", description="Use random calibration"
)
inference_scheme¶
inference_scheme: str = Field(
default="single", alias="inferenceScheme", description="NPU inference scheme"
)
cpu_offload¶
cpu_offload: bool = Field(
default=False,
alias="cpuOffload",
description="Enable CPU offload for unsupported operators",
)
force_npu_input_reposition¶
force_npu_input_reposition: bool = Field(
default=False,
alias="forceNpuInputReposition",
description="Force input reposition operations to run on NPU instead of CPU",
)
force_npu_output_reposition¶
force_npu_output_reposition: bool = Field(
default=False,
alias="forceNpuOutputReposition",
description="Force output reposition operations to run on NPU instead of CPU",
)
optimize_option¶
optimize_option: int = Field(
default=1,
alias="optimizeOption",
description="Compiler optimization selector",
ge=0,
)
buffer_mode¶
device¶
dtype¶
debug¶
trace¶
image_channels¶
image_channels: int = Field(
default=0,
alias="imageChannels",
description="Number of image channels (0 for auto-detect)",
)
config_version¶
config_version: str = Field(
default="1.0.0", alias="configVersion", description="Config schema version"
)
split_blocks¶
split_blocks: List[int] = Field(
default=[],
alias="splitBlocks",
description="Multi-MXQ split points by transformer block index",
)
split_parts¶
split_parts: int = Field(
default=0,
alias="splitParts",
description="Evenly split transformer blocks into N MXQ parts",
)
uint8_input¶
preprocessing¶
resource_management¶
resource_management: ResourceManagementConfig = Field(
default_factory=ResourceManagementConfig, alias="resourceManagement"
)
calibration¶
bit¶
hessian_quant¶
hessian_quant: HessianQuantConfig = Field(
default_factory=HessianQuantConfig, alias="hessianQuant"
)
bias_correction¶
bias_correction: BiasCorrectionConfig = Field(
default_factory=BiasCorrectionConfig, alias="biasCorrection"
)
mod¶
llm¶
moe¶
equivalent_transformation¶
equivalent_transformation: EquivalentTransformationConfig = Field(
default_factory=EquivalentTransformationConfig, alias="equivalentTransformation"
)
search_weight_scale¶
search_weight_scale: SearchWeightScaleConfig = Field(
default_factory=SearchWeightScaleConfig, alias="searchWeightScale"
)
load_scale¶
runtime_options¶
save_sample¶
extra_output¶
group_wise¶
Methods¶
| Name | Description |
|---|---|
with_uint8_input |
Return a copy with uint8_input settings enabled. |
with_preprocessing |
Return a copy with preprocessing settings enabled. |
with_hessian_quant |
Return a copy with hessian_quant settings enabled. |
with_bias_correction |
Return a copy with bias_correction settings enabled. |
with_mod |
Return a copy with mod settings enabled. |
with_llm |
Return a copy with llm settings enabled. |
with_search_weight_scale |
Return a copy with search_weight_scale settings enabled. |
with_load_scale |
Return a copy with load_scale settings enabled. |
with_save_sample |
Return a copy with save_sample settings enabled. |
with_extra_output |
Return a copy with extra_output settings enabled. |
with_group_wise |
Return a copy with group_wise settings enabled. |
from_file |
Load config from YAML or JSON file. |
from_preset |
Load config from a preset. |
to_file |
Save config to YAML or JSON file. |
with_uint8_input¶
Return a copy with uint8_input settings enabled.
with_preprocessing¶
Return a copy with preprocessing settings enabled.
with_hessian_quant¶
Return a copy with hessian_quant settings enabled.
with_bias_correction¶
Return a copy with bias_correction settings enabled.
with_mod¶
Return a copy with mod settings enabled.
with_llm¶
Return a copy with llm settings enabled.
with_search_weight_scale¶
Return a copy with search_weight_scale settings enabled.
with_load_scale¶
Return a copy with load_scale settings enabled.
with_save_sample¶
Return a copy with save_sample settings enabled.
with_extra_output¶
Return a copy with extra_output settings enabled.
with_group_wise¶
Return a copy with group_wise settings enabled.
from_file¶
Load config from YAML or JSON file.
from_preset¶
Load config from a preset.
to_file¶
Save config to YAML or JSON file.