Skip to content

qbcompiler.configs.models

Auto-generated Pydantic models from config_schema.yaml.

Attributes

Name Description
SCHEMA_VERSION

SCHEMA_VERSION

SCHEMA_VERSION = "1.0.0"

Uint8InputConfig

class Uint8InputConfig(BaseModel)

Configuration for uint8 input handling

Defines whether inputs should be treated as uint8 and which specific inputs to apply this to.

Parameters

  • apply — bool. If true, treat specified inputs as uint8
  • inputs — List[str]. List of input names to treat as uint8. If empty and apply is true, applies to all inputs
  • division_factor — float. Division factor for uint8 to float conversion (e.g., 255.0 for [0,1], 127.5 for [0,2])

Attributes

Name Description
model_config
apply
inputs
division_factor

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False, description="If true, treat specified inputs as uint8"
)

inputs

inputs: List[str] = Field(
    default=[],
    description="List of input names to treat as uint8. If empty and apply is true, applies to all inputs",
)

division_factor

division_factor: float = Field(
    default=255.0,
    alias="divisionFactor",
    description="Division factor for uint8 to float conversion (e.g., 255.0 for [0,1], 127.5 for [0,2])",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "Uint8InputConfig"

Return a copy with updated fields.

PreprocessingConfig

class PreprocessingConfig(BaseModel)

Configuration for input preprocessing pipeline

Defines preprocessing operations to be applied to model inputs, including operations like resize, normalize, color conversion, etc.

Parameters

  • apply — bool. If true, apply preprocessing pipeline
  • auto_convert_format — bool. If true, automatically convert input format
  • pipeline — List[Dict[str, Any]]. List of preprocessing operations to apply globally
  • input_configs — Dict[str, Any]. Per-input preprocessing configurations. Keys are input names

Attributes

Name Description
model_config
apply
auto_convert_format
pipeline
input_configs

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False, description="If true, apply preprocessing pipeline"
)

auto_convert_format

auto_convert_format: bool = Field(
    default=False,
    alias="autoConvertFormat",
    description="If true, automatically convert input format",
)

pipeline

pipeline: Any = Field(
    default=[], description="List of preprocessing operations to apply globally"
)

input_configs

input_configs: Any = Field(
    default={},
    alias="inputConfigs",
    description="Per-input preprocessing configurations. Keys are input names",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "PreprocessingConfig"

Return a copy with updated fields.

ResourceManagementConfig

class ResourceManagementConfig(BaseModel)

Configuration for resource management during model compilation

Controls GPU and memory management settings during the compilation process.

Parameters

  • weight_dtype — str. Weight data type for calibration (e.g., 'float32', 'float16')
  • use_gpu_only_for_calibration — bool. If True, use GPU only during the calibration phase
  • weight_memory — WeightMemory. Weight memory management configuration

Attributes

Name Description
model_config
weight_dtype
use_gpu_only_for_calibration
weight_memory

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

weight_dtype

weight_dtype: str = Field(
    default="float32",
    alias="weightDtype",
    description="Weight data type for calibration (e.g., 'float32', 'float16')",
)

use_gpu_only_for_calibration

use_gpu_only_for_calibration: bool = Field(
    default=True,
    alias="useGPUOnlyForCalibration",
    description="If True, use GPU only during the calibration phase",
)

weight_memory

weight_memory: WeightMemory = Field(
    default_factory=WeightMemory, alias="weightMemory"
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "ResourceManagementConfig"

Return a copy with updated fields.

ResourceManagementConfig.WeightMemory

class WeightMemory(BaseModel)

Weight memory management configuration

Parameters

  • method — int. Weight memory management method index:
    0: DeleteFloat - Delete float weights after quantization.
    1: SaveFloat - Save float weights to disk.
    2: MoveFloat - Move float weights to CPU.
    3: KeepFloat - Keep float weights in memory.
    4: KeepAll - Keep all weights in memory.

Attributes

Name Description
method_list
method

method_list

method_list: List[str] = Field(
    default=["DeleteFloat", "SaveFloat", "MoveFloat", "KeepFloat", "KeepAll"],
    alias="methodList",
)

method

method: int = Field(default=0, alias="method")

CalibrationConfig

class CalibrationConfig(BaseModel)

Configuration for calibration during quantization

Defines calibration and quantization parameterization used to derive activation/weight scales and related statistics during quantized compilation.

Parameters

  • method — int. Calibration method index:
    0: WChALayer - Weight per-channel, Activation per-layer.
    1: WChAMulti - Weight per-channel, Activation multi-layer.
    2: WChALayerZeropoint - Weight per-channel, Activation per-layer with zeropoint.
    3: WChAMultiZeropoint - Weight per-channel, Activation multi-layer with zeropoint.
  • output — int. Output quantization type index:
    0: Layer - Per-layer quantization.
    1: Ch - Per-channel quantization.
    2: Sigmoid - Sigmoid-based quantization.
  • mode — int. Quantization mode index:
    0: Max - Maximum value calibration.
    1: MaxPercentile - Maximum percentile calibration.
    2: Histogram - Histogram-based calibration.
  • act_scale_min — float. Minimum allowed activation scale (lower bound clamp)
  • act16_scale_min — float. Minimum 16-bit activation scale (actScaleMin / 256)
  • weight_scale_min — float. Minimum allowed weight scale (lower bound clamp)
  • weight16_scale_min — float. Minimum 16-bit weight scale (weightScaleMin / 256)
  • min_clip_ratio — float. Minimum clip ratio constraint applied during calibration
  • max_calib_data_size — int. Maximum number of calibration samples kept after loading or generation
  • max_sample_size_for_quant_scheme — int. Maximum number of calibration samples used per quant scheme stage
  • max_percentile — MaxPercentile. MaxPercentile mode configuration
  • fast_dist — FastDist. Fast distribution calibration configuration
  • histogram — Histogram. Histogram-based calibration configuration
  • layer_overrides — LayerOverrides. Layer-specific override settings for calibration
  • statistics — Statistics. Statistics save/load configuration with percentile selection
  • group_lut — GroupLut. LUT grouping algorithm configuration
  • optimize_lut — OptimizeLut. LUT optimization configuration

Attributes

Name Description
model_config
method_list
method
output_list
output
mode_list
mode
act_scale_min
act16_scale_min
weight_scale_min
weight16_scale_min
min_clip_ratio
max_calib_data_size
max_sample_size_for_quant_scheme
max_percentile
fast_dist
histogram
layer_overrides
statistics
group_lut
optimize_lut

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

method_list

method_list: List[str] = Field(
    default=["WChALayer", "WChAMulti", "WChALayerZeropoint", "WChAMultiZeropoint"],
    alias="methodList",
)

method

method: int = Field(default=1, alias="method")

output_list

output_list: List[str] = Field(
    default=["Layer", "Ch", "Sigmoid"], alias="outputList"
)

output

output: int = Field(default=0, alias="output")

mode_list

mode_list: List[str] = Field(
    default=["Max", "MaxPercentile", "Histogram"], alias="modeList"
)

mode

mode: int = Field(default=1, alias="mode")

act_scale_min

act_scale_min: float = Field(
    default=0.0005,
    alias="actScaleMin",
    description="Minimum allowed activation scale (lower bound clamp)",
    ge=0,
    le=1,
)

act16_scale_min

act16_scale_min: float = Field(
    default=1.953125e-06,
    alias="act16ScaleMin",
    description="Minimum 16-bit activation scale (actScaleMin / 256)",
)

weight_scale_min

weight_scale_min: float = Field(
    default=1e-06,
    alias="weightScaleMin",
    description="Minimum allowed weight scale (lower bound clamp)",
    ge=0,
    le=1,
)

weight16_scale_min

weight16_scale_min: float = Field(
    default=3.90625e-09,
    alias="weight16ScaleMin",
    description="Minimum 16-bit weight scale (weightScaleMin / 256)",
)

min_clip_ratio

min_clip_ratio: float = Field(
    default=-1,
    alias="minClipRatio",
    description="Minimum clip ratio constraint applied during calibration",
    ge=-1,
    le=1,
)

max_calib_data_size

max_calib_data_size: int = Field(
    default=-1,
    alias="maxCalibDataSize",
    description="Maximum number of calibration samples kept after loading or generation",
    ge=-1,
)

max_sample_size_for_quant_scheme

max_sample_size_for_quant_scheme: int = Field(
    default=16,
    alias="maxSampleSizeForQuantScheme",
    description="Maximum number of calibration samples used per quant scheme stage",
    ge=1,
)

max_percentile

max_percentile: MaxPercentile = Field(
    default_factory=MaxPercentile, alias="maxPercentile"
)

fast_dist

fast_dist: FastDist = Field(default_factory=FastDist, alias="fastDist")

histogram

histogram: Histogram = Field(default_factory=Histogram, alias="histogram")

layer_overrides

layer_overrides: LayerOverrides = Field(
    default_factory=LayerOverrides, alias="layerOverrides"
)

statistics

statistics: Statistics = Field(default_factory=Statistics, alias="statistics")

group_lut

group_lut: GroupLut = Field(default_factory=GroupLut, alias="groupLut")

optimize_lut

optimize_lut: OptimizeLut = Field(default_factory=OptimizeLut, alias="optimizeLut")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "CalibrationConfig"

Return a copy with updated fields.

CalibrationConfig.MaxPercentile

class MaxPercentile(BaseModel)

Attributes

Name Description
model_config MaxPercentile mode configuration
percentile
topk_ratio
max_each
max_total
per_ch_divisor

model_config

model_config = ConfigDict(populate_by_name=True)

MaxPercentile mode configuration

Parameters

  • percentile — float. Percentile value for maxPercentile mode
  • topk_ratio — float. Top-k ratio used in maxPercentile mode
  • max_each — int. Maximum number of samples processed per iteration
  • max_total — int. Total maximum number of samples
  • per_ch_divisor — int. Divisor for per-channel buffer capacity (bufferCap = max(maxTotal / perChDivisor, maxEach))

percentile

percentile: float = Field(
    default=0.9999, description="Percentile value for maxPercentile mode"
)

topk_ratio

topk_ratio: float = Field(
    default=0.01,
    alias="topKRatio",
    description="Top-k ratio used in maxPercentile mode",
)

max_each

max_each: int = Field(
    default=128,
    alias="maxEach",
    description="Maximum number of samples processed per iteration",
)

max_total

max_total: int = Field(
    default=65536,
    alias="maxTotal",
    description="Total maximum number of samples",
)

per_ch_divisor

per_ch_divisor: int = Field(
    default=16,
    alias="perChDivisor",
    description="Divisor for per-channel buffer capacity (bufferCap = max(maxTotal / perChDivisor, maxEach))",
    ge=1,
)

CalibrationConfig.FastDist

class FastDist(BaseModel)

Attributes

Name Description
model_config Fast distribution calibration configuration
size_cali
kernel_size
stack_size

model_config

model_config = ConfigDict(populate_by_name=True)

Fast distribution calibration configuration

Parameters

  • size_cali — int.
  • kernel_size — int.
  • stack_size — int.

size_cali

size_cali: int = Field(default=100, alias="sizeCali")

kernel_size

kernel_size: int = Field(default=9, alias="kernelSize")

stack_size

stack_size: int = Field(default=32768, alias="stackSize")

CalibrationConfig.Histogram

class Histogram(BaseModel)

Attributes

Name Description
model_config Histogram-based calibration configuration
search_type_list
search_type
percentile
use_gpu
num_bins
num_samples
buffer_size
min_bin_width
search_percentile_min
search_percentile_max
num_search

model_config

model_config = ConfigDict(populate_by_name=True)

Histogram-based calibration configuration

Parameters

  • search_type — int. Search type for histogram calibration:
    0: Percentile.
    1: MSE.
    2: KL.
  • percentile — float. Percentile value for histogram calibration
  • use_gpu — bool. Use GPU for histogram computation
  • num_bins — int. Number of bins for histogram
  • num_samples — int. Number of samples for histogram calibration
  • buffer_size — int. Buffer size for histogram computation (-1 for auto)
  • min_bin_width — float. Minimum bin width for histogram
  • search_percentile_min — float. Minimum search percentile
  • search_percentile_max — float. Maximum search percentile
  • num_search — int. Number of search iterations

search_type_list

search_type_list: List[str] = Field(
    default=["Percentile", "MSE", "KL"], alias="searchTypeList"
)

search_type

search_type: int = Field(default=0, alias="searchType")

percentile

percentile: float = Field(
    default=0.9999, description="Percentile value for histogram calibration"
)

use_gpu

use_gpu: bool = Field(
    default=True,
    alias="useGPU",
    description="Use GPU for histogram computation",
)

num_bins

num_bins: int = Field(
    default=256, alias="numBins", description="Number of bins for histogram"
)

num_samples

num_samples: int = Field(
    default=128,
    alias="numSamples",
    description="Number of samples for histogram calibration",
)

buffer_size

buffer_size: int = Field(
    default=-1,
    alias="bufferSize",
    description="Buffer size for histogram computation (-1 for auto)",
)

min_bin_width

min_bin_width: float = Field(
    default=1e-06,
    alias="minBinWidth",
    description="Minimum bin width for histogram",
)

search_percentile_min

search_percentile_min: float = Field(
    default=0.9999,
    alias="searchPercentileMin",
    description="Minimum search percentile",
)

search_percentile_max

search_percentile_max: float = Field(
    default=1.0,
    alias="searchPercentileMax",
    description="Maximum search percentile",
)
num_search: int = Field(
    default=128, alias="numSearch", description="Number of search iterations"
)

CalibrationConfig.LayerOverrides

class LayerOverrides(BaseModel)

Attributes

Name Description
model_config Layer-specific override settings for calibration
act_scale_min
percentile
method

model_config

model_config = ConfigDict(populate_by_name=True)

Layer-specific override settings for calibration

Parameters

  • act_scale_min — dict. Per-layer activation scale minimum overrides. Keys are actScaleMin values (e.g. '0.0005'), values are lists of layer names to apply that override. (e.g. {'0.0005' : ['layer1', 'layer2']})
  • percentile — dict. Per-layer maxPercentile percentile overrides. Keys are layer names, values are percentile floats. (e.g. {'/model/layer0/conv': 0.999, '/model/layer5/attn': 0.99})
  • method — dict. Per-layer calibration method overrides. Keys are layer names, values are method ints (0=WChALayer, 1=WChAMulti, 2=WChALayerZeropoint, 3=WChAMultiZeropoint). (e.g. {'/model/layer0/conv': 1, '/model/layer5/attn': 3})

act_scale_min

act_scale_min: Any = Field(
    default={},
    alias="actScaleMin",
    description="Per-layer activation scale minimum overrides. Keys are actScaleMin values (e.g. '0.0005'), values are lists of layer names to apply that override. (e.g. {'0.0005' : ['layer1', 'layer2']})",
)

percentile

percentile: Any = Field(
    default={},
    description="Per-layer maxPercentile percentile overrides. Keys are layer names, values are percentile floats. (e.g. {'/model/layer0/conv': 0.999, '/model/layer5/attn': 0.99})",
)

method

method: Any = Field(
    default={},
    description="Per-layer calibration method overrides. Keys are layer names, values are method ints (0=WChALayer, 1=WChAMulti, 2=WChALayerZeropoint, 3=WChAMultiZeropoint). (e.g. {'/model/layer0/conv': 1, '/model/layer5/attn': 3})",
)

CalibrationConfig.Statistics

class Statistics(BaseModel)

Attributes

Name Description
model_config Statistics save/load configuration with percentile selection
apply
save_path
load_path
percentiles
percentile_index

model_config

model_config = ConfigDict(populate_by_name=True)

Statistics save/load configuration with percentile selection

Parameters

  • apply — bool. Enable statistics save/load
  • save_path — str. Path to save statistics. If empty, not saved
  • load_path — str. Path to load statistics. If empty, not loaded
  • percentiles — List[float]. List of percentile candidates
  • percentile_index — int. Index into percentiles list to select active percentile

apply

apply: bool = Field(default=False, description="Enable statistics save/load")

save_path

save_path: str = Field(
    default="",
    alias="savePath",
    description="Path to save statistics. If empty, not saved",
)

load_path

load_path: str = Field(
    default="",
    alias="loadPath",
    description="Path to load statistics. If empty, not loaded",
)

percentiles

percentiles: List[float] = Field(
    default=[0.9999, 0.999, 0.99, 0.9],
    description="List of percentile candidates",
)

percentile_index

percentile_index: int = Field(
    default=0,
    alias="percentileIndex",
    description="Index into percentiles list to select active percentile",
)

CalibrationConfig.GroupLut

class GroupLut(BaseModel)

Attributes

Name Description
model_config LUT grouping algorithm configuration
irls_iter
dro_eps
cover_floor

model_config

model_config = ConfigDict(populate_by_name=True)

LUT grouping algorithm configuration

Parameters

  • irls_iter — int. Number of IRLS iterations for optimal scale computation (1 = single WLS)
  • dro_eps — float. DRO worst-point coefficient; larger = more conservative clipping (0 = disable)
  • cover_floor — float. LUTAM cluster scale lower bound as fraction of coverage

irls_iter

irls_iter: int = Field(
    default=3,
    alias="irlsIter",
    description="Number of IRLS iterations for optimal scale computation (1 = single WLS)",
    ge=1,
)

dro_eps

dro_eps: float = Field(
    default=0,
    alias="droEps",
    description="DRO worst-point coefficient; larger = more conservative clipping (0 = disable)",
    ge=0,
)

cover_floor

cover_floor: float = Field(
    default=0.85,
    alias="coverFloor",
    description="LUTAM cluster scale lower bound as fraction of coverage",
    ge=0,
    le=1,
)

CalibrationConfig.OptimizeLut

class OptimizeLut(BaseModel)

Attributes

Name Description
model_config LUT optimization configuration
optimization_level

model_config

model_config = ConfigDict(populate_by_name=True)

LUT optimization configuration

Parameters

  • optimization_level — int. Optimizer search depth (0 = surrogate-only fast, 1 = surrogate + true-objective refinement)

optimization_level

optimization_level: int = Field(
    default=1,
    alias="optimizationLevel",
    description="Optimizer search depth (0 = surrogate-only fast, 1 = surrogate + true-objective refinement)",
    ge=0,
    le=1,
)

BitConfig

class BitConfig(BaseModel)

Configuration for bit precision

Defines bit-width parameterization for activations and weights used in mixed-precision quantization (e.g., attention and FFN components).

Parameters

Attributes

Name Description
model_config
transformer
save_info
layer_overrides

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

transformer

transformer: Transformer = Field(default_factory=Transformer, alias="transformer")

save_info

save_info: SaveInfo = Field(default_factory=SaveInfo, alias="saveInfo")

layer_overrides

layer_overrides: LayerOverrides = Field(
    default_factory=LayerOverrides, alias="layerOverrides"
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "BitConfig"

Return a copy with updated fields.

BitConfig.Transformer

class Transformer(BaseModel)

Attributes

Name Description
model_config Transformer-specific bit-width configuration
activation
weight
mixed_precision

model_config

model_config = ConfigDict(populate_by_name=True)

Transformer-specific bit-width configuration

Parameters

activation

activation: Activation = Field(default_factory=Activation, alias="activation")

weight

weight: Weight = Field(default_factory=Weight, alias="weight")

mixed_precision

mixed_precision: MixedPrecision = Field(
    default_factory=MixedPrecision, alias="mixedPrecision"
)

BitConfig.Transformer.Activation

class Activation(BaseModel)

Attributes

Name Description
model_config Activation bit-widths for transformer components
query
key
value
output
head
router
ffn

model_config

model_config = ConfigDict(populate_by_name=True)

Activation bit-widths for transformer components

Parameters

  • query — int. Query activation bit-width
  • key — int. Key activation bit-width
  • value — int. Value activation bit-width
  • output — int. Output activation bit-width
  • head — int. Head activation bit-width
  • router — int. MoE router gate activation bit-width
  • ffn — Ffn. FFN activation bit-widths (int shorthand sets all sublayers)

query

query: int = Field(default=8, description="Query activation bit-width")

key

key: int = Field(default=8, description="Key activation bit-width")

value

value: int = Field(default=8, description="Value activation bit-width")

output

output: int = Field(default=16, description="Output activation bit-width")

head

head: int = Field(default=8, description="Head activation bit-width")

router

router: int = Field(
    default=8, description="MoE router gate activation bit-width"
)

ffn

ffn: Ffn = Field(default_factory=Ffn, alias="ffn")

BitConfig.Transformer.Activation.Ffn

class Ffn(BaseModel)

FFN activation bit-widths (int shorthand sets all sublayers)

Parameters

  • up — int. FFN up-projection activation bit-width
  • gate — int. FFN gate (SwiGLU) activation bit-width
  • down — int. FFN down-projection activation bit-width

Attributes

Name Description
up
gate
down

up

up: int = Field(
    default=16, description="FFN up-projection activation bit-width"
)

gate

gate: int = Field(
    default=16, description="FFN gate (SwiGLU) activation bit-width"
)

down

down: int = Field(
    default=16, description="FFN down-projection activation bit-width"
)

Methods

Name Description
expand_int_shorthand

expand_int_shorthand

def expand_int_shorthand(cls, v)

BitConfig.Transformer.Weight

class Weight(BaseModel)

Attributes

Name Description
model_config Weight bit-widths for transformer components
query
key
value
output
head
router
ffn

model_config

model_config = ConfigDict(populate_by_name=True)

Weight bit-widths for transformer components

Parameters

  • query — int. Query weight bit-width
  • key — int. Key weight bit-width
  • value — int. Value weight bit-width
  • output — int. Output weight bit-width
  • head — int. Head weight bit-width
  • router — int. MoE router gate weight bit-width
  • ffn — Ffn. FFN weight bit-widths (int shorthand sets all sublayers)

query

query: int = Field(default=8, description="Query weight bit-width")

key

key: int = Field(default=8, description="Key weight bit-width")

value

value: int = Field(default=8, description="Value weight bit-width")

output

output: int = Field(default=8, description="Output weight bit-width")

head

head: int = Field(default=8, description="Head weight bit-width")

router

router: int = Field(
    default=8, description="MoE router gate weight bit-width"
)

ffn

ffn: Ffn = Field(default_factory=Ffn, alias="ffn")

BitConfig.Transformer.Weight.Ffn

class Ffn(BaseModel)

FFN weight bit-widths (int shorthand sets all sublayers)

Parameters

  • up — int. FFN up-projection weight bit-width
  • gate — int. FFN gate (SwiGLU) weight bit-width
  • down — int. FFN down-projection weight bit-width

Attributes

Name Description
up
gate
down

up

up: int = Field(
    default=8, description="FFN up-projection weight bit-width"
)

gate

gate: int = Field(
    default=8, description="FFN gate (SwiGLU) weight bit-width"
)

down

down: int = Field(
    default=8, description="FFN down-projection weight bit-width"
)

Methods

Name Description
expand_int_shorthand

expand_int_shorthand

def expand_int_shorthand(cls, v)

BitConfig.Transformer.MixedPrecision

class MixedPrecision(BaseModel)

Attributes

Name Description
model_config Mixed precision configuration
weight
activation

model_config

model_config = ConfigDict(populate_by_name=True)

Mixed precision configuration

Parameters

  • weight — Weight. Mixed precision configuration for weights
  • activation — Activation. Per-layer activation mixed precision configuration

weight

weight: Weight = Field(default_factory=Weight, alias="weight")

activation

activation: Activation = Field(
    default_factory=Activation, alias="activation"
)

BitConfig.Transformer.MixedPrecision.Weight

class Weight(BaseModel)

Attributes

Name Description
model_config Mixed precision configuration for weights
apply
type_wise
prune
bit_2
bit_4
bit_8
importance_threshold_low
importance_threshold_high

model_config

model_config = ConfigDict(populate_by_name=True)

Mixed precision configuration for weights

Parameters

  • apply — bool. If true, apply mixed-precision according to the specified bit-widths
  • type_wise — bool. Apply type-wise mixed precision
  • prune — float. Pruning ratio
  • bit_2 — float. Ratio of 2-bit quantization
  • bit_4 — float. Ratio of 4-bit quantization
  • bit_8 — float. Ratio of 8-bit quantization
  • importance_threshold_low — float. Low importance threshold
  • importance_threshold_high — float. High importance threshold

apply

apply: bool = Field(
    default=False,
    description="If true, apply mixed-precision according to the specified bit-widths",
)

type_wise

type_wise: bool = Field(
    default=True,
    alias="typeWise",
    description="Apply type-wise mixed precision",
)

prune

prune: float = Field(default=0, description="Pruning ratio")

bit_2

bit_2: float = Field(
    default=0, alias="bit2", description="Ratio of 2-bit quantization"
)

bit_4

bit_4: float = Field(
    default=0, alias="bit4", description="Ratio of 4-bit quantization"
)

bit_8

bit_8: float = Field(
    default=1, alias="bit8", description="Ratio of 8-bit quantization"
)

importance_threshold_low

importance_threshold_low: float = Field(
    default=-1,
    alias="importanceThreshold_low",
    description="Low importance threshold",
)

importance_threshold_high

importance_threshold_high: float = Field(
    default=-1,
    alias="importanceThreshold_high",
    description="High importance threshold",
)

BitConfig.Transformer.MixedPrecision.Activation

class Activation(BaseModel)

Attributes

Name Description
model_config Per-layer activation mixed precision configuration
apply
ratio_16bit
importance_threshold
search_range

model_config

model_config = ConfigDict(populate_by_name=True)

Per-layer activation mixed precision configuration

Parameters

  • apply — bool. If true, apply per-layer activation mixed precision
  • ratio_16bit — float. Ratio of layers assigned 16-bit (used when importanceThreshold < 0)
  • importance_threshold — float. Normalized importance threshold for 16-bit assignment (negative = use ratio_16bit)
  • search_range — int. Target layer range: -1=all layers, N=first N layers

apply

apply: bool = Field(
    default=False,
    description="If true, apply per-layer activation mixed precision",
)

ratio_16bit

ratio_16bit: float = Field(
    default=0.45,
    alias="ratio16Bit",
    description="Ratio of layers assigned 16-bit (used when importanceThreshold < 0)",
)

importance_threshold

importance_threshold: float = Field(
    default=-1,
    alias="importanceThreshold",
    description="Normalized importance threshold for 16-bit assignment (negative = use ratio_16bit)",
)

search_range

search_range: int = Field(
    default=-1,
    alias="searchRange",
    description="Target layer range: -1=all layers, N=first N layers",
)

BitConfig.SaveInfo

class SaveInfo(BaseModel)

Attributes

Name Description
model_config Bit allocation save/load configuration
save_path
load_path

model_config

model_config = ConfigDict(populate_by_name=True)

Bit allocation save/load configuration

Parameters

  • save_path — str. Path to save the bit allocation. If empty, not saved
  • load_path — str. Path to load the bit allocation. If empty, not loaded

save_path

save_path: str = Field(
    default="",
    alias="savePath",
    description="Path to save the bit allocation. If empty, not saved",
)

load_path

load_path: str = Field(
    default="",
    alias="loadPath",
    description="Path to load the bit allocation. If empty, not loaded",
)

BitConfig.LayerOverrides

class LayerOverrides(BaseModel)

Attributes

Name Description
model_config Layer-specific bit-width override settings
activation_16bits
weight_16bits
weight_8bits

model_config

model_config = ConfigDict(populate_by_name=True)

Layer-specific bit-width override settings

Parameters

  • activation_16bits — list[string]. Layer names to force 16-bit activations
  • weight_16bits — list[string]. Layer names to force 16-bit weights
  • weight_8bits — list[string]. Layer names to force 8-bit weights

activation_16bits

activation_16bits: List[str] = Field(
    default=[],
    alias="activation16Bits",
    description="Layer names to force 16-bit activations",
)

weight_16bits

weight_16bits: List[str] = Field(
    default=[],
    alias="weight16Bits",
    description="Layer names to force 16-bit weights",
)

weight_8bits

weight_8bits: List[str] = Field(
    default=[],
    alias="weight8Bits",
    description="Layer names to force 8-bit weights",
)

HessianQuantConfig

class HessianQuantConfig(BaseModel)

Configuration for HessianQuant algorithm

Defines parameters controlling whether and how HessianQuant is applied during quantization, including layer-level inclusion/exclusion lists.

Parameters

  • apply — bool. If true, apply HessianQuant
  • hessian_dtype — str. Storage dtype for the accumulated HessianQuant Hessian. bf16 halves its memory footprint (host RAM when the Hessian lives on CPU, VRAM when on GPU) — critical for large models such as MoE with thousands of expert FFN Hessians. Compute stays float32 regardless: the per-batch matmul and accumulation run in float32 and the solve upcasts back to float32; only the persistent accumulator is bfloat16.:
    0: fp32 - Store the Hessian in float32.
    1: bf16 - Store the Hessian in bfloat16 (half the memory).
  • accumulation_device — str. Device the HessianQuant Hessian is accumulated on during calibration. gpu accumulates on the calibration device, which is 3-5x faster because the whole d x d accumulator is otherwise copied off the device once per layer per batch; the Hessian is parked on the host as soon as the last batch is in, so this does not raise peak VRAM for the rest of the compile. cpu accumulates on the host instead, bounding VRAM during calibration itself for models whose summed Hessian does not fit alongside the activations. auto follows resourceManagement.useGPUOnlyForCalibration, which is the knob that already says whether this compile is VRAM-bound: false picks gpu, true picks cpu. Resolved once at config load and logged; a CPU compile always accumulates on the host regardless.:
    0: auto - Follow useGPUOnlyForCalibration: false picks gpu, true picks cpu.
    1: cpu - Accumulate on the host, bounding VRAM during calibration itself.
    2: gpu - Accumulate on the calibration device, then park on the host.
  • attributes — Attributes. HessianQuant algorithm attributes

Attributes

Name Description
model_config
apply
hessian_dtype
accumulation_device
attributes

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(default=False, description="If true, apply HessianQuant")

hessian_dtype

hessian_dtype: str = Field(
    default="fp32",
    alias="hessianDtype",
    description="Storage dtype for the accumulated HessianQuant Hessian. bf16 halves its memory footprint (host RAM when the Hessian lives on CPU, VRAM when on GPU) — critical for large models such as MoE with thousands of expert FFN Hessians. Compute stays float32 regardless: the per-batch matmul and accumulation run in float32 and the solve upcasts back to float32; only the persistent accumulator is bfloat16.",
)

accumulation_device

accumulation_device: str = Field(
    default="auto",
    alias="accumulationDevice",
    description="Device the HessianQuant Hessian is accumulated on during calibration. gpu accumulates on the calibration device, which is 3-5x faster because the whole d x d accumulator is otherwise copied off the device once per layer per batch; the Hessian is parked on the host as soon as the last batch is in, so this does not raise peak VRAM for the rest of the compile. cpu accumulates on the host instead, bounding VRAM during calibration itself for models whose summed Hessian does not fit alongside the activations. auto follows resourceManagement.useGPUOnlyForCalibration, which is the knob that already says whether this compile is VRAM-bound: false picks gpu, true picks cpu. Resolved once at config load and logged; a CPU compile always accumulates on the host regardless.",
)

attributes

attributes: Attributes = Field(default_factory=Attributes, alias="attributes")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "HessianQuantConfig"

Return a copy with updated fields.

HessianQuantConfig.Attributes

class Attributes(BaseModel)

Attributes

Name Description
model_config HessianQuant algorithm attributes
act_order
block_size
perc_damp
apply_layers
exclude_layers

model_config

model_config = ConfigDict(populate_by_name=True)

HessianQuant algorithm attributes

Parameters

  • act_order — bool. If true, use activation order
  • block_size — int. Block size used for HessianQuant
  • perc_damp — float. Percentage dampening factor
  • apply_layers — List[str]. Layer names to apply HessianQuant. If empty, applies to all eligible layers
  • exclude_layers — List[str]. Layer names to exclude from HessianQuant

act_order

act_order: bool = Field(
    default=True, alias="actOrder", description="If true, use activation order"
)

block_size

block_size: int = Field(
    default=256,
    alias="blockSize",
    description="Block size used for HessianQuant",
)

perc_damp

perc_damp: float = Field(
    default=0.01, alias="percDamp", description="Percentage dampening factor"
)

apply_layers

apply_layers: List[str] = Field(
    default=[],
    alias="applyLayers",
    description="Layer names to apply HessianQuant. If empty, applies to all eligible layers",
)

exclude_layers

exclude_layers: List[str] = Field(
    default=[],
    alias="excludeLayers",
    description="Layer names to exclude from HessianQuant",
)

LayerBiasCorrectionConfig

class LayerBiasCorrectionConfig(BaseModel)

Configuration for calibration-derived layer bias correction

Measures systematic per-channel float-to-integer activation error throughout the quantized network and folds damped corrections into integer convolution biases. Corrections are recomputed between iterations to account for upstream changes. This is calibration-only bias correction and does not use labels or Minimum Output Difference optimization.

Parameters

  • apply — bool. If true, correct systematic per-layer quantization bias
  • attributes — Attributes. Layer bias correction attributes

Attributes

Name Description
model_config
apply
attributes

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False,
    description="If true, correct systematic per-layer quantization bias",
)

attributes

attributes: Attributes = Field(default_factory=Attributes, alias="attributes")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "LayerBiasCorrectionConfig"

Return a copy with updated fields.

LayerBiasCorrectionConfig.Attributes

class Attributes(BaseModel)

Attributes

Name Description
model_config Layer bias correction attributes
num_samples
iterations
correction_rate

model_config

model_config = ConfigDict(populate_by_name=True)

Layer bias correction attributes

Parameters

  • num_samples — int. Maximum number of calibration samples used per iteration
  • iterations — int. Number of damped correction iterations
  • correction_rate — float. Fraction of the measured bias error applied per iteration

num_samples

num_samples: int = Field(
    default=256,
    alias="numSamples",
    description="Maximum number of calibration samples used per iteration",
    ge=1,
)

iterations

iterations: int = Field(
    default=5, description="Number of damped correction iterations", ge=1
)

correction_rate

correction_rate: float = Field(
    default=0.05,
    alias="correctionRate",
    description="Fraction of the measured bias error applied per iteration",
    ge=1e-06,
    le=1.0,
)

ModConfig

class ModConfig(BaseModel)

Configuration for Minimum Output Difference algorithm

Defines parameters controlling whether and how MOD is applied during quantization, including layer-level inclusion/exclusion lists.

Parameters

  • apply — bool. If true, apply MOD
  • attributes — Attributes. MOD algorithm attributes

Attributes

Name Description
model_config
apply
attributes

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(default=False, description="If true, apply MOD")

attributes

attributes: Attributes = Field(default_factory=Attributes, alias="attributes")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "ModConfig"

Return a copy with updated fields.

ModConfig.Attributes

class Attributes(BaseModel)

Attributes

Name Description
model_config MOD algorithm attributes
epochs
warmup_epochs
lr_min_ratio
save_dir
seed
apply_layers
exclude_layers
mod_after_layer_name
anchors
use_xyxy
learning_rates
training
loss
post_processing

model_config

model_config = ConfigDict(populate_by_name=True)

MOD algorithm attributes

Parameters

  • epochs — int. Number of training epochs
  • warmup_epochs — int. Number of warmup epochs
  • lr_min_ratio — float. Minimum learning rate ratio
  • save_dir — str. Directory to save MOD results
  • seed — int. Random seed for MOD
  • apply_layers — List[str]. Layer names to apply MOD. If empty, applies to all eligible layers
  • exclude_layers — List[str]. Layer names to exclude from MOD
  • mod_after_layer_name — str. Apply MOD after this layer
  • anchors — List. Anchor configurations for detection models. Nested list of anchor box sizes for detection models. Structure: List[List[List[int]]] where outer list is per detection head (e.g. small/medium/large), middle list is anchors per head, inner list is [width, height]. Example: [[[12,16],[19,36],[40,28]], [[36,75],[76,55],[72,146]], [[142,110],[192,243],[459,401]]]
  • use_xyxy — bool. Use XYXY format for bounding boxes
  • learning_rates — LearningRates. Learning rate configuration for MOD
  • training — Training. MOD training configuration
  • loss — Loss. MOD loss configuration
  • post_processing — PostProcessing. Post-processing configuration for detection models

epochs

epochs: int = Field(default=4, description="Number of training epochs")

warmup_epochs

warmup_epochs: int = Field(
    default=1, alias="warmupEpochs", description="Number of warmup epochs"
)

lr_min_ratio

lr_min_ratio: float = Field(
    default=0.0001,
    alias="lrMinRatio",
    description="Minimum learning rate ratio",
)

save_dir

save_dir: str = Field(
    default="", alias="saveDir", description="Directory to save MOD results"
)

seed

seed: int = Field(default=0, description="Random seed for MOD")

apply_layers

apply_layers: List[str] = Field(
    default=[],
    alias="applyLayers",
    description="Layer names to apply MOD. If empty, applies to all eligible layers",
)

exclude_layers

exclude_layers: List[str] = Field(
    default=[],
    alias="excludeLayers",
    description="Layer names to exclude from MOD",
)

mod_after_layer_name

mod_after_layer_name: str = Field(
    default="",
    alias="modAfterLayerName",
    description="Apply MOD after this layer",
)

anchors

anchors: Any = Field(
    default=[],
    description="Anchor configurations for detection models. Nested list of anchor box sizes for detection models. Structure: List[List[List[int]]] where outer list is per detection head (e.g. small/medium/large), middle list is anchors per head, inner list is [width, height]. Example: [[[12,16],[19,36],[40,28]], [[36,75],[76,55],[72,146]], [[142,110],[192,243],[459,401]]]",
)

use_xyxy

use_xyxy: bool = Field(
    default=False,
    alias="useXYXY",
    description="Use XYXY format for bounding boxes",
)

learning_rates

learning_rates: LearningRates = Field(
    default_factory=LearningRates, alias="learningRates"
)

training

training: Training = Field(default_factory=Training, alias="training")

loss

loss: Loss = Field(default_factory=Loss, alias="loss")

post_processing

post_processing: PostProcessing = Field(
    default_factory=PostProcessing, alias="postProcessing"
)

ModConfig.Attributes.LearningRates

class LearningRates(BaseModel)

Attributes

Name Description
model_config Learning rate configuration for MOD
act_scale
zeropoint
weight_scale
weight
bias

model_config

model_config = ConfigDict(populate_by_name=True)

Learning rate configuration for MOD

Parameters

  • act_scale — float. Learning rate for activation scale
  • zeropoint — float. Learning rate for zeropoint
  • weight_scale — float. Learning rate for weight scale
  • weight — float. Learning rate for weight
  • bias — float. Learning rate for bias

act_scale

act_scale: float = Field(
    default=0.0,
    alias="actScale",
    description="Learning rate for activation scale",
)

zeropoint

zeropoint: float = Field(
    default=0.0, description="Learning rate for zeropoint"
)

weight_scale

weight_scale: float = Field(
    default=0.0,
    alias="weightScale",
    description="Learning rate for weight scale",
)

weight

weight: float = Field(default=4e-06, description="Learning rate for weight")

bias

bias: float = Field(default=4e-06, description="Learning rate for bias")

ModConfig.Attributes.Training

class Training(BaseModel)

Attributes

Name Description
model_config MOD training configuration
batch_size
q_drop
quantize_weight
weight_scale_init
downresol_mode
scheduler_type

model_config

model_config = ConfigDict(populate_by_name=True)

MOD training configuration

Parameters

  • batch_size — int. Batch size for MOD training
  • q_drop — float. Quantization drop probability
  • quantize_weight — bool. Whether to quantize weights
  • weight_scale_init — str. Weight scale initialization method
  • downresol_mode — str. Downresolution mode
  • scheduler_type — str. LR scheduler type

batch_size

batch_size: int = Field(
    default=1, alias="batchSize", description="Batch size for MOD training"
)

q_drop

q_drop: float = Field(
    default=0.0, alias="qDrop", description="Quantization drop probability"
)

quantize_weight

quantize_weight: bool = Field(
    default=True,
    alias="quantizeWeight",
    description="Whether to quantize weights",
)

weight_scale_init

weight_scale_init: str = Field(
    default="MinMax",
    alias="weightScaleInit",
    description="Weight scale initialization method",
)

downresol_mode

downresol_mode: str = Field(
    default="STE", alias="downresolMode", description="Downresolution mode"
)

scheduler_type

scheduler_type: str = Field(
    default="Cosine", alias="schedulerType", description="LR scheduler type"
)

ModConfig.Attributes.Loss

class Loss(BaseModel)

Attributes

Name Description
model_config MOD loss configuration
type
use_outputs
kl_temperature
recon_prob
recon_coeff
lambda_0
lambda_1
lambda_2
lambda_3
custom_loss_jit_path

model_config

model_config = ConfigDict(populate_by_name=True)

MOD loss configuration

Parameters

  • type — str. Loss type (MSE, KL, etc.)
  • use_outputs — bool. Use model outputs for loss computation
  • kl_temperature — float. KL divergence temperature
  • recon_prob — float. Reconstruction probability
  • recon_coeff — float. Reconstruction coefficient
  • lambda_0 — float. Loss weight lambda_0
  • lambda_1 — float. Loss weight lambda_1
  • lambda_2 — float. Loss weight lambda_2
  • lambda_3 — float. Loss weight lambda_3
  • custom_loss_jit_path — str. Path to custom JIT-compiled loss function. Refer to /workspace/quantizer/pyutils/mel.pt

type

type: str = Field(default="MSE", description="Loss type (MSE, KL, etc.)")

use_outputs

use_outputs: bool = Field(
    default=False,
    alias="useOutputs",
    description="Use model outputs for loss computation",
)

kl_temperature

kl_temperature: float = Field(
    default=1.0,
    alias="KLTemperature",
    description="KL divergence temperature",
)

recon_prob

recon_prob: float = Field(
    default=1.0, alias="reconProb", description="Reconstruction probability"
)

recon_coeff

recon_coeff: float = Field(
    default=1.0,
    alias="reconCoeff",
    description="Reconstruction coefficient",
)

lambda_0

lambda_0: float = Field(
    default=1.0, alias="lambda0", description="Loss weight lambda_0"
)

lambda_1

lambda_1: float = Field(
    default=1.0, alias="lambda1", description="Loss weight lambda_1"
)

lambda_2

lambda_2: float = Field(
    default=1.0, alias="lambda2", description="Loss weight lambda_2"
)

lambda_3

lambda_3: float = Field(
    default=1.0, alias="lambda3", description="Loss weight lambda_3"
)

custom_loss_jit_path

custom_loss_jit_path: str = Field(
    default="",
    alias="customLossJITPath",
    description="Path to custom JIT-compiled loss function. Refer to /workspace/quantizer/pyutils/mel.pt",
)

ModConfig.Attributes.PostProcessing

class PostProcessing(BaseModel)

Attributes

Name Description
model_config Post-processing configuration for detection models
post
box_conf_thres
box_iou_thres

model_config

model_config = ConfigDict(populate_by_name=True)

Post-processing configuration for detection models

Parameters

  • post — str. Post-processing type
  • box_conf_thres — float. Box confidence threshold
  • box_iou_thres — float. Box IoU threshold

post

post: str = Field(default="", description="Post-processing type")

box_conf_thres

box_conf_thres: float = Field(
    default=0, alias="boxConfThres", description="Box confidence threshold"
)

box_iou_thres

box_iou_thres: float = Field(
    default=0, alias="boxIoUThres", description="Box IoU threshold"
)

LlmConfig

class LlmConfig(BaseModel)

Configuration for Large Language Model (LLM) compilation

Defines LLM-specific settings including sequence lengths, cache configurations, and runtime parameters for efficient LLM inference.

Parameters

  • apply — bool. If True, apply LLM-specific configurations
  • npu_parallel_degree — int. Number of NPU partitions for FFN tensor parallelism (1 = disabled). Applied before OptimizeFFN if both active.
  • attributes — Attributes. LLM attributes configuration

Attributes

Name Description
model_config
apply
npu_parallel_degree
attributes

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False, description="If True, apply LLM-specific configurations"
)

npu_parallel_degree

npu_parallel_degree: int = Field(
    default=1,
    alias="npuParallelDegree",
    description="Number of NPU partitions for FFN tensor parallelism (1 = disabled). Applied before OptimizeFFN if both active.",
)

attributes

attributes: Attributes = Field(default_factory=Attributes, alias="attributes")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "LlmConfig"

Return a copy with updated fields.

LlmConfig.Attributes

class Attributes(BaseModel)

Attributes

Name Description
model_config LLM attributes configuration
max_data_length
max_sequence_length
max_cache_length
max_core_data_length
calibration
runtime
debug

model_config

model_config = ConfigDict(populate_by_name=True)

LLM attributes configuration

Parameters

  • max_data_length — int. Maximum data length
  • max_sequence_length — int. Maximum sequence length
  • max_cache_length — int. Maximum cache length
  • max_core_data_length — int. Maximum core data length
  • calibration — Calibration. LLM calibration settings
  • runtime — Runtime. LLM runtime settings
  • debug — Debug. LLM debug settings

max_data_length

max_data_length: int = Field(
    default=4096, alias="maxDataLength", description="Maximum data length"
)

max_sequence_length

max_sequence_length: int = Field(
    default=4096,
    alias="maxSequenceLength",
    description="Maximum sequence length",
)

max_cache_length

max_cache_length: int = Field(
    default=4096, alias="maxCacheLength", description="Maximum cache length"
)

max_core_data_length

max_core_data_length: int = Field(
    default=128,
    alias="maxCoreDataLength",
    description="Maximum core data length",
)

calibration

calibration: Calibration = Field(
    default_factory=Calibration, alias="calibration"
)

runtime

runtime: Runtime = Field(default_factory=Runtime, alias="runtime")

debug

debug: Debug = Field(default_factory=Debug, alias="debug")

LlmConfig.Attributes.Calibration

class Calibration(BaseModel)

Attributes

Name Description
model_config LLM calibration settings
random_seq_length
use_full_seq_length

model_config

model_config = ConfigDict(populate_by_name=True)

LLM calibration settings

Parameters

  • random_seq_length — int. Random sequence length used for calibration
  • use_full_seq_length — bool. If True, use the full sequence length for calibration

random_seq_length

random_seq_length: int = Field(
    default=80,
    alias="randomSeqLength",
    description="Random sequence length used for calibration",
)

use_full_seq_length

use_full_seq_length: bool = Field(
    default=False,
    alias="useFullSeqLength",
    description="If True, use the full sequence length for calibration",
)

LlmConfig.Attributes.Runtime

class Runtime(BaseModel)

Attributes

Name Description
model_config LLM runtime settings
use_global_core
batch_size
npu_core_ids
dynamic_rope
dynamic_mask

model_config

model_config = ConfigDict(populate_by_name=True)

LLM runtime settings

Parameters

  • use_global_core — bool. If True, use a global core
  • batch_size — int. Batch size
  • npu_core_ids — List[int]. List of NPU core IDs
  • dynamic_rope — bool. If True, enable dynamic RoPE (rotary position embedding)
  • dynamic_mask — bool. If True, enable dynamic mask (attention mask as runtime input)

use_global_core

use_global_core: bool = Field(
    default=False,
    alias="useGlobalCore",
    description="If True, use a global core",
)

batch_size

batch_size: int = Field(
    default=1, alias="batchSize", description="Batch size"
)

npu_core_ids

npu_core_ids: List[int] = Field(
    default=[0], alias="npuCoreIds", description="List of NPU core IDs"
)

dynamic_rope

dynamic_rope: bool = Field(
    default=False,
    alias="dynamicRope",
    description="If True, enable dynamic RoPE (rotary position embedding)",
)

dynamic_mask

dynamic_mask: bool = Field(
    default=False,
    alias="dynamicMask",
    description="If True, enable dynamic mask (attention mask as runtime input)",
)

LlmConfig.Attributes.Debug

class Debug(BaseModel)

Attributes

Name Description
model_config LLM debug settings
apply
batch_debug_bundle_size

model_config

model_config = ConfigDict(populate_by_name=True)

LLM debug settings

Parameters

  • apply — bool. Enable LLM debug mode
  • batch_debug_bundle_size — int. Batch debug bundle size

apply

apply: bool = Field(default=False, description="Enable LLM debug mode")

batch_debug_bundle_size

batch_debug_bundle_size: int = Field(
    default=0,
    alias="batchDebugBundleSize",
    description="Batch debug bundle size",
)

MoeConfig

class MoeConfig(BaseModel)

Sparse MoE expert-selection configuration (calibration only)

Controls which experts are calibrated inside SparseMoe modules. selectionMode is a calibration-only knob: it picks which experts collect statistics (and, for TopK, on which tokens). It does NOT change inference routing — the forward path always routes the router's top-K experts regardless of this setting. scoreThreshold is only used when selectionMode is Threshold.

Parameters

  • selection_mode — int. Expert selection mode index (calibration only; inference is always top-K):
    0: TopK - Calibrate only the router's top-K experts per token (matches inference routing).
    1: All - Calibrate every expert on the full sequence.
    2: Threshold - Calibrate all experts whose routing score exceeds scoreThreshold.
  • score_threshold — float. Routing score threshold used when selectionMode is Threshold (calibration only)

Attributes

Name Description
model_config
selection_mode_list
selection_mode
score_threshold

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

selection_mode_list

selection_mode_list: List[str] = Field(
    default=["TopK", "All", "Threshold"], alias="selectionModeList"
)

selection_mode

selection_mode: int = Field(default=0, alias="selectionMode")

score_threshold

score_threshold: float = Field(
    default=0.0,
    alias="scoreThreshold",
    description="Routing score threshold used when selectionMode is Threshold (calibration only)",
    ge=0,
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "MoeConfig"

Return a copy with updated fields.

EquivalentTransformationConfig

class EquivalentTransformationConfig(BaseModel)

Configuration for equivalent transformation techniques

Defines parameters for various equivalent transformation methods including NormConv, QK smoothing, and rotation matrices for improved quantization.

Parameters

  • seed — int. Random seed for transformation
  • apply_hadamard_rotation_matrix — bool. Apply Hadamard rotation matrix
  • norm_conv — NormConv. NormConv equivalent transformation
  • qk — Qk. QK smoothing transformation
  • ud — Ud. UD transformation
  • vo — Vo. VO transformation
  • feed_forward_multi_lut — FeedForwardMultiLut. Feed-forward multi-LUT transformation
  • spin_r1 — SpinR1. SpinR1 rotation transformation
  • head_out_ch_rotation — HeadOutChRotation. Head output channel rotation transformation
  • in_rotation — InRotation. Input rotation transformation
  • spin_r2 — SpinR2. SpinR2 rotation transformation
  • qk_rotation — QkRotation. QK rotation transformation
  • flatten_quant — FlattenQuant. Flatten quantization transformation
  • optimize_ffn — OptimizeFfn. FFN optimization

Attributes

Name Description
model_config
seed
apply_hadamard_rotation_matrix
norm_conv
qk
ud
vo
feed_forward_multi_lut
spin_r1
head_out_ch_rotation
in_rotation
spin_r2
qk_rotation
flatten_quant
optimize_ffn

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

seed

seed: int = Field(default=0, description="Random seed for transformation")

apply_hadamard_rotation_matrix

apply_hadamard_rotation_matrix: bool = Field(
    default=True,
    alias="applyHadamardRotationMatrix",
    description="Apply Hadamard rotation matrix",
)

norm_conv

norm_conv: NormConv = Field(default_factory=NormConv, alias="NormConv")

qk

qk: Qk = Field(default_factory=Qk, alias="QK")

ud

ud: Ud = Field(default_factory=Ud, alias="UD")

vo

vo: Vo = Field(default_factory=Vo, alias="VO")

feed_forward_multi_lut

feed_forward_multi_lut: FeedForwardMultiLut = Field(
    default_factory=FeedForwardMultiLut, alias="FeedForwardMultiLUT"
)

spin_r1

spin_r1: SpinR1 = Field(default_factory=SpinR1, alias="SpinR1")

head_out_ch_rotation

head_out_ch_rotation: HeadOutChRotation = Field(
    default_factory=HeadOutChRotation, alias="HeadOutChRotation"
)

in_rotation

in_rotation: InRotation = Field(default_factory=InRotation, alias="InRotation")

spin_r2

spin_r2: SpinR2 = Field(default_factory=SpinR2, alias="SpinR2")

qk_rotation

qk_rotation: QkRotation = Field(default_factory=QkRotation, alias="QKRotation")

flatten_quant

flatten_quant: FlattenQuant = Field(
    default_factory=FlattenQuant, alias="FlattenQuant"
)

optimize_ffn

optimize_ffn: OptimizeFfn = Field(default_factory=OptimizeFfn, alias="OptimizeFFN")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "EquivalentTransformationConfig"

Return a copy with updated fields.

EquivalentTransformationConfig.NormConv

class NormConv(BaseModel)

Attributes

Name Description
model_config NormConv equivalent transformation
apply
learn
smoothing_factor
min_gamma
max_gamma

model_config

model_config = ConfigDict(populate_by_name=True)

NormConv equivalent transformation

Parameters

  • apply — bool. Apply NormConv transformation
  • learn — bool. Learn transformation parameters
  • smoothing_factor — float. Smoothing factor
  • min_gamma — float. Minimum gamma value
  • max_gamma — float. Maximum gamma value

apply

apply: bool = Field(default=False, description="Apply NormConv transformation")

learn

learn: bool = Field(
    default=False, description="Learn transformation parameters"
)

smoothing_factor

smoothing_factor: float = Field(
    default=0.5, alias="smoothingFactor", description="Smoothing factor"
)

min_gamma

min_gamma: float = Field(
    default=0.0001, alias="minGamma", description="Minimum gamma value"
)

max_gamma

max_gamma: float = Field(
    default=10000.0, alias="maxGamma", description="Maximum gamma value"
)

EquivalentTransformationConfig.Qk

class Qk(BaseModel)

Attributes

Name Description
model_config QK smoothing transformation
apply
smoothing_factor
min_gamma
max_gamma

model_config

model_config = ConfigDict(populate_by_name=True)

QK smoothing transformation

Parameters

  • apply — bool. Apply QK transformation
  • smoothing_factor — float. Smoothing factor
  • min_gamma — float. Minimum gamma value
  • max_gamma — float. Maximum gamma value

apply

apply: bool = Field(default=False, description="Apply QK transformation")

smoothing_factor

smoothing_factor: float = Field(
    default=0.5, alias="smoothingFactor", description="Smoothing factor"
)

min_gamma

min_gamma: float = Field(
    default=0.0001, alias="minGamma", description="Minimum gamma value"
)

max_gamma

max_gamma: float = Field(
    default=10000.0, alias="maxGamma", description="Maximum gamma value"
)

EquivalentTransformationConfig.Ud

class Ud(BaseModel)

Attributes

Name Description
model_config UD transformation
apply
learn
smoothing_factor
min_gamma
max_gamma

model_config

model_config = ConfigDict(populate_by_name=True)

UD transformation

Parameters

  • apply — bool. Apply UD transformation
  • learn — bool. Learn transformation parameters
  • smoothing_factor — float. Smoothing factor
  • min_gamma — float. Minimum gamma value
  • max_gamma — float. Maximum gamma value

apply

apply: bool = Field(default=False, description="Apply UD transformation")

learn

learn: bool = Field(
    default=False, description="Learn transformation parameters"
)

smoothing_factor

smoothing_factor: float = Field(
    default=0.5, alias="smoothingFactor", description="Smoothing factor"
)

min_gamma

min_gamma: float = Field(
    default=0.0001, alias="minGamma", description="Minimum gamma value"
)

max_gamma

max_gamma: float = Field(
    default=10000.0, alias="maxGamma", description="Maximum gamma value"
)

EquivalentTransformationConfig.Vo

class Vo(BaseModel)

Attributes

Name Description
model_config VO transformation
apply
smoothing_factor
min_gamma
max_gamma

model_config

model_config = ConfigDict(populate_by_name=True)

VO transformation

Parameters

  • apply — bool. Apply VO transformation
  • smoothing_factor — float. Smoothing factor
  • min_gamma — float. Minimum gamma value
  • max_gamma — float. Maximum gamma value

apply

apply: bool = Field(default=False, description="Apply VO transformation")

smoothing_factor

smoothing_factor: float = Field(
    default=0.5, alias="smoothingFactor", description="Smoothing factor"
)

min_gamma

min_gamma: float = Field(
    default=0.0001, alias="minGamma", description="Minimum gamma value"
)

max_gamma

max_gamma: float = Field(
    default=10000.0, alias="maxGamma", description="Maximum gamma value"
)

EquivalentTransformationConfig.FeedForwardMultiLut

class FeedForwardMultiLut(BaseModel)

Feed-forward multi-LUT transformation

Parameters

  • apply — bool. Apply feed-forward multi-LUT transformation
  • breakpoints — List[float]. Breakpoints for multi-LUT

Attributes

Name Description
apply
breakpoints

apply

apply: bool = Field(
    default=False, description="Apply feed-forward multi-LUT transformation"
)

breakpoints

breakpoints: List[float] = Field(
    default=[-8.0, -4.0, 0], description="Breakpoints for multi-LUT"
)

EquivalentTransformationConfig.SpinR1

class SpinR1(BaseModel)

Attributes

Name Description
model_config SpinR1 rotation transformation
apply
matrix_path

model_config

model_config = ConfigDict(populate_by_name=True)

SpinR1 rotation transformation

Parameters

  • apply — bool. Apply SpinR1 transformation
  • matrix_path — str. Path to rotation matrix file

apply

apply: bool = Field(default=False, description="Apply SpinR1 transformation")

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

EquivalentTransformationConfig.HeadOutChRotation

class HeadOutChRotation(BaseModel)

Attributes

Name Description
model_config Head output channel rotation transformation
apply
matrix_path

model_config

model_config = ConfigDict(populate_by_name=True)

Head output channel rotation transformation

Parameters

  • apply — bool. Apply head output channel rotation
  • matrix_path — str. Path to rotation matrix file

apply

apply: bool = Field(
    default=False, description="Apply head output channel rotation"
)

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

EquivalentTransformationConfig.InRotation

class InRotation(BaseModel)

Attributes

Name Description
model_config Input rotation transformation
apply
matrix_path
input_names

model_config

model_config = ConfigDict(populate_by_name=True)

Input rotation transformation

Parameters

  • apply — bool. Apply input rotation
  • matrix_path — str. Path to rotation matrix file
  • input_names — List[str]. Names of the input layers to rotate

apply

apply: bool = Field(default=False, description="Apply input rotation")

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

input_names

input_names: List[str] = Field(
    default=[],
    alias="inputNames",
    description="Names of the input layers to rotate",
)

EquivalentTransformationConfig.SpinR2

class SpinR2(BaseModel)

Attributes

Name Description
model_config SpinR2 rotation transformation
apply
learn
matrix_path

model_config

model_config = ConfigDict(populate_by_name=True)

SpinR2 rotation transformation

Parameters

  • apply — bool. Apply SpinR2 transformation
  • learn — bool. Learn rotation matrix
  • matrix_path — str. Path to rotation matrix file

apply

apply: bool = Field(default=False, description="Apply SpinR2 transformation")

learn

learn: bool = Field(default=False, description="Learn rotation matrix")

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

EquivalentTransformationConfig.QkRotation

class QkRotation(BaseModel)

Attributes

Name Description
model_config QK rotation transformation
apply
matrix_path

model_config

model_config = ConfigDict(populate_by_name=True)

QK rotation transformation

Parameters

  • apply — bool. Apply QK rotation transformation
  • matrix_path — str. Path to rotation matrix file

apply

apply: bool = Field(
    default=False, description="Apply QK rotation transformation"
)

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

EquivalentTransformationConfig.FlattenQuant

class FlattenQuant(BaseModel)

Attributes

Name Description
model_config Flatten quantization transformation
apply
learn
apply_threshold
max_overhead

model_config

model_config = ConfigDict(populate_by_name=True)

Flatten quantization transformation

Parameters

  • apply — bool. Apply flatten quantization
  • learn — bool. Learn flattening parameters
  • apply_threshold — float. Threshold for applying flatten quantization
  • max_overhead — float. Maximum overhead allowed for flattening

apply

apply: bool = Field(default=False, description="Apply flatten quantization")

learn

learn: bool = Field(default=False, description="Learn flattening parameters")

apply_threshold

apply_threshold: float = Field(
    default=0.33,
    alias="applyThreshold",
    description="Threshold for applying flatten quantization",
)

max_overhead

max_overhead: float = Field(
    default=0.02,
    alias="maxOverhead",
    description="Maximum overhead allowed for flattening",
)

EquivalentTransformationConfig.OptimizeFfn

class OptimizeFfn(BaseModel)

Attributes

Name Description
model_config FFN optimization
apply
ch_per_ffn

model_config

model_config = ConfigDict(populate_by_name=True)

FFN optimization

Parameters

  • apply — bool. Apply FFN optimization
  • ch_per_ffn — int. Optimize FFN split (-1 for auto)

apply

apply: bool = Field(default=False, description="Apply FFN optimization")

ch_per_ffn

ch_per_ffn: int = Field(
    default=-1, alias="chPerFFN", description="Optimize FFN split (-1 for auto)"
)

SearchWeightScaleConfig

class SearchWeightScaleConfig(BaseModel)

Configuration for weight scale search

Defines which transformer components should have their weight scales searched for optimal quantization.

Parameters

  • apply — bool. If true, apply weight scale search
  • transformer — Transformer. Transformer components for weight scale search

Attributes

Name Description
model_config
apply
transformer

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(default=False, description="If true, apply weight scale search")

transformer

transformer: Transformer = Field(default_factory=Transformer, alias="transformer")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "SearchWeightScaleConfig"

Return a copy with updated fields.

SearchWeightScaleConfig.Transformer

class Transformer(BaseModel)

Transformer components for weight scale search

Parameters

  • query — bool. Search weight scale for query
  • key — bool. Search weight scale for key
  • value — bool. Search weight scale for value
  • out — bool. Search weight scale for output
  • ffn — bool. Search weight scale for FFN

Attributes

Name Description
query
key
value
out
ffn

query

query: bool = Field(default=False, description="Search weight scale for query")

key

key: bool = Field(default=False, description="Search weight scale for key")

value

value: bool = Field(default=False, description="Search weight scale for value")

out

out: bool = Field(default=False, description="Search weight scale for output")

ffn

ffn: bool = Field(default=False, description="Search weight scale for FFN")

LoadScaleConfig

class LoadScaleConfig(BaseModel)

QAT activation scale loading configuration

Loads pre-trained activation scales from safetensors files and applies them to specified layers before scale/zeropoint computation. NOTE: entries is stored as raw JSON (type: list) because the generator does not support list[CustomStruct]. The schema records the field shape; parsing is done manually in applyQATLoadScaleQuantType / applyQATLoadScales.

Parameters

  • apply — bool. If true, load and apply QAT scales from safetensors files
  • entries — List. List of file entries. Each entry is a dict: { path: str, scales: [ { key: str, layers: [str] } ] } path: safetensors file path; key: tensor name in the file; layers: layer names whose activation scale will be overridden.

Attributes

Name Description
model_config
apply
entries

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False,
    description="If true, load and apply QAT scales from safetensors files",
)

entries

entries: Any = Field(
    default=[],
    description="List of file entries. Each entry is a dict: { path: str, scales: [ { key: str, layers: [str] } ] } path: safetensors file path; key: tensor name in the file; layers: layer names whose activation scale will be overridden.",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "LoadScaleConfig"

Return a copy with updated fields.

RuntimeOptions

class RuntimeOptions(BaseModel)

Runtime options for compilation

Contains runtime-specific settings like version info and cache options.

Parameters

  • version — str. Compiler version string (e.g., 0.0.0)

Attributes

Name Description
model_config
version

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

version

version: str = Field(
    default="0.0.0", description="Compiler version string (e.g., 0.0.0)"
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "RuntimeOptions"

Return a copy with updated fields.

SaveSampleConfig

class SaveSampleConfig(BaseModel)

Sample data generation and saving configuration

Parameters

  • apply — bool. Enable sample data saving
  • mode — str. Inference mode: infer (standard) or inferWithCache (LLM cache models)
  • batch_size — int. Number of inference batches to generate
  • batch_seq_lens — List. Per-batch step-wise sequence lengths for inferWithCache mode. e.g. [[80, 1], [240, 10]]
  • save_folder — str. Output folder for sample data
  • dtype — str. Data type for saved samples: float or int8

Attributes

Name Description
model_config
apply
mode
batch_size
batch_seq_lens
save_folder
dtype

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(default=False, description="Enable sample data saving")

mode

mode: str = Field(
    default="infer",
    description="Inference mode: infer (standard) or inferWithCache (LLM cache models)",
)

batch_size

batch_size: int = Field(
    default=1,
    alias="batchSize",
    description="Number of inference batches to generate",
)

batch_seq_lens

batch_seq_lens: Any = Field(
    default=[],
    alias="batchSeqLens",
    description="Per-batch step-wise sequence lengths for inferWithCache mode. e.g. [[80, 1], [240, 10]]",
)

save_folder

save_folder: str = Field(
    default="sampleInout",
    alias="saveFolder",
    description="Output folder for sample data",
)

dtype

dtype: str = Field(
    default="float", description="Data type for saved samples: float or int8"
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "SaveSampleConfig"

Return a copy with updated fields.

GroupWiseConfig

class GroupWiseConfig(BaseModel)

Group-wise streaming quantization configuration for large LLMs

Configuration for the group-wise (streaming) quantization pipeline used for LLMs that do not fit fully on GPU. Partitions the model into groups of transformer blocks and quantizes each group independently.

Parameters

  • apply — bool. Enable group-wise streaming quantization pipeline
  • group_size — int. Group size in number of transformer blocks (0 = auto)
  • gpu_budget_gb — float. GPU memory budget in GiB for group-wise execution (0 = auto-detect)
  • gpu_safety_margin_gb — float. Safety margin in GiB subtracted from detected GPU budget
  • cache_dir — str. Directory for group-wise activation/state cache (empty = system temp)
  • keep_cache — bool. Retain group-wise cache after run completes (for debugging)
  • partition_policy — str. Partitioning policy (e.g. transformer_block, moe_expert_subgroup)
  • expert_groups — List[List[int]]. MoE fallback partitioner: list of expert-index lists (List[List[int]]).
  • retain_topology_weights — bool. Keep inflated FP weights across groups (false = release after each group for tight memory budgets).
  • checkpoint — bool. Save per-group checkpoint so quantization can resume from the last completed group after a crash.

Attributes

Name Description
model_config
apply
group_size
gpu_budget_gb
gpu_safety_margin_gb
cache_dir
keep_cache
partition_policy
expert_groups
retain_topology_weights
checkpoint

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False, description="Enable group-wise streaming quantization pipeline"
)

group_size

group_size: int = Field(
    default=0,
    alias="groupSize",
    description="Group size in number of transformer blocks (0 = auto)",
)

gpu_budget_gb

gpu_budget_gb: float = Field(
    default=0,
    alias="gpuBudgetGb",
    description="GPU memory budget in GiB for group-wise execution (0 = auto-detect)",
)

gpu_safety_margin_gb

gpu_safety_margin_gb: float = Field(
    default=4,
    alias="gpuSafetyMarginGb",
    description="Safety margin in GiB subtracted from detected GPU budget",
)

cache_dir

cache_dir: str = Field(
    default="",
    alias="cacheDir",
    description="Directory for group-wise activation/state cache (empty = system temp)",
)

keep_cache

keep_cache: bool = Field(
    default=False,
    alias="keepCache",
    description="Retain group-wise cache after run completes (for debugging)",
)

partition_policy

partition_policy: str = Field(
    default="transformer_block",
    alias="partitionPolicy",
    description="Partitioning policy (e.g. transformer_block, moe_expert_subgroup)",
)

expert_groups

expert_groups: Any = Field(
    default=[],
    alias="expertGroups",
    description="MoE fallback partitioner: list of expert-index lists (List[List[int]]).",
)

retain_topology_weights

retain_topology_weights: bool = Field(
    default=True,
    alias="retainTopologyWeights",
    description="Keep inflated FP weights across groups (false = release after each group for tight memory budgets).",
)

checkpoint

checkpoint: bool = Field(
    default=False,
    description="Save per-group checkpoint so quantization can resume from the last completed group after a crash.",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "GroupWiseConfig"

Return a copy with updated fields.

CompileConfig

class CompileConfig(BaseModel)

Unified compilation configuration for Mobilint MXQ compilation.

Attributes

Name Description
model_config
model_paths
calib_data_path
save_paths
use_random_calib
inference_scheme
cpu_offload
force_npu_input_reposition
force_npu_output_reposition
optimize_option
buffer_mode
input_shape_dict
device
dtype
debug
trace
image_channels
config_version
split_blocks
split_parts
uint8_input
preprocessing
resource_management
calibration
bit
hessian_quant
layer_bias_correction
mod
llm
moe
equivalent_transformation
search_weight_scale
load_scale
runtime_options
save_sample
group_wise

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

model_paths

model_paths: List[str] = Field(
    default=[], alias="modelPaths", description="Paths to model files"
)

calib_data_path

calib_data_path: List[str] = Field(
    default=[], alias="calibDataPaths", description="Paths to calibration datasets"
)

save_paths

save_paths: List[str] = Field(
    default=["./tmp.mxq"],
    alias="savePaths",
    description="Output MXQ filename/paths",
)

use_random_calib

use_random_calib: bool = Field(
    default=False, alias="useRandomCalib", description="Use random calibration"
)

inference_scheme

inference_scheme: str = Field(
    default="single", alias="inferenceScheme", description="NPU inference scheme"
)

cpu_offload

cpu_offload: bool = Field(
    default=False,
    alias="cpuOffload",
    description="Enable CPU offload for unsupported operators",
)

force_npu_input_reposition

force_npu_input_reposition: bool = Field(
    default=False,
    alias="forceNpuInputReposition",
    description="Force input reposition operations to run on NPU instead of CPU",
)

force_npu_output_reposition

force_npu_output_reposition: bool = Field(
    default=False,
    alias="forceNpuOutputReposition",
    description="Force output reposition operations to run on NPU instead of CPU",
)

optimize_option

optimize_option: int = Field(
    default=1,
    alias="optimizeOption",
    description="Compiler optimization selector",
    ge=0,
)

buffer_mode

buffer_mode: int = Field(
    default=1, alias="bufferMode", description="Buffer serialization mode"
)

input_shape_dict

input_shape_dict: Any = Field(
    default={}, alias="inputShapeDict", description="Dictionary of input shapes"
)

device

device: str = Field(default="gpu", description="Device for computation")

dtype

dtype: str = Field(default="float", description="Data type for computation")

debug

debug: bool = Field(default=False, description="Enable debug mode")

trace

trace: bool = Field(default=False, description="Enable trace mode")

image_channels

image_channels: int = Field(
    default=0,
    alias="imageChannels",
    description="Number of image channels (0 for auto-detect)",
)

config_version

config_version: str = Field(
    default="1.0.0", alias="configVersion", description="Config schema version"
)

split_blocks

split_blocks: List[int] = Field(
    default=[],
    alias="splitBlocks",
    description="Multi-MXQ split points by transformer block index",
)

split_parts

split_parts: int = Field(
    default=0,
    alias="splitParts",
    description="Evenly split transformer blocks into N MXQ parts",
)

uint8_input

uint8_input: Uint8InputConfig = Field(
    default_factory=Uint8InputConfig, alias="uint8Input"
)

preprocessing

preprocessing: PreprocessingConfig = Field(default_factory=PreprocessingConfig)

resource_management

resource_management: ResourceManagementConfig = Field(
    default_factory=ResourceManagementConfig, alias="resourceManagement"
)

calibration

calibration: CalibrationConfig = Field(default_factory=CalibrationConfig)

bit

bit: BitConfig = Field(default_factory=BitConfig)

hessian_quant

hessian_quant: HessianQuantConfig = Field(
    default_factory=HessianQuantConfig, alias="hessianQuant"
)

layer_bias_correction

layer_bias_correction: LayerBiasCorrectionConfig = Field(
    default_factory=LayerBiasCorrectionConfig, alias="layerBiasCorrection"
)

mod

mod: ModConfig = Field(default_factory=ModConfig)

llm

llm: LlmConfig = Field(default_factory=LlmConfig)

moe

moe: MoeConfig = Field(default_factory=MoeConfig)

equivalent_transformation

equivalent_transformation: EquivalentTransformationConfig = Field(
    default_factory=EquivalentTransformationConfig, alias="equivalentTransformation"
)

search_weight_scale

search_weight_scale: SearchWeightScaleConfig = Field(
    default_factory=SearchWeightScaleConfig, alias="searchWeightScale"
)

load_scale

load_scale: LoadScaleConfig = Field(
    default_factory=LoadScaleConfig, alias="loadScale"
)

runtime_options

runtime_options: RuntimeOptions = Field(
    default_factory=RuntimeOptions, alias="runtimeOptions"
)

save_sample

save_sample: SaveSampleConfig = Field(
    default_factory=SaveSampleConfig, alias="saveSample"
)

group_wise

group_wise: GroupWiseConfig = Field(
    default_factory=GroupWiseConfig, alias="groupWise"
)

Methods

Name Description
with_uint8_input Return a copy with uint8_input settings enabled.
with_preprocessing Return a copy with preprocessing settings enabled.
with_hessian_quant Return a copy with hessian_quant settings enabled.
with_layer_bias_correction Return a copy with layer_bias_correction settings enabled.
with_mod Return a copy with mod settings enabled.
with_llm Return a copy with llm settings enabled.
with_search_weight_scale Return a copy with search_weight_scale settings enabled.
with_load_scale Return a copy with load_scale settings enabled.
with_save_sample Return a copy with save_sample settings enabled.
with_group_wise Return a copy with group_wise settings enabled.
from_file Load config from YAML or JSON file.
from_preset Load config from a preset.
to_file Save config to YAML or JSON file.

with_uint8_input

def with_uint8_input(self, **kwargs) -> "CompileConfig"

Return a copy with uint8_input settings enabled.

with_preprocessing

def with_preprocessing(self, **kwargs) -> "CompileConfig"

Return a copy with preprocessing settings enabled.

with_hessian_quant

def with_hessian_quant(self, **kwargs) -> "CompileConfig"

Return a copy with hessian_quant settings enabled.

with_layer_bias_correction

def with_layer_bias_correction(self, **kwargs) -> "CompileConfig"

Return a copy with layer_bias_correction settings enabled.

with_mod

def with_mod(self, **kwargs) -> "CompileConfig"

Return a copy with mod settings enabled.

with_llm

def with_llm(self, **kwargs) -> "CompileConfig"

Return a copy with llm settings enabled.

with_search_weight_scale

def with_search_weight_scale(self, **kwargs) -> "CompileConfig"

Return a copy with search_weight_scale settings enabled.

with_load_scale

def with_load_scale(self, **kwargs) -> "CompileConfig"

Return a copy with load_scale settings enabled.

with_save_sample

def with_save_sample(self, **kwargs) -> "CompileConfig"

Return a copy with save_sample settings enabled.

with_group_wise

def with_group_wise(self, **kwargs) -> "CompileConfig"

Return a copy with group_wise settings enabled.

from_file

def from_file(cls, path: Union[str, Path]) -> "CompileConfig"

Load config from YAML or JSON file.

from_preset

def from_preset(cls, name: str) -> "CompileConfig"

Load config from a preset.

to_file

def to_file(self, path: Union[str, Path]) -> None

Save config to YAML or JSON file.