콘텐츠로 이동

qbcompiler.configs.models

Auto-generated Pydantic models from config_schema.yaml.

Attributes

Name Description
SCHEMA_VERSION

SCHEMA_VERSION

SCHEMA_VERSION = "1.0.0"

Uint8InputConfig

class Uint8InputConfig(BaseModel)

Configuration for uint8 input handling

Defines whether inputs should be treated as uint8 and which specific inputs to apply this to.

Parameters

  • apply — bool. If true, treat specified inputs as uint8
  • inputs — List[str]. List of input names to treat as uint8. If empty and apply is true, applies to all inputs
  • division_factor — float. Division factor for uint8 to float conversion (e.g., 255.0 for [0,1], 127.5 for [0,2])

Attributes

Name Description
model_config
apply
inputs
division_factor

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False, description="If true, treat specified inputs as uint8"
)

inputs

inputs: List[str] = Field(
    default=[],
    description="List of input names to treat as uint8. If empty and apply is true, applies to all inputs",
)

division_factor

division_factor: float = Field(
    default=255.0,
    alias="divisionFactor",
    description="Division factor for uint8 to float conversion (e.g., 255.0 for [0,1], 127.5 for [0,2])",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "Uint8InputConfig"

Return a copy with updated fields.

PreprocessingConfig

class PreprocessingConfig(BaseModel)

Configuration for input preprocessing pipeline

Defines preprocessing operations to be applied to model inputs, including operations like resize, normalize, color conversion, etc. Each pipeline entry is {"op": <name>, ...op keys}; unknown keys are rejected. resize/letterbox take "backend": "torch" (default, float interpolate), "pil" (Pillow, bit-exact, bilinear/bicubic) or "opencv" (cv2 INTER_LINEAR, bit-exact), so calibration sees the same pixels as an evaluator built on that library. letterbox "alignType" is 0 (default: center, padding split around the image) or 1 (top-left: padding only at the bottom and right, as YOLOX and DAMO-YOLO use). letterbox "roundType" is 0 (default: each backend's rounding of the resized size) or 1 (floor: upstream YOLOX and DAMO-YOLO's int(w * r)); a fuseIntoFirstLayer letterbox ignores it, since its exact odd decimation has no fraction to round.

Parameters

  • apply — bool. If true, apply preprocessing pipeline
  • auto_convert_format — bool. If true, automatically convert input format
  • pipeline — List[Dict[str, Any]]. List of preprocessing operations to apply globally
  • input_configs — Dict[str, Any]. Per-input preprocessing configurations. Keys are input names

Attributes

Name Description
model_config
apply
auto_convert_format
pipeline
input_configs

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False, description="If true, apply preprocessing pipeline"
)

auto_convert_format

auto_convert_format: bool = Field(
    default=False,
    alias="autoConvertFormat",
    description="If true, automatically convert input format",
)

pipeline

pipeline: Any = Field(
    default=[], description="List of preprocessing operations to apply globally"
)

input_configs

input_configs: Any = Field(
    default={},
    alias="inputConfigs",
    description="Per-input preprocessing configurations. Keys are input names",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "PreprocessingConfig"

Return a copy with updated fields.

ResourceManagementConfig

class ResourceManagementConfig(BaseModel)

Configuration for resource management during model compilation

Controls GPU and memory management settings during the compilation process.

Parameters

  • weight_dtype — str. Weight data type for calibration (e.g., 'float32', 'float16')
  • gpu_memory_budget_mb — int. Device memory budget for weight quantization in MiB; 0 = unlimited, -1 = auto
  • weight_memory — WeightMemory. Weight memory management configuration

Attributes

Name Description
model_config
weight_dtype
gpu_memory_budget_mb
weight_memory

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

weight_dtype

weight_dtype: str = Field(
    default="float32",
    alias="weightDtype",
    description="Weight data type for calibration (e.g., 'float32', 'float16')",
)

gpu_memory_budget_mb

gpu_memory_budget_mb: int = Field(
    default=-1,
    alias="gpuMemoryBudgetMB",
    description="Device memory budget for weight quantization in MiB; 0 = unlimited, -1 = auto",
)

weight_memory

weight_memory: WeightMemory = Field(
    default_factory=WeightMemory, alias="weightMemory"
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "ResourceManagementConfig"

Return a copy with updated fields.

ResourceManagementConfig.WeightMemory

class WeightMemory(BaseModel)

Weight memory management configuration

Parameters

  • method — int. Weight memory management method index:
    0: DeleteFloat - Delete float weights after quantization.
    1: SaveFloat - Save float weights to disk.
    2: MoveFloat - Move float weights to CPU.
    3: KeepFloat - Keep float weights in memory.
    4: KeepAll - Keep all weights in memory.

Attributes

Name Description
method_list
method

method_list

method_list: List[str] = Field(
    default=["DeleteFloat", "SaveFloat", "MoveFloat", "KeepFloat", "KeepAll"],
    alias="methodList",
)

method

method: int = Field(default=0, alias="method")

CalibrationConfig

class CalibrationConfig(BaseModel)

Configuration for calibration during quantization

Defines calibration and quantization parameterization used to derive activation/weight scales and related statistics during quantized compilation.

Parameters

  • method — int. Calibration method index:
    0: WChALayer - Weight per-channel, Activation per-layer.
    1: WChAMulti - Weight per-channel, Activation multi-layer.
    2: WChALayerZeropoint - Weight per-channel, Activation per-layer with zeropoint.
    3: WChAMultiZeropoint - Weight per-channel, Activation multi-layer with zeropoint.
  • output — int. Output quantization type index:
    0: Layer - Per-layer quantization.
    1: Ch - Per-channel quantization.
    2: Sigmoid - Sigmoid-based quantization.
  • mode — int. Quantization mode index:
    0: Max - Maximum value calibration.
    1: MaxPercentile - Maximum percentile calibration.
    2: Histogram - Histogram-based calibration.
    3: GeometricSketch - Geometric-bin relative-error sketch (DDSketch family).
  • act_scale_min — float. Minimum allowed activation scale (lower bound clamp)
  • act16_scale_min — float. Minimum 16-bit activation scale (actScaleMin / 256)
  • weight_scale_min — float. Minimum allowed weight scale (lower bound clamp)
  • weight16_scale_min — float. Minimum 16-bit weight scale (weightScaleMin / 256)
  • min_clip_ratio — float. Minimum clip ratio constraint applied during calibration
  • max_calib_data_size — int. Maximum number of calibration samples kept after loading or generation
  • max_sample_size_for_quant_scheme — int. Maximum number of calibration samples used per quant scheme stage
  • max_percentile — MaxPercentile. MaxPercentile mode configuration
  • fast_dist — FastDist. Fast distribution calibration configuration
  • histogram — Histogram. Histogram-based calibration configuration
  • geometric_sketch — GeometricSketch. GeometricSketch mode configuration
  • layer_overrides — LayerOverrides. Layer-specific override settings for calibration
  • statistics — Statistics. Statistics save/load configuration with percentile selection
  • group_lut — GroupLut. LUT grouping algorithm configuration

Attributes

Name Description
model_config
method_list
method
output_list
output
mode_list
mode
act_scale_min
act16_scale_min
weight_scale_min
weight16_scale_min
min_clip_ratio
max_calib_data_size
max_sample_size_for_quant_scheme
max_percentile
fast_dist
histogram
geometric_sketch
layer_overrides
statistics
group_lut

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

method_list

method_list: List[str] = Field(
    default=["WChALayer", "WChAMulti", "WChALayerZeropoint", "WChAMultiZeropoint"],
    alias="methodList",
)

method

method: int = Field(default=1, alias="method")

output_list

output_list: List[str] = Field(
    default=["Layer", "Ch", "Sigmoid"], alias="outputList"
)

output

output: int = Field(default=0, alias="output")

mode_list

mode_list: List[str] = Field(
    default=["Max", "MaxPercentile", "Histogram", "GeometricSketch"],
    alias="modeList",
)

mode

mode: int = Field(default=1, alias="mode")

act_scale_min

act_scale_min: float = Field(
    default=0.0005,
    alias="actScaleMin",
    description="Minimum allowed activation scale (lower bound clamp)",
    ge=0,
    le=1,
)

act16_scale_min

act16_scale_min: float = Field(
    default=1.953125e-06,
    alias="act16ScaleMin",
    description="Minimum 16-bit activation scale (actScaleMin / 256)",
)

weight_scale_min

weight_scale_min: float = Field(
    default=1e-06,
    alias="weightScaleMin",
    description="Minimum allowed weight scale (lower bound clamp)",
    ge=0,
    le=1,
)

weight16_scale_min

weight16_scale_min: float = Field(
    default=3.90625e-09,
    alias="weight16ScaleMin",
    description="Minimum 16-bit weight scale (weightScaleMin / 256)",
)

min_clip_ratio

min_clip_ratio: float = Field(
    default=-1,
    alias="minClipRatio",
    description="Minimum clip ratio constraint applied during calibration",
    ge=-1,
    le=1,
)

max_calib_data_size

max_calib_data_size: int = Field(
    default=-1,
    alias="maxCalibDataSize",
    description="Maximum number of calibration samples kept after loading or generation",
    ge=-1,
)

max_sample_size_for_quant_scheme

max_sample_size_for_quant_scheme: int = Field(
    default=16,
    alias="maxSampleSizeForQuantScheme",
    description="Maximum number of calibration samples used per quant scheme stage",
    ge=1,
)

max_percentile

max_percentile: MaxPercentile = Field(
    default_factory=MaxPercentile, alias="maxPercentile"
)

fast_dist

fast_dist: FastDist = Field(default_factory=FastDist, alias="fastDist")

histogram

histogram: Histogram = Field(default_factory=Histogram, alias="histogram")

geometric_sketch

geometric_sketch: GeometricSketch = Field(
    default_factory=GeometricSketch, alias="geometricSketch"
)

layer_overrides

layer_overrides: LayerOverrides = Field(
    default_factory=LayerOverrides, alias="layerOverrides"
)

statistics

statistics: Statistics = Field(default_factory=Statistics, alias="statistics")

group_lut

group_lut: GroupLut = Field(default_factory=GroupLut, alias="groupLut")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "CalibrationConfig"

Return a copy with updated fields.

CalibrationConfig.MaxPercentile

class MaxPercentile(BaseModel)

Attributes

Name Description
model_config MaxPercentile mode configuration
percentile
topk_ratio
max_each
max_total
per_ch_divisor

model_config

model_config = ConfigDict(populate_by_name=True)

MaxPercentile mode configuration

Parameters

  • percentile — float. Percentile value for maxPercentile mode
  • topk_ratio — float. Top-k ratio used in maxPercentile mode
  • max_each — int. Maximum number of samples processed per iteration
  • max_total — int. Total maximum number of samples
  • per_ch_divisor — int. Divisor for per-channel buffer capacity (bufferCap = max(maxTotal / perChDivisor, maxEach))

percentile

percentile: float = Field(
    default=0.9999, description="Percentile value for maxPercentile mode"
)

topk_ratio

topk_ratio: float = Field(
    default=0.01,
    alias="topKRatio",
    description="Top-k ratio used in maxPercentile mode",
)

max_each

max_each: int = Field(
    default=128,
    alias="maxEach",
    description="Maximum number of samples processed per iteration",
)

max_total

max_total: int = Field(
    default=65536,
    alias="maxTotal",
    description="Total maximum number of samples",
)

per_ch_divisor

per_ch_divisor: int = Field(
    default=16,
    alias="perChDivisor",
    description="Divisor for per-channel buffer capacity (bufferCap = max(maxTotal / perChDivisor, maxEach))",
    ge=1,
)

CalibrationConfig.FastDist

class FastDist(BaseModel)

Attributes

Name Description
model_config Fast distribution calibration configuration
size_cali
kernel_size
stack_size

model_config

model_config = ConfigDict(populate_by_name=True)

Fast distribution calibration configuration

Parameters

  • size_cali — int.
  • kernel_size — int.
  • stack_size — int.

size_cali

size_cali: int = Field(default=100, alias="sizeCali")

kernel_size

kernel_size: int = Field(default=9, alias="kernelSize")

stack_size

stack_size: int = Field(default=32768, alias="stackSize")

CalibrationConfig.Histogram

class Histogram(BaseModel)

Attributes

Name Description
model_config Histogram-based calibration configuration
search_type_list
search_type
percentile
use_gpu
num_bins
num_samples
buffer_size
min_bin_width
search_percentile_min
search_percentile_max
num_search

model_config

model_config = ConfigDict(populate_by_name=True)

Histogram-based calibration configuration

Parameters

  • search_type — int. Search type for histogram calibration:
    0: Percentile.
    1: MSE.
    2: KL.
  • percentile — float. Percentile value for histogram calibration
  • use_gpu — bool. Use GPU for histogram computation
  • num_bins — int. Number of bins for histogram
  • num_samples — int. Number of samples for histogram calibration
  • buffer_size — int. Buffer size for histogram computation (-1 for auto)
  • min_bin_width — float. Minimum bin width for histogram
  • search_percentile_min — float. Minimum search percentile
  • search_percentile_max — float. Maximum search percentile
  • num_search — int. Number of search iterations

search_type_list

search_type_list: List[str] = Field(
    default=["Percentile", "MSE", "KL"], alias="searchTypeList"
)

search_type

search_type: int = Field(default=0, alias="searchType")

percentile

percentile: float = Field(
    default=0.9999, description="Percentile value for histogram calibration"
)

use_gpu

use_gpu: bool = Field(
    default=True,
    alias="useGPU",
    description="Use GPU for histogram computation",
)

num_bins

num_bins: int = Field(
    default=256, alias="numBins", description="Number of bins for histogram"
)

num_samples

num_samples: int = Field(
    default=128,
    alias="numSamples",
    description="Number of samples for histogram calibration",
)

buffer_size

buffer_size: int = Field(
    default=-1,
    alias="bufferSize",
    description="Buffer size for histogram computation (-1 for auto)",
)

min_bin_width

min_bin_width: float = Field(
    default=1e-06,
    alias="minBinWidth",
    description="Minimum bin width for histogram",
)

search_percentile_min

search_percentile_min: float = Field(
    default=0.9999,
    alias="searchPercentileMin",
    description="Minimum search percentile",
)

search_percentile_max

search_percentile_max: float = Field(
    default=1.0,
    alias="searchPercentileMax",
    description="Maximum search percentile",
)
num_search: int = Field(
    default=128, alias="numSearch", description="Number of search iterations"
)

CalibrationConfig.GeometricSketch

class GeometricSketch(BaseModel)

Attributes

Name Description
model_config GeometricSketch mode configuration
clip_ratio8
clip_ratio16
relative_error
exp_min
exp_max
bin_budget

model_config

model_config = ConfigDict(populate_by_name=True)

GeometricSketch mode configuration

Parameters

  • clip_ratio8 — float. Fraction clipped from each tail for INT8; must be >= 0 and < 0.5
  • clip_ratio16 — float. Fraction clipped from each tail for INT16; must be >= 0 and < 0.5
  • relative_error — float. Geometric bin relative error; bin k holds magnitudes in ((1+r)^(k-1), (1+r)^k]
  • exp_min — int. Smallest power-of-two exponent the magnitude grid covers; values below clamp to the end bin
  • exp_max — int. Largest power-of-two exponent the magnitude grid covers; values above clamp to the end bin
  • bin_budget — int. Stored bins per sign ladder and channel; magnitudes more than that many bins below the channel's largest fall into one floor bin

clip_ratio8

clip_ratio8: float = Field(
    default=5e-06,
    alias="clipRatio8",
    description="Fraction clipped from each tail for INT8; must be >= 0 and < 0.5",
    ge=0,
    lt=0.5,
)

clip_ratio16

clip_ratio16: float = Field(
    default=1e-06,
    alias="clipRatio16",
    description="Fraction clipped from each tail for INT16; must be >= 0 and < 0.5",
    ge=0,
    lt=0.5,
)

relative_error

relative_error: float = Field(
    default=0.05,
    alias="relativeError",
    description="Geometric bin relative error; bin k holds magnitudes in ((1+r)^(k-1), (1+r)^k]",
    gt=0,
    lt=1,
)

exp_min

exp_min: int = Field(
    default=-14,
    alias="expMin",
    description="Smallest power-of-two exponent the magnitude grid covers; values below clamp to the end bin",
)

exp_max

exp_max: int = Field(
    default=24,
    alias="expMax",
    description="Largest power-of-two exponent the magnitude grid covers; values above clamp to the end bin",
)

bin_budget

bin_budget: int = Field(
    default=256,
    alias="binBudget",
    description="Stored bins per sign ladder and channel; magnitudes more than that many bins below the channel's largest fall into one floor bin",
    ge=8,
)

CalibrationConfig.LayerOverrides

class LayerOverrides(BaseModel)

Attributes

Name Description
model_config Layer-specific override settings for calibration
act_scale_min
percentile
method

model_config

model_config = ConfigDict(populate_by_name=True)

Layer-specific override settings for calibration

Parameters

  • act_scale_min — dict. Per-layer activation scale minimum overrides. Keys are actScaleMin values (e.g. '0.0005'), values are lists of layer names to apply that override. (e.g. {'0.0005' : ['layer1', 'layer2']})
  • percentile — dict. Per-layer maxPercentile percentile overrides. Keys are layer names, values are percentile floats. (e.g. {'/model/layer0/conv': 0.999, '/model/layer5/attn': 0.99})
  • method — dict. Per-layer calibration method overrides. Keys are layer names, values are method ints (0=WChALayer, 1=WChAMulti, 2=WChALayerZeropoint, 3=WChAMultiZeropoint). (e.g. {'/model/layer0/conv': 1, '/model/layer5/attn': 3})

act_scale_min

act_scale_min: Any = Field(
    default={},
    alias="actScaleMin",
    description="Per-layer activation scale minimum overrides. Keys are actScaleMin values (e.g. '0.0005'), values are lists of layer names to apply that override. (e.g. {'0.0005' : ['layer1', 'layer2']})",
)

percentile

percentile: Any = Field(
    default={},
    description="Per-layer maxPercentile percentile overrides. Keys are layer names, values are percentile floats. (e.g. {'/model/layer0/conv': 0.999, '/model/layer5/attn': 0.99})",
)

method

method: Any = Field(
    default={},
    description="Per-layer calibration method overrides. Keys are layer names, values are method ints (0=WChALayer, 1=WChAMulti, 2=WChALayerZeropoint, 3=WChAMultiZeropoint). (e.g. {'/model/layer0/conv': 1, '/model/layer5/attn': 3})",
)

CalibrationConfig.Statistics

class Statistics(BaseModel)

Attributes

Name Description
model_config Statistics save/load configuration with percentile selection
apply
save_path
load_path
percentiles
percentile_index

model_config

model_config = ConfigDict(populate_by_name=True)

Statistics save/load configuration with percentile selection

Parameters

  • apply — bool. Enable statistics save/load
  • save_path — str. Path to save statistics. If empty, not saved
  • load_path — str. Path to load statistics. If empty, not loaded
  • percentiles — List[float]. List of percentile candidates
  • percentile_index — int. Index into percentiles list to select active percentile

apply

apply: bool = Field(default=False, description="Enable statistics save/load")

save_path

save_path: str = Field(
    default="",
    alias="savePath",
    description="Path to save statistics. If empty, not saved",
)

load_path

load_path: str = Field(
    default="",
    alias="loadPath",
    description="Path to load statistics. If empty, not loaded",
)

percentiles

percentiles: List[float] = Field(
    default=[0.9999, 0.999, 0.99, 0.9],
    description="List of percentile candidates",
)

percentile_index

percentile_index: int = Field(
    default=0,
    alias="percentileIndex",
    description="Index into percentiles list to select active percentile",
)

CalibrationConfig.GroupLut

class GroupLut(BaseModel)

Attributes

Name Description
model_config LUT grouping algorithm configuration
irls_iter
dro_eps
cover_floor

model_config

model_config = ConfigDict(populate_by_name=True)

LUT grouping algorithm configuration

Parameters

  • irls_iter — int. Number of IRLS iterations for optimal scale computation (1 = single WLS)
  • dro_eps — float. DRO worst-point coefficient; larger = more conservative clipping (0 = disable)
  • cover_floor — float. LUTAM cluster scale lower bound as fraction of coverage

irls_iter

irls_iter: int = Field(
    default=3,
    alias="irlsIter",
    description="Number of IRLS iterations for optimal scale computation (1 = single WLS)",
    ge=1,
)

dro_eps

dro_eps: float = Field(
    default=0,
    alias="droEps",
    description="DRO worst-point coefficient; larger = more conservative clipping (0 = disable)",
    ge=0,
)

cover_floor

cover_floor: float = Field(
    default=0.85,
    alias="coverFloor",
    description="LUTAM cluster scale lower bound as fraction of coverage",
    ge=0,
    le=1,
)

BitConfig

class BitConfig(BaseModel)

Configuration for bit precision

Defines bit-width parameterization for activations and weights used in mixed-precision quantization (e.g., attention and FFN components).

Parameters

Attributes

Name Description
model_config
transformer
save_info
layer_overrides

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

transformer

transformer: Transformer = Field(default_factory=Transformer, alias="transformer")

save_info

save_info: SaveInfo = Field(default_factory=SaveInfo, alias="saveInfo")

layer_overrides

layer_overrides: LayerOverrides = Field(
    default_factory=LayerOverrides, alias="layerOverrides"
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "BitConfig"

Return a copy with updated fields.

BitConfig.Transformer

class Transformer(BaseModel)

Attributes

Name Description
model_config Transformer-specific bit-width configuration
activation
weight
mixed_precision

model_config

model_config = ConfigDict(populate_by_name=True)

Transformer-specific bit-width configuration

Parameters

activation

activation: Activation = Field(default_factory=Activation, alias="activation")

weight

weight: Weight = Field(default_factory=Weight, alias="weight")

mixed_precision

mixed_precision: MixedPrecision = Field(
    default_factory=MixedPrecision, alias="mixedPrecision"
)

BitConfig.Transformer.Activation

class Activation(BaseModel)

Attributes

Name Description
model_config Activation bit-widths for transformer components
query
key
value
output
head
router
ffn

model_config

model_config = ConfigDict(populate_by_name=True)

Activation bit-widths for transformer components

Parameters

  • query — int. Query activation bit-width
  • key — int. Key activation bit-width
  • value — int. Value activation bit-width
  • output — int. Output activation bit-width
  • head — int. Head activation bit-width
  • router — int. MoE router gate activation bit-width
  • ffn — Ffn. FFN activation bit-widths (int shorthand sets all sublayers)

query

query: int = Field(default=8, description="Query activation bit-width")

key

key: int = Field(default=8, description="Key activation bit-width")

value

value: int = Field(default=8, description="Value activation bit-width")

output

output: int = Field(default=16, description="Output activation bit-width")

head

head: int = Field(default=8, description="Head activation bit-width")

router

router: int = Field(
    default=8, description="MoE router gate activation bit-width"
)

ffn

ffn: Ffn = Field(default_factory=Ffn, alias="ffn")

BitConfig.Transformer.Activation.Ffn

class Ffn(BaseModel)

FFN activation bit-widths (int shorthand sets all sublayers)

Parameters

  • up — int. FFN up-projection activation bit-width
  • gate — int. FFN gate (SwiGLU) activation bit-width
  • down — int. FFN down-projection activation bit-width

Attributes

Name Description
up
gate
down

up

up: int = Field(
    default=16, description="FFN up-projection activation bit-width"
)

gate

gate: int = Field(
    default=16, description="FFN gate (SwiGLU) activation bit-width"
)

down

down: int = Field(
    default=16, description="FFN down-projection activation bit-width"
)

Methods

Name Description
expand_int_shorthand

expand_int_shorthand

def expand_int_shorthand(cls, v)

BitConfig.Transformer.Weight

class Weight(BaseModel)

Attributes

Name Description
model_config Weight bit-widths for transformer components
query
key
value
output
head
router
ffn

model_config

model_config = ConfigDict(populate_by_name=True)

Weight bit-widths for transformer components

Parameters

  • query — int. Query weight bit-width
  • key — int. Key weight bit-width
  • value — int. Value weight bit-width
  • output — int. Output weight bit-width
  • head — int. Head weight bit-width
  • router — int. MoE router gate weight bit-width
  • ffn — Ffn. FFN weight bit-widths (int shorthand sets all sublayers)

query

query: int = Field(default=8, description="Query weight bit-width")

key

key: int = Field(default=8, description="Key weight bit-width")

value

value: int = Field(default=8, description="Value weight bit-width")

output

output: int = Field(default=8, description="Output weight bit-width")

head

head: int = Field(default=8, description="Head weight bit-width")

router

router: int = Field(
    default=8, description="MoE router gate weight bit-width"
)

ffn

ffn: Ffn = Field(default_factory=Ffn, alias="ffn")

BitConfig.Transformer.Weight.Ffn

class Ffn(BaseModel)

FFN weight bit-widths (int shorthand sets all sublayers)

Parameters

  • up — int. FFN up-projection weight bit-width
  • gate — int. FFN gate (SwiGLU) weight bit-width
  • down — int. FFN down-projection weight bit-width

Attributes

Name Description
up
gate
down

up

up: int = Field(
    default=8, description="FFN up-projection weight bit-width"
)

gate

gate: int = Field(
    default=8, description="FFN gate (SwiGLU) weight bit-width"
)

down

down: int = Field(
    default=8, description="FFN down-projection weight bit-width"
)

Methods

Name Description
expand_int_shorthand

expand_int_shorthand

def expand_int_shorthand(cls, v)

BitConfig.Transformer.MixedPrecision

class MixedPrecision(BaseModel)

Attributes

Name Description
model_config Mixed precision configuration
weight
activation

model_config

model_config = ConfigDict(populate_by_name=True)

Mixed precision configuration

Parameters

  • weight — Weight. Mixed precision configuration for weights
  • activation — Activation. Per-layer activation mixed precision configuration

weight

weight: Weight = Field(default_factory=Weight, alias="weight")

activation

activation: Activation = Field(
    default_factory=Activation, alias="activation"
)

BitConfig.Transformer.MixedPrecision.Weight

class Weight(BaseModel)

Attributes

Name Description
model_config Mixed precision configuration for weights
apply
type_wise
prune
bit_2
bit_4
bit_8
importance_threshold_low
importance_threshold_high

model_config

model_config = ConfigDict(populate_by_name=True)

Mixed precision configuration for weights

Parameters

  • apply — bool. If true, apply mixed-precision according to the specified bit-widths
  • type_wise — bool. Apply type-wise mixed precision
  • prune — float. Pruning ratio
  • bit_2 — float. Ratio of 2-bit quantization
  • bit_4 — float. Ratio of 4-bit quantization
  • bit_8 — float. Ratio of 8-bit quantization
  • importance_threshold_low — float. Low importance threshold
  • importance_threshold_high — float. High importance threshold

apply

apply: bool = Field(
    default=False,
    description="If true, apply mixed-precision according to the specified bit-widths",
)

type_wise

type_wise: bool = Field(
    default=True,
    alias="typeWise",
    description="Apply type-wise mixed precision",
)

prune

prune: float = Field(default=0, description="Pruning ratio")

bit_2

bit_2: float = Field(
    default=0,
    validation_alias=AliasChoices("bit_2", "bit2"),
    description="Ratio of 2-bit quantization",
)

bit_4

bit_4: float = Field(
    default=0,
    validation_alias=AliasChoices("bit_4", "bit4"),
    description="Ratio of 4-bit quantization",
)

bit_8

bit_8: float = Field(
    default=1,
    validation_alias=AliasChoices("bit_8", "bit8"),
    description="Ratio of 8-bit quantization",
)

importance_threshold_low

importance_threshold_low: float = Field(
    default=-1,
    alias="importanceThreshold_low",
    description="Low importance threshold",
)

importance_threshold_high

importance_threshold_high: float = Field(
    default=-1,
    alias="importanceThreshold_high",
    description="High importance threshold",
)

BitConfig.Transformer.MixedPrecision.Activation

class Activation(BaseModel)

Attributes

Name Description
model_config Per-layer activation mixed precision configuration
apply
ratio_16bit
importance_threshold
search_range

model_config

model_config = ConfigDict(populate_by_name=True)

Per-layer activation mixed precision configuration

Parameters

  • apply — bool. If true, apply per-layer activation mixed precision
  • ratio_16bit — float. Ratio of layers assigned 16-bit (used when importanceThreshold < 0)
  • importance_threshold — float. Normalized importance threshold for 16-bit assignment (negative = use ratio_16bit)
  • search_range — int. Target layer range: -1=all layers, N=first N layers

apply

apply: bool = Field(
    default=False,
    description="If true, apply per-layer activation mixed precision",
)

ratio_16bit

ratio_16bit: float = Field(
    default=0.45,
    alias="ratio16Bit",
    description="Ratio of layers assigned 16-bit (used when importanceThreshold < 0)",
)

importance_threshold

importance_threshold: float = Field(
    default=-1,
    alias="importanceThreshold",
    description="Normalized importance threshold for 16-bit assignment (negative = use ratio_16bit)",
)

search_range

search_range: int = Field(
    default=-1,
    alias="searchRange",
    description="Target layer range: -1=all layers, N=first N layers",
)

BitConfig.SaveInfo

class SaveInfo(BaseModel)

Attributes

Name Description
model_config Bit allocation save/load configuration
save_path
load_path

model_config

model_config = ConfigDict(populate_by_name=True)

Bit allocation save/load configuration

Parameters

  • save_path — str. Path to save the bit allocation. If empty, not saved
  • load_path — str. Path to load the bit allocation. If empty, not loaded

save_path

save_path: str = Field(
    default="",
    alias="savePath",
    description="Path to save the bit allocation. If empty, not saved",
)

load_path

load_path: str = Field(
    default="",
    alias="loadPath",
    description="Path to load the bit allocation. If empty, not loaded",
)

BitConfig.LayerOverrides

class LayerOverrides(BaseModel)

Attributes

Name Description
model_config Layer-specific bit-width override settings
activation_16bits
weight_16bits
weight_8bits

model_config

model_config = ConfigDict(populate_by_name=True)

Layer-specific bit-width override settings

Parameters

  • activation_16bits — list[string]. Layer names to force 16-bit activations
  • weight_16bits — list[string]. Layer names to force 16-bit weights
  • weight_8bits — list[string]. Layer names to force 8-bit weights

activation_16bits

activation_16bits: List[str] = Field(
    default=[],
    alias="activation16Bits",
    description="Layer names to force 16-bit activations",
)

weight_16bits

weight_16bits: List[str] = Field(
    default=[],
    alias="weight16Bits",
    description="Layer names to force 16-bit weights",
)

weight_8bits

weight_8bits: List[str] = Field(
    default=[],
    alias="weight8Bits",
    description="Layer names to force 8-bit weights",
)

HessianQuantConfig

class HessianQuantConfig(BaseModel)

Configuration for HessianQuant algorithm

Defines parameters controlling whether and how HessianQuant is applied during quantization, including layer-level inclusion/exclusion lists.

Parameters

  • apply — bool. If true, apply HessianQuant
  • hessian_dtype — str. Storage dtype for the accumulated HessianQuant Hessian. bf16 halves its memory footprint (host RAM when the Hessian lives on CPU, VRAM when on GPU) — critical for large models such as MoE with thousands of expert FFN Hessians. Compute stays float32 regardless: the per-batch matmul and accumulation run in float32 and the solve upcasts back to float32; only the persistent accumulator is bfloat16. Accepted values:
    "fp32" - Store the Hessian in float32. The default.
    "bf16" - Store the Hessian in bfloat16 (half the memory).
  • solver — str. HessianQuant solve algorithm used to compute per-layer integer weights. symmetric targets the output on the quantized input; the asymmetric solvers target the original-weight float output and differ in how they absorb the input mismatch. Accepted values:
    "symmetric" - The default. min ||(W - Q) X_q||^2: sequential greedy rounding via a Cholesky-based Hessian solve.
    "asymmetric_causal" - min ||W X_fp - Q X_q||^2: each column's input-mismatch residual is fed only to the columns after it, scaled by alpha.
    "asymmetric_refit" - min ||W X_fp - Q X_q||^2: least-squares refit W^T = H^-1 G W^T of every column, then a symmetric solve around W.
  • rescomp — bool. Enable residual-compensation correction. When true, each solver additionally applies an R correction term that compensates for accumulated weight drift from inter-block error propagation. Requires crossGram collection even for solver=symmetric.
  • attributes — Attributes. HessianQuant algorithm attributes

Attributes

Name Description
model_config
apply
hessian_dtype
solver
rescomp
attributes

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(default=False, description="If true, apply HessianQuant")

hessian_dtype

hessian_dtype: str = Field(
    default="fp32",
    alias="hessianDtype",
    description="Storage dtype for the accumulated HessianQuant Hessian. bf16 halves its memory footprint (host RAM when the Hessian lives on CPU, VRAM when on GPU) — critical for large models such as MoE with thousands of expert FFN Hessians. Compute stays float32 regardless: the per-batch matmul and accumulation run in float32 and the solve upcasts back to float32; only the persistent accumulator is bfloat16.",
)

solver

solver: str = Field(
    default="symmetric",
    description="HessianQuant solve algorithm used to compute per-layer integer weights. symmetric targets the output on the quantized input; the asymmetric solvers target the original-weight float output and differ in how they absorb the input mismatch.",
)

rescomp

rescomp: bool = Field(
    default=False,
    description="Enable residual-compensation correction. When true, each solver additionally applies an R correction term that compensates for accumulated weight drift from inter-block error propagation. Requires crossGram collection even for solver=symmetric.",
)

attributes

attributes: Attributes = Field(default_factory=Attributes, alias="attributes")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "HessianQuantConfig"

Return a copy with updated fields.

HessianQuantConfig.Attributes

class Attributes(BaseModel)

Attributes

Name Description
model_config HessianQuant algorithm attributes
act_order
block_size
perc_damp
apply_layers
exclude_layers
alpha

model_config

model_config = ConfigDict(populate_by_name=True)

HessianQuant algorithm attributes

Parameters

  • act_order — bool. If true, use activation order
  • block_size — int. Block size used for HessianQuant
  • perc_damp — float. Percentage dampening factor
  • apply_layers — List[str]. Layer names to apply HessianQuant. If empty, applies to all eligible layers
  • exclude_layers — List[str]. Layer names to exclude from HessianQuant
  • alpha — float. Scale of the secondary correction terms: the asymmetric_causal P term and the rescomp R term. symmetric and asymmetric_refit use it only through rescomp.

act_order

act_order: bool = Field(
    default=True, alias="actOrder", description="If true, use activation order"
)

block_size

block_size: int = Field(
    default=256,
    alias="blockSize",
    description="Block size used for HessianQuant",
)

perc_damp

perc_damp: float = Field(
    default=0.01, alias="percDamp", description="Percentage dampening factor"
)

apply_layers

apply_layers: List[str] = Field(
    default=[],
    alias="applyLayers",
    description="Layer names to apply HessianQuant. If empty, applies to all eligible layers",
)

exclude_layers

exclude_layers: List[str] = Field(
    default=[],
    alias="excludeLayers",
    description="Layer names to exclude from HessianQuant",
)

alpha

alpha: float = Field(
    default=0.25,
    description="Scale of the secondary correction terms: the asymmetric_causal P term and the rescomp R term. symmetric and asymmetric_refit use it only through rescomp.",
)

BiasCorrectionConfig

class BiasCorrectionConfig(BaseModel)

Configuration for in-schedule integer bias correction

During weight quantization, measures the per-channel activation error of each quantized convolution (plain, depthwise or transposed; not grouped) against the original-weight float path over every calibration sample, folds the correction into the integer bias within its range, skipping a channel whose slope is too flat to invert, and patches the already-produced integer activations so downstream layers quantize against corrected outputs. Corrections propagate layer by layer in one pass; there is no damping rate and no iteration count. Under group-wise quantization both streams start each group from the float boundary batch, so the correction covers the error the group itself adds. A run the engine cannot drive leaf by leaf (expert aggregation, an external input narrower than its per-channel qparam) keeps the calibrated bias. A layer whose LUT window is re-chosen is measured on its accumulator, ahead of the window; any other layer on its requantized codes.

Parameters

  • apply — bool. If true, correct systematic per-layer quantization bias
  • attributes — Attributes. Bias correction attributes

Attributes

Name Description
model_config
apply
attributes

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=True,
    description="If true, correct systematic per-layer quantization bias",
)

attributes

attributes: Attributes = Field(default_factory=Attributes, alias="attributes")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "BiasCorrectionConfig"

Return a copy with updated fields.

BiasCorrectionConfig.Attributes

class Attributes(BaseModel)

Attributes

Name Description
model_config Bias correction attributes
apply_layers
exclude_layers

model_config

model_config = ConfigDict(populate_by_name=True)

Bias correction attributes

Parameters

  • apply_layers — List[str]. Layer names to apply bias correction. If empty, applies to all eligible layers
  • exclude_layers — List[str]. Layer names to exclude from bias correction

apply_layers

apply_layers: List[str] = Field(
    default=[],
    alias="applyLayers",
    description="Layer names to apply bias correction. If empty, applies to all eligible layers",
)

exclude_layers

exclude_layers: List[str] = Field(
    default=[],
    alias="excludeLayers",
    description="Layer names to exclude from bias correction",
)

ModConfig

class ModConfig(BaseModel)

Configuration for Minimum Output Difference algorithm

Defines parameters controlling whether and how MOD is applied during quantization, including layer-level inclusion/exclusion lists.

Parameters

  • apply — bool. If true, apply MOD
  • attributes — Attributes. MOD algorithm attributes

Attributes

Name Description
model_config
apply
attributes

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(default=False, description="If true, apply MOD")

attributes

attributes: Attributes = Field(default_factory=Attributes, alias="attributes")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "ModConfig"

Return a copy with updated fields.

ModConfig.Attributes

class Attributes(BaseModel)

Attributes

Name Description
model_config MOD algorithm attributes
epochs
warmup_epochs
lr_min_ratio
save_dir
seed
apply_layers
exclude_layers
mod_after_layer_name
anchors
use_xyxy
learning_rates
training
loss
post_processing

model_config

model_config = ConfigDict(populate_by_name=True)

MOD algorithm attributes

Parameters

  • epochs — int. Number of training epochs
  • warmup_epochs — int. Number of warmup epochs
  • lr_min_ratio — float. Minimum learning rate ratio
  • save_dir — str. Directory to save MOD results
  • seed — int. Random seed for MOD
  • apply_layers — List[str]. Layer names to apply MOD. If empty, applies to all eligible layers
  • exclude_layers — List[str]. Layer names to exclude from MOD
  • mod_after_layer_name — str. Apply MOD after this layer
  • anchors — List. Anchor configurations for detection models. Nested list of anchor box sizes for detection models. Structure: List[List[List[int]]] where outer list is per detection head (e.g. small/medium/large), middle list is anchors per head, inner list is [width, height]. Example: [[[12,16],[19,36],[40,28]], [[36,75],[76,55],[72,146]], [[142,110],[192,243],[459,401]]]
  • use_xyxy — bool. Use XYXY format for bounding boxes
  • learning_rates — LearningRates. Learning rate configuration for MOD
  • training — Training. MOD training configuration
  • loss — Loss. MOD loss configuration
  • post_processing — PostProcessing. Post-processing configuration for detection models

epochs

epochs: int = Field(default=4, description="Number of training epochs")

warmup_epochs

warmup_epochs: int = Field(
    default=1, alias="warmupEpochs", description="Number of warmup epochs"
)

lr_min_ratio

lr_min_ratio: float = Field(
    default=0.0001,
    alias="lrMinRatio",
    description="Minimum learning rate ratio",
)

save_dir

save_dir: str = Field(
    default="", alias="saveDir", description="Directory to save MOD results"
)

seed

seed: int = Field(default=0, description="Random seed for MOD")

apply_layers

apply_layers: List[str] = Field(
    default=[],
    alias="applyLayers",
    description="Layer names to apply MOD. If empty, applies to all eligible layers",
)

exclude_layers

exclude_layers: List[str] = Field(
    default=[],
    alias="excludeLayers",
    description="Layer names to exclude from MOD",
)

mod_after_layer_name

mod_after_layer_name: str = Field(
    default="",
    alias="modAfterLayerName",
    description="Apply MOD after this layer",
)

anchors

anchors: Any = Field(
    default=[],
    description="Anchor configurations for detection models. Nested list of anchor box sizes for detection models. Structure: List[List[List[int]]] where outer list is per detection head (e.g. small/medium/large), middle list is anchors per head, inner list is [width, height]. Example: [[[12,16],[19,36],[40,28]], [[36,75],[76,55],[72,146]], [[142,110],[192,243],[459,401]]]",
)

use_xyxy

use_xyxy: bool = Field(
    default=False,
    alias="useXYXY",
    description="Use XYXY format for bounding boxes",
)

learning_rates

learning_rates: LearningRates = Field(
    default_factory=LearningRates, alias="learningRates"
)

training

training: Training = Field(default_factory=Training, alias="training")

loss

loss: Loss = Field(default_factory=Loss, alias="loss")

post_processing

post_processing: PostProcessing = Field(
    default_factory=PostProcessing, alias="postProcessing"
)

ModConfig.Attributes.LearningRates

class LearningRates(BaseModel)

Attributes

Name Description
model_config Learning rate configuration for MOD
act_scale
zeropoint
weight_scale
weight
bias

model_config

model_config = ConfigDict(populate_by_name=True)

Learning rate configuration for MOD

Parameters

  • act_scale — float. Learning rate for activation scale
  • zeropoint — float. Learning rate for zeropoint
  • weight_scale — float. Learning rate for weight scale
  • weight — float. Learning rate for weight
  • bias — float. Learning rate for bias

act_scale

act_scale: float = Field(
    default=0.0,
    alias="actScale",
    description="Learning rate for activation scale",
)

zeropoint

zeropoint: float = Field(
    default=0.0, description="Learning rate for zeropoint"
)

weight_scale

weight_scale: float = Field(
    default=0.0,
    alias="weightScale",
    description="Learning rate for weight scale",
)

weight

weight: float = Field(default=4e-06, description="Learning rate for weight")

bias

bias: float = Field(default=4e-06, description="Learning rate for bias")

ModConfig.Attributes.Training

class Training(BaseModel)

Attributes

Name Description
model_config MOD training configuration
batch_size
q_drop
quantize_weight
weight_scale_init
downresol_mode
scheduler_type

model_config

model_config = ConfigDict(populate_by_name=True)

MOD training configuration

Parameters

  • batch_size — int. Batch size for MOD training
  • q_drop — float. Quantization drop probability
  • quantize_weight — bool. Whether to quantize weights
  • weight_scale_init — str. Weight scale initialization method
  • downresol_mode — str. Downresolution mode
  • scheduler_type — str. LR scheduler type

batch_size

batch_size: int = Field(
    default=1, alias="batchSize", description="Batch size for MOD training"
)

q_drop

q_drop: float = Field(
    default=0.0, alias="qDrop", description="Quantization drop probability"
)

quantize_weight

quantize_weight: bool = Field(
    default=True,
    alias="quantizeWeight",
    description="Whether to quantize weights",
)

weight_scale_init

weight_scale_init: str = Field(
    default="MinMax",
    alias="weightScaleInit",
    description="Weight scale initialization method",
)

downresol_mode

downresol_mode: str = Field(
    default="STE", alias="downresolMode", description="Downresolution mode"
)

scheduler_type

scheduler_type: str = Field(
    default="Cosine", alias="schedulerType", description="LR scheduler type"
)

ModConfig.Attributes.Loss

class Loss(BaseModel)

Attributes

Name Description
model_config MOD loss configuration
type
use_outputs
kl_temperature
recon_prob
recon_coeff
lambda_0
lambda_1
lambda_2
lambda_3
custom_loss_jit_path

model_config

model_config = ConfigDict(populate_by_name=True)

MOD loss configuration

Parameters

  • type — str. Loss type (MSE, KL, etc.)
  • use_outputs — bool. Use model outputs for loss computation
  • kl_temperature — float. KL divergence temperature
  • recon_prob — float. Reconstruction probability
  • recon_coeff — float. Reconstruction coefficient
  • lambda_0 — float. Loss weight lambda_0
  • lambda_1 — float. Loss weight lambda_1
  • lambda_2 — float. Loss weight lambda_2
  • lambda_3 — float. Loss weight lambda_3
  • custom_loss_jit_path — str. Path to custom JIT-compiled loss function. Refer to /workspace/quantizer/pyutils/mel.pt

type

type: str = Field(default="MSE", description="Loss type (MSE, KL, etc.)")

use_outputs

use_outputs: bool = Field(
    default=False,
    alias="useOutputs",
    description="Use model outputs for loss computation",
)

kl_temperature

kl_temperature: float = Field(
    default=1.0,
    alias="KLTemperature",
    description="KL divergence temperature",
)

recon_prob

recon_prob: float = Field(
    default=1.0, alias="reconProb", description="Reconstruction probability"
)

recon_coeff

recon_coeff: float = Field(
    default=1.0,
    alias="reconCoeff",
    description="Reconstruction coefficient",
)

lambda_0

lambda_0: float = Field(
    default=1.0,
    validation_alias=AliasChoices("lambda_0", "lambda0"),
    description="Loss weight lambda_0",
)

lambda_1

lambda_1: float = Field(
    default=1.0,
    validation_alias=AliasChoices("lambda_1", "lambda1"),
    description="Loss weight lambda_1",
)

lambda_2

lambda_2: float = Field(
    default=1.0,
    validation_alias=AliasChoices("lambda_2", "lambda2"),
    description="Loss weight lambda_2",
)

lambda_3

lambda_3: float = Field(
    default=1.0,
    validation_alias=AliasChoices("lambda_3", "lambda3"),
    description="Loss weight lambda_3",
)

custom_loss_jit_path

custom_loss_jit_path: str = Field(
    default="",
    alias="customLossJITPath",
    description="Path to custom JIT-compiled loss function. Refer to /workspace/quantizer/pyutils/mel.pt",
)

ModConfig.Attributes.PostProcessing

class PostProcessing(BaseModel)

Attributes

Name Description
model_config Post-processing configuration for detection models
post
box_conf_thres
box_iou_thres

model_config

model_config = ConfigDict(populate_by_name=True)

Post-processing configuration for detection models

Parameters

  • post — str. Post-processing type
  • box_conf_thres — float. Box confidence threshold
  • box_iou_thres — float. Box IoU threshold

post

post: str = Field(default="", description="Post-processing type")

box_conf_thres

box_conf_thres: float = Field(
    default=0, alias="boxConfThres", description="Box confidence threshold"
)

box_iou_thres

box_iou_thres: float = Field(
    default=0, alias="boxIoUThres", description="Box IoU threshold"
)

LlmConfig

class LlmConfig(BaseModel)

Configuration for Large Language Model (LLM) compilation

Defines LLM-specific settings including sequence lengths, cache configurations, and runtime parameters for efficient LLM inference.

Parameters

  • apply — bool. If True, apply LLM-specific configurations
  • npu_parallel_degree — int. Number of NPU partitions for FFN tensor parallelism (1 = disabled). Applied before OptimizeFFN if both active.
  • attributes — Attributes. LLM attributes configuration

Attributes

Name Description
model_config
apply
npu_parallel_degree
attributes

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False, description="If True, apply LLM-specific configurations"
)

npu_parallel_degree

npu_parallel_degree: int = Field(
    default=1,
    alias="npuParallelDegree",
    description="Number of NPU partitions for FFN tensor parallelism (1 = disabled). Applied before OptimizeFFN if both active.",
)

attributes

attributes: Attributes = Field(default_factory=Attributes, alias="attributes")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "LlmConfig"

Return a copy with updated fields.

LlmConfig.Attributes

class Attributes(BaseModel)

Attributes

Name Description
model_config LLM attributes configuration
max_data_length
max_sequence_length
max_cache_length
max_core_data_length
calibration
runtime
debug

model_config

model_config = ConfigDict(populate_by_name=True)

LLM attributes configuration

Parameters

  • max_data_length — int. Maximum data length
  • max_sequence_length — int. Maximum sequence length
  • max_cache_length — int. Maximum cache length
  • max_core_data_length — int. Maximum core data length
  • calibration — Calibration. LLM calibration settings
  • runtime — Runtime. LLM runtime settings
  • debug — Debug. LLM debug settings

max_data_length

max_data_length: int = Field(
    default=4096, alias="maxDataLength", description="Maximum data length"
)

max_sequence_length

max_sequence_length: int = Field(
    default=4096,
    alias="maxSequenceLength",
    description="Maximum sequence length",
)

max_cache_length

max_cache_length: int = Field(
    default=4096, alias="maxCacheLength", description="Maximum cache length"
)

max_core_data_length

max_core_data_length: int = Field(
    default=128,
    alias="maxCoreDataLength",
    description="Maximum core data length",
)

calibration

calibration: Calibration = Field(
    default_factory=Calibration, alias="calibration"
)

runtime

runtime: Runtime = Field(default_factory=Runtime, alias="runtime")

debug

debug: Debug = Field(default_factory=Debug, alias="debug")

LlmConfig.Attributes.Calibration

class Calibration(BaseModel)

Attributes

Name Description
model_config LLM calibration settings
random_seq_length
use_full_seq_length

model_config

model_config = ConfigDict(populate_by_name=True)

LLM calibration settings

Parameters

  • random_seq_length — int. Random sequence length used for calibration
  • use_full_seq_length — bool. If True, use the full sequence length for calibration

random_seq_length

random_seq_length: int = Field(
    default=80,
    alias="randomSeqLength",
    description="Random sequence length used for calibration",
)

use_full_seq_length

use_full_seq_length: bool = Field(
    default=False,
    alias="useFullSeqLength",
    description="If True, use the full sequence length for calibration",
)

LlmConfig.Attributes.Runtime

class Runtime(BaseModel)

Attributes

Name Description
model_config LLM runtime settings
use_global_core
batch_size
npu_core_ids
dynamic_rope
dynamic_mask

model_config

model_config = ConfigDict(populate_by_name=True)

LLM runtime settings

Parameters

  • use_global_core — bool. If True, use a global core
  • batch_size — int. Batch size
  • npu_core_ids — List[int]. List of NPU core IDs
  • dynamic_rope — bool. If True, enable dynamic RoPE (rotary position embedding)
  • dynamic_mask — bool. If True, enable dynamic mask (attention mask as runtime input)

use_global_core

use_global_core: bool = Field(
    default=False,
    alias="useGlobalCore",
    description="If True, use a global core",
)

batch_size

batch_size: int = Field(
    default=1, alias="batchSize", description="Batch size"
)

npu_core_ids

npu_core_ids: List[int] = Field(
    default=[0], alias="npuCoreIds", description="List of NPU core IDs"
)

dynamic_rope

dynamic_rope: bool = Field(
    default=False,
    alias="dynamicRope",
    description="If True, enable dynamic RoPE (rotary position embedding)",
)

dynamic_mask

dynamic_mask: bool = Field(
    default=False,
    alias="dynamicMask",
    description="If True, enable dynamic mask (attention mask as runtime input)",
)

LlmConfig.Attributes.Debug

class Debug(BaseModel)

Attributes

Name Description
model_config LLM debug settings
apply
batch_debug_bundle_size

model_config

model_config = ConfigDict(populate_by_name=True)

LLM debug settings

Parameters

  • apply — bool. Enable LLM debug mode
  • batch_debug_bundle_size — int. Batch debug bundle size

apply

apply: bool = Field(default=False, description="Enable LLM debug mode")

batch_debug_bundle_size

batch_debug_bundle_size: int = Field(
    default=0,
    alias="batchDebugBundleSize",
    description="Batch debug bundle size",
)

MoeConfig

class MoeConfig(BaseModel)

Sparse MoE expert-selection configuration (calibration only)

Controls which experts are calibrated inside SparseMoe modules. selectionMode is a calibration-only knob: it picks which experts collect statistics (and, for TopK, on which tokens). It does NOT change inference routing — the forward path always routes the router's top-K experts regardless of this setting. scoreThreshold is only used when selectionMode is Threshold.

Parameters

  • selection_mode — int. Expert selection mode index (calibration only; inference is always top-K):
    0: TopK - Calibrate only the router's top-K experts per token (matches inference routing).
    1: All - Calibrate every expert on the full sequence.
    2: Threshold - Calibrate all experts whose routing score exceeds scoreThreshold.
  • score_threshold — float. Routing score threshold used when selectionMode is Threshold (calibration only)

Attributes

Name Description
model_config
selection_mode_list
selection_mode
score_threshold

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

selection_mode_list

selection_mode_list: List[str] = Field(
    default=["TopK", "All", "Threshold"], alias="selectionModeList"
)

selection_mode

selection_mode: int = Field(default=0, alias="selectionMode")

score_threshold

score_threshold: float = Field(
    default=0.0,
    alias="scoreThreshold",
    description="Routing score threshold used when selectionMode is Threshold (calibration only)",
    ge=0,
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "MoeConfig"

Return a copy with updated fields.

EquivalentTransformationConfig

class EquivalentTransformationConfig(BaseModel)

Configuration for equivalent transformation techniques

Defines parameters for various equivalent transformation methods including NormConv, QK smoothing, and rotation matrices for improved quantization.

Parameters

  • seed — int. Random seed for transformation
  • apply_hadamard_rotation_matrix — bool. Apply Hadamard rotation matrix
  • norm_conv — NormConv. NormConv equivalent transformation
  • qk — Qk. QK smoothing transformation
  • ud — Ud. UD transformation
  • vo — Vo. VO transformation
  • feed_forward_multi_lut — FeedForwardMultiLut. Feed-forward multi-LUT transformation
  • spin_r1 — SpinR1. SpinR1 rotation transformation
  • head_out_ch_rotation — HeadOutChRotation. Head output channel rotation transformation
  • in_rotation — InRotation. Input rotation transformation
  • spin_r2 — SpinR2. SpinR2 rotation transformation
  • qk_rotation — QkRotation. QK rotation transformation
  • flatten_quant — FlattenQuant. Flatten quantization transformation
  • optimize_ffn — OptimizeFfn. FFN optimization

Attributes

Name Description
model_config
seed
apply_hadamard_rotation_matrix
norm_conv
qk
ud
vo
feed_forward_multi_lut
spin_r1
head_out_ch_rotation
in_rotation
spin_r2
qk_rotation
flatten_quant
optimize_ffn

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

seed

seed: int = Field(default=0, description="Random seed for transformation")

apply_hadamard_rotation_matrix

apply_hadamard_rotation_matrix: bool = Field(
    default=True,
    alias="applyHadamardRotationMatrix",
    description="Apply Hadamard rotation matrix",
)

norm_conv

norm_conv: NormConv = Field(default_factory=NormConv, alias="NormConv")

qk

qk: Qk = Field(default_factory=Qk, alias="QK")

ud

ud: Ud = Field(default_factory=Ud, alias="UD")

vo

vo: Vo = Field(default_factory=Vo, alias="VO")

feed_forward_multi_lut

feed_forward_multi_lut: FeedForwardMultiLut = Field(
    default_factory=FeedForwardMultiLut, alias="FeedForwardMultiLUT"
)

spin_r1

spin_r1: SpinR1 = Field(default_factory=SpinR1, alias="SpinR1")

head_out_ch_rotation

head_out_ch_rotation: HeadOutChRotation = Field(
    default_factory=HeadOutChRotation, alias="HeadOutChRotation"
)

in_rotation

in_rotation: InRotation = Field(default_factory=InRotation, alias="InRotation")

spin_r2

spin_r2: SpinR2 = Field(default_factory=SpinR2, alias="SpinR2")

qk_rotation

qk_rotation: QkRotation = Field(default_factory=QkRotation, alias="QKRotation")

flatten_quant

flatten_quant: FlattenQuant = Field(
    default_factory=FlattenQuant, alias="FlattenQuant"
)

optimize_ffn

optimize_ffn: OptimizeFfn = Field(default_factory=OptimizeFfn, alias="OptimizeFFN")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "EquivalentTransformationConfig"

Return a copy with updated fields.

EquivalentTransformationConfig.NormConv

class NormConv(BaseModel)

Attributes

Name Description
model_config NormConv equivalent transformation
apply
learn
smoothing_factor
min_gamma
max_gamma

model_config

model_config = ConfigDict(populate_by_name=True)

NormConv equivalent transformation

Parameters

  • apply — bool. Apply NormConv transformation
  • learn — bool. Learn transformation parameters
  • smoothing_factor — float. Smoothing factor
  • min_gamma — float. Minimum gamma value
  • max_gamma — float. Maximum gamma value

apply

apply: bool = Field(default=False, description="Apply NormConv transformation")

learn

learn: bool = Field(
    default=False, description="Learn transformation parameters"
)

smoothing_factor

smoothing_factor: float = Field(
    default=0.5, alias="smoothingFactor", description="Smoothing factor"
)

min_gamma

min_gamma: float = Field(
    default=0.0001, alias="minGamma", description="Minimum gamma value"
)

max_gamma

max_gamma: float = Field(
    default=10000.0, alias="maxGamma", description="Maximum gamma value"
)

EquivalentTransformationConfig.Qk

class Qk(BaseModel)

Attributes

Name Description
model_config QK smoothing transformation
apply
smoothing_factor
min_gamma
max_gamma

model_config

model_config = ConfigDict(populate_by_name=True)

QK smoothing transformation

Parameters

  • apply — bool. Apply QK transformation
  • smoothing_factor — float. Smoothing factor
  • min_gamma — float. Minimum gamma value
  • max_gamma — float. Maximum gamma value

apply

apply: bool = Field(default=False, description="Apply QK transformation")

smoothing_factor

smoothing_factor: float = Field(
    default=0.5, alias="smoothingFactor", description="Smoothing factor"
)

min_gamma

min_gamma: float = Field(
    default=0.0001, alias="minGamma", description="Minimum gamma value"
)

max_gamma

max_gamma: float = Field(
    default=10000.0, alias="maxGamma", description="Maximum gamma value"
)

EquivalentTransformationConfig.Ud

class Ud(BaseModel)

Attributes

Name Description
model_config UD transformation
apply
learn
smoothing_factor
min_gamma
max_gamma

model_config

model_config = ConfigDict(populate_by_name=True)

UD transformation

Parameters

  • apply — bool. Apply UD transformation
  • learn — bool. Learn transformation parameters
  • smoothing_factor — float. Smoothing factor
  • min_gamma — float. Minimum gamma value
  • max_gamma — float. Maximum gamma value

apply

apply: bool = Field(default=False, description="Apply UD transformation")

learn

learn: bool = Field(
    default=False, description="Learn transformation parameters"
)

smoothing_factor

smoothing_factor: float = Field(
    default=0.5, alias="smoothingFactor", description="Smoothing factor"
)

min_gamma

min_gamma: float = Field(
    default=0.0001, alias="minGamma", description="Minimum gamma value"
)

max_gamma

max_gamma: float = Field(
    default=10000.0, alias="maxGamma", description="Maximum gamma value"
)

EquivalentTransformationConfig.Vo

class Vo(BaseModel)

Attributes

Name Description
model_config VO transformation
apply
smoothing_factor
min_gamma
max_gamma

model_config

model_config = ConfigDict(populate_by_name=True)

VO transformation

Parameters

  • apply — bool. Apply VO transformation
  • smoothing_factor — float. Smoothing factor
  • min_gamma — float. Minimum gamma value
  • max_gamma — float. Maximum gamma value

apply

apply: bool = Field(default=False, description="Apply VO transformation")

smoothing_factor

smoothing_factor: float = Field(
    default=0.5, alias="smoothingFactor", description="Smoothing factor"
)

min_gamma

min_gamma: float = Field(
    default=0.0001, alias="minGamma", description="Minimum gamma value"
)

max_gamma

max_gamma: float = Field(
    default=10000.0, alias="maxGamma", description="Maximum gamma value"
)

EquivalentTransformationConfig.FeedForwardMultiLut

class FeedForwardMultiLut(BaseModel)

Feed-forward multi-LUT transformation

Parameters

  • apply — bool. Apply feed-forward multi-LUT transformation
  • breakpoints — List[float]. Breakpoints for multi-LUT

Attributes

Name Description
apply
breakpoints

apply

apply: bool = Field(
    default=False, description="Apply feed-forward multi-LUT transformation"
)

breakpoints

breakpoints: List[float] = Field(
    default=[-8.0, -4.0, 0], description="Breakpoints for multi-LUT"
)

EquivalentTransformationConfig.SpinR1

class SpinR1(BaseModel)

Attributes

Name Description
model_config SpinR1 rotation transformation
apply
matrix_path
insert_output_conv

model_config

model_config = ConfigDict(populate_by_name=True)

SpinR1 rotation transformation

Parameters

  • apply — bool. Apply SpinR1 transformation
  • matrix_path — str. Path to rotation matrix file
  • insert_output_conv — bool. Insert an identity output conv when the residual stream ends without a head conv

apply

apply: bool = Field(default=False, description="Apply SpinR1 transformation")

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

insert_output_conv

insert_output_conv: bool = Field(
    default=False,
    alias="insertOutputConv",
    description="Insert an identity output conv when the residual stream ends without a head conv",
)

EquivalentTransformationConfig.HeadOutChRotation

class HeadOutChRotation(BaseModel)

Attributes

Name Description
model_config Head output channel rotation transformation
apply
matrix_path

model_config

model_config = ConfigDict(populate_by_name=True)

Head output channel rotation transformation

Parameters

  • apply — bool. Apply head output channel rotation
  • matrix_path — str. Path to rotation matrix file

apply

apply: bool = Field(
    default=False, description="Apply head output channel rotation"
)

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

EquivalentTransformationConfig.InRotation

class InRotation(BaseModel)

Attributes

Name Description
model_config Input rotation transformation
apply
matrix_path
input_names

model_config

model_config = ConfigDict(populate_by_name=True)

Input rotation transformation

Parameters

  • apply — bool. Apply input rotation
  • matrix_path — str. Path to rotation matrix file
  • input_names — List[str]. Names of the input layers to rotate

apply

apply: bool = Field(default=False, description="Apply input rotation")

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

input_names

input_names: List[str] = Field(
    default=[],
    alias="inputNames",
    description="Names of the input layers to rotate",
)

EquivalentTransformationConfig.SpinR2

class SpinR2(BaseModel)

Attributes

Name Description
model_config SpinR2 rotation transformation
apply
learn
matrix_path

model_config

model_config = ConfigDict(populate_by_name=True)

SpinR2 rotation transformation

Parameters

  • apply — bool. Apply SpinR2 transformation
  • learn — bool. Learn rotation matrix
  • matrix_path — str. Path to rotation matrix file

apply

apply: bool = Field(default=False, description="Apply SpinR2 transformation")

learn

learn: bool = Field(default=False, description="Learn rotation matrix")

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

EquivalentTransformationConfig.QkRotation

class QkRotation(BaseModel)

Attributes

Name Description
model_config QK rotation transformation
apply
matrix_path

model_config

model_config = ConfigDict(populate_by_name=True)

QK rotation transformation

Parameters

  • apply — bool. Apply QK rotation transformation
  • matrix_path — str. Path to rotation matrix file

apply

apply: bool = Field(
    default=False, description="Apply QK rotation transformation"
)

matrix_path

matrix_path: str = Field(
    default="", alias="matrixPath", description="Path to rotation matrix file"
)

EquivalentTransformationConfig.FlattenQuant

class FlattenQuant(BaseModel)

Attributes

Name Description
model_config Flatten quantization transformation
apply
learn
apply_threshold
max_overhead

model_config

model_config = ConfigDict(populate_by_name=True)

Flatten quantization transformation

Parameters

  • apply — bool. Apply flatten quantization
  • learn — bool. Learn flattening parameters
  • apply_threshold — float. Threshold for applying flatten quantization
  • max_overhead — float. Maximum overhead allowed for flattening

apply

apply: bool = Field(default=False, description="Apply flatten quantization")

learn

learn: bool = Field(default=False, description="Learn flattening parameters")

apply_threshold

apply_threshold: float = Field(
    default=0.33,
    alias="applyThreshold",
    description="Threshold for applying flatten quantization",
)

max_overhead

max_overhead: float = Field(
    default=0.02,
    alias="maxOverhead",
    description="Maximum overhead allowed for flattening",
)

EquivalentTransformationConfig.OptimizeFfn

class OptimizeFfn(BaseModel)

Attributes

Name Description
model_config FFN optimization
apply
ch_per_ffn

model_config

model_config = ConfigDict(populate_by_name=True)

FFN optimization

Parameters

  • apply — bool. Apply FFN optimization
  • ch_per_ffn — int. Optimize FFN split (-1 for auto)

apply

apply: bool = Field(default=False, description="Apply FFN optimization")

ch_per_ffn

ch_per_ffn: int = Field(
    default=-1, alias="chPerFFN", description="Optimize FFN split (-1 for auto)"
)

SearchWeightScaleConfig

class SearchWeightScaleConfig(BaseModel)

Configuration for weight scale search

Defines which transformer components should have their weight scales searched for optimal quantization.

Parameters

  • apply — bool. If true, apply weight scale search
  • transformer — Transformer. Transformer components for weight scale search

Attributes

Name Description
model_config
apply
transformer

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(default=False, description="If true, apply weight scale search")

transformer

transformer: Transformer = Field(default_factory=Transformer, alias="transformer")

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "SearchWeightScaleConfig"

Return a copy with updated fields.

SearchWeightScaleConfig.Transformer

class Transformer(BaseModel)

Transformer components for weight scale search

Parameters

  • query — bool. Search weight scale for query
  • key — bool. Search weight scale for key
  • value — bool. Search weight scale for value
  • out — bool. Search weight scale for output
  • ffn — bool. Search weight scale for FFN

Attributes

Name Description
query
key
value
out
ffn

query

query: bool = Field(default=False, description="Search weight scale for query")

key

key: bool = Field(default=False, description="Search weight scale for key")

value

value: bool = Field(default=False, description="Search weight scale for value")

out

out: bool = Field(default=False, description="Search weight scale for output")

ffn

ffn: bool = Field(default=False, description="Search weight scale for FFN")

LoadScaleConfig

class LoadScaleConfig(BaseModel)

QAT activation scale loading configuration

Loads activation scales from .safetensors, torch.save tensor dictionaries, or libtorch OutputArchive .pt/.pth files. Scalars select per-layer quantization; channel vectors select per-channel quantization. Invalid imports fail compilation. Sigmoid/Compare imports must match the hardware-fixed scale and zero-point (within 1e-6 relative scale tolerance). actScaleMin and enabled minClipRatio clipping take precedence over imported values; applied scales and clipping adjustments are logged, with quantization ranges updated to match. NOTE: entries is stored as raw JSON (type: list) because the generator does not support list[CustomStruct]. The schema records the field shape; parsing is done manually in activation_scale_loader::read.

Parameters

  • apply — bool. If true, load and apply QAT activation scales; invalid imports are errors
  • entries — List. List of file entries. Each entry is a dict: { path: str, scales: [ { key: str, layers: [str], zeropointKey?: str } ] } path: .safetensors or .pt/.pth file; key: floating scalar or channel vector. zeropointKey: optional I32/I64 tensor; .actScale keys automatically use a sibling .zeropoint if present, otherwise zero. Scalar zeropoints broadcast. layers: compiler layer names whose independent activation scales are overridden.

Attributes

Name Description
model_config
apply
entries

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False,
    description="If true, load and apply QAT activation scales; invalid imports are errors",
)

entries

entries: Any = Field(
    default=[],
    description="List of file entries. Each entry is a dict: { path: str, scales: [ { key: str, layers: [str], zeropointKey?: str } ] } path: .safetensors or .pt/.pth file; key: floating scalar or channel vector. zeropointKey: optional I32/I64 tensor; .actScale keys automatically use a sibling .zeropoint if present, otherwise zero. Scalar zeropoints broadcast. layers: compiler layer names whose independent activation scales are overridden.",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "LoadScaleConfig"

Return a copy with updated fields.

RuntimeOptions

class RuntimeOptions(BaseModel)

Runtime options for compilation

Contains runtime-specific settings like version info and cache options.

Parameters

  • version — str. Compiler version string (e.g., 0.0.0)
  • deterministic_algorithms — bool. Run LUT grouping, the only compile stage found to vary between runs on one GPU, with PyTorch deterministic algorithms; false gives a faster grouping whose result can differ between runs

Attributes

Name Description
model_config
version
deterministic_algorithms

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

version

version: str = Field(
    default="0.0.0", description="Compiler version string (e.g., 0.0.0)"
)

deterministic_algorithms

deterministic_algorithms: bool = Field(
    default=True,
    alias="deterministicAlgorithms",
    description="Run LUT grouping, the only compile stage found to vary between runs on one GPU, with PyTorch deterministic algorithms; false gives a faster grouping whose result can differ between runs",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "RuntimeOptions"

Return a copy with updated fields.

SaveSampleConfig

class SaveSampleConfig(BaseModel)

Sample data generation and saving configuration

Parameters

  • apply — bool. Enable sample data saving
  • mode — str. Inference mode: infer (standard) or inferWithCache (LLM cache models)
  • batch_size — int. Number of inference batches to generate
  • batch_seq_lens — List. Per-batch step-wise sequence lengths for inferWithCache mode. e.g. [[80, 1], [240, 10]]
  • save_folder — str. Output folder for sample data
  • dtype — str. Data type for saved samples: float or int8

Attributes

Name Description
model_config
apply
mode
batch_size
batch_seq_lens
save_folder
dtype

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(default=False, description="Enable sample data saving")

mode

mode: str = Field(
    default="infer",
    description="Inference mode: infer (standard) or inferWithCache (LLM cache models)",
)

batch_size

batch_size: int = Field(
    default=1,
    alias="batchSize",
    description="Number of inference batches to generate",
)

batch_seq_lens

batch_seq_lens: Any = Field(
    default=[],
    alias="batchSeqLens",
    description="Per-batch step-wise sequence lengths for inferWithCache mode. e.g. [[80, 1], [240, 10]]",
)

save_folder

save_folder: str = Field(
    default="sampleInout",
    alias="saveFolder",
    description="Output folder for sample data",
)

dtype

dtype: str = Field(
    default="float", description="Data type for saved samples: float or int8"
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "SaveSampleConfig"

Return a copy with updated fields.

ExtraOutputConfig

class ExtraOutputConfig(BaseModel)

Promote named intermediate layers to additional model outputs

Applied after quantization, so the rest of the bundle is unchanged. Names are matched against the post-fusion graph, and an unknown one is an error.

Parameters

  • apply — bool. Promote the named layers. Off by default: promotion is an explicit opt-in.
  • layers — List[str]. Layer names to promote; one that is already an output is left alone

Attributes

Name Description
model_config
apply
layers

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False,
    description="Promote the named layers. Off by default: promotion is an explicit opt-in.",
)

layers

layers: List[str] = Field(
    default=[],
    description="Layer names to promote; one that is already an output is left alone",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "ExtraOutputConfig"

Return a copy with updated fields.

GroupWiseConfig

class GroupWiseConfig(BaseModel)

Group-wise streaming quantization configuration for large LLMs

Configuration for the group-wise (streaming) quantization pipeline used for LLMs that do not fit fully on GPU. Partitions the model into groups of transformer blocks and quantizes each group independently.

Parameters

  • apply — bool. Enable group-wise streaming quantization pipeline
  • group_size — int. Group size in number of transformer blocks (0 = auto)
  • gpu_budget_gb — float. GPU memory budget in GiB for group-wise execution (0 = auto-detect)
  • gpu_safety_margin_gb — float. Safety margin in GiB subtracted from detected GPU budget
  • cache_dir — str. Directory for group-wise activation/state cache (empty = system temp)
  • keep_cache — bool. Retain group-wise cache after run completes (for debugging)
  • partition_policy — str. Partitioning policy (e.g. transformer_block, moe_expert_subgroup)
  • expert_groups — List[List[int]]. MoE fallback partitioner: list of expert-index lists (List[List[int]]).
  • retain_topology_weights — bool. Keep inflated FP weights across groups (false = release after each group for tight memory budgets).
  • checkpoint — bool. Save per-group checkpoint so quantization can resume from the last completed group after a crash.

Attributes

Name Description
model_config
apply
group_size
gpu_budget_gb
gpu_safety_margin_gb
cache_dir
keep_cache
partition_policy
expert_groups
retain_topology_weights
checkpoint

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

apply

apply: bool = Field(
    default=False, description="Enable group-wise streaming quantization pipeline"
)

group_size

group_size: int = Field(
    default=0,
    alias="groupSize",
    description="Group size in number of transformer blocks (0 = auto)",
)

gpu_budget_gb

gpu_budget_gb: float = Field(
    default=0,
    alias="gpuBudgetGb",
    description="GPU memory budget in GiB for group-wise execution (0 = auto-detect)",
)

gpu_safety_margin_gb

gpu_safety_margin_gb: float = Field(
    default=4,
    alias="gpuSafetyMarginGb",
    description="Safety margin in GiB subtracted from detected GPU budget",
)

cache_dir

cache_dir: str = Field(
    default="",
    alias="cacheDir",
    description="Directory for group-wise activation/state cache (empty = system temp)",
)

keep_cache

keep_cache: bool = Field(
    default=False,
    alias="keepCache",
    description="Retain group-wise cache after run completes (for debugging)",
)

partition_policy

partition_policy: str = Field(
    default="transformer_block",
    alias="partitionPolicy",
    description="Partitioning policy (e.g. transformer_block, moe_expert_subgroup)",
)

expert_groups

expert_groups: Any = Field(
    default=[],
    alias="expertGroups",
    description="MoE fallback partitioner: list of expert-index lists (List[List[int]]).",
)

retain_topology_weights

retain_topology_weights: bool = Field(
    default=True,
    alias="retainTopologyWeights",
    description="Keep inflated FP weights across groups (false = release after each group for tight memory budgets).",
)

checkpoint

checkpoint: bool = Field(
    default=False,
    description="Save per-group checkpoint so quantization can resume from the last completed group after a crash.",
)

Methods

Name Description
with_updates Return a copy with updated fields.

with_updates

def with_updates(self, **kwargs) -> "GroupWiseConfig"

Return a copy with updated fields.

CompileConfig

class CompileConfig(BaseModel)

Unified compilation configuration for Mobilint MXQ compilation.

Attributes

Name Description
model_config
model_paths
calib_data_path
save_paths
use_random_calib
inference_scheme
cpu_offload
force_npu_input_reposition
force_npu_output_reposition
optimize_option
buffer_mode
device
dtype
debug
trace
image_channels
config_version
split_blocks
split_parts
uint8_input
preprocessing
resource_management
calibration
bit
hessian_quant
bias_correction
mod
llm
moe
equivalent_transformation
search_weight_scale
load_scale
runtime_options
save_sample
extra_output
group_wise

model_config

model_config = ConfigDict(
    populate_by_name=True,
    extra="forbid",
)

model_paths

model_paths: List[str] = Field(
    default=[], alias="modelPaths", description="Paths to model files"
)

calib_data_path

calib_data_path: List[str] = Field(
    default=[], alias="calibDataPaths", description="Paths to calibration datasets"
)

save_paths

save_paths: List[str] = Field(
    default=["./tmp.mxq"],
    alias="savePaths",
    description="Output MXQ filename/paths",
)

use_random_calib

use_random_calib: bool = Field(
    default=False, alias="useRandomCalib", description="Use random calibration"
)

inference_scheme

inference_scheme: str = Field(
    default="single", alias="inferenceScheme", description="NPU inference scheme"
)

cpu_offload

cpu_offload: bool = Field(
    default=False,
    alias="cpuOffload",
    description="Enable CPU offload for unsupported operators",
)

force_npu_input_reposition

force_npu_input_reposition: bool = Field(
    default=False,
    alias="forceNpuInputReposition",
    description="Force input reposition operations to run on NPU instead of CPU",
)

force_npu_output_reposition

force_npu_output_reposition: bool = Field(
    default=False,
    alias="forceNpuOutputReposition",
    description="Force output reposition operations to run on NPU instead of CPU",
)

optimize_option

optimize_option: int = Field(
    default=1,
    alias="optimizeOption",
    description="Compiler optimization selector",
    ge=0,
)

buffer_mode

buffer_mode: int = Field(
    default=1, alias="bufferMode", description="Buffer serialization mode"
)

device

device: str = Field(default="gpu", description="Device for computation")

dtype

dtype: str = Field(default="float", description="Data type for computation")

debug

debug: bool = Field(default=False, description="Enable debug mode")

trace

trace: bool = Field(default=False, description="Enable trace mode")

image_channels

image_channels: int = Field(
    default=0,
    alias="imageChannels",
    description="Number of image channels (0 for auto-detect)",
)

config_version

config_version: str = Field(
    default="1.0.0", alias="configVersion", description="Config schema version"
)

split_blocks

split_blocks: List[int] = Field(
    default=[],
    alias="splitBlocks",
    description="Multi-MXQ split points by transformer block index",
)

split_parts

split_parts: int = Field(
    default=0,
    alias="splitParts",
    description="Evenly split transformer blocks into N MXQ parts",
)

uint8_input

uint8_input: Uint8InputConfig = Field(
    default_factory=Uint8InputConfig, alias="uint8Input"
)

preprocessing

preprocessing: PreprocessingConfig = Field(default_factory=PreprocessingConfig)

resource_management

resource_management: ResourceManagementConfig = Field(
    default_factory=ResourceManagementConfig, alias="resourceManagement"
)

calibration

calibration: CalibrationConfig = Field(default_factory=CalibrationConfig)

bit

bit: BitConfig = Field(default_factory=BitConfig)

hessian_quant

hessian_quant: HessianQuantConfig = Field(
    default_factory=HessianQuantConfig, alias="hessianQuant"
)

bias_correction

bias_correction: BiasCorrectionConfig = Field(
    default_factory=BiasCorrectionConfig, alias="biasCorrection"
)

mod

mod: ModConfig = Field(default_factory=ModConfig)

llm

llm: LlmConfig = Field(default_factory=LlmConfig)

moe

moe: MoeConfig = Field(default_factory=MoeConfig)

equivalent_transformation

equivalent_transformation: EquivalentTransformationConfig = Field(
    default_factory=EquivalentTransformationConfig, alias="equivalentTransformation"
)

search_weight_scale

search_weight_scale: SearchWeightScaleConfig = Field(
    default_factory=SearchWeightScaleConfig, alias="searchWeightScale"
)

load_scale

load_scale: LoadScaleConfig = Field(
    default_factory=LoadScaleConfig, alias="loadScale"
)

runtime_options

runtime_options: RuntimeOptions = Field(
    default_factory=RuntimeOptions, alias="runtimeOptions"
)

save_sample

save_sample: SaveSampleConfig = Field(
    default_factory=SaveSampleConfig, alias="saveSample"
)

extra_output

extra_output: ExtraOutputConfig = Field(
    default_factory=ExtraOutputConfig, alias="extraOutput"
)

group_wise

group_wise: GroupWiseConfig = Field(
    default_factory=GroupWiseConfig, alias="groupWise"
)

Methods

Name Description
with_uint8_input Return a copy with uint8_input settings enabled.
with_preprocessing Return a copy with preprocessing settings enabled.
with_hessian_quant Return a copy with hessian_quant settings enabled.
with_bias_correction Return a copy with bias_correction settings enabled.
with_mod Return a copy with mod settings enabled.
with_llm Return a copy with llm settings enabled.
with_search_weight_scale Return a copy with search_weight_scale settings enabled.
with_load_scale Return a copy with load_scale settings enabled.
with_save_sample Return a copy with save_sample settings enabled.
with_extra_output Return a copy with extra_output settings enabled.
with_group_wise Return a copy with group_wise settings enabled.
from_file Load config from YAML or JSON file.
from_preset Load config from a preset.
to_file Save config to YAML or JSON file.

with_uint8_input

def with_uint8_input(self, **kwargs) -> "CompileConfig"

Return a copy with uint8_input settings enabled.

with_preprocessing

def with_preprocessing(self, **kwargs) -> "CompileConfig"

Return a copy with preprocessing settings enabled.

with_hessian_quant

def with_hessian_quant(self, **kwargs) -> "CompileConfig"

Return a copy with hessian_quant settings enabled.

with_bias_correction

def with_bias_correction(self, **kwargs) -> "CompileConfig"

Return a copy with bias_correction settings enabled.

with_mod

def with_mod(self, **kwargs) -> "CompileConfig"

Return a copy with mod settings enabled.

with_llm

def with_llm(self, **kwargs) -> "CompileConfig"

Return a copy with llm settings enabled.

with_search_weight_scale

def with_search_weight_scale(self, **kwargs) -> "CompileConfig"

Return a copy with search_weight_scale settings enabled.

with_load_scale

def with_load_scale(self, **kwargs) -> "CompileConfig"

Return a copy with load_scale settings enabled.

with_save_sample

def with_save_sample(self, **kwargs) -> "CompileConfig"

Return a copy with save_sample settings enabled.

with_extra_output

def with_extra_output(self, **kwargs) -> "CompileConfig"

Return a copy with extra_output settings enabled.

with_group_wise

def with_group_wise(self, **kwargs) -> "CompileConfig"

Return a copy with group_wise settings enabled.

from_file

def from_file(cls, path: Union[str, Path]) -> "CompileConfig"

Load config from YAML or JSON file.

from_preset

def from_preset(cls, name: str) -> "CompileConfig"

Load config from a preset.

to_file

def to_file(self, path: Union[str, Path]) -> None

Save config to YAML or JSON file.