Skip to content

qbcompiler.calibration.make_calib_data

Functions

Name Description
get_yaml
make_calib_man From images and a user-provided preprocessing function, generate calibration NumPy files and a manifest.
make_calib Create calibration NumPy files and a manifest using a YAML or dict preprocessing configuration.
make_calib_llm Generate calibration embeddings for LLM models using Hugging Face datasets.

get_yaml

def get_yaml(yaml_path: str)

make_calib_man

def make_calib_man(
    pre_ftn,
    data_dir: str,
    save_dir: str,
    save_name: str,
    anno_json: str = None,
    file_format: str = "%012d.jpg",
    max_size: int = -1,
    remove_npy: bool = False,
    seed: int = 2023,
    save_calib_msg: bool = False,
    msg_path: str = None,
)

From images and a user-provided preprocessing function, generate calibration NumPy files and a manifest.

Parameters

  • pre_ftn — Callable. Pre-processing function that accepts an image path and returns a NumPy array.
  • data_dir — string. Directory containing the raw calibration images.
  • save_dir — string. Directory in which to store the generated NumPy files and metadata text file.
  • save_name — string. Basename for the saved assets. NumPy files are written to {save_dir}/{save_name}_npy and the manifest to {save_dir}/{save_name}.txt. Defaults to the basename of data_dir when omitted.
  • anno_json — string. Optional COCO-style annotation file. When provided, samples are randomly selected while maintaining class balance.
  • file_format — string. Filename format that uses image_idx (for example "%012d.jpg").
  • max_size — int. Maximum number of calibration samples. A value of -1 disables the limit.
  • remove_npy — bool. Remove existing NumPy outputs before generating new data.
  • seed — int. Random seed applied to sampling and class-balanced selection. Defaults to 2023.
  • save_calib_msg — bool. Persist the calibration metadata as an MSGpack file in addition to the manifest.
  • msg_path — string. Explicit MSGpack output path. When omitted, the path is auto-generated by dataset name and sample count.

Returns

string. Path to the generated text file that lists the preprocessed NumPy samples.

Note

Calibration samples are written as float32 NumPy arrays derived from the pre_ftn output. When max_size is positive and the dataset is larger, a random subset is used. COCO annotations override this sampling to balance categories.

make_calib

def make_calib(
    args_pre: Union[str, dict],
    data_dir: str,
    save_dir: str,
    save_name: str = None,
    anno_json: str = None,
    file_format: str = "%012d.jpg",
    max_size: int = -1,
    remove_npy: bool = False,
    seed: int = 2023,
    save_calib_msg: bool = False,
    msg_path: str = None,
) -> str

Create calibration NumPy files and a manifest using a YAML or dict preprocessing configuration.

Parameters

  • args_pre — string or dict. Path to a YAML file or an in-memory dictionary describing the preprocessing pipeline. See the preproc_builder class for all supported operators and parameters.
  • data_dir — string. Directory of calibration images.
  • save_dir — string. Directory where the generated NumPy files and manifest text file are stored.
  • save_name — string. Optional basename for outputs. Defaults to the basename of data_dir.
  • anno_json — string. Optional COCO-format annotation file used to sample images with class balance.
  • file_format — string. Filename template built from image_idx (defaults to "%012d.jpg").
  • max_size — int. Maximum number of calibration samples to generate; -1 keeps all available items.
  • remove_npy — bool. Remove existing NumPy outputs before running preprocessing.
  • seed — int. Random seed used for sampling. Defaults to 2023.
  • save_calib_msg — bool. Save calibration metadata as MSGpack alongside NumPy and text outputs.
  • msg_path — string. Explicit MSGpack output path. Auto-generated from dataset name and sample count when omitted.

Returns

string. Path to the manifest text file that lists the generated NumPy samples.

Note

The preprocessing pipeline is constructed through preproc_builder and executed via make_calib_man. Use the YAML schema to combine primitives such as GetImage, Resize, Normalize, Pad, CenterCrop, and SetOrder in a Pre-Order list.

make_calib_llm

def make_calib_llm(
    args_pre: Union[str, dict],
    dataset_name: str,
    subset_list: Union[List[str], str],
    text_field: str,
    split: str,
    save_dir: str,
    max_calib: int = 512,
    save_prefix: Optional[str] = None,
) -> List[str]

Generate calibration embeddings for LLM models using Hugging Face datasets.

Args: args_pre: YAML path or preprocessing configuration dictionary. Must set Datatype to "LLM". dataset_name: Name of the Hugging Face dataset to load. subset_list: List of dataset subsets (e.g. language splits) to iterate. split: HF dataset split to use. text_field: Key containing text in dataset entries. save_dir: Root directory where calibration samples are stored. max_calib: Maximum number of calibration samples per subset. save_prefix: Override prefix used in output directory naming. If None, derives from tokenizer directory name when available.

Returns: List of paths to generated calibration txt files, one per subset.