qbcompiler.calibration.make_calib_data¶
Functions¶
| Name | Description |
|---|---|
get_yaml |
|
make_calib_man |
From images and a user-provided preprocessing function, generate calibration NumPy files and a manifest. |
make_calib |
Create calibration NumPy files and a manifest using a YAML or dict preprocessing configuration. |
make_calib_llm |
Generate calibration embeddings for LLM models using Hugging Face datasets. |
get_yaml¶
make_calib_man¶
def make_calib_man(
pre_ftn,
data_dir: str,
save_dir: str,
save_name: str,
anno_json: str = None,
file_format: str = "%012d.jpg",
max_size: int = -1,
remove_npy: bool = False,
seed: int = 2023,
save_calib_msg: bool = False,
msg_path: str = None,
)
From images and a user-provided preprocessing function, generate calibration NumPy files and a manifest.
Parameters
pre_ftn— Callable. Pre-processing function that accepts an image path and returns a NumPy array.data_dir— string. Directory containing the raw calibration images.save_dir— string. Directory in which to store the generated NumPy files and metadata text file.save_name— string. Basename for the saved assets. NumPy files are written to{save_dir}/{save_name}_npyand the manifest to{save_dir}/{save_name}.txt. Defaults to the basename ofdata_dirwhen omitted.anno_json— string. Optional COCO-style annotation file. When provided, samples are randomly selected while maintaining class balance.file_format— string. Filename format that usesimage_idx(for example "%012d.jpg").max_size— int. Maximum number of calibration samples. A value of -1 disables the limit.remove_npy— bool. Remove existing NumPy outputs before generating new data.seed— int. Random seed applied to sampling and class-balanced selection. Defaults to 2023.save_calib_msg— bool. Persist the calibration metadata as an MSGpack file in addition to the manifest.msg_path— string. Explicit MSGpack output path. When omitted, the path is auto-generated by dataset name and sample count.
Returns
string. Path to the generated text file that lists the preprocessed NumPy samples.
Note
Calibration samples are written as float32 NumPy arrays derived from the pre_ftn output. When max_size is positive and the dataset is larger, a random subset is used. COCO annotations override this sampling to balance categories.
make_calib¶
def make_calib(
args_pre: Union[str, dict],
data_dir: str,
save_dir: str,
save_name: str = None,
anno_json: str = None,
file_format: str = "%012d.jpg",
max_size: int = -1,
remove_npy: bool = False,
seed: int = 2023,
save_calib_msg: bool = False,
msg_path: str = None,
) -> str
Create calibration NumPy files and a manifest using a YAML or dict preprocessing configuration.
Parameters
args_pre— string or dict. Path to a YAML file or an in-memory dictionary describing the preprocessing pipeline. See the preproc_builder class for all supported operators and parameters.data_dir— string. Directory of calibration images.save_dir— string. Directory where the generated NumPy files and manifest text file are stored.save_name— string. Optional basename for outputs. Defaults to the basename ofdata_dir.anno_json— string. Optional COCO-format annotation file used to sample images with class balance.file_format— string. Filename template built fromimage_idx(defaults to "%012d.jpg").max_size— int. Maximum number of calibration samples to generate; -1 keeps all available items.remove_npy— bool. Remove existing NumPy outputs before running preprocessing.seed— int. Random seed used for sampling. Defaults to 2023.save_calib_msg— bool. Save calibration metadata as MSGpack alongside NumPy and text outputs.msg_path— string. Explicit MSGpack output path. Auto-generated from dataset name and sample count when omitted.
Returns
string. Path to the manifest text file that lists the generated NumPy samples.
Note
The preprocessing pipeline is constructed through preproc_builder and executed via make_calib_man. Use the YAML schema to combine primitives such as GetImage, Resize, Normalize, Pad, CenterCrop, and SetOrder in a Pre-Order list.
make_calib_llm¶
def make_calib_llm(
args_pre: Union[str, dict],
dataset_name: str,
subset_list: Union[List[str], str],
text_field: str,
split: str,
save_dir: str,
max_calib: int = 512,
save_prefix: Optional[str] = None,
) -> List[str]
Generate calibration embeddings for LLM models using Hugging Face datasets.
Args: args_pre: YAML path or preprocessing configuration dictionary. Must set Datatype to "LLM". dataset_name: Name of the Hugging Face dataset to load. subset_list: List of dataset subsets (e.g. language splits) to iterate. split: HF dataset split to use. text_field: Key containing text in dataset entries. save_dir: Root directory where calibration samples are stored. max_calib: Maximum number of calibration samples per subset. save_prefix: Override prefix used in output directory naming. If None, derives from tokenizer directory name when available.
Returns: List of paths to generated calibration txt files, one per subset.