# qbcompiler.calibration.make\_calib\_data

## Functions

| Name | Description |
| --- | --- |
| [`get_yaml`](<#get-yaml>) |  |
| [`make_calib_man`](<#make-calib-man>) | From images and a user-provided preprocessing function, generate calibration NumPy files and a manifest. |
| [`make_calib`](<#make-calib>) | Create calibration NumPy files and a manifest using a YAML or dict preprocessing configuration. |
| [`make_calib_llm`](<#make-calib-llm>) | Generate calibration embeddings for LLM models using Hugging Face datasets. |

### get\_yaml

```python
def get_yaml(yaml_path: str)
```

### make\_calib\_man

```python
def make_calib_man(
    pre_ftn,
    data_dir: str,
    save_dir: str,
    save_name: str,
    anno_json: str = None,
    file_format: str = "%012d.jpg",
    max_size: int = -1,
    remove_npy: bool = False,
    seed: int = 2023,
    save_calib_msg: bool = False,
    msg_path: str = None,
)
```

From images and a user-provided preprocessing function, generate calibration NumPy files and a manifest.

**Parameters**

- `pre_ftn` — Callable. Pre-processing function that accepts an image path and returns a NumPy array.
- `data_dir` — string. Directory containing the raw calibration images.
- `save_dir` — string. Directory in which to store the generated NumPy files and metadata text file.
- `save_name` — string. Basename for the saved assets. NumPy files are written to `{save_dir}/{save_name}_npy` and the manifest to `{save_dir}/{save_name}.txt`. Defaults to the basename of `data_dir` when omitted.
- `anno_json` — string. Optional COCO-style annotation file. When provided, samples are randomly selected while maintaining class balance.
- `file_format` — string. Filename format that uses `image_idx` (for example "%012d.jpg").
- `max_size` — int. Maximum number of calibration samples. A value of -1 disables the limit.
- `remove_npy` — bool. Remove existing NumPy outputs before generating new data.
- `seed` — int. Random seed applied to sampling and class-balanced selection. Defaults to 2023.
- `save_calib_msg` — bool. Persist the calibration metadata as an MSGpack file in addition to the manifest.
- `msg_path` — string. Explicit MSGpack output path. When omitted, the path is auto-generated by dataset name and sample count.

**Returns**

string. Path to the generated text file that lists the preprocessed NumPy samples.

> [!NOTE]
>
> Calibration samples are written as float32 NumPy arrays derived from the `pre_ftn` output. When `max_size` is positive and the dataset is larger, a random subset is used. COCO annotations override this sampling to balance categories.

### make\_calib

```python
def make_calib(
    args_pre: Union[str, dict],
    data_dir: str,
    save_dir: str,
    save_name: str = None,
    anno_json: str = None,
    file_format: str = "%012d.jpg",
    max_size: int = -1,
    remove_npy: bool = False,
    seed: int = 2023,
    save_calib_msg: bool = False,
    msg_path: str = None,
) -> str
```

Create calibration NumPy files and a manifest using a YAML or dict preprocessing configuration.

**Parameters**

- `args_pre` — string or dict. Path to a YAML file or an in-memory dictionary describing the preprocessing pipeline. See the [preproc\_builder](<https://docs.mobilint.com/compiler/en/v1.3/api/python/qbcompiler.calibration.preprocess/#preproc-builder>) class for all supported operators and parameters.
- `data_dir` — string. Directory of calibration images.
- `save_dir` — string. Directory where the generated NumPy files and manifest text file are stored.
- `save_name` — string. Optional basename for outputs. Defaults to the basename of `data_dir`.
- `anno_json` — string. Optional COCO-format annotation file used to sample images with class balance.
- `file_format` — string. Filename template built from `image_idx` (defaults to "%012d.jpg").
- `max_size` — int. Maximum number of calibration samples to generate; -1 keeps all available items.
- `remove_npy` — bool. Remove existing NumPy outputs before running preprocessing.
- `seed` — int. Random seed used for sampling. Defaults to 2023.
- `save_calib_msg` — bool. Save calibration metadata as MSGpack alongside NumPy and text outputs.
- `msg_path` — string. Explicit MSGpack output path. Auto-generated from dataset name and sample count when omitted.

**Returns**

string. Path to the manifest text file that lists the generated NumPy samples.

> [!NOTE]
>
> The preprocessing pipeline is constructed through [`preproc_builder`](<https://docs.mobilint.com/compiler/en/v1.3/api/python/qbcompiler.calibration.preprocess/#preproc-builder>) and executed via `make_calib_man`. Use the YAML schema to combine primitives such as [GetImage](<https://docs.mobilint.com/compiler/en/v1.3/api/python/qbcompiler.calibration.preprocess/#getimage>), [Resize](<https://docs.mobilint.com/compiler/en/v1.3/api/python/qbcompiler.calibration.preprocess/#resize>), [Normalize](<https://docs.mobilint.com/compiler/en/v1.3/api/python/qbcompiler.calibration.preprocess/#normalize>), [Pad](<https://docs.mobilint.com/compiler/en/v1.3/api/python/qbcompiler.calibration.preprocess/#pad>), [CenterCrop](<https://docs.mobilint.com/compiler/en/v1.3/api/python/qbcompiler.calibration.preprocess/#centercrop>), and [SetOrder](<https://docs.mobilint.com/compiler/en/v1.3/api/python/qbcompiler.calibration.preprocess/#setorder>) in a Pre-Order list.

### make\_calib\_llm

```python
def make_calib_llm(
    args_pre: Union[str, dict],
    dataset_name: str,
    subset_list: Union[List[str], str],
    text_field: str,
    split: str,
    save_dir: str,
    max_calib: int = 512,
    save_prefix: Optional[str] = None,
) -> List[str]
```

Generate calibration embeddings for LLM models using Hugging Face datasets.

Args: args\_pre: YAML path or preprocessing configuration dictionary. Must set Datatype to "LLM". dataset\_name: Name of the Hugging Face dataset to load. subset\_list: List of dataset subsets (e.g. language splits) to iterate. split: HF dataset split to use. text\_field: Key containing text in dataset entries. save\_dir: Root directory where calibration samples are stored. max\_calib: Maximum number of calibration samples per subset. save\_prefix: Override prefix used in output directory naming. If None, derives from tokenizer directory name when available.

Returns: List of paths to generated calibration txt files, one per subset.
