Quick Start

Quick Start#

The article below go through the steps for preparing the calibration dataset, model compiling, and inference. Further examples and details are available in the tutorial section.

Preparing Calibration Data#

To compile the model, you should prepare the calibration dataset (the pre-processed inputs for the model) for quantization. There are two recommended ways to make the calibration dataset as follows:

  1. Utilize a pre-processing configuration YAML file.

  2. Use a manually defined pre-processing function.

Remark The process of making a calibration dataset may vary depending on whether you compile the model for CPU offloading or not. Currently, qbcompiler compiles the model without CPU offloading by default. In this scenario, the pre-processed input shape should be in the format (H, W, C). On the other hand, when CPU offloading is employed, the pre-processed input shape should match the input shape that the original model takes.

We are providing example calibration dataset for classification models that are trained on ImageNet.

Using a pre-processing configuration YAML file#

Image pre-processing techniques such as resizing, cropping, and normalization are often applied in machine vision tasks. Users can construct a pre-processing configuration using a YAML file and prepare the calibration dataset via the API provided by qbcompiler, make_calib. Please be aware that this method can only be employed when the raw data is an image. An example code is shown below. The following code assumes that images for calibration are prepared in the directory /workspace/calibration/cali_1000.

First, create an executable Python script (e.g., prepare_calib.py) with the following content:

from qbcompiler.calibration import make_calib
make_calib(
    args_pre="/workspace/resnet50.yaml", # path to pre-processing configuration yaml file
    data_dir="/workspace/calibration/cali_1000", # path to folder of original calibration data files such as images
    save_dir="/workspace/calibration/", # path to folder to save pre-processed calibration data files
    save_name="resnet50", # tag for the generated calibration dataset
    max_size=100 # Maximum number of data to use for calibration
)

Then, write the pre-processing configuration YAML file (e.g., resnet50.yaml) in the directory that you specified in the code above. The YAML file should contain the pre-processing functions and their parameters. Below is an example of a YAML file for ResNet50 pre-processing.

# resnet50.yaml
Datatype: Image
GetImage:
    to_float32: false
    channel_order: RGB

Pre-Order: [ResizeTorch, CenterCrop, Normalize, SetOrder]
Pre-processing:
    ResizeTorch:
        size: 256
        interpolation: bilinear
    CenterCrop:
        size: [224, 224]
    Normalize:
        mean: [0.485, 0.456, 0.406]
        std: [0.229, 0.224, 0.225]
        to_float_div255: true
    SetOrder:
        shape: HWC

The above results are in a directory containing the pre-processed calibration dataset (NumPy tensors), located at /workspace/calibration/resnet50. In addition, a calibration meta txt file containing the paths to the pre-processed NumPy files is created, named /workspace/calibration/resnet50.txt.

Remark The sample dataset for calibration should be composed of images with the same format. If some are in color images and others are in grayscale images, the calibration dataset will not be created properly.

Using a manually defined pre-processing function#

You can use your pre-processing function to make the calibration dataset via the API provided by qbcompiler, make_calib_man. In this case, the pre-processing function should take the image path as input and return a NumPy tensor. An example of the code is shown below. The following code assumes that images for calibration are prepared in the directory /workspace/calibration/cali_1000.

Like the previous method, create an executable Python script (e.g., prepare_calib.py) with the following content:

import torch
import numpy as np
from PIL import Image
import torchvision.transforms.functional as F
from torchvision.transforms import InterpolationMode
from qbcompiler.calibration import make_calib_man

def preprocess_resnet50(img_path: str):
    img = Image.open(img_path)
    resize_size=256
    crop_size=(224, 224)
    mean=[0.485, 0.456, 0.406]
    std=[0.229, 0.224, 0.225]
    out = F.pil_to_tensor(img)
    out = F.resize(out, size=resize_size, interpolation = InterpolationMode.BILINEAR)
    out = F.center_crop(out, output_size=crop_size)
    out = out.to(torch.float, copy=False) / 255.
    out = F.normalize(out, mean, std)
    out = np.transpose(out.numpy(), axes=[1, 2, 0])
    return out

make_calib_man(
    pre_ftn=preprocess_resnet50, # callable function to pre-process the calibration data
    data_dir="/workspace/calibration/cali_1000", # path to folder of original calibration data files such as images
    save_dir="/workspace/calibration/", # path to folder to save pre-processed calibration data files
    save_name="resnet50", # tag for the generated calibration dataset
    max_size=100 # Maximum number of data to use for calibration
)

The above results are in a directory containing the pre-processed calibration dataset (NumPy tensors), located at /workspace/calibration/resnet50. In addition, a calibration meta txt file containing the paths to the pre-processed NumPy files is created, named /workspace/calibration/resnet50.txt.

Remark Unless the custom pre-processing function contains proper exception handling, the sample dataset for calibration should be composed of images with the same format. Like the previous method, the calibration dataset will not be created properly if some are in color images and others are in grayscale images.

Compiling Deep Learning Models#

Once the calibration dataset is prepared, you can compile the model with the qbcompiler. The qbcompiler supports various deep learning frameworks such as ONNX, PyTorch, Keras, TensorFlow, and TensorFlow Lite. The compilation process involves parsing the model, quantizing it, and generating Mobilint IR (MXQ) files.

Compiling ONNX Models#

ONNX is the most recommended framework to be used for compiling the trained model. With simple code, the ONNX model can be directly parsed to obtain Mobilint IR. The example code is shown below. The following code assumes that the calibration dataset and the model are prepared in the directory /workspace/calibration/resnet50 and /workspace/resnet50.onnx, respectively.

To compile the ONNX model, create an executable Python script (e.g., compile_onnx.py) with the following content and run it inside the Docker container:

""" Compile ONNX model""" 
from qbcompiler import mxq_compile
onnx_model_path = "/workspace/resnet50.onnx"
calib_data_path = "/workspace/calibration/resnet50"
# calib_data_path can be replaced with the path to the calibration meta file such as "/workspace/calibration/resnet50.txt"

mxq_compile(
    model=onnx_model_path,
    calib_data_path=calib_data_path,
    save_path="resnet50.mxq",
    backend="onnx"
)

Compiling PyTorch Models#

The qbcompiler supports direct compilation of PyTorch models. Before compiling, you must prepare a feed_dict, which is a dictionary whose keys are the model’s input names and whose values are the corresponding torch.Tensors. The parameter names of the model’s forward function must exactly match the keys in feed_dict. For example, in torchvision’s resnet50, the default input name is “x”. The tensors in feed_dict must match the model’s expected input and must not cause errors when calling forward with that shape. For resnet50, a tensor like (1, 3, 224, 224) is valid, but (1, 8, 224, 224) with an incorrect channel count will fail to compile.

""" Compile PyTorch model"""
from qbcompiler import mxq_compile
### get resnet50 from torchvision 
import torchvision
import torch
import numpy as np
calib_data_path = "/workspace/calibration/resnet50"
# A calibration meta file such as "/workspace/calibration/resnet50.txt" can be used instead.

torch_model = torchvision.models.resnet50(pretrained=True)
torch_model.eval().cpu()
feed_dict = {"x": torch.randn(1, 3, 224, 224).cpu()}

mxq_compile(
    model=torch_model,
    calib_data_path=calib_data_path,
    backend="torch",
    save_path="resnet50.mxq",
    feed_dict=feed_dict,
)

Compiling Torchscript Models#

The qbcompiler supports compiling models in the TorchScript format. The example below saves a torchvision ResNet-50 model to TorchScript, reloads it, and compiles it with qbcompiler.

""" Compile PyTorch model"""
from qbcompiler import mxq_compile
### get resnet50 from torchvision 
import torchvision
import torch
import numpy as np
calib_data_path = "/workspace/calibration/resnet50"
# A calibration meta file such as "/workspace/calibration/resnet50.txt" can be used instead.

### get resnet50 from torchvision and convert it to torchscript
torch_model = torchvision.models.resnet50(pretrained=True) 
torchscript_model_path = "/workspace/resnet50.pt"

dummy_input = np.random.randn(1,3,224,224).astype(np.float32)
feed_dict = {"input": dummy_input}
#When compiling with TorchScript, feed_dict information is required, which contains the temporary input name and a sample input.
scripted_model = torch.jit.script(torch_model, torch.tensor(dummy_input))
torch.jit.save(scripted_model, torchscript_model_path)

mxq_compile(
    model=torchscript_model_path,
    calib_data_path=calib_data_path,
    backend="torchscript",
    save_path="resnet50.mxq",
    feed_dict=feed_dict,
)

Compiling TensorFlow/Keras Models#

Since Keras works as an interface for TensorFlow, models on the Keras framework can be converted to Mobilint IR via TensorFlow. Currently, the qbcompiler supports TensorFlow models saved in the format of the SavedModel or high-level save format(.keras or h5). The following code assumes that the calibration dataset and the model are prepared in the directory /workspace/calibration/resnet50 and /workspace/tf_models/resnet50.h5, respectively.

To compile the Keras/TensorFlow model, create an executable Python script (e.g., compile_tf.py) with the following content and run it inside the Docker container:

""" Compile Keras/TensorFlow model in SavedModel format """
from qbcompiler import mxq_compile
import tensorflow as tf

keras_model = tf.keras.applications.resnet50.ResNet50() # Load a Keras model 
calib_data_path = "/workspace/calibration/resnet50"
# A calibration metadata file such as "/workspace/calibration/resnet50.txt" can be used instead.

keras_model_save_path = "/workspace/tf_models/resnet50.h5" # path to save .h5 keras model
keras_model.save(keras_model_save_path)

mxq_compile(
    model=keras_model_save_path,
    calib_data_path=calib_data_path,
    backend="tf",
    save_path="resnet50.mxq",
)

Compiling TensorFlow Lite Models#

The qbcompiler supports compiling TensorFlow Lite models. The example below converts a Keras ResNet50 model to a TFLite model and then compiles it with qbcompiler.

""" Compile Tensorflow Lite model """ 
from qbcompiler import mxq_compile
import tensorflow as tf

keras_model = tf.keras.applications.resnet50.ResNet50() # Load a pre-trained Keras model
calib_data_path = "/workspace/calibration/resnet50"
# A calibration metadata file such as "/workspace/calibration/resnet50.txt" can be used instead.

keras_model_save_path = "/workspace/tf_models/resnet50.h5"# path to save .h5 keras model
tflite_model_path = "/workspace/tflite_models/resnet50.tflite"
keras_model.save(keras_model_save_path)

loaded_keras_model = tf.keras.models.load_model(keras_model_save_path)
tflite_model = tf.lite.TFLiteConverter.from_keras_model(loaded_keras_model).convert() # Convert the model to TFLite format
with open(tflite_model_path, 'wb') as f:
    f.write(tflite_model)

mxq_compile(
    model=tflite_model_path,
    calib_data_path=calib_data_path,
    backend="tflite",
    save_path="resnet50.mxq",
)

(Optional) Setting Quantization Methods and Parameters#

After the model preparation, the qbcompiler quantizes the model to reduce its size and improve inference speed. The qbcompiler is equipped with carefully tuned default quantization methods, which are suitable for most models. However, users can also customize the quantization methods and parameters to achieve better performance for their specific models.

The example code below shows how to compile the model with customized quantization methods and parameters. Further details about the quantization methods and parameters can be found in the mxq_compile function.

""" Quantization Methods and Parameters """ 
from qbcompiler import mxq_compile
onnx_model_path = "/workspace/resnet50.onnx"
calib_data_path = "/workspace/calibration/resnet50"

mxq_compile(
    model=onnx_model_path,
    calib_data_path=calib_data_path,
    quantization_method=1, # per channel quantization
    quantization_mode=1, # max percentile quantization 
    percentile=0.999,
    quantization_output=0, # per layer quantization for the output layer
    save_path="resnet50.mxq",
    backend="onnx"
)