Skip to content

ModelVariantHandle

#include <qbruntime/model_variant_handle.h>

class mobilint::ModelVariantHandle

Handle to a specific variant of a loaded model.

This class provides access to variant-specific information such as input/output shapes, buffer information, and quantization scales. It also offers APIs for managing inference buffers, consistent with the interface of the Model class.

Objects of this class are obtained via Model::getModelVariantHandle.

Buffer Management APIs

These APIs are used when performing inference with Model::inferBuffer or Model::inferBufferToFloat, using this variant’s input and output shapes.

Buffers are acquired using:

  • acquireInputBuffer
  • acquireOutputBuffer

Any acquired buffer must be released using:

  • releaseBuffer
  • releaseBuffers

Repositioning is handled by:

  • repositionInputs
  • repositionOutputs

Note

These APIs are intended for advanced use and follow the same buffer management interface as the Model class.

Member Description
acquireInputBuffer
acquireOutputBuffer
acquireInputBuffers
acquireOutputBuffers
acquireInputNPUData Acquires an NPUData for this variant's input at the given index.
acquireOutputNPUData Acquires an NPUData for this variant's output at the given index.
releaseBuffer
releaseBuffers
repositionInputs ×4
repositionOutputs ×4

acquireInputBuffer

std::vector<Buffer> acquireInputBuffer(const std::vector<std::vector<int>> &seqlens={}) const

acquireOutputBuffer

std::vector<Buffer> acquireOutputBuffer(const std::vector<std::vector<int>> &seqlens={}) const

acquireInputBuffers

std::vector<std::vector<Buffer>> acquireInputBuffers(int batch_size, const std::vector<std::vector<int>> &seqlens={}) const

acquireOutputBuffers

std::vector<std::vector<Buffer>> acquireOutputBuffers(int batch_size, const std::vector<std::vector<int>> &seqlens={}) const

acquireInputNPUData

NPUData acquireInputNPUData(const std::vector<int64_t> &shape, int idx, bool upload, DataType dtype, StatusCode &sc) const

Acquires an NPUData for this variant's input at the given index.

Equivalent to Model::acquireInputNPUData, except that the tensor layout comes from this variant instead of variant 0. Use this when running inference on a variant other than the first one: Model::infer selects the variant from the input shapes, so the NPUData must be bound to that same variant.

Warning

This API is in beta: it may still contain bugs, and its behavior may change in a future release.

Parameters

  • shape [in] — The shape of the input tensor, in NHWC/HWC or NCHW/CHW layout.
  • idx [in] — The index of the model input.
  • upload [in] — If true, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).
  • dtype [in] — The element type of the data. Must be one of the types the model accepts for this input; otherwise sc is set to Model_DtypeMismatched.
  • sc [out] — A reference to a status code that will be updated to indicate whether the operation was successful or encountered an error.

Returns

The acquired NPUData. Its contents are valid only when sc is OK.

acquireOutputNPUData

NPUData acquireOutputNPUData(const std::vector<int64_t> &shape, int idx, bool upload, DataType dtype, StatusCode &sc) const

Acquires an NPUData for this variant's output at the given index.

Equivalent to Model::acquireOutputNPUData, except that the tensor layout comes from this variant instead of variant 0. See acquireInputNPUData.

Warning

This API is in beta: it may still contain bugs, and its behavior may change in a future release.

Parameters

  • shape [in] — The shape of the output tensor, in NHWC/HWC or NCHW/CHW layout.
  • idx [in] — The index of the model output.
  • upload [in] — If true, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).
  • dtype [in] — The element type of the data. Must be one of the types the model produces for this output; otherwise sc is set to Model_DtypeMismatched.
  • sc [out] — A reference to a status code that will be updated to indicate whether the operation was successful or encountered an error.

Returns

The acquired NPUData. Its contents are valid only when sc is OK.

releaseBuffer

StatusCode releaseBuffer(std::vector<Buffer> &buffer) const

releaseBuffers

StatusCode releaseBuffers(std::vector<std::vector<Buffer>> &buffers) const

repositionInputs

StatusCode repositionInputs(const std::vector<float *> &input, std::vector<Buffer> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<uint8_t *> &input, std::vector<Buffer> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<float *> &input, std::vector<std::vector<Buffer>> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<uint8_t *> &input, std::vector<std::vector<Buffer>> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const

repositionOutputs

StatusCode repositionOutputs(const std::vector<Buffer> &output_buf, std::vector<float *> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<Buffer> &output_buf, std::vector<std::vector<float>> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<std::vector<Buffer>> &output_buf, std::vector<float *> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<std::vector<Buffer>> &output_buf, std::vector<std::vector<float>> &output, const std::vector<std::vector<int>> &seqlens={}) const

Member functions

Member Description
ModelVariantHandle ×2
operator= ×2
getVariantIdx Returns the index of this model variant.
getModelInputShape Returns the input shape for this model variant.
getModelOutputShape Returns the output shape for this model variant.
getInputBufferInfo Returns the input buffer information for this variant.
getOutputBufferInfo Returns the output buffer information for this variant.
getInputScale Returns the input quantization scale(s) for this variant.
getOutputScale Returns the output quantization scale(s) for this variant.
getModelInputDataType Returns a data type for model inputs.
getModelOutputDataType Returns a data type for model outputs.

ModelVariantHandle

ModelVariantHandle(const ModelVariantHandle &other)=delete
ModelVariantHandle(ModelVariantHandle &&other)=delete

operator=

ModelVariantHandle & operator=(const ModelVariantHandle &rhs)=delete
ModelVariantHandle & operator=(ModelVariantHandle &&rhs) noexcept=delete

getVariantIdx

int getVariantIdx() const

Returns the index of this model variant.

Returns

Index of the model variant.

getModelInputShape

const std::vector<std::vector<int64_t>> & getModelInputShape() const

Returns the input shape for this model variant.

Returns

Reference to the input shape.

getModelOutputShape

const std::vector<std::vector<int64_t>> & getModelOutputShape() const

Returns the output shape for this model variant.

Returns

Reference to the output shape.

getInputBufferInfo

const std::vector<BufferInfo> & getInputBufferInfo() const

Returns the input buffer information for this variant.

Returns

Reference to a vector of input buffer information.

getOutputBufferInfo

const std::vector<BufferInfo> & getOutputBufferInfo() const

Returns the output buffer information for this variant.

Returns

Reference to a vector of output buffer information.

getInputScale

std::vector<Scale> getInputScale() const

Returns the input quantization scale(s) for this variant.

Returns

Vector of input scales.

getOutputScale

std::vector<Scale> getOutputScale() const

Returns the output quantization scale(s) for this variant.

Returns

Vector of output scales.

getModelInputDataType

DataType getModelInputDataType() const

Returns a data type for model inputs.

Returns

An input data type.

getModelOutputDataType

DataType getModelOutputDataType() const

Returns a data type for model outputs.

Returns

An output data type.