ModelVariantHandle¶
Handle to a specific variant of a loaded model.
This class provides access to variant-specific information such as input/output shapes, buffer information, and quantization scales. It also offers APIs for managing inference buffers, consistent with the interface of the Model class.
Objects of this class are obtained via Model::getModelVariantHandle.
Buffer Management APIs¶
These APIs are used when performing inference with Model::inferBuffer or Model::inferBufferToFloat, using this variant’s input and output shapes.
Buffers are acquired using:
acquireInputBufferacquireOutputBuffer
Any acquired buffer must be released using:
releaseBufferreleaseBuffers
Repositioning is handled by:
repositionInputsrepositionOutputs
Note
These APIs are intended for advanced use and follow the same buffer management interface as the Model class.
| Member | Description |
|---|---|
acquireInputBuffer |
|
acquireOutputBuffer |
|
acquireInputBuffers |
|
acquireOutputBuffers |
|
releaseBuffer |
|
releaseBuffers |
|
repositionInputs ×4 |
|
repositionOutputs ×4 |
acquireInputBuffer¶
acquireOutputBuffer¶
acquireInputBuffers¶
std::vector<std::vector<Buffer>> acquireInputBuffers(int batch_size, const std::vector<std::vector<int>> &seqlens={}) const
acquireOutputBuffers¶
std::vector<std::vector<Buffer>> acquireOutputBuffers(int batch_size, const std::vector<std::vector<int>> &seqlens={}) const
releaseBuffer¶
releaseBuffers¶
repositionInputs¶
StatusCode repositionInputs(const std::vector<float *> &input, std::vector<Buffer> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<uint8_t *> &input, std::vector<Buffer> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<float *> &input, std::vector<std::vector<Buffer>> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<uint8_t *> &input, std::vector<std::vector<Buffer>> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
repositionOutputs¶
StatusCode repositionOutputs(const std::vector<Buffer> &output_buf, std::vector<float *> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<Buffer> &output_buf, std::vector<std::vector<float>> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<std::vector<Buffer>> &output_buf, std::vector<float *> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<std::vector<Buffer>> &output_buf, std::vector<std::vector<float>> &output, const std::vector<std::vector<int>> &seqlens={}) const
Member functions¶
| Member | Description |
|---|---|
ModelVariantHandle ×2 |
|
operator= ×2 |
|
getVariantIdx |
Returns the index of this model variant. |
getModelInputShape |
Returns the input shape for this model variant. |
getModelOutputShape |
Returns the output shape for this model variant. |
getInputBufferInfo |
Returns the input buffer information for this variant. |
getOutputBufferInfo |
Returns the output buffer information for this variant. |
getInputScale |
Returns the input quantization scale(s) for this variant. |
getOutputScale |
Returns the output quantization scale(s) for this variant. |
getModelInputDataType |
Returns a data type for model inputs. |
getModelOutputDataType |
Returns a data type for model outputs. |
ModelVariantHandle¶
operator=¶
getVariantIdx¶
Returns the index of this model variant.
Returns
Index of the model variant.
getModelInputShape¶
Returns the input shape for this model variant.
Returns
Reference to the input shape.
getModelOutputShape¶
Returns the output shape for this model variant.
Returns
Reference to the output shape.
getInputBufferInfo¶
Returns the input buffer information for this variant.
Returns
Reference to a vector of input buffer information.
getOutputBufferInfo¶
Returns the output buffer information for this variant.
Returns
Reference to a vector of output buffer information.
getInputScale¶
Returns the input quantization scale(s) for this variant.
Returns
Vector of input scales.
getOutputScale¶
Returns the output quantization scale(s) for this variant.
Returns
Vector of output scales.
getModelInputDataType¶
Returns a data type for model inputs.
Returns
An input data type.
getModelOutputDataType¶
Returns a data type for model outputs.
Returns
An output data type.