ModelVariantHandle¶
Handle to a specific variant of a loaded model.
This class provides access to variant-specific information such as input/output shapes, buffer information, and quantization scales. It also offers APIs for managing inference buffers, consistent with the interface of the Model class.
Objects of this class are obtained via Model::getModelVariantHandle.
Buffer Management APIs¶
These APIs are used when performing inference with Model::inferBuffer or Model::inferBufferToFloat, using this variant’s input and output shapes.
Buffers are acquired using:
acquireInputBufferacquireOutputBuffer
Any acquired buffer must be released using:
releaseBufferreleaseBuffers
Repositioning is handled by:
repositionInputsrepositionOutputs
Note
These APIs are intended for advanced use and follow the same buffer management interface as the Model class.
| Member | Description |
|---|---|
acquireInputBuffer |
|
acquireOutputBuffer |
|
acquireInputBuffers |
|
acquireOutputBuffers |
|
acquireInputNPUData |
Acquires an NPUData for this variant's input at the given index. |
acquireOutputNPUData |
Acquires an NPUData for this variant's output at the given index. |
releaseBuffer |
|
releaseBuffers |
|
repositionInputs ×4 |
|
repositionOutputs ×4 |
acquireInputBuffer¶
acquireOutputBuffer¶
acquireInputBuffers¶
std::vector<std::vector<Buffer>> acquireInputBuffers(int batch_size, const std::vector<std::vector<int>> &seqlens={}) const
acquireOutputBuffers¶
std::vector<std::vector<Buffer>> acquireOutputBuffers(int batch_size, const std::vector<std::vector<int>> &seqlens={}) const
acquireInputNPUData¶
NPUData acquireInputNPUData(const std::vector<int64_t> &shape, int idx, bool upload, DataType dtype, StatusCode &sc) const
Acquires an NPUData for this variant's input at the given index.
Equivalent to Model::acquireInputNPUData, except that the tensor layout comes from this variant instead of variant 0. Use this when running inference on a variant other than the first one: Model::infer selects the variant from the input shapes, so the NPUData must be bound to that same variant.
Warning
This API is in beta: it may still contain bugs, and its behavior may change in a future release.
Parameters
shape[in] — The shape of the input tensor, in NHWC/HWC or NCHW/CHW layout.idx[in] — The index of the model input.upload[in] — If true, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).dtype[in] — The element type of the data. Must be one of the types the model accepts for this input; otherwisescis set toModel_DtypeMismatched.sc[out] — A reference to a status code that will be updated to indicate whether the operation was successful or encountered an error.
Returns
The acquired NPUData. Its contents are valid only when sc is OK.
acquireOutputNPUData¶
NPUData acquireOutputNPUData(const std::vector<int64_t> &shape, int idx, bool upload, DataType dtype, StatusCode &sc) const
Acquires an NPUData for this variant's output at the given index.
Equivalent to Model::acquireOutputNPUData, except that the tensor layout comes from this variant instead of variant 0. See acquireInputNPUData.
Warning
This API is in beta: it may still contain bugs, and its behavior may change in a future release.
Parameters
shape[in] — The shape of the output tensor, in NHWC/HWC or NCHW/CHW layout.idx[in] — The index of the model output.upload[in] — If true, NPU memory is allocated and the data is placed on the NPU; otherwise the data is kept on the host (CPU).dtype[in] — The element type of the data. Must be one of the types the model produces for this output; otherwisescis set toModel_DtypeMismatched.sc[out] — A reference to a status code that will be updated to indicate whether the operation was successful or encountered an error.
Returns
The acquired NPUData. Its contents are valid only when sc is OK.
releaseBuffer¶
releaseBuffers¶
repositionInputs¶
StatusCode repositionInputs(const std::vector<float *> &input, std::vector<Buffer> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<uint8_t *> &input, std::vector<Buffer> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<float *> &input, std::vector<std::vector<Buffer>> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionInputs(const std::vector<uint8_t *> &input, std::vector<std::vector<Buffer>> &input_buf, const std::vector<std::vector<int>> &seqlens={}) const
repositionOutputs¶
StatusCode repositionOutputs(const std::vector<Buffer> &output_buf, std::vector<float *> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<Buffer> &output_buf, std::vector<std::vector<float>> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<std::vector<Buffer>> &output_buf, std::vector<float *> &output, const std::vector<std::vector<int>> &seqlens={}) const
StatusCode repositionOutputs(const std::vector<std::vector<Buffer>> &output_buf, std::vector<std::vector<float>> &output, const std::vector<std::vector<int>> &seqlens={}) const
Member functions¶
| Member | Description |
|---|---|
ModelVariantHandle ×2 |
|
operator= ×2 |
|
getVariantIdx |
Returns the index of this model variant. |
getModelInputShape |
Returns the input shape for this model variant. |
getModelOutputShape |
Returns the output shape for this model variant. |
getInputBufferInfo |
Returns the input buffer information for this variant. |
getOutputBufferInfo |
Returns the output buffer information for this variant. |
getInputScale |
Returns the input quantization scale(s) for this variant. |
getOutputScale |
Returns the output quantization scale(s) for this variant. |
getModelInputDataType |
Returns a data type for model inputs. |
getModelOutputDataType |
Returns a data type for model outputs. |
ModelVariantHandle¶
operator=¶
getVariantIdx¶
Returns the index of this model variant.
Returns
Index of the model variant.
getModelInputShape¶
Returns the input shape for this model variant.
Returns
Reference to the input shape.
getModelOutputShape¶
Returns the output shape for this model variant.
Returns
Reference to the output shape.
getInputBufferInfo¶
Returns the input buffer information for this variant.
Returns
Reference to a vector of input buffer information.
getOutputBufferInfo¶
Returns the output buffer information for this variant.
Returns
Reference to a vector of output buffer information.
getInputScale¶
Returns the input quantization scale(s) for this variant.
Returns
Vector of input scales.
getOutputScale¶
Returns the output quantization scale(s) for this variant.
Returns
Vector of output scales.
getModelInputDataType¶
Returns a data type for model inputs.
Returns
An input data type.
getModelOutputDataType¶
Returns a data type for model outputs.
Returns
An output data type.