TokenizeToEmbedding Class Reference

TokenizeToEmbedding Class Reference#

Mobilint SDK qb Compiler: TokenizeToEmbedding Class Reference

Mobilint SDK qb Compiler v1.2

MCS002-EN

Convert tokenized text into embeddings using a Hugging Face model. More...

Convert tokenized text into embeddings using a Hugging Face model.

Parameters

tokenizer_dir	Directory containing tokenizer resources.
model_dir	Optional directory for the model weights; defaults to tokenizer_dir.
device	Device string determining where inference runs.
trust_remote_code	Whether to load custom code from the model repo.
min_seqlen	Minimum token length required for a sample.
max_seqlen	Maximum token length; longer sequences are truncated.
pad_to_max	When true, pad sequences up to max_seqlen with zeros.
dtype	NumPy dtype used for exported embeddings.

Definition at line 321 of file preprocess.py.

The documentation for this class was generated from the following file: