LLM Configuration

LLM Configuration#

Mobilint SDK qb Compiler: LLM Configuration

Mobilint SDK qb Compiler v1.0

MCS002-EN

LLM configuration definitions used by Mobilint compiler for LLM model quantization. More...

Classes
class	qbcompiler.configs.llm_config.LlmConfig
	Configuration for Large Language Model (LLM) compilation. More...

Functions
	qbcompiler.configs.llm_config.LlmConfig.__init__ (self, bool llm_config_apply, int max_data_length, int max_sequence_length, int max_cache_length, int max_core_data_length, int random_seq_length, bool use_full_seq_length, bool use_global_core, int batch_size, List[int] npu_core_ids, bool dynamic_rope, bool optional=optional)
	Initialize the LlmConfig:
	qbcompiler.configs.llm_config.LlmConfig.default_config (cls)
	qbcompiler.configs.llm_config.LlmConfig.from_kwargs (cls, **kwargs)
	qbcompiler.configs.llm_config.LlmConfig.from_dict (cls, dict config_dict)
	Create LlmConfig from dictionary (JSON structure)
	qbcompiler.configs.llm_config.LlmConfig.to_dict (self)
LlmConfig	qbcompiler.configs.llm_config.get_llm_config (**kwargs)
	Create LlmConfig with partial parameters merged with defaults.

Variables
bool	qbcompiler.configs.llm_config.LlmConfig.optional = True
dict	qbcompiler.configs.llm_config.LlmConfig.DEFAULTS
dict	qbcompiler.configs.llm_config.LlmConfig.py_to_cpp_name
dict	qbcompiler.configs.llm_config.LlmConfig.min_max_check
	qbcompiler.configs.llm_config.LlmConfig.llm_config_apply = llm_config_apply
	qbcompiler.configs.llm_config.LlmConfig.max_data_length = max_data_length
	qbcompiler.configs.llm_config.LlmConfig.max_sequence_length = max_sequence_length
	qbcompiler.configs.llm_config.LlmConfig.max_cache_length = max_cache_length
	qbcompiler.configs.llm_config.LlmConfig.max_core_data_length = max_core_data_length
	qbcompiler.configs.llm_config.LlmConfig.random_seq_length = random_seq_length
	qbcompiler.configs.llm_config.LlmConfig.use_full_seq_length = use_full_seq_length
	qbcompiler.configs.llm_config.LlmConfig.use_global_core = use_global_core
	qbcompiler.configs.llm_config.LlmConfig.batch_size = batch_size
	qbcompiler.configs.llm_config.LlmConfig.npu_core_ids = npu_core_ids
	qbcompiler.configs.llm_config.LlmConfig.dynamic_rope = dynamic_rope

Detailed Description

LLM configuration definitions used by Mobilint compiler for LLM model quantization.

Function Documentation

◆ init()

qbcompiler.configs.llm_config.LlmConfig.__init__	(		self,
		bool	llm_config_apply,
		int	max_data_length,
		int	max_sequence_length,
		int	max_cache_length,
		int	max_core_data_length,
		int	random_seq_length,
		bool	use_full_seq_length,
		bool	use_global_core,
		int	batch_size,
		List[int]	npu_core_ids,
		bool	dynamic_rope,
		bool	optional = optional )

Initialize the LlmConfig:

Parameters

llm_config_apply	bool. If True, apply LLM-specific configurations.
max_data_length	int. Maximum data length.
max_sequence_length	int. Maximum sequence length.
max_cache_length	int. Maximum cache length.
max_core_data_length	int. Maximum core data length.
random_seq_length	int. Random sequence length used for calibration.
use_full_seq_length	bool. If True, use the full sequence length for calibration.
use_global_core	bool. If True, use a global core.
batch_size	int. Batch size.
npu_core_ids	List[int]. List of NPU core IDs.
dynamic_rope	bool. If True, enable dynamic RoPE.
optional	bool. Indicates whether this config is optional. Use the default value defined as a class variable.

Definition at line 54 of file llm_config.py.

◆ default_config()

qbcompiler.configs.llm_config.LlmConfig.default_config ( cls )

Definition at line 99 of file llm_config.py.

◆ from_kwargs()

qbcompiler.configs.llm_config.LlmConfig.from_kwargs	(		cls,
		**	kwargs )

Definition at line 103 of file llm_config.py.

◆ from_dict()

qbcompiler.configs.llm_config.LlmConfig.from_dict	(		cls,
		dict	config_dict )

Create LlmConfig from dictionary (JSON structure)

Definition at line 119 of file llm_config.py.

◆ to_dict()

qbcompiler.configs.llm_config.LlmConfig.to_dict ( self )

Definition at line 158 of file llm_config.py.

◆ get_llm_config()

LlmConfig qbcompiler.configs.llm_config.get_llm_config ( ** kwargs )

Create LlmConfig with partial parameters merged with defaults.

Definition at line 186 of file llm_config.py.

Variable Documentation

◆ optional

bool qbcompiler.configs.llm_config.LlmConfig.optional = True

static

Definition at line 17 of file llm_config.py.

◆ DEFAULTS

qbcompiler.configs.llm_config.LlmConfig.DEFAULTS

static

Initial value:

=  {
        "llm_config_apply": False,
        "max_data_length": 4096,
        "max_sequence_length": 4096,
        "max_cache_length": 4096,
        "max_core_data_length": 128,
        "random_seq_length": 80,
        "use_full_seq_length": False,
        "use_global_core": False,
        "batch_size": 1,
        "npu_core_ids": [0],
        "dynamic_rope": False,
    }

Definition at line 18 of file llm_config.py.

◆ py_to_cpp_name

dict qbcompiler.configs.llm_config.LlmConfig.py_to_cpp_name

static

Initial value:

=  {
        "llm_config_apply": "apply",
        "max_data_length": "maxDataLength",
        "max_sequence_length": "maxSequenceLength",
        "max_cache_length": "maxCacheLength",
        "max_core_data_length": "maxCoreDataLength",
        "random_seq_length": "randomSeqLength",
        "use_full_seq_length": "useFullSeqLength",
        "use_global_core": "useGlobalCore",
        "batch_size": "batchSize",
        "dynamic_rope": "dynamicRope",
    }

Definition at line 32 of file llm_config.py.

◆ min_max_check

dict qbcompiler.configs.llm_config.LlmConfig.min_max_check

static

Initial value:

=  {
        "max_data_length": [0, None],
        "max_sequence_length": [0, None],
        "max_cache_length": [0, None],
        "max_core_data_length": [0, None],
        "random_seq_length": [0, None],
        "batch_size": [0, None],
    }

Definition at line 45 of file llm_config.py.

◆ llm_config_apply

qbcompiler.configs.llm_config.LlmConfig.llm_config_apply = llm_config_apply

Definition at line 85 of file llm_config.py.

◆ max_data_length

qbcompiler.configs.llm_config.LlmConfig.max_data_length = max_data_length

Definition at line 86 of file llm_config.py.

◆ max_sequence_length

qbcompiler.configs.llm_config.LlmConfig.max_sequence_length = max_sequence_length

Definition at line 87 of file llm_config.py.

◆ max_cache_length

qbcompiler.configs.llm_config.LlmConfig.max_cache_length = max_cache_length

Definition at line 88 of file llm_config.py.

◆ max_core_data_length

qbcompiler.configs.llm_config.LlmConfig.max_core_data_length = max_core_data_length

Definition at line 89 of file llm_config.py.

◆ random_seq_length

qbcompiler.configs.llm_config.LlmConfig.random_seq_length = random_seq_length

Definition at line 90 of file llm_config.py.

◆ use_full_seq_length

qbcompiler.configs.llm_config.LlmConfig.use_full_seq_length = use_full_seq_length

Definition at line 91 of file llm_config.py.

◆ use_global_core

qbcompiler.configs.llm_config.LlmConfig.use_global_core = use_global_core

Definition at line 92 of file llm_config.py.

◆ batch_size

qbcompiler.configs.llm_config.LlmConfig.batch_size = batch_size

Definition at line 93 of file llm_config.py.

◆ npu_core_ids

qbcompiler.configs.llm_config.LlmConfig.npu_core_ids = npu_core_ids

Definition at line 94 of file llm_config.py.

◆ dynamic_rope

qbcompiler.configs.llm_config.LlmConfig.dynamic_rope = dynamic_rope

Definition at line 95 of file llm_config.py.

LLM Configuration

LLM Configuration#

Classes

Functions

Variables

Detailed Description

Function Documentation

◆ __init__()

◆ default_config()

◆ from_kwargs()

◆ from_dict()

◆ to_dict()

◆ get_llm_config()

Variable Documentation

◆ optional

◆ DEFAULTS

◆ py_to_cpp_name

◆ min_max_check

◆ llm_config_apply

◆ max_data_length

◆ max_sequence_length

◆ max_cache_length

◆ max_core_data_length

◆ random_seq_length

◆ use_full_seq_length

◆ use_global_core

◆ batch_size

◆ npu_core_ids

◆ dynamic_rope

◆ init()