LLM Configuration

Mobilint SDK qb Compiler v0.12.0.0 MCS002-EN

Classes | Functions | Variables

LLM Configuration

LLM configuration definitions used by Mobilint compiler for LLM model quantization. More...

Classes
class	qubee.configs.llm_config.LlmConfig
	Configuration for Large Language Model (LLM) compilation. More...

Functions
	qubee.configs.llm_config.LlmConfig.__init__ (self, bool llm_config_apply, int max_data_length, int max_sequence_length, int max_cache_length, int max_core_data_length, int random_seq_length, bool use_full_seq_length, bool use_custom_mask_input, int start_dram_offset, bool use_global_core, int batch_size, List[int] npu_core_ids, bool dynamic_rope, bool optional=optional)
	Initialize the LlmConfig:
	qubee.configs.llm_config.LlmConfig.default_config (cls)
	qubee.configs.llm_config.LlmConfig.from_kwargs (cls, **kwargs)
	qubee.configs.llm_config.LlmConfig.from_dict (cls, dict config_dict)
	Create LlmConfig from dictionary (JSON structure)
	qubee.configs.llm_config.LlmConfig.to_dict (self)
LlmConfig	qubee.configs.llm_config.get_llm_config (**kwargs)
	Create LlmConfig with partial parameters merged with defaults.

Variables
bool	qubee.configs.llm_config.LlmConfig.optional = True
dict	qubee.configs.llm_config.LlmConfig.DEFAULTS
dict	qubee.configs.llm_config.LlmConfig.py_to_cpp_name
dict	qubee.configs.llm_config.LlmConfig.min_max_check
	qubee.configs.llm_config.LlmConfig.llm_config_apply = llm_config_apply
	qubee.configs.llm_config.LlmConfig.max_data_length = max_data_length
	qubee.configs.llm_config.LlmConfig.max_sequence_length = max_sequence_length
	qubee.configs.llm_config.LlmConfig.max_cache_length = max_cache_length
	qubee.configs.llm_config.LlmConfig.max_core_data_length = max_core_data_length
	qubee.configs.llm_config.LlmConfig.random_seq_length = random_seq_length
	qubee.configs.llm_config.LlmConfig.use_full_seq_length = use_full_seq_length
	qubee.configs.llm_config.LlmConfig.use_custom_mask_input = use_custom_mask_input
	qubee.configs.llm_config.LlmConfig.start_dram_offset = start_dram_offset
	qubee.configs.llm_config.LlmConfig.use_global_core = use_global_core
	qubee.configs.llm_config.LlmConfig.batch_size = batch_size
	qubee.configs.llm_config.LlmConfig.npu_core_ids = npu_core_ids
	qubee.configs.llm_config.LlmConfig.dynamic_rope = dynamic_rope

Detailed Description

LLM configuration definitions used by Mobilint compiler for LLM model quantization.

Function Documentation

◆ __init__()

qubee.configs.llm_config.LlmConfig.__init__	(		self,
		bool	llm_config_apply,
		int	max_data_length,
		int	max_sequence_length,
		int	max_cache_length,
		int	max_core_data_length,
		int	random_seq_length,
		bool	use_full_seq_length,
		bool	use_custom_mask_input,
		int	start_dram_offset,
		bool	use_global_core,
		int	batch_size,
		List[int]	npu_core_ids,
		bool	dynamic_rope,
		bool	optional = optional )

Initialize the LlmConfig:

Parameters

llm_config_apply	bool. If True, apply LLM-specific configurations.
max_data_length	int. Maximum data length.
max_sequence_length	int. Maximum sequence length.
max_cache_length	int. Maximum cache length.
max_core_data_length	int. Maximum core data length.
random_seq_length	int. Random sequence length used for calibration.
use_full_seq_length	bool. If True, use the full sequence length for calibration.
use_custom_mask_input	bool. If True, use a custom mask input.
start_dram_offset	int. Starting DRAM offset.
use_global_core	bool. If True, use a global core.
batch_size	int. Batch size.
npu_core_ids	List[int]. List of NPU core IDs.
dynamic_rope	bool. If True, enable dynamic RoPE.
optional	bool. Indicates whether this config is optional. Use the default value defined as a class variable.

Definition at line 59 of file llm_config.py.

◆ default_config()

qubee.configs.llm_config.LlmConfig.default_config

(

cls

)

Definition at line 110 of file llm_config.py.

◆ from_kwargs()

qubee.configs.llm_config.LlmConfig.from_kwargs	(		cls,
		**	kwargs )

Definition at line 114 of file llm_config.py.

◆ from_dict()

qubee.configs.llm_config.LlmConfig.from_dict	(		cls,
		dict	config_dict )

Create LlmConfig from dictionary (JSON structure)

Definition at line 130 of file llm_config.py.

◆ to_dict()

qubee.configs.llm_config.LlmConfig.to_dict

(

self

)

Definition at line 173 of file llm_config.py.

◆ get_llm_config()

LlmConfig qubee.configs.llm_config.get_llm_config

(

**

kwargs

)

Create LlmConfig with partial parameters merged with defaults.

Definition at line 203 of file llm_config.py.

Variable Documentation

◆ optional

bool qubee.configs.llm_config.LlmConfig.optional = True

static

Definition at line 17 of file llm_config.py.

◆ DEFAULTS

qubee.configs.llm_config.LlmConfig.DEFAULTS

static

Initial value:

= {

"llm_config_apply": False,

"max_data_length": 4096,

"max_sequence_length": 4096,

"max_cache_length": 4096,

"max_core_data_length": 128,

"random_seq_length": 80,

"use_full_seq_length": False,

"use_custom_mask_input": False,

"start_dram_offset": 0,

"use_global_core": False,

"batch_size": 1,

"npu_core_ids": [0],

"dynamic_rope": False,

}

Definition at line 18 of file llm_config.py.

◆ py_to_cpp_name

dict qubee.configs.llm_config.LlmConfig.py_to_cpp_name

static

Initial value:

= {

"llm_config_apply": "apply",

"max_data_length": "maxDataLength",

"max_sequence_length": "maxSequenceLength",

"max_cache_length": "maxCacheLength",

"max_core_data_length": "maxCoreDataLength",

"random_seq_length": "randomSeqLength",

"use_full_seq_length": "useFullSeqLength",

"use_custom_mask_input": "useCustomMaskInput",

"start_dram_offset": "startDramOffset",

"use_global_core": "useGlobalCore",

"batch_size": "batchSize",

"dynamic_rope": "dynamicRope",

}

Definition at line 34 of file llm_config.py.

◆ min_max_check

dict qubee.configs.llm_config.LlmConfig.min_max_check

static

Initial value:

= {

"max_data_length": [0, None],

"max_sequence_length": [0, None],

"max_cache_length": [0, None],

"max_core_data_length": [0, None],

"random_seq_length": [0, None],

"start_dram_offset": [0, None],

"batch_size": [0, None],

}

Definition at line 49 of file llm_config.py.

◆ llm_config_apply

qubee.configs.llm_config.LlmConfig.llm_config_apply = llm_config_apply

Definition at line 94 of file llm_config.py.

◆ max_data_length

qubee.configs.llm_config.LlmConfig.max_data_length = max_data_length

Definition at line 95 of file llm_config.py.

◆ max_sequence_length

qubee.configs.llm_config.LlmConfig.max_sequence_length = max_sequence_length

Definition at line 96 of file llm_config.py.

◆ max_cache_length

qubee.configs.llm_config.LlmConfig.max_cache_length = max_cache_length

Definition at line 97 of file llm_config.py.

◆ max_core_data_length

qubee.configs.llm_config.LlmConfig.max_core_data_length = max_core_data_length

Definition at line 98 of file llm_config.py.

◆ random_seq_length

qubee.configs.llm_config.LlmConfig.random_seq_length = random_seq_length

Definition at line 99 of file llm_config.py.

◆ use_full_seq_length

qubee.configs.llm_config.LlmConfig.use_full_seq_length = use_full_seq_length

Definition at line 100 of file llm_config.py.

◆ use_custom_mask_input

qubee.configs.llm_config.LlmConfig.use_custom_mask_input = use_custom_mask_input

Definition at line 101 of file llm_config.py.

◆ start_dram_offset

qubee.configs.llm_config.LlmConfig.start_dram_offset = start_dram_offset

Definition at line 102 of file llm_config.py.

◆ use_global_core

qubee.configs.llm_config.LlmConfig.use_global_core = use_global_core

Definition at line 103 of file llm_config.py.

◆ batch_size

qubee.configs.llm_config.LlmConfig.batch_size = batch_size

Definition at line 104 of file llm_config.py.

◆ npu_core_ids

qubee.configs.llm_config.LlmConfig.npu_core_ids = npu_core_ids

Definition at line 105 of file llm_config.py.

◆ dynamic_rope

qubee.configs.llm_config.LlmConfig.dynamic_rope = dynamic_rope

Definition at line 106 of file llm_config.py.

---

Generated by  1.14.0