# Transformer

This chapter describes LLM compilation for HuggingFace `transformers` models through the PyTorch path, using `backend="torch"`.

The LLM flow follows the same qb Compiler pipeline used by other models:

```text
HuggingFace transformers model
  -> MBLT
  -> MXQ
```

For LLMs, the important difference is that the compiled model is treated as a transformer decoder body with runtime state. The runtime keeps the KV cache while the NPU executes the repeated transformer blocks.

For Vision Transformer (ViT) models, see [Vision — Vision Transformer](vision.md#vision-transformer).

## LLM Compile Overview

A typical HuggingFace LLM compile has four parts:

1. Prepare the model in HuggingFace `transformers` format.
2. Prepare calibration data that matches the tensor input used by the compiled decoder body.
3. Set `backend="torch"` and provide HuggingFace loader metadata through `hf_config`.
4. Enable LLM settings with `LlmConfig`, either directly or through the `llm` / `llm_fast` preset.

Use GPU compilation for practical LLM workloads. Transformer decoder models are large, and calibration plus quantization can require substantial memory.

## Supported Model Structure: Transformer + KV Cache

qb Compiler targets causal decoder-style transformer models with KV cache. The expected structure is:

- token embedding and tokenizer are handled outside the compiled NPU body;
- the compiled body receives embedded token tensors or the model input form selected by the frontend;
- each transformer block contains attention, RoPE or another positional encoding path, MLP/FFN, residual connections, and normalization;
- attention uses past key/value state so generation can append new tokens without recomputing the full prefix;
- the output head can be quantized separately from the internal transformer blocks.

The model should be loadable by HuggingFace `transformers`, usually with `AutoModelForCausalLM`. Architectures such as LLaMA, Qwen, Gemma, and similar decoder-only families are the main target. Models with unusual custom attention, sliding-window behavior, MoE routing, or nonstandard cache layout may require model-specific handling and should be validated before relying on generated MXQ output.

## HuggingFace LLM Compilation

HuggingFace models are compiled through the PyTorch frontend:

```python
hf_config = {
    "library": "transformers",
    "loader": "AutoModelForCausalLM",
    "tokenizer": "AutoTokenizer",
    "model_args": (),
    "model_kwargs": {
        "trust_remote_code": True,
    },
    "tokenizer_args": (),
    "tokenizer_kwargs": {},
}
```

Pass the model id or local model directory as `model`, set `backend="torch"`, and pass `hf_config` to `mxq_compile`. For gated HuggingFace models, authenticate with `huggingface-cli login` before compiling.

Set `weight_dtype` to match the original model weights, for example `bfloat16` or `float16`. The value is usually visible in the model's `config.json` or model card.

## LlmConfig

`LlmConfig` is the LLM-specific sub-config. It enables LLM graph handling and records sequence, cache, calibration, and runtime metadata.

```python
from qbcompiler.configs import LlmConfig

llm_config = LlmConfig(
    apply=True,
    attributes=LlmConfig.Attributes(
        maxSequenceLength=4096,
        maxCacheLength=4096,
        maxDataLength=4096,
        maxCoreDataLength=128,
        calibration=LlmConfig.Attributes.Calibration(
            useFullSeqLength=True,
        ),
        runtime=LlmConfig.Attributes.Runtime(
            batchSize=1,
            dynamicRope=False,
            dynamicMask=False,
        ),
    ),
)
```

The same fields can be written in JSON/YAML compile config files with their schema names:

```json
{
  "llm": {
    "apply": true,
    "attributes": {
      "maxSequenceLength": 4096,
      "maxCacheLength": 4096,
      "calibration": {
        "useFullSeqLength": true
      },
      "runtime": {
        "batchSize": 1,
        "dynamicRope": false,
        "dynamicMask": false
      }
    }
  }
}
```

Some compatibility examples use helper functions such as `get_llm_config(...)`. The schema-backed form above is preferred when writing reusable config files because it maps directly to `CompileConfig`.

## Sequence Length

`maxSequenceLength` is the maximum token sequence length represented by the compiled LLM graph. It affects attention shapes, RoPE/mask generation, calibration shape, memory use, compile time, and the maximum prompt or decode window supported by the resulting MXQ.

Choose the smallest value that covers the deployment requirement. Increasing it from 4K to 8K or beyond can sharply increase compilation cost and runtime memory requirements.

For current LLM workflows, set calibration samples to the same shape policy used at compile time. If `useFullSeqLength=True`, calibration uses full-length sequences and better represents the worst-case activation range.

## Cache Length

`maxCacheLength` is the maximum KV cache length recorded for runtime generation. It should usually match `maxSequenceLength`.

When `maxCacheLength` is smaller than the intended prompt plus generated context, runtime generation will hit the cache limit. When it is much larger than required, the compiled package and runtime memory budget can become unnecessarily large.

## Batch Size

The LLM runtime batch size is configured in `llm.attributes.runtime.batchSize`. Most edge deployments use `batchSize=1` for interactive generation. When `batchSize` is greater than `1`, qb Compiler automatically fixes compile-time `inferenceScheme` to `single`.

This is different from vision-model batch inference that may use `multi` mode. For LLM and other KV-cache transformer models, the NPU distributes batched transformer work by splitting operations into one-core units. Because the transformer runtime state and KV cache must be managed per sequence, batched LLM compilation is supported only with `single` mode. `multi`, `global4`, and `global8` cannot be used for `LlmConfig` batch compilation. For what the core modes mean in general, see {external+aries:doc}`ARIES Core Mode <core-mode>`.

The qb Runtime infer API also changes for Batch LLM. Instead of passing independent vision-style batch tensors, runtime code passes batch metadata such as `BatchParam` with the concatenated LLM input. See the Batch LLM entry in {external+runtime:doc}`qb Runtime Release Notes <en/release_note>` for the current API shape.

When using a batch size greater than `1`, keep these points in mind:

- `runtime.batchSize`;
- runtime input packing and `BatchParam` metadata;
- target device memory budget, because KV cache memory is reserved for the configured batch size.

## KV Cache

The KV cache stores the key and value tensors produced by previous tokens. During autoregressive generation, the runtime appends the new token's K/V tensors to the cache and attention reads both the cached prefix and the current token.

This avoids recomputing the whole prompt for every generated token. It also means the compiled model and runtime metadata must agree on cache length, batch size, head layout, and positional encoding behavior.

If a model uses sliding-window attention or another bounded-cache mechanism, verify that the HuggingFace model path and generated MBLT express that cache policy correctly before producing a final MXQ.

## `llm` and `llm_fast` Presets

The built-in `llm` preset enables the LLM config and sets the default sequence/cache length to 4096:

| Setting | Value |
| --- | --- |
| `llm.attributes.maxSequenceLength` | `4096` |
| `llm.attributes.maxCacheLength` | `4096` |
| `calibration.mode` | `0` |
| `calibration.output` | `0` |

```python
mxq_compile(
    model=model_id,
    backend="torch",
    config_preset="llm",
    hf_config=hf_config,
    calib_data_path=calib_path,
    save_path="model.mxq",
)
```

The `llm_fast` preset extends `llm` and skips some accuracy-oriented optimization steps in exchange for faster compilation. It disables HessianQuant and enables several equivalent-transform passes used for transformer blocks, including QK, UD, VO, SpinR1, SpinR2, and OptimizeFFN. It also enables full-sequence calibration.

Use `llm` as the conservative starting point. Use `llm_fast` when compile time matters more, then validate model quality on representative prompts.

## `multimodal` Preset

The `multimodal` preset is the configuration reference point for models with more than one input family, such as image tensors plus token tensors. It sets `calibration.method=3` and enables `llm.apply=true`.

The preset does not prescribe image or text preprocessing, input sizes, sequence lengths, or cache lengths. Define those values to match the exported vision and language submodels. For calibration, prepare data for every compiled input; for image-plus-text models, this may mean image tensors for the vision path and token, embedding, mask, or cache tensors for the language path.

Use the model-specific Python script rather than generic CLI examples because the graph usually combines multiple input contracts. The `multimodal` preset provides a starting configuration, but it does not replace model-specific export, preprocessing, and calibration code.

## LLM Quantization Settings

For detailed quantization sub-config descriptions and recommended configuration recipes, see [Model Quantization — Quantization Configuration](model_quantization.md#quantization-configuration). For how to use presets, config files, and `dump-config`, see [Compile Configuration](compile_configuration.md).

LLM quantization usually needs different precision choices for different transformer projections. The transformer weight fields under `BitConfig` let you control query, key, value, output, FFN, and head weights independently:

```python
from qbcompiler.configs import BitConfig, CalibrationConfig, SearchWeightScaleConfig

calibration_config = CalibrationConfig(mode=0, output=0)

bit_config = BitConfig(
    transformer=BitConfig.Transformer(
        weight=BitConfig.Transformer.Weight(
            query=4,
            key=4,
            value=8,
            output=4,
            ffn=4,
            head=8,
        ),
    ),
)

search_weight_scale_config = SearchWeightScaleConfig(
    apply=True,
    transformer=SearchWeightScaleConfig.Transformer(
        query=True,
        key=True,
        value=True,
        out=True,
        ffn=True,
    ),
)
```

A practical progression is:

- start with 8-bit transformer weights to establish a quality baseline;
- move query/key/output/FFN to 4-bit for size and bandwidth reduction;
- keep value and head at 8-bit if perplexity or generation quality drops;
- enable search weight scale when using aggressive low-bit settings;
- validate with task-level prompts, not only compile success.

## Dynamic RoPE and Dynamic Mask

`dynamicRope` and `dynamicMask` are runtime options inside `LlmConfig.Attributes.Runtime`.

`dynamicRope=True` makes RoPE position data dynamic at runtime instead of fully fixed at compile time. Use it when the deployment needs variable decode positions that cannot be represented well by a fixed compile-time position path.

`dynamicMask=True` makes the attention mask a runtime input. Use it when prompt lengths, padding patterns, or causal/padding mask combinations vary at runtime.

Dynamic settings can make one compiled package cover more runtime cases, but they also add runtime input requirements and can reduce optimization opportunities. Keep them disabled unless the deployment needs that flexibility.

## `splitBlocks` and `splitParts`

Large LLMs may be split into multiple MXQ files. This can help with very large transformer stacks, packaging constraints, or runtime scheduling.

`splitBlocks` gives explicit split points by transformer block index:

```json
{
  "splitBlocks": [8, 16, 24]
}
```

`splitParts` evenly divides transformer blocks into the requested number of parts:

```json
{
  "splitParts": 4
}
```

Use only one split strategy for a compile job. Prefer `splitParts` when blocks are uniform and you only need a simple even split. Prefer `splitBlocks` when the model has nonuniform blocks or when target memory measurements show that specific boundaries work better.

## End-to-End Example

The following example compiles a HuggingFace LLaMA-style model through `backend="torch"` using an explicit `LlmConfig`.

```python
from qbcompiler import mxq_compile
from qbcompiler.configs import (
    BitConfig,
    CalibrationConfig,
    LlmConfig,
    SearchWeightScaleConfig,
)

model_id = "meta-llama/Llama-3.2-1B-Instruct"
calib_path = "/workspace/Llama-3.2-1B-Instruct-Wikipedia-en"
save_path = "./Llama-3.2-1B-Instruct.mxq"

hf_config = {
    "library": "transformers",
    "loader": "AutoModelForCausalLM",
    "tokenizer": "AutoTokenizer",
    "model_args": (),
    "model_kwargs": {
        "trust_remote_code": True,
    },
    "tokenizer_args": (),
    "tokenizer_kwargs": {},
}

llm_config = LlmConfig(
    apply=True,
    attributes=LlmConfig.Attributes(
        maxSequenceLength=4096,
        maxCacheLength=4096,
        calibration=LlmConfig.Attributes.Calibration(
            useFullSeqLength=True,
        ),
        runtime=LlmConfig.Attributes.Runtime(
            batchSize=1,
            dynamicRope=False,
            dynamicMask=False,
        ),
    ),
)

calibration_config = CalibrationConfig(mode=0, output=0)

bit_config = BitConfig(
    transformer=BitConfig.Transformer(
        weight=BitConfig.Transformer.Weight(
            query=8,
            key=8,
            value=8,
            output=8,
            ffn=8,
            head=8,
        ),
    ),
)

search_weight_scale_config = SearchWeightScaleConfig(
    apply=False,
    transformer=SearchWeightScaleConfig.Transformer(
        query=True,
        key=True,
        value=True,
        out=True,
        ffn=True,
    ),
)

mxq_compile(
    model=model_id,
    backend="torch",
    target_device="aries-rb",
    hf_config=hf_config,
    calib_data_path=calib_path,
    save_path=save_path,
    device="gpu",
    weight_dtype="bfloat16",
    use_gpu_only_for_calibration=True,
    calibration_config=calibration_config,
    bit_config=bit_config,
    search_weight_scale_config=search_weight_scale_config,
    llm_config=llm_config,
)
```

For larger models, add `splitParts` or `splitBlocks` through a `CompileConfig` or a JSON/YAML config file, then validate each produced MXQ with the runtime flow used by the application.

## LLM Calibration Data

For many LLMs, calibration uses embedded tensors rather than token IDs because embedding can be handled outside the NPU graph. The typical workflow is:

1. Prepare text representative of the deployment prompts.
2. Tokenize the text with the model tokenizer.
3. Pass the tokenized input through the model embedding layer.
4. Save the resulting tensors as the calibration dataset.

The calibration tensor shapes must match the compiled decoder body input. If the model frontend selects a specific input form, the calibration tensors must follow the same contract.

If compilation fails, reduce sequence and cache length first to confirm that the model path is valid, then scale back up within memory limits.

## Multimodal (VLM) Compilation

HuggingFace multimodal (VLM) models are compiled with dedicated Python scripts through `mblt_compile` and `mxq_compile` rather than generic CLI examples. Start from the tutorial code or reference implementation for each model, then adapt the script and configuration to the exported model contract. See the reference implementations at https://github.com/mobilint.

The [`multimodal` preset](#multimodal-preset) is the configuration starting point for these scripts.

Practical guidance:

- Compile and validate the vision encoder alone when possible.
- Compile and validate the LLM path alone when possible.
- Combine the graph only after each subpath has a clear shape and preprocessing contract.
- Move tokenizer logic, image decoding, prompt formatting, and other Python-level preprocessing outside the compiled graph.
- Prepare calibration data for every compiled input. Sample names, shapes, and ordering must match the original model inputs.
- Use CPU Offloading only for bounded graph regions that the deployment runtime supports.