# DSP Programming Guide

This page walks through building an application against the SP500 DSP API, deploying it to a REGULUS device, and using each operation. For the concepts behind the API, see [DSP Overview](dsp_overview.md). For every function, struct, and constant, see the DSP API Reference. For complete example applications, see [Examples](#dsp-examples).

## Before You Begin

The following must be in place on the device before an application calls the API:

- **DSP firmware is loaded.** The DSP has no firmware after the device boots. Run `dsp_firmware_load` once after every boot; see [Loading the DSP Firmware](#loading-the-dsp-firmware).
- **`libopencv_core.so.409` is installed.** `libsp500.so` links against it, and the loader resolves it at start-up even though no OpenCV type appears in the API.

To check what the library needs on the device:

```bash
readelf -d libsp500.so | grep NEEDED
ldd libsp500.so
```

(loading-the-dsp-firmware)=

## Loading the DSP Firmware

The Vision DSP runs its own firmware: FreeRTOS and the DSP side of the SP500 operations. Right after the device boots, no firmware is loaded on the DSP, so load it before starting an application that uses the DSP:

```bash
dsp_firmware_load
```

```text
SP500 reset assert
SP500 ptcm.bin loading complete
SP500 dtcm.bin loading complete
SP500 external.bin loading complete
SP500 reset de-assert
```

- Run it once after boot. The firmware stays loaded until the device is powered off or rebooted, so there is no need to load it again when an application restarts or another application starts.
- `dsp_firmware_load` is installed on REGULUS images that include the SP500 package. It loads the following firmware images:

  ```text
  /usr/lib/firmware/mobilint/sp500/ptcm.bin
  /usr/lib/firmware/mobilint/sp500/dtcm.bin
  /usr/lib/firmware/mobilint/sp500/external.bin
  ```

- If the firmware is not loaded, `sp500_dsp_open()` waits for it until its timeout, 10 seconds by default, and returns `SP500_ERR_TIMEOUT`.

## Building an Application

Applications are cross-compiled on the host PC with the REGULUS toolchain. Set up the toolchain environment first:

```bash
source /opt/crosstools/mobilint/2.0.0/<VERSION>/environment-setup-cortexa53-mobilint-linux
```

Then compile with the SDK headers and link `libsp500`:

```bash
$CXX my_app.cpp -I<sdk>/include -L<sdk>/lib -lsp500 -o my_app
```

Nothing else needs to be named on the link line. Include the headers by their `sp500/` path:

```c
#include <sp500/letterbox.h>
#include <sp500/log_mel.h>
#include <sp500/yolo.h>
```

## Deploying to the Device

REGULUS images that include the SP500 package already install `libsp500.so` and its links under `/usr/lib`. Copy the library yourself only to ship a different version of it with the application.

The application binary records the SONAME `libsp500.so.1`, not `libsp500.so`. Copy all three library names and keep the symbolic links:

```bash
cp -a lib/libsp500.so* /target/usr/lib/
```

Copying only `libsp500.so`, or copying with plain `cp` so that the links become file copies, causes the following error at start-up, or leaves three full copies of the library on the root file system:

```text
error while loading shared libraries: libsp500.so.1: cannot open shared object file
```

If the application and the library are shipped separately, check at start-up that their major versions match:

```c
if (sp500_version() / 10000 != SP500_VERSION_MAJOR) {
    /* the header and the library disagree */
}
```

## Running the Application

Copy the application to the device, load the DSP firmware if it has not been loaded since the last boot, and start the application:

```bash
dsp_firmware_load
./my_app
```

(dsp-examples)=

## Examples

[regulus-dsp-examples](https://github.com/mobilint/regulus-dsp-examples) on GitHub collects complete example applications for the SP500 DSP API, covering the preprocessing and postprocessing operations on this page. Each example comes with the steps to build it, copy it to the device, load the DSP firmware, and run it.

## First Application

The following program letterboxes a 1920x1080 camera frame into a 640x640 model input, and decodes the output of an anchor-free YOLO detection head into boxes in source-frame coordinates.

```c
#include <sp500/letterbox.h>
#include <sp500/yolo.h>
#include <stdio.h>

static unsigned char frame[1920 * 1080 * 3];   /* captured BGR frame */

int main(void)
{
    /* Attach once, at start-up. This waits for the DSP firmware to register. */
    sp500_status rc = sp500_dsp_open(0);
    if (rc != SP500_OK) {
        fprintf(stderr, "SP500: %s\n", sp500_status_string(rc));
        return 1;
    }

    /* 1. 1920x1080 BGR frame -> 640x640 model input */
    sp500_letterbox_params lb;
    sp500_letterbox_params_init(&lb);
    lb.src_w = 1920;
    lb.src_h = 1080;

    sp500_buffer src = sp500_buffer_wrap(frame, 1920 * 1080 * 3);
    sp500_buffer dst = sp500_buffer_alloc(640 * 640 * 3, SP500_ALIGN_NPU);

    sp500_letterbox_result geom;
    rc = sp500_letterbox(&src, &lb, &dst, &geom);

    /* ... run the network on dst ... */

    /* 2. raw head tensors -> detections */
    sp500_yolo_params yp;
    sp500_yolo_params_init(&yp);
    yp.max_detections = 300;

    sp500_yolo_detection dets[300];
    sp500_yolo_output out;
    sp500_yolo_output_init(&out);
    out.records = sp500_buffer_wrap(dets, sizeof dets);

    sp500_yolo_raw raw;
    sp500_yolo_raw_init(&raw);
    /* set raw.box[i] and raw.cls[i] for i = 0, 1, 2 (strides 8, 16, 32) */

    sp500_yolo_result res;
    rc = sp500_yolo_decode(SP500_YOLO_HEAD_ANCHORLESS, SP500_YOLO_TASK_DETECT,
                           &raw, &yp, &out, &res);
    if (rc == SP500_OK) {
        for (uint32_t i = 0; i < res.count; ++i) {
            /* boxes are in the 640x640 network frame; undo the letterbox
               to get back to the 1920x1080 source */
            float x1 = (dets[i].x1 - geom.pad_x) / geom.scale;
            float y1 = (dets[i].y1 - geom.pad_y) / geom.scale;
            (void)x1; (void)y1;
        }
    }

    sp500_buffer_release(&dst);
    return 0;
}
```

The rest of this page describes each step in more detail.

## Attaching to the DSP

`sp500_dsp_open(timeout_ms)` must succeed before any other DSP call. It blocks until the DSP firmware has registered, for at most `timeout_ms` milliseconds; pass 0 for the default of 10 seconds. Calling it again after it has succeeded is a cheap no-op, so every module of an application can call it.

A successful open also maps the shared-memory window, so `sp500_shmem_contains()` and `sp500_shmem_phys_addr()` work from an NPU setup path that runs before the first DSP call.

Treat `SP500_ERR_TIMEOUT` from `sp500_dsp_open()` as a fatal start-up error rather than retrying in a loop. A timeout leaves a worker thread waiting inside the link layer until the DSP appears.

## Letterbox

`sp500_letterbox()` scales the source frame by `r = min(dst_w / src_w, dst_h / src_h)` so that it fits the destination without distortion, and fills the remaining border with a constant value.

Frames are 8-bit interleaved HWC with no row padding: a row is exactly `width * channels` bytes. `src_w` and `src_h` must be set; the other fields default as follows.

| Field | Default | Description |
| --- | --- | --- |
| `dst_w`, `dst_h` | 640, 640 | Destination size in pixels. |
| `channels` | 3 | Interleaved channels. |
| `interp` | `SP500_RESIZE_BILINEAR` | `SP500_RESIZE_NEAREST`, `_BILINEAR`, `_BICUBIC`, or `_LANCZOS`. |
| `pad_value` | 114 | Border fill, applied to every channel. |
| `center` | 1 | 1 splits the border evenly; 0 pads bottom-right only. |
| `scale_up` | 1 | 1 allows upscaling a smaller source; 0 never scales above 1.0. |
| `swap_rb` | 0 | 1 swaps channels 0 and 2 while writing (BGR to RGB or back). Requires 3 channels. |

The result carries the applied scale and the top-left padding. Use them to map a point in the model input back to the source frame:

```c
src_x = (dst_x - result.pad_x) / result.scale;
src_y = (dst_y - result.pad_y) / result.scale;
```

Allocate the destination with `sp500_buffer_alloc(size, SP500_ALIGN_NPU)` so the NPU can read the model input directly.

## Log-Mel Spectrogram

`sp500_log_mel()` turns float32 PCM into the log-mel spectrogram that Whisper and Qwen3-ASR encoders take. The computation matches the Hugging Face `WhisperFeatureExtractor`.

```c
sp500_log_mel_params p;
sp500_log_mel_params_init(&p);
p.n_samples = 16000 * 30;          /* 30 s at 16 kHz */

const int32_t frames = sp500_log_mel_num_frames(p.n_samples, p.hop_length);

sp500_buffer audio = sp500_buffer_wrap(pcm, p.n_samples * sizeof(float));
sp500_buffer feats = sp500_buffer_alloc(frames * p.n_mels * sizeof(float),
                                        SP500_ALIGN_DSP);

sp500_log_mel_result r;
if (sp500_log_mel(&audio, &p, &feats, &r) == SP500_OK) {
    /* feats.data holds r.n_frames * p.n_mels floats, frame-major */
}
sp500_buffer_release(&feats);
```

- Only `n_samples` must be set. The defaults are the Whisper 80-mel front end: 16 kHz, `n_fft` 400, `hop_length` 160, 0 to 8000 Hz, with Whisper normalization. Qwen3-ASR and Whisper large-v3 use `n_mels = 128`.
- The output is frame-major, `features[t * n_mels + m]`, with `n_frames = n_samples / hop_length`. This is the `[n_frames, n_mels]` layout the MXQ Whisper models take; Hugging Face's own `input_features` is its transpose.
- `sp500_log_mel_cpu()` computes the same result on the host CPU without the DSP. Use it to check the DSP result during bring-up, or as a fallback.

A 30-second clip is several thousand frames of DSP work. Raise the call deadline before submitting it; see [Call Deadline](#call-deadline).

## YOLO Decode

A YOLO network returns raw numbers for every candidate location rather than boxes. The YOLO functions decode the coordinates, discard candidates below the confidence threshold, and remove duplicates in one DSP round trip.

### Choosing the Head Family

YOLO generations encode boxes and remove duplicates differently. Pick the family the network belongs to:

| Family | Box encoding | Duplicate removal | Tasks | Models |
| --- | --- | --- | --- | --- |
| `SP500_YOLO_HEAD_ANCHORLESS` | DFL, 4 x 16 logits | Per-class NMS | Detect, segment, pose | v3u, v5u, v8, v9, GELAN, 11, 12 |
| `SP500_YOLO_HEAD_ANCHOR` | Anchor offsets + objectness | Per-class NMS | Detect, segment | v3, v5, v7 |
| `SP500_YOLO_HEAD_NMSFREE` | DFL, 4 x 16 logits | Dual top-k | Detect | v10 |
| `SP500_YOLO_HEAD_DFLFREE` | 4 direct distances | Dual top-k | Detect, segment, pose | 26 |

YOLOv3u and YOLOv5u are retrained with an anchor-free head and belong to the anchor-less family, not to the anchor family of plain YOLOv3 and YOLOv5. `sp500_yolo_head_for_model()` maps a model name such as `"yolov8n"` or `"yolo11"` to its family.

### Choosing the Entry Point

| Function | Input |
| --- | --- |
| `sp500_yolo_decode()` | Raw head tensors as separate box, class, and task planes for the three levels (strides 8, 16, 32). For the anchor-less, NMS-free, and DFL-free families. |
| `sp500_yolo_decode_anchor()` | One combined tensor per level. For the anchor family. |
| `sp500_yolo_postprocess()` | A table that the network has already decoded inside its graph. For any family. |

### Parameters and Results

`sp500_yolo_params_init()` sets defaults for a 640x640, 80-class detection model: `max_detections` 100, `conf_threshold` 0.25, and `iou_threshold` 0.7. A plain detector needs no field set. Segmentation additionally requires `proto`, `mask_coeffs`, `proto_h`, `proto_w`, and `mask_threshold`.

`output.records` receives up to `max_detections` fixed-size records. Size it with `sp500_yolo_records_bytes()`, and read it as `sp500_yolo_detection`, `sp500_yolo_segmentation`, or `sp500_yolo_pose` according to the task. For segmentation, `output.mask` receives the per-box masks; read each record's mask offset with `sp500_yolo_mask_offset()`.

Box coordinates are pixels in the network input frame. If the source was letterboxed, map them back with the scale and padding from `sp500_letterbox()`.

## Best Practices

### Initialize Every Parameter Struct

Call `sp500_letterbox_params_init()`, `sp500_log_mel_params_init()`, or `sp500_yolo_params_init()` before setting fields, then override only what you need. A struct that was zeroed and given only its required fields runs with every other option at zero: a letterbox would pad with black instead of 114 and would not center the image.

### Prefer Zero-Copy Buffers

A buffer from `sp500_buffer_wrap()` is copied in and out around every call; for a 640x640x3 frame that is 1.2 MB of copying per call. A buffer from `sp500_buffer_alloc()` lives in the shared-memory window and is not copied. `sp500_shmem_phys_addr()` returns its physical address so the NPU can read and write the same memory.

### Allocate the Working Set Once

The shared-memory window uses a first-fit allocator without compaction. Freed memory is reclaimed and merged with its neighbors, but allocating and freeing buffers of mixed lifetimes every frame fragments the window: it then holds roughly a quarter more than the live data, and a large request can fail while smaller gaps go unused. Allocate buffers once at start-up and reuse them.

Do not keep a shared-memory buffer in a global or static object. It would be released after the DSP link has already been torn down.

(call-deadline)=

### Call Deadline

Every DSP call is bounded by a deadline, 3 seconds by default. The deadline guards against a DSP that has stopped responding; it is not a latency budget, and it must be longer than the longest operation submitted. A long log-mel clip is the case to check.

An expired deadline does not cancel the work. The DSP keeps working on buffers the call has already released, and the next call can return the previous call's result; recovering from that requires reloading the DSP. For that reason `sp500_set_dsp_timeout_ms()` can only raise the deadline: a value below the 3-second default is clamped up to it, and `sp500_get_dsp_timeout_ms()` reports the value in effect.

Raise the deadline before a long call. If a call returns `SP500_ERR_TIMEOUT`, raise the deadline rather than retrying.

### Calls Are Serialized System-Wide

Every DSP call holds one lock shared by all threads and processes for the length of its round trip, so concurrent calls are safe but not parallel. To make a sequence such as allocate, letterbox, read, and free atomic as a whole, wrap it in `sp500_dsp_lock()` and `sp500_dsp_unlock()`. Every lock must be paired with exactly one unlock; an unbalanced lock stalls the DSP and the shared-memory allocator for every process on the system.