DSP Programming Guide

DSP Programming Guide#

This page walks through building an application against the SP500 DSP API, deploying it to a REGULUS device, and using each operation. For the concepts behind the API, see DSP Overview. For every function, struct, and constant, see the DSP API Reference. For complete example applications, see Examples.

Before You Begin#

The following must be in place on the device before an application calls the API:

DSP firmware is loaded. The DSP has no firmware after the device boots. Run dsp_firmware_load once after every boot; see Loading the DSP Firmware.
libopencv_core.so.409 is installed. libsp500.so links against it, and the loader resolves it at start-up even though no OpenCV type appears in the API.

To check what the library needs on the device:

readelf -d libsp500.so | grep NEEDED
ldd libsp500.so

Loading the DSP Firmware#

The Vision DSP runs its own firmware: FreeRTOS and the DSP side of the SP500 operations. Right after the device boots, no firmware is loaded on the DSP, so load it before starting an application that uses the DSP:

dsp_firmware_load

SP500 reset assert
SP500 ptcm.bin loading complete
SP500 dtcm.bin loading complete
SP500 external.bin loading complete
SP500 reset de-assert

Run it once after boot. The firmware stays loaded until the device is powered off or rebooted, so there is no need to load it again when an application restarts or another application starts.

dsp_firmware_load is installed on REGULUS images that include the SP500 package. It loads the following firmware images:

/usr/lib/firmware/mobilint/sp500/ptcm.bin
/usr/lib/firmware/mobilint/sp500/dtcm.bin
/usr/lib/firmware/mobilint/sp500/external.bin

If the firmware is not loaded, sp500_dsp_open() waits for it until its timeout, 10 seconds by default, and returns SP500_ERR_TIMEOUT.

Building an Application#

Applications are cross-compiled on the host PC with the REGULUS toolchain. Set up the toolchain environment first:

source /opt/crosstools/mobilint/2.0.0/<VERSION>/environment-setup-cortexa53-mobilint-linux

Then compile with the SDK headers and link libsp500:

$CXX my_app.cpp -I<sdk>/include -L<sdk>/lib -lsp500 -o my_app

Nothing else needs to be named on the link line. Include the headers by their sp500/ path:

#include <sp500/letterbox.h>
#include <sp500/log_mel.h>
#include <sp500/yolo.h>

Deploying to the Device#

REGULUS images that include the SP500 package already install libsp500.so and its links under /usr/lib. Copy the library yourself only to ship a different version of it with the application.

The application binary records the SONAME libsp500.so.1, not libsp500.so. Copy all three library names and keep the symbolic links:

cp -a lib/libsp500.so* /target/usr/lib/

Copying only libsp500.so, or copying with plain cp so that the links become file copies, causes the following error at start-up, or leaves three full copies of the library on the root file system:

error while loading shared libraries: libsp500.so.1: cannot open shared object file

If the application and the library are shipped separately, check at start-up that their major versions match:

if (sp500_version() / 10000 != SP500_VERSION_MAJOR) {
    /* the header and the library disagree */
}

Running the Application#

Copy the application to the device, load the DSP firmware if it has not been loaded since the last boot, and start the application:

dsp_firmware_load
./my_app

Examples#

regulus-dsp-examples on GitHub collects complete example applications for the SP500 DSP API, covering the preprocessing and postprocessing operations on this page. Each example comes with the steps to build it, copy it to the device, load the DSP firmware, and run it.

First Application#

The following program letterboxes a 1920x1080 camera frame into a 640x640 model input, and decodes the output of an anchor-free YOLO detection head into boxes in source-frame coordinates.

#include <sp500/letterbox.h>
#include <sp500/yolo.h>
#include <stdio.h>

static unsigned char frame[1920 * 1080 * 3];   /* captured BGR frame */

int main(void)
{
    /* Attach once, at start-up. This waits for the DSP firmware to register. */
    sp500_status rc = sp500_dsp_open(0);
    if (rc != SP500_OK) {
        fprintf(stderr, "SP500: %s\n", sp500_status_string(rc));
        return 1;
    }

    /* 1. 1920x1080 BGR frame -> 640x640 model input */
    sp500_letterbox_params lb;
    sp500_letterbox_params_init(&lb);
    lb.src_w = 1920;
    lb.src_h = 1080;

    sp500_buffer src = sp500_buffer_wrap(frame, 1920 * 1080 * 3);
    sp500_buffer dst = sp500_buffer_alloc(640 * 640 * 3, SP500_ALIGN_NPU);

    sp500_letterbox_result geom;
    rc = sp500_letterbox(&src, &lb, &dst, &geom);

    /* ... run the network on dst ... */

    /* 2. raw head tensors -> detections */
    sp500_yolo_params yp;
    sp500_yolo_params_init(&yp);
    yp.max_detections = 300;

    sp500_yolo_detection dets[300];
    sp500_yolo_output out;
    sp500_yolo_output_init(&out);
    out.records = sp500_buffer_wrap(dets, sizeof dets);

    sp500_yolo_raw raw;
    sp500_yolo_raw_init(&raw);
    /* set raw.box[i] and raw.cls[i] for i = 0, 1, 2 (strides 8, 16, 32) */

    sp500_yolo_result res;
    rc = sp500_yolo_decode(SP500_YOLO_HEAD_ANCHORLESS, SP500_YOLO_TASK_DETECT,
                           &raw, &yp, &out, &res);
    if (rc == SP500_OK) {
        for (uint32_t i = 0; i < res.count; ++i) {
            /* boxes are in the 640x640 network frame; undo the letterbox
               to get back to the 1920x1080 source */
            float x1 = (dets[i].x1 - geom.pad_x) / geom.scale;
            float y1 = (dets[i].y1 - geom.pad_y) / geom.scale;
            (void)x1; (void)y1;
        }
    }

    sp500_buffer_release(&dst);
    return 0;
}

The rest of this page describes each step in more detail.

Attaching to the DSP#

sp500_dsp_open(timeout_ms) must succeed before any other DSP call. It blocks until the DSP firmware has registered, for at most timeout_ms milliseconds; pass 0 for the default of 10 seconds. Calling it again after it has succeeded is a cheap no-op, so every module of an application can call it.

A successful open also maps the shared-memory window, so sp500_shmem_contains() and sp500_shmem_phys_addr() work from an NPU setup path that runs before the first DSP call.

Treat SP500_ERR_TIMEOUT from sp500_dsp_open() as a fatal start-up error rather than retrying in a loop. A timeout leaves a worker thread waiting inside the link layer until the DSP appears.

Letterbox#

sp500_letterbox() scales the source frame by r = min(dst_w / src_w, dst_h / src_h) so that it fits the destination without distortion, and fills the remaining border with a constant value.

Frames are 8-bit interleaved HWC with no row padding: a row is exactly width * channels bytes. src_w and src_h must be set; the other fields default as follows.

Field	Default	Description
`dst_w`, `dst_h`	640, 640	Destination size in pixels.
`channels`	3	Interleaved channels.
`interp`	`SP500_RESIZE_BILINEAR`	`SP500_RESIZE_NEAREST`, `_BILINEAR`, `_BICUBIC`, or `_LANCZOS`.
`pad_value`	114	Border fill, applied to every channel.
`center`	1	1 splits the border evenly; 0 pads bottom-right only.
`scale_up`	1	1 allows upscaling a smaller source; 0 never scales above 1.0.
`swap_rb`	0	1 swaps channels 0 and 2 while writing (BGR to RGB or back). Requires 3 channels.

The result carries the applied scale and the top-left padding. Use them to map a point in the model input back to the source frame:

src_x = (dst_x - result.pad_x) / result.scale;
src_y = (dst_y - result.pad_y) / result.scale;

Allocate the destination with sp500_buffer_alloc(size, SP500_ALIGN_NPU) so the NPU can read the model input directly.

Log-Mel Spectrogram#

sp500_log_mel() turns float32 PCM into the log-mel spectrogram that Whisper and Qwen3-ASR encoders take. The computation matches the Hugging Face WhisperFeatureExtractor.

sp500_log_mel_params p;
sp500_log_mel_params_init(&p);
p.n_samples = 16000 * 30;          /* 30 s at 16 kHz */

const int32_t frames = sp500_log_mel_num_frames(p.n_samples, p.hop_length);

sp500_buffer audio = sp500_buffer_wrap(pcm, p.n_samples * sizeof(float));
sp500_buffer feats = sp500_buffer_alloc(frames * p.n_mels * sizeof(float),
                                        SP500_ALIGN_DSP);

sp500_log_mel_result r;
if (sp500_log_mel(&audio, &p, &feats, &r) == SP500_OK) {
    /* feats.data holds r.n_frames * p.n_mels floats, frame-major */
}
sp500_buffer_release(&feats);

Only n_samples must be set. The defaults are the Whisper 80-mel front end: 16 kHz, n_fft 400, hop_length 160, 0 to 8000 Hz, with Whisper normalization. Qwen3-ASR and Whisper large-v3 use n_mels = 128.
The output is frame-major, features[t * n_mels + m], with n_frames = n_samples / hop_length. This is the [n_frames, n_mels] layout the MXQ Whisper models take; Hugging Face’s own input_features is its transpose.
sp500_log_mel_cpu() computes the same result on the host CPU without the DSP. Use it to check the DSP result during bring-up, or as a fallback.

A 30-second clip is several thousand frames of DSP work. Raise the call deadline before submitting it; see Call Deadline.

YOLO Decode#

A YOLO network returns raw numbers for every candidate location rather than boxes. The YOLO functions decode the coordinates, discard candidates below the confidence threshold, and remove duplicates in one DSP round trip.

Choosing the Head Family#

YOLO generations encode boxes and remove duplicates differently. Pick the family the network belongs to:

Family	Box encoding	Duplicate removal	Tasks	Models
`SP500_YOLO_HEAD_ANCHORLESS`	DFL, 4 x 16 logits	Per-class NMS	Detect, segment, pose	v3u, v5u, v8, v9, GELAN, 11, 12
`SP500_YOLO_HEAD_ANCHOR`	Anchor offsets + objectness	Per-class NMS	Detect, segment	v3, v5, v7
`SP500_YOLO_HEAD_NMSFREE`	DFL, 4 x 16 logits	Dual top-k	Detect	v10
`SP500_YOLO_HEAD_DFLFREE`	4 direct distances	Dual top-k	Detect, segment, pose	26

YOLOv3u and YOLOv5u are retrained with an anchor-free head and belong to the anchor-less family, not to the anchor family of plain YOLOv3 and YOLOv5. sp500_yolo_head_for_model() maps a model name such as "yolov8n" or "yolo11" to its family.

Choosing the Entry Point#

Function	Input
`sp500_yolo_decode()`	Raw head tensors as separate box, class, and task planes for the three levels (strides 8, 16, 32). For the anchor-less, NMS-free, and DFL-free families.
`sp500_yolo_decode_anchor()`	One combined tensor per level. For the anchor family.
`sp500_yolo_postprocess()`	A table that the network has already decoded inside its graph. For any family.

Parameters and Results#

sp500_yolo_params_init() sets defaults for a 640x640, 80-class detection model: max_detections 100, conf_threshold 0.25, and iou_threshold 0.7. A plain detector needs no field set. Segmentation additionally requires proto, mask_coeffs, proto_h, proto_w, and mask_threshold.

output.records receives up to max_detections fixed-size records. Size it with sp500_yolo_records_bytes(), and read it as sp500_yolo_detection, sp500_yolo_segmentation, or sp500_yolo_pose according to the task. For segmentation, output.mask receives the per-box masks; read each record’s mask offset with sp500_yolo_mask_offset().

Box coordinates are pixels in the network input frame. If the source was letterboxed, map them back with the scale and padding from sp500_letterbox().

Best Practices#

Initialize Every Parameter Struct#

Call sp500_letterbox_params_init(), sp500_log_mel_params_init(), or sp500_yolo_params_init() before setting fields, then override only what you need. A struct that was zeroed and given only its required fields runs with every other option at zero: a letterbox would pad with black instead of 114 and would not center the image.

Prefer Zero-Copy Buffers#

A buffer from sp500_buffer_wrap() is copied in and out around every call; for a 640x640x3 frame that is 1.2 MB of copying per call. A buffer from sp500_buffer_alloc() lives in the shared-memory window and is not copied. sp500_shmem_phys_addr() returns its physical address so the NPU can read and write the same memory.

Allocate the Working Set Once#

The shared-memory window uses a first-fit allocator without compaction. Freed memory is reclaimed and merged with its neighbors, but allocating and freeing buffers of mixed lifetimes every frame fragments the window: it then holds roughly a quarter more than the live data, and a large request can fail while smaller gaps go unused. Allocate buffers once at start-up and reuse them.

Do not keep a shared-memory buffer in a global or static object. It would be released after the DSP link has already been torn down.

Call Deadline#

Every DSP call is bounded by a deadline, 3 seconds by default. The deadline guards against a DSP that has stopped responding; it is not a latency budget, and it must be longer than the longest operation submitted. A long log-mel clip is the case to check.

An expired deadline does not cancel the work. The DSP keeps working on buffers the call has already released, and the next call can return the previous call’s result; recovering from that requires reloading the DSP. For that reason sp500_set_dsp_timeout_ms() can only raise the deadline: a value below the 3-second default is clamped up to it, and sp500_get_dsp_timeout_ms() reports the value in effect.

Raise the deadline before a long call. If a call returns SP500_ERR_TIMEOUT, raise the deadline rather than retrying.

Calls Are Serialized System-Wide#

Every DSP call holds one lock shared by all threads and processes for the length of its round trip, so concurrent calls are safe but not parallel. To make a sequence such as allocate, letterbox, read, and free atomic as a whole, wrap it in sp500_dsp_lock() and sp500_dsp_unlock(). Every lock must be paired with exactly one unlock; an unbalanced lock stalls the DSP and the shared-memory allocator for every process on the system.