BadBob / Playground
Browser · ONNX · WASM · MIT

05 / JavaScript / TypeScript

JavaScript / TypeScript. From model to camera.

Run a supported ONNX detector locally in your browser. Choose your own files and keep each image paired with its predictions.

These examples focus on objects and animals in controlled settings. They do not cover people surveillance, identification or behavioral analysis.

05.01

Prepare your project

Install the listed dependencies and run the tests. Try an offline image before opening the camera.

Basics

npm ci · npm run build
Configures dependencies and compiles the project.
npm test · npm run check
Runs automated checks without opening the camera.
Terminalbash
unzip badbob-playground-javascript.zip
cd badbob-playground-javascript
npm ci
npm test
npm run check
npm run build
05.02

Connect your model

Validate the model and labels before opening a camera.

Basics

parseLabels(text)
Reads class names in model order; rejects empty or duplicate labels.
OnnxDetector.create(model, labelsText)
Loads the ONNX model and validates its input, output and label count.
OnnxDetector.detect(frame)
Prepares the image, runs the model and returns detections.
Keep in mind
Symptom
Boxes drift or the model fails to load.
Cause
Preprocessing, output layout or label order does not match.
Remedy
This adapter supports only the contract below. Core ML files and other ONNX layouts need a different adapter.

This adapter supports only the contract below. Core ML files and other ONNX layouts need a different adapter.

In depth

Coordinates describe box centers and dimensions in the 640×640 input pixels. Labels follow the exact model class order.

ONNXyaml
input:
  color: RGB
  dtype: float32
  shape: [1, 3, 640, 640]
  letterbox: 114
  scale: 1/255
output:
  dtype: float32
  shape: [1, 4+C, N]
  boxes: [cx, cy, width, height]
  class_scores: probabilities
  objectness: false
  embedded_nms: false
onnx-detector.tstypescript
import * as ort from "onnxruntime-web/wasm";
import {
    decode,
    INPUT_EDGE,
    letterbox,
    parseLabels,
    rgbaToNchw,
    validateOutputShape,
} from "./core.ts";
import type { Detection, Detector, Letterbox } from "./types.ts";

export function validateModelMetadata(
    inputs: readonly ort.InferenceSession.ValueMetadata[],
    outputs: readonly ort.InferenceSession.ValueMetadata[],
    classes: number,
): void {
    const input = inputs[0];
    const output = outputs[0];

    if (
        inputs.length !== 1 ||
        !input?.isTensor ||
        input.type !== "float32" ||
        input.shape.length !== 4 ||
        input.shape.some((dimension, index) => dimension !== [1, 3, 640, 640][index])
    ) {
        throw new Error("The model must have one static float32 RGB input [1,3,640,640].");
    }

    if (
        outputs.length !== 1 ||
        !output?.isTensor ||
        output.type !== "float32" ||
        output.shape.length !== 3 ||
        output.shape[0] !== 1 ||
        output.shape[1] !== 4 + classes
    ) {
        throw new Error(
            "The model must have one float32 output [1,4+C,N], with C matching labels.",
        );
    }

    const count = output.shape[2];

    if (typeof count === "number") {
        validateOutputShape([1, 4 + classes, count], classes);
    }
}

export class OnnxDetector implements Detector {
    private disposed = false;

    private constructor(
        private readonly session: ort.InferenceSession,
        private readonly classes: number,
    ) {}

    static async create(model: ArrayBuffer, labelsText: string): Promise<OnnxDetector> {
        const labels = parseLabels(labelsText);
        ort.env.wasm.numThreads = 1;
        ort.env.wasm.proxy = false;
        const session = await ort.InferenceSession.create(new Uint8Array(model), {
            executionProviders: ["wasm"],
        });
        const detector = new OnnxDetector(session, labels.length);

        try {
            validateModelMetadata(session.inputMetadata, session.outputMetadata, labels.length);
            const neutral = new Float32Array(3 * INPUT_EDGE * INPUT_EDGE).fill(114 / 255);
            await detector.predict(neutral, letterbox({ width: 640, height: 640 }));
            return detector;
        } catch (error: unknown) {
            await detector.dispose();
            throw error;
        }
    }

    async detect(frame: ImageBitmap): Promise<Detection[]> {
        const transform = letterbox(frame);
        const canvas = new OffscreenCanvas(INPUT_EDGE, INPUT_EDGE);
        const context = canvas.getContext("2d", { willReadFrequently: true });

        if (!context) {
            throw new Error("OffscreenCanvas 2D is unavailable in this browser's worker.");
        }

        context.fillStyle = "rgb(114, 114, 114)";
        context.fillRect(0, 0, INPUT_EDGE, INPUT_EDGE);
        context.drawImage(
            frame,
            transform.left,
            transform.top,
            transform.resized.width,
            transform.resized.height,
        );
        const pixels = context.getImageData(0, 0, INPUT_EDGE, INPUT_EDGE);
        return this.predict(rgbaToNchw(pixels.data), transform);
    }

    async predict(input: Float32Array, transform: Letterbox): Promise<Detection[]> {
        if (this.disposed || input.length !== 3 * INPUT_EDGE * INPUT_EDGE) {
            throw new Error("Session is disposed or input is not [1,3,640,640].");
        }

        if (input.some((value) => !Number.isFinite(value) || value < 0 || value > 1)) {
            throw new Error("Input pixels must be finite normalized RGB values in [0,1].");
        }

        const inputName = this.session.inputNames[0];
        const outputName = this.session.outputNames[0];

        if (!inputName || !outputName) {
            throw new Error("Model tensor names are missing.");
        }

        const tensor = new ort.Tensor("float32", input, [1, 3, INPUT_EDGE, INPUT_EDGE]);
        let outputs: ort.InferenceSession.ReturnType | undefined;

        try {
            outputs = await this.session.run({ [inputName]: tensor });
            const output = outputs[outputName];

            if (
                !(output instanceof ort.Tensor) ||
                output.type !== "float32" ||
                !(output.data instanceof Float32Array)
            ) {
                throw new Error("Expected one CPU float32 output tensor.");
            }

            return decode(output.data, output.dims, this.classes, transform);
        } finally {
            tensor.dispose();

            for (const output of Object.values(outputs ?? {})) {
                if (output instanceof ort.Tensor) {
                    output.dispose();
                }
            }
        }
    }

    async dispose(): Promise<void> {
        if (this.disposed) {
            return;
        }

        this.disposed = true;
        await this.session.release();
    }
}
05.03

Bring in the camera

The browser captures frames while a worker runs inference. Only one frame is processed at a time, so a backlog cannot grow.

Basics

CameraSource.open(signal)
Requests video permission and opens the camera after an explicit start.
CameraSource.next()
Captures the next frame, or reports that the camera stream has stopped.
WorkerDetector.infer(frame)
Prepares the image, runs the model and returns detections.
WorkerDetector.dispose()
Stops accepting new results and releases resources after active work finishes.
Keep in mind
Symptom
The preview lags and memory use grows.
Cause
The camera produces frames faster than inference handles them, so an unbounded queue builds up.
Remedy
The browser captures frames while a worker runs inference. Only one frame is processed at a time, so a backlog cannot grow.
In depth

Request stop, reject late results and wait for active work before releasing its resources. Backend cancellation depends on platform support.

frame-source.tstypescript
import { validateSize } from "./core.ts";

export interface FrameSource {
    next(): Promise<ImageBitmap | null>;
    stop(): void;
}

export class ImageSource implements FrameSource {
    private constructor(private image: ImageBitmap | undefined) {}

    static async open(file: File): Promise<ImageSource> {
        const image = await createImageBitmap(file);

        try {
            validateSize(image);
            return new ImageSource(image);
        } catch (error: unknown) {
            image.close();
            throw error;
        }
    }

    async next(): Promise<ImageBitmap | null> {
        const image = this.image;
        this.image = undefined;
        return image ?? null;
    }

    stop(): void {
        this.image?.close();
        this.image = undefined;
    }
}

export class CameraSource implements FrameSource {
    private stopped = false;

    private constructor(
        private readonly stream: MediaStream,
        private readonly video: HTMLVideoElement,
    ) {}

    static async open(signal: AbortSignal): Promise<CameraSource> {
        if (!navigator.mediaDevices?.getUserMedia) {
            throw new Error(
                "Camera API unavailable. Use HTTPS or localhost in a supported browser.",
            );
        }

        const stream = await navigator.mediaDevices.getUserMedia({
            video: { width: { ideal: 1280 }, height: { ideal: 720 } },
            audio: false,
        });
        const video = document.createElement("video");
        const source = new CameraSource(stream, video);

        if (signal.aborted) {
            source.stop();
            throw new Error("Camera startup stopped.");
        }

        const abort = (): void => source.stop();
        signal.addEventListener("abort", abort, { once: true });
        video.muted = true;
        video.playsInline = true;
        video.srcObject = stream;

        try {
            await video.play();

            if (signal.aborted || source.stopped) {
                throw new Error("Camera startup stopped.");
            }

            validateSize({ width: video.videoWidth, height: video.videoHeight });
            return source;
        } catch (error: unknown) {
            source.stop();
            throw error;
        } finally {
            signal.removeEventListener("abort", abort);
        }
    }

    async next(): Promise<ImageBitmap | null> {
        if (this.stopped) {
            return null;
        }

        if (!this.stream.getVideoTracks().some((track) => track.readyState === "live")) {
            throw new Error("Camera stream ended. Stop, check the device and start again.");
        }

        return createImageBitmap(this.video);
    }

    stop(): void {
        if (this.stopped) {
            return;
        }

        this.stopped = true;

        for (const track of this.stream.getTracks()) {
            track.stop();
        }

        this.video.pause();
        this.video.srcObject = null;
        this.video.removeAttribute("src");
        this.video.load();
    }
}
worker-client.tstypescript
import { isWorkerResponse } from "./protocol.ts";
import type { WorkerRequest, WorkerResponse } from "./protocol.ts";
import type { Detection } from "./types.ts";

export interface PairedResult {
    frame: ImageBitmap;
    detections: Detection[];
}

interface PendingFrame {
    id: number;
    resolve: (result: PairedResult) => void;
    reject: (error: Error) => void;
}

let nextGeneration = 1;

export class WorkerDetector {
    readonly ready: Promise<void>;
    private readonly worker = new Worker(new URL("./worker.ts", import.meta.url), {
        type: "module",
    });
    private readonly generation = nextGeneration++;
    private nextId = 1;
    private pending: PendingFrame | undefined;
    private stopping = false;
    private resolveReady!: () => void;
    private rejectReady!: (error: Error) => void;
    private readonly closed: Promise<void>;
    private resolveClosed!: () => void;
    private shutdownTimer: ReturnType<typeof setTimeout> | undefined;

    constructor(model: ArrayBuffer, labels: string) {
        this.ready = new Promise((resolve, reject) => {
            this.resolveReady = resolve;
            this.rejectReady = reject;
        });
        this.closed = new Promise((resolve) => {
            this.resolveClosed = resolve;
        });
        this.worker.addEventListener("message", (event: MessageEvent<unknown>) => {
            if (!isWorkerResponse(event.data)) {
                this.fail(new Error("Worker sent an invalid response."));
                return;
            }

            this.receive(event.data);
        });
        this.worker.addEventListener("error", (event: ErrorEvent) => {
            event.preventDefault();
            this.fail(new Error(event.message || "Inference worker failed."));
        });
        this.worker.addEventListener("messageerror", () =>
            this.fail(new Error("Worker message could not be decoded.")),
        );
        try {
            this.send({ kind: "initialize", generation: this.generation, model, labels }, [model]);
        } catch (error: unknown) {
            this.fail(error instanceof Error ? error : new Error(String(error)));
        }
    }

    infer(frame: ImageBitmap): Promise<PairedResult> {
        if (this.stopping || this.pending) {
            frame.close();
            return Promise.reject(new Error("Worker is stopping or already processing a frame."));
        }

        return new Promise((resolve, reject) => {
            const id = this.nextId++;
            this.pending = { id, resolve, reject };

            try {
                this.send({ kind: "infer", generation: this.generation, id, frame }, [frame]);
            } catch (error: unknown) {
                frame.close();
                this.pending = undefined;
                reject(error);
            }
        });
    }

    async dispose(): Promise<void> {
        if (this.stopping) {
            return this.closed;
        }

        this.stopping = true;
        const error = new Error("Inference stopped.");
        this.rejectReady(error);
        this.pending?.reject(error);
        this.pending = undefined;
        this.shutdownTimer = setTimeout(() => this.finishShutdown(), 5000);
        this.send({ kind: "stop", generation: this.generation });
        await this.closed;
    }

    private receive(message: WorkerResponse): void {
        if (message.generation !== this.generation) {
            if (message.kind === "result") {
                message.frame.close();
            }

            return;
        }

        if (message.kind === "disposed") {
            this.finishShutdown();
            return;
        }

        if (this.stopping) {
            if (message.kind === "result") {
                message.frame.close();
            }

            return;
        }

        if (message.kind === "ready") {
            this.resolveReady();
            return;
        }

        if (message.kind === "error") {
            const error = new Error(message.message);

            if (message.id === null) {
                this.rejectReady(error);
                return;
            }

            if (message.id === this.pending?.id) {
                this.pending.reject(error);
                this.pending = undefined;
            }

            return;
        }

        if (message.id !== this.pending?.id) {
            message.frame.close();
            return;
        }

        this.pending.resolve({ frame: message.frame, detections: message.detections });
        this.pending = undefined;
    }

    private send(message: WorkerRequest, transfer: Transferable[] = []): void {
        this.worker.postMessage(message, transfer);
    }

    private fail(error: Error): void {
        this.stopping = true;
        this.rejectReady(error);
        this.pending?.reject(error);
        this.pending = undefined;
        this.finishShutdown();
    }

    private finishShutdown(): void {
        clearTimeout(this.shutdownTimer);
        this.worker.terminate();
        this.resolveClosed();
    }
}
05.04

Put predictions in place

Undo letterbox scaling and padding, then clip boxes before drawing.

Basics

letterbox(size) · unproject(...)
Preserves the aspect ratio and maps boxes back to the original image.
decode(data, dims, classes, transform)
Validates predictions and removes overlapping boxes of the same class.
Keep in mind

Treat image size, crop, orientation and coordinates as one contract. Test it with known boxes before trusting an overlay.

core.tstypescript
import type { Box, Detection, Letterbox, Size } from "./types.ts";

export const INPUT_EDGE = 640;
const MAX_CANDIDATES = 100_000;

export function parseLabels(text: string): string[] {
    const normalized = text
        .replace(/^\uFEFF/, "")
        .replace(/\r\n/g, "\n")
        .replace(/\n$/, "");
    const labels = normalized.split("\n").map((line) => line.trim());

    if (labels.length > 1024 || labels.some((label) => label.length === 0 || label.length > 256)) {
        throw new Error(
            "Supply 1–1024 nonempty labels, one per line (maximum 256 characters each).",
        );
    }

    if (new Set(labels).size !== labels.length) {
        throw new Error("Labels must be distinct and in the model's exact class order.");
    }

    return labels;
}

export function validateSize(size: Size): void {
    if (
        !Number.isSafeInteger(size.width) ||
        !Number.isSafeInteger(size.height) ||
        size.width <= 0 ||
        size.height <= 0 ||
        size.width > 16384 ||
        size.height > 16384 ||
        size.width * size.height > 64_000_000
    ) {
        throw new Error(
            "Use an image with positive integer dimensions, at most 16384 per side and 64 MP.",
        );
    }
}

export function letterbox(size: Size): Letterbox {
    validateSize(size);
    const ratio = Math.min(INPUT_EDGE / size.width, INPUT_EDGE / size.height);
    const resized = {
        width: Math.max(1, Math.round(size.width * ratio)),
        height: Math.max(1, Math.round(size.height * ratio)),
    };

    return {
        original: { width: size.width, height: size.height },
        resized,
        left: Math.floor((INPUT_EDGE - resized.width) / 2),
        top: Math.floor((INPUT_EDGE - resized.height) / 2),
        edge: INPUT_EDGE,
    };
}

function validateTransform(transform: Letterbox): void {
    const expected = letterbox(transform.original);

    if (
        transform.edge !== expected.edge ||
        transform.left !== expected.left ||
        transform.top !== expected.top ||
        transform.resized.width !== expected.resized.width ||
        transform.resized.height !== expected.resized.height
    ) {
        throw new Error("Invalid letterbox geometry.");
    }
}

export function rgbaToNchw(rgba: Uint8ClampedArray): Float32Array {
    const pixels = INPUT_EDGE * INPUT_EDGE;

    if (rgba.length !== pixels * 4) {
        throw new Error("Expected a letterboxed 640×640 RGBA image.");
    }

    const result = new Float32Array(pixels * 3);

    for (let pixel = 0; pixel < pixels; pixel += 1) {
        result[pixel] = rgba[pixel * 4]! / 255;
        result[pixels + pixel] = rgba[pixel * 4 + 1]! / 255;
        result[pixels * 2 + pixel] = rgba[pixel * 4 + 2]! / 255;
    }

    return result;
}

function unproject(
    centerX: number,
    centerY: number,
    width: number,
    height: number,
    transform: Letterbox,
): Box {
    const { original, resized, left, top } = transform;
    const projectX = (x: number): number =>
        Math.max(0, Math.min(1, (x - left) / resized.width)) * original.width;
    const projectY = (y: number): number =>
        Math.max(0, Math.min(1, (y - top) / resized.height)) * original.height;
    const x = projectX(centerX - width / 2);
    const y = projectY(centerY - height / 2);

    return {
        x,
        y,
        width: projectX(centerX + width / 2) - x,
        height: projectY(centerY + height / 2) - y,
    };
}

function intersectionOverUnion(a: Box, b: Box): number {
    const width = Math.max(0, Math.min(a.x + a.width, b.x + b.width) - Math.max(a.x, b.x));
    const height = Math.max(0, Math.min(a.y + a.height, b.y + b.height) - Math.max(a.y, b.y));
    const intersection = width * height;
    const union = a.width * a.height + b.width * b.height - intersection;

    return union > 0 ? intersection / union : 0;
}

export function validateOutputShape(dims: readonly number[], classes: number): number {
    const count = dims[2];

    if (
        !Number.isSafeInteger(classes) ||
        classes < 1 ||
        classes > 1024 ||
        dims.length !== 3 ||
        dims[0] !== 1 ||
        dims[1] !== 4 + classes ||
        count === undefined ||
        !Number.isSafeInteger(count) ||
        count < 1 ||
        count > MAX_CANDIDATES
    ) {
        throw new Error(
            "Expected float32 [1,4+C,N], C matching labels and 1≤N≤100000. No objectness/NMS.",
        );
    }

    return count;
}

export function decode(
    data: Float32Array,
    dims: readonly number[],
    classes: number,
    transform: Letterbox,
): Detection[] {
    validateTransform(transform);
    const count = validateOutputShape(dims, classes);

    if (data.length !== (4 + classes) * count) {
        throw new Error("Output tensor data does not match its dimensions.");
    }

    const candidates: Detection[] = [];

    for (let index = 0; index < count; index += 1) {
        const centerX = data[index]!;
        const centerY = data[count + index]!;
        const width = data[count * 2 + index]!;
        const height = data[count * 3 + index]!;

        if (![centerX, centerY, width, height].every(Number.isFinite) || width < 0 || height < 0) {
            throw new Error(
                "Detector coordinates must be finite, with nonnegative width and height.",
            );
        }

        const { label, score } = bestClass(data, count, classes, index);

        if (score < 0.25) {
            continue;
        }

        const box = unproject(centerX, centerY, width, height, transform);

        if (box.width > 0 && box.height > 0) {
            candidates.push({ box, label, score });
        }
    }

    return suppressOverlaps(candidates);
}

function bestClass(
    data: Float32Array,
    count: number,
    classes: number,
    index: number,
): { label: number; score: number } {
    let label = 0;
    let score = 0;

    for (let candidate = 0; candidate < classes; candidate += 1) {
        const probability = data[(4 + candidate) * count + index]!;

        if (!Number.isFinite(probability) || probability < 0 || probability > 1) {
            throw new Error("Expected class probabilities in [0,1], not logits or objectness.");
        }

        if (probability > score) {
            score = probability;
            label = candidate;
        }
    }

    return { label, score };
}

function suppressOverlaps(candidates: Detection[]): Detection[] {
    candidates.sort((a, b) => b.score - a.score);
    const selected: Detection[] = [];

    for (const candidate of candidates.slice(0, 3000)) {
        const suppressed = selected.some(
            (previous) =>
                previous.label === candidate.label &&
                intersectionOverUnion(previous.box, candidate.box) > 0.45,
        );

        if (!suppressed) {
            selected.push(candidate);
        }

        if (selected.length === 300) {
            break;
        }
    }

    return selected;
}
05.05

Build. Run. Try your model.

Load your model and labels, then try an offline image. Start the camera only when you choose, and stop to release its tracks and worker.

Basics

CameraApp.loadModel()
Reads your selected local files and validates the model and labels before processing.
CameraApp.start(camera)
Validates the selected source and delegates image or camera processing.
CameraApp.stop()
Stops accepting new results and releases resources after active work finishes.
Keep in mind
Symptom
Old boxes return after stopping or changing models.
Cause
An earlier asynchronous operation can finish after stop or after a new model has been selected.
Remedy
Request stop, reject late results and wait for active work before releasing its resources. Backend cancellation depends on platform support.
Terminalbash
npm run preview -- --port 4176

05.06 / Best practices

Small details. Better examples.

  • Keep capture and inference off the UI thread. Bound the queue instead of collecting a backlog of old frames.
  • Treat image size, crop, orientation and coordinates as one contract. Test it with known boxes before trusting an overlay.
  • Request stop, reject late results and wait for active work before releasing its resources. Backend cancellation depends on platform support.