BadBob / Playground
Browser · ONNX · WASM · MIT
05.01

Bereid je project voor

Installeer de vermelde afhankelijkheden en voer de tests uit. Probeer eerst een lokale afbeelding voor je de camera opent.

De basis

npm ci · npm run build
Configureert afhankelijkheden en compileert het project.
npm test · npm run check
Voert automatische controles uit zonder de camera te openen.
Terminalbash
unzip badbob-playground-javascript.zip
cd badbob-playground-javascript
npm ci
npm test
npm run check
npm run build
05.02

Koppel je model

Valideer het model en de labels voordat je een camera opent.

De basis

parseLabels(text)
Leest klassenamen in modelvolgorde; weigert lege of dubbele labels.
OnnxDetector.create(model, labelsText)
Laadt het ONNX-model en controleert de invoer, uitvoer en het aantal labels.
OnnxDetector.detect(frame)
Bereidt de afbeelding voor, voert het model uit en geeft detecties terug.
Goed om te weten
Symptoom
Kaders verschuiven of het model wordt niet geladen.
Oorzaak
Voorbewerking, uitvoerindeling of labelvolgorde komt niet overeen.
Oplossing
Deze adapter ondersteunt alleen het onderstaande contract. Core ML-bestanden en andere ONNX-indelingen hebben een andere adapter nodig.

Deze adapter ondersteunt alleen het onderstaande contract. Core ML-bestanden en andere ONNX-indelingen hebben een andere adapter nodig.

Verdieping

Coördinaten beschrijven het middelpunt en de afmetingen van kaders in de 640×640 invoerpixels. Labels volgen exact de klassevolgorde van het model.

ONNXyaml
input:
  color: RGB
  dtype: float32
  shape: [1, 3, 640, 640]
  letterbox: 114
  scale: 1/255
output:
  dtype: float32
  shape: [1, 4+C, N]
  boxes: [cx, cy, width, height]
  class_scores: probabilities
  objectness: false
  embedded_nms: false
onnx-detector.tstypescript
import * as ort from "onnxruntime-web/wasm";
import {
    decode,
    INPUT_EDGE,
    letterbox,
    parseLabels,
    rgbaToNchw,
    validateOutputShape,
} from "./core.ts";
import type { Detection, Detector, Letterbox } from "./types.ts";

export function validateModelMetadata(
    inputs: readonly ort.InferenceSession.ValueMetadata[],
    outputs: readonly ort.InferenceSession.ValueMetadata[],
    classes: number,
): void {
    const input = inputs[0];
    const output = outputs[0];

    if (
        inputs.length !== 1 ||
        !input?.isTensor ||
        input.type !== "float32" ||
        input.shape.length !== 4 ||
        input.shape.some((dimension, index) => dimension !== [1, 3, 640, 640][index])
    ) {
        throw new Error("The model must have one static float32 RGB input [1,3,640,640].");
    }

    if (
        outputs.length !== 1 ||
        !output?.isTensor ||
        output.type !== "float32" ||
        output.shape.length !== 3 ||
        output.shape[0] !== 1 ||
        output.shape[1] !== 4 + classes
    ) {
        throw new Error(
            "The model must have one float32 output [1,4+C,N], with C matching labels.",
        );
    }

    const count = output.shape[2];

    if (typeof count === "number") {
        validateOutputShape([1, 4 + classes, count], classes);
    }
}

export class OnnxDetector implements Detector {
    private disposed = false;

    private constructor(
        private readonly session: ort.InferenceSession,
        private readonly classes: number,
    ) {}

    static async create(model: ArrayBuffer, labelsText: string): Promise<OnnxDetector> {
        const labels = parseLabels(labelsText);
        ort.env.wasm.numThreads = 1;
        ort.env.wasm.proxy = false;
        const session = await ort.InferenceSession.create(new Uint8Array(model), {
            executionProviders: ["wasm"],
        });
        const detector = new OnnxDetector(session, labels.length);

        try {
            validateModelMetadata(session.inputMetadata, session.outputMetadata, labels.length);
            const neutral = new Float32Array(3 * INPUT_EDGE * INPUT_EDGE).fill(114 / 255);
            await detector.predict(neutral, letterbox({ width: 640, height: 640 }));
            return detector;
        } catch (error: unknown) {
            await detector.dispose();
            throw error;
        }
    }

    async detect(frame: ImageBitmap): Promise<Detection[]> {
        const transform = letterbox(frame);
        const canvas = new OffscreenCanvas(INPUT_EDGE, INPUT_EDGE);
        const context = canvas.getContext("2d", { willReadFrequently: true });

        if (!context) {
            throw new Error("OffscreenCanvas 2D is unavailable in this browser's worker.");
        }

        context.fillStyle = "rgb(114, 114, 114)";
        context.fillRect(0, 0, INPUT_EDGE, INPUT_EDGE);
        context.drawImage(
            frame,
            transform.left,
            transform.top,
            transform.resized.width,
            transform.resized.height,
        );
        const pixels = context.getImageData(0, 0, INPUT_EDGE, INPUT_EDGE);
        return this.predict(rgbaToNchw(pixels.data), transform);
    }

    async predict(input: Float32Array, transform: Letterbox): Promise<Detection[]> {
        if (this.disposed || input.length !== 3 * INPUT_EDGE * INPUT_EDGE) {
            throw new Error("Session is disposed or input is not [1,3,640,640].");
        }

        if (input.some((value) => !Number.isFinite(value) || value < 0 || value > 1)) {
            throw new Error("Input pixels must be finite normalized RGB values in [0,1].");
        }

        const inputName = this.session.inputNames[0];
        const outputName = this.session.outputNames[0];

        if (!inputName || !outputName) {
            throw new Error("Model tensor names are missing.");
        }

        const tensor = new ort.Tensor("float32", input, [1, 3, INPUT_EDGE, INPUT_EDGE]);
        let outputs: ort.InferenceSession.ReturnType | undefined;

        try {
            outputs = await this.session.run({ [inputName]: tensor });
            const output = outputs[outputName];

            if (
                !(output instanceof ort.Tensor) ||
                output.type !== "float32" ||
                !(output.data instanceof Float32Array)
            ) {
                throw new Error("Expected one CPU float32 output tensor.");
            }

            return decode(output.data, output.dims, this.classes, transform);
        } finally {
            tensor.dispose();

            for (const output of Object.values(outputs ?? {})) {
                if (output instanceof ort.Tensor) {
                    output.dispose();
                }
            }
        }
    }

    async dispose(): Promise<void> {
        if (this.disposed) {
            return;
        }

        this.disposed = true;
        await this.session.release();
    }
}
05.03

Voeg de camera toe

De browser legt beelden vast terwijl een achtergrondtaak de inferentie uitvoert. Er wordt slechts één beeld tegelijk verwerkt, zodat er geen groeiende wachtrij ontstaat.

De basis

CameraSource.open(signal)
Vraagt toestemming voor video en opent de camera na een expliciete start.
CameraSource.next()
Legt het volgende beeld vast of meldt dat de camerastream is gestopt.
WorkerDetector.infer(frame)
Bereidt de afbeelding voor, voert het model uit en geeft detecties terug.
WorkerDetector.dispose()
Accepteert geen nieuwe resultaten meer en geeft middelen vrij zodra actief werk klaar is.
Goed om te weten
Symptoom
De voorvertoning loopt achter en het geheugengebruik groeit.
Oorzaak
De camera levert beelden sneller aan dan de inferentie ze verwerkt, waardoor een onbeperkte wachtrij groeit.
Oplossing
De browser legt beelden vast terwijl een achtergrondtaak de inferentie uitvoert. Er wordt slechts één beeld tegelijk verwerkt, zodat er geen groeiende wachtrij ontstaat.
Verdieping

Vraag om te stoppen, wijs late resultaten af en wacht tot actief werk klaar is voor je de bijbehorende middelen vrijgeeft. Annulering in de onderliggende engine hangt af van de platformondersteuning.

frame-source.tstypescript
import { validateSize } from "./core.ts";

export interface FrameSource {
    next(): Promise<ImageBitmap | null>;
    stop(): void;
}

export class ImageSource implements FrameSource {
    private constructor(private image: ImageBitmap | undefined) {}

    static async open(file: File): Promise<ImageSource> {
        const image = await createImageBitmap(file);

        try {
            validateSize(image);
            return new ImageSource(image);
        } catch (error: unknown) {
            image.close();
            throw error;
        }
    }

    async next(): Promise<ImageBitmap | null> {
        const image = this.image;
        this.image = undefined;
        return image ?? null;
    }

    stop(): void {
        this.image?.close();
        this.image = undefined;
    }
}

export class CameraSource implements FrameSource {
    private stopped = false;

    private constructor(
        private readonly stream: MediaStream,
        private readonly video: HTMLVideoElement,
    ) {}

    static async open(signal: AbortSignal): Promise<CameraSource> {
        if (!navigator.mediaDevices?.getUserMedia) {
            throw new Error(
                "Camera API unavailable. Use HTTPS or localhost in a supported browser.",
            );
        }

        const stream = await navigator.mediaDevices.getUserMedia({
            video: { width: { ideal: 1280 }, height: { ideal: 720 } },
            audio: false,
        });
        const video = document.createElement("video");
        const source = new CameraSource(stream, video);

        if (signal.aborted) {
            source.stop();
            throw new Error("Camera startup stopped.");
        }

        const abort = (): void => source.stop();
        signal.addEventListener("abort", abort, { once: true });
        video.muted = true;
        video.playsInline = true;
        video.srcObject = stream;

        try {
            await video.play();

            if (signal.aborted || source.stopped) {
                throw new Error("Camera startup stopped.");
            }

            validateSize({ width: video.videoWidth, height: video.videoHeight });
            return source;
        } catch (error: unknown) {
            source.stop();
            throw error;
        } finally {
            signal.removeEventListener("abort", abort);
        }
    }

    async next(): Promise<ImageBitmap | null> {
        if (this.stopped) {
            return null;
        }

        if (!this.stream.getVideoTracks().some((track) => track.readyState === "live")) {
            throw new Error("Camera stream ended. Stop, check the device and start again.");
        }

        return createImageBitmap(this.video);
    }

    stop(): void {
        if (this.stopped) {
            return;
        }

        this.stopped = true;

        for (const track of this.stream.getTracks()) {
            track.stop();
        }

        this.video.pause();
        this.video.srcObject = null;
        this.video.removeAttribute("src");
        this.video.load();
    }
}
worker-client.tstypescript
import { isWorkerResponse } from "./protocol.ts";
import type { WorkerRequest, WorkerResponse } from "./protocol.ts";
import type { Detection } from "./types.ts";

export interface PairedResult {
    frame: ImageBitmap;
    detections: Detection[];
}

interface PendingFrame {
    id: number;
    resolve: (result: PairedResult) => void;
    reject: (error: Error) => void;
}

let nextGeneration = 1;

export class WorkerDetector {
    readonly ready: Promise<void>;
    private readonly worker = new Worker(new URL("./worker.ts", import.meta.url), {
        type: "module",
    });
    private readonly generation = nextGeneration++;
    private nextId = 1;
    private pending: PendingFrame | undefined;
    private stopping = false;
    private resolveReady!: () => void;
    private rejectReady!: (error: Error) => void;
    private readonly closed: Promise<void>;
    private resolveClosed!: () => void;
    private shutdownTimer: ReturnType<typeof setTimeout> | undefined;

    constructor(model: ArrayBuffer, labels: string) {
        this.ready = new Promise((resolve, reject) => {
            this.resolveReady = resolve;
            this.rejectReady = reject;
        });
        this.closed = new Promise((resolve) => {
            this.resolveClosed = resolve;
        });
        this.worker.addEventListener("message", (event: MessageEvent<unknown>) => {
            if (!isWorkerResponse(event.data)) {
                this.fail(new Error("Worker sent an invalid response."));
                return;
            }

            this.receive(event.data);
        });
        this.worker.addEventListener("error", (event: ErrorEvent) => {
            event.preventDefault();
            this.fail(new Error(event.message || "Inference worker failed."));
        });
        this.worker.addEventListener("messageerror", () =>
            this.fail(new Error("Worker message could not be decoded.")),
        );
        try {
            this.send({ kind: "initialize", generation: this.generation, model, labels }, [model]);
        } catch (error: unknown) {
            this.fail(error instanceof Error ? error : new Error(String(error)));
        }
    }

    infer(frame: ImageBitmap): Promise<PairedResult> {
        if (this.stopping || this.pending) {
            frame.close();
            return Promise.reject(new Error("Worker is stopping or already processing a frame."));
        }

        return new Promise((resolve, reject) => {
            const id = this.nextId++;
            this.pending = { id, resolve, reject };

            try {
                this.send({ kind: "infer", generation: this.generation, id, frame }, [frame]);
            } catch (error: unknown) {
                frame.close();
                this.pending = undefined;
                reject(error);
            }
        });
    }

    async dispose(): Promise<void> {
        if (this.stopping) {
            return this.closed;
        }

        this.stopping = true;
        const error = new Error("Inference stopped.");
        this.rejectReady(error);
        this.pending?.reject(error);
        this.pending = undefined;
        this.shutdownTimer = setTimeout(() => this.finishShutdown(), 5000);
        this.send({ kind: "stop", generation: this.generation });
        await this.closed;
    }

    private receive(message: WorkerResponse): void {
        if (message.generation !== this.generation) {
            if (message.kind === "result") {
                message.frame.close();
            }

            return;
        }

        if (message.kind === "disposed") {
            this.finishShutdown();
            return;
        }

        if (this.stopping) {
            if (message.kind === "result") {
                message.frame.close();
            }

            return;
        }

        if (message.kind === "ready") {
            this.resolveReady();
            return;
        }

        if (message.kind === "error") {
            const error = new Error(message.message);

            if (message.id === null) {
                this.rejectReady(error);
                return;
            }

            if (message.id === this.pending?.id) {
                this.pending.reject(error);
                this.pending = undefined;
            }

            return;
        }

        if (message.id !== this.pending?.id) {
            message.frame.close();
            return;
        }

        this.pending.resolve({ frame: message.frame, detections: message.detections });
        this.pending = undefined;
    }

    private send(message: WorkerRequest, transfer: Transferable[] = []): void {
        this.worker.postMessage(message, transfer);
    }

    private fail(error: Error): void {
        this.stopping = true;
        this.rejectReady(error);
        this.pending?.reject(error);
        this.pending = undefined;
        this.finishShutdown();
    }

    private finishShutdown(): void {
        clearTimeout(this.shutdownTimer);
        this.worker.terminate();
        this.resolveClosed();
    }
}
05.04

Plaats de voorspellingen

Draai letterbox-schaling en opvulling terug en begrens de kaders tot het beeld voordat je tekent.

De basis

letterbox(size) · unproject(...)
Behoudt de beeldverhouding en rekent kaders terug naar de oorspronkelijke afbeelding.
decode(data, dims, classes, transform)
Valideert voorspellingen en verwijdert overlappende kaders van dezelfde klasse.
Goed om te weten

Behandel beeldformaat, uitsnede, oriëntatie en coördinaten als één contract. Test bekende kaders voordat je op de markeringen vertrouwt.

core.tstypescript
import type { Box, Detection, Letterbox, Size } from "./types.ts";

export const INPUT_EDGE = 640;
const MAX_CANDIDATES = 100_000;

export function parseLabels(text: string): string[] {
    const normalized = text
        .replace(/^\uFEFF/, "")
        .replace(/\r\n/g, "\n")
        .replace(/\n$/, "");
    const labels = normalized.split("\n").map((line) => line.trim());

    if (labels.length > 1024 || labels.some((label) => label.length === 0 || label.length > 256)) {
        throw new Error(
            "Supply 1–1024 nonempty labels, one per line (maximum 256 characters each).",
        );
    }

    if (new Set(labels).size !== labels.length) {
        throw new Error("Labels must be distinct and in the model's exact class order.");
    }

    return labels;
}

export function validateSize(size: Size): void {
    if (
        !Number.isSafeInteger(size.width) ||
        !Number.isSafeInteger(size.height) ||
        size.width <= 0 ||
        size.height <= 0 ||
        size.width > 16384 ||
        size.height > 16384 ||
        size.width * size.height > 64_000_000
    ) {
        throw new Error(
            "Use an image with positive integer dimensions, at most 16384 per side and 64 MP.",
        );
    }
}

export function letterbox(size: Size): Letterbox {
    validateSize(size);
    const ratio = Math.min(INPUT_EDGE / size.width, INPUT_EDGE / size.height);
    const resized = {
        width: Math.max(1, Math.round(size.width * ratio)),
        height: Math.max(1, Math.round(size.height * ratio)),
    };

    return {
        original: { width: size.width, height: size.height },
        resized,
        left: Math.floor((INPUT_EDGE - resized.width) / 2),
        top: Math.floor((INPUT_EDGE - resized.height) / 2),
        edge: INPUT_EDGE,
    };
}

function validateTransform(transform: Letterbox): void {
    const expected = letterbox(transform.original);

    if (
        transform.edge !== expected.edge ||
        transform.left !== expected.left ||
        transform.top !== expected.top ||
        transform.resized.width !== expected.resized.width ||
        transform.resized.height !== expected.resized.height
    ) {
        throw new Error("Invalid letterbox geometry.");
    }
}

export function rgbaToNchw(rgba: Uint8ClampedArray): Float32Array {
    const pixels = INPUT_EDGE * INPUT_EDGE;

    if (rgba.length !== pixels * 4) {
        throw new Error("Expected a letterboxed 640×640 RGBA image.");
    }

    const result = new Float32Array(pixels * 3);

    for (let pixel = 0; pixel < pixels; pixel += 1) {
        result[pixel] = rgba[pixel * 4]! / 255;
        result[pixels + pixel] = rgba[pixel * 4 + 1]! / 255;
        result[pixels * 2 + pixel] = rgba[pixel * 4 + 2]! / 255;
    }

    return result;
}

function unproject(
    centerX: number,
    centerY: number,
    width: number,
    height: number,
    transform: Letterbox,
): Box {
    const { original, resized, left, top } = transform;
    const projectX = (x: number): number =>
        Math.max(0, Math.min(1, (x - left) / resized.width)) * original.width;
    const projectY = (y: number): number =>
        Math.max(0, Math.min(1, (y - top) / resized.height)) * original.height;
    const x = projectX(centerX - width / 2);
    const y = projectY(centerY - height / 2);

    return {
        x,
        y,
        width: projectX(centerX + width / 2) - x,
        height: projectY(centerY + height / 2) - y,
    };
}

function intersectionOverUnion(a: Box, b: Box): number {
    const width = Math.max(0, Math.min(a.x + a.width, b.x + b.width) - Math.max(a.x, b.x));
    const height = Math.max(0, Math.min(a.y + a.height, b.y + b.height) - Math.max(a.y, b.y));
    const intersection = width * height;
    const union = a.width * a.height + b.width * b.height - intersection;

    return union > 0 ? intersection / union : 0;
}

export function validateOutputShape(dims: readonly number[], classes: number): number {
    const count = dims[2];

    if (
        !Number.isSafeInteger(classes) ||
        classes < 1 ||
        classes > 1024 ||
        dims.length !== 3 ||
        dims[0] !== 1 ||
        dims[1] !== 4 + classes ||
        count === undefined ||
        !Number.isSafeInteger(count) ||
        count < 1 ||
        count > MAX_CANDIDATES
    ) {
        throw new Error(
            "Expected float32 [1,4+C,N], C matching labels and 1≤N≤100000. No objectness/NMS.",
        );
    }

    return count;
}

export function decode(
    data: Float32Array,
    dims: readonly number[],
    classes: number,
    transform: Letterbox,
): Detection[] {
    validateTransform(transform);
    const count = validateOutputShape(dims, classes);

    if (data.length !== (4 + classes) * count) {
        throw new Error("Output tensor data does not match its dimensions.");
    }

    const candidates: Detection[] = [];

    for (let index = 0; index < count; index += 1) {
        const centerX = data[index]!;
        const centerY = data[count + index]!;
        const width = data[count * 2 + index]!;
        const height = data[count * 3 + index]!;

        if (![centerX, centerY, width, height].every(Number.isFinite) || width < 0 || height < 0) {
            throw new Error(
                "Detector coordinates must be finite, with nonnegative width and height.",
            );
        }

        const { label, score } = bestClass(data, count, classes, index);

        if (score < 0.25) {
            continue;
        }

        const box = unproject(centerX, centerY, width, height, transform);

        if (box.width > 0 && box.height > 0) {
            candidates.push({ box, label, score });
        }
    }

    return suppressOverlaps(candidates);
}

function bestClass(
    data: Float32Array,
    count: number,
    classes: number,
    index: number,
): { label: number; score: number } {
    let label = 0;
    let score = 0;

    for (let candidate = 0; candidate < classes; candidate += 1) {
        const probability = data[(4 + candidate) * count + index]!;

        if (!Number.isFinite(probability) || probability < 0 || probability > 1) {
            throw new Error("Expected class probabilities in [0,1], not logits or objectness.");
        }

        if (probability > score) {
            score = probability;
            label = candidate;
        }
    }

    return { label, score };
}

function suppressOverlaps(candidates: Detection[]): Detection[] {
    candidates.sort((a, b) => b.score - a.score);
    const selected: Detection[] = [];

    for (const candidate of candidates.slice(0, 3000)) {
        const suppressed = selected.some(
            (previous) =>
                previous.label === candidate.label &&
                intersectionOverUnion(previous.box, candidate.box) > 0.45,
        );

        if (!suppressed) {
            selected.push(candidate);
        }

        if (selected.length === 300) {
            break;
        }
    }

    return selected;
}
05.05

Bouw. Start. Probeer je model.

Laad je model en labels en probeer daarna een lokale afbeelding. Start de camera wanneer je dat zelf wilt en stop de verwerking om de camerasporen en achtergrondtaak vrij te geven.

De basis

CameraApp.loadModel()
Leest je geselecteerde lokale bestanden en controleert het model en de labels vóór de verwerking.
CameraApp.start(camera)
Controleert de gekozen bron en delegeert de verwerking van afbeeldingen of camerabeelden.
CameraApp.stop()
Accepteert geen nieuwe resultaten meer en geeft middelen vrij zodra actief werk klaar is.
Goed om te weten
Symptoom
Oude kaders verschijnen opnieuw na stoppen of wisselen van model.
Oorzaak
Een eerdere asynchrone bewerking kan na het stoppen of na de keuze van een nieuw model worden voltooid.
Oplossing
Vraag om te stoppen, wijs late resultaten af en wacht tot actief werk klaar is voor je de bijbehorende middelen vrijgeeft. Annulering in de onderliggende engine hangt af van de platformondersteuning.
Terminalbash
npm run preview -- --port 4176

05.06 / Goede werkwijzen

Kleine details. Betere voorbeelden.

  • Houd opname en inferentie buiten de UI-thread. Begrens de wachtrij in plaats van oude frames op te stapelen.
  • Behandel beeldformaat, uitsnede, oriëntatie en coördinaten als één contract. Test bekende kaders voordat je op de markeringen vertrouwt.
  • Vraag om te stoppen, wijs late resultaten af en wacht tot actief werk klaar is voor je de bijbehorende middelen vrijgeeft. Annulering in de onderliggende engine hangt af van de platformondersteuning.