BadBob / Playground
Browser · ONNX · WASM · MIT
05.01

Siapkan proyek

Instal dependensi yang tercantum dan jalankan pengujian. Coba gambar lokal sebelum membuka kamera.

Dasar

npm ci · npm run build
Mengonfigurasi dependensi dan mengompilasi proyek.
npm test · npm run check
Menjalankan pemeriksaan otomatis tanpa membuka kamera.
Terminalbash
unzip badbob-playground-javascript.zip
cd badbob-playground-javascript
npm ci
npm test
npm run check
npm run build
05.02

Hubungkan model

Validasi model dan label sebelum membuka kamera.

Dasar

parseLabels(text)
Membaca nama kelas sesuai urutan model; menolak label kosong atau duplikat.
OnnxDetector.create(model, labelsText)
Memuat model ONNX dan memvalidasi masukan, keluaran, serta jumlah label.
OnnxDetector.detect(frame)
Menyiapkan gambar, menjalankan model, dan mengembalikan deteksi.
Perlu diingat
Gejala
Kotak bergeser atau model gagal dimuat.
Penyebab
Prapemrosesan, tata letak keluaran, atau urutan label tidak sesuai.
Penanganan
Adaptor ini hanya mendukung kontrak berikut. Berkas Core ML dan tata letak ONNX lain memerlukan adaptor berbeda.

Adaptor ini hanya mendukung kontrak berikut. Berkas Core ML dan tata letak ONNX lain memerlukan adaptor berbeda.

Lebih mendalam

Koordinat menyatakan pusat dan ukuran kotak dalam piksel masukan 640×640. Label harus mengikuti urutan kelas model secara persis.

ONNXyaml
input:
  color: RGB
  dtype: float32
  shape: [1, 3, 640, 640]
  letterbox: 114
  scale: 1/255
output:
  dtype: float32
  shape: [1, 4+C, N]
  boxes: [cx, cy, width, height]
  class_scores: probabilities
  objectness: false
  embedded_nms: false
onnx-detector.tstypescript
import * as ort from "onnxruntime-web/wasm";
import {
    decode,
    INPUT_EDGE,
    letterbox,
    parseLabels,
    rgbaToNchw,
    validateOutputShape,
} from "./core.ts";
import type { Detection, Detector, Letterbox } from "./types.ts";

export function validateModelMetadata(
    inputs: readonly ort.InferenceSession.ValueMetadata[],
    outputs: readonly ort.InferenceSession.ValueMetadata[],
    classes: number,
): void {
    const input = inputs[0];
    const output = outputs[0];

    if (
        inputs.length !== 1 ||
        !input?.isTensor ||
        input.type !== "float32" ||
        input.shape.length !== 4 ||
        input.shape.some((dimension, index) => dimension !== [1, 3, 640, 640][index])
    ) {
        throw new Error("The model must have one static float32 RGB input [1,3,640,640].");
    }

    if (
        outputs.length !== 1 ||
        !output?.isTensor ||
        output.type !== "float32" ||
        output.shape.length !== 3 ||
        output.shape[0] !== 1 ||
        output.shape[1] !== 4 + classes
    ) {
        throw new Error(
            "The model must have one float32 output [1,4+C,N], with C matching labels.",
        );
    }

    const count = output.shape[2];

    if (typeof count === "number") {
        validateOutputShape([1, 4 + classes, count], classes);
    }
}

export class OnnxDetector implements Detector {
    private disposed = false;

    private constructor(
        private readonly session: ort.InferenceSession,
        private readonly classes: number,
    ) {}

    static async create(model: ArrayBuffer, labelsText: string): Promise<OnnxDetector> {
        const labels = parseLabels(labelsText);
        ort.env.wasm.numThreads = 1;
        ort.env.wasm.proxy = false;
        const session = await ort.InferenceSession.create(new Uint8Array(model), {
            executionProviders: ["wasm"],
        });
        const detector = new OnnxDetector(session, labels.length);

        try {
            validateModelMetadata(session.inputMetadata, session.outputMetadata, labels.length);
            const neutral = new Float32Array(3 * INPUT_EDGE * INPUT_EDGE).fill(114 / 255);
            await detector.predict(neutral, letterbox({ width: 640, height: 640 }));
            return detector;
        } catch (error: unknown) {
            await detector.dispose();
            throw error;
        }
    }

    async detect(frame: ImageBitmap): Promise<Detection[]> {
        const transform = letterbox(frame);
        const canvas = new OffscreenCanvas(INPUT_EDGE, INPUT_EDGE);
        const context = canvas.getContext("2d", { willReadFrequently: true });

        if (!context) {
            throw new Error("OffscreenCanvas 2D is unavailable in this browser's worker.");
        }

        context.fillStyle = "rgb(114, 114, 114)";
        context.fillRect(0, 0, INPUT_EDGE, INPUT_EDGE);
        context.drawImage(
            frame,
            transform.left,
            transform.top,
            transform.resized.width,
            transform.resized.height,
        );
        const pixels = context.getImageData(0, 0, INPUT_EDGE, INPUT_EDGE);
        return this.predict(rgbaToNchw(pixels.data), transform);
    }

    async predict(input: Float32Array, transform: Letterbox): Promise<Detection[]> {
        if (this.disposed || input.length !== 3 * INPUT_EDGE * INPUT_EDGE) {
            throw new Error("Session is disposed or input is not [1,3,640,640].");
        }

        if (input.some((value) => !Number.isFinite(value) || value < 0 || value > 1)) {
            throw new Error("Input pixels must be finite normalized RGB values in [0,1].");
        }

        const inputName = this.session.inputNames[0];
        const outputName = this.session.outputNames[0];

        if (!inputName || !outputName) {
            throw new Error("Model tensor names are missing.");
        }

        const tensor = new ort.Tensor("float32", input, [1, 3, INPUT_EDGE, INPUT_EDGE]);
        let outputs: ort.InferenceSession.ReturnType | undefined;

        try {
            outputs = await this.session.run({ [inputName]: tensor });
            const output = outputs[outputName];

            if (
                !(output instanceof ort.Tensor) ||
                output.type !== "float32" ||
                !(output.data instanceof Float32Array)
            ) {
                throw new Error("Expected one CPU float32 output tensor.");
            }

            return decode(output.data, output.dims, this.classes, transform);
        } finally {
            tensor.dispose();

            for (const output of Object.values(outputs ?? {})) {
                if (output instanceof ort.Tensor) {
                    output.dispose();
                }
            }
        }
    }

    async dispose(): Promise<void> {
        if (this.disposed) {
            return;
        }

        this.disposed = true;
        await this.session.release();
    }
}
05.03

Tambahkan kamera

Browser menangkap bingkai sementara tugas latar belakang menjalankan inferensi. Hanya satu bingkai diproses pada satu waktu, sehingga antrean tidak menumpuk.

Dasar

CameraSource.open(signal)
Meminta izin video dan membuka kamera setelah perintah mulai yang eksplisit.
CameraSource.next()
Mengambil bingkai berikutnya atau melaporkan bahwa aliran kamera telah berhenti.
WorkerDetector.infer(frame)
Menyiapkan gambar, menjalankan model, dan mengembalikan deteksi.
WorkerDetector.dispose()
Berhenti menerima hasil baru dan melepaskan sumber daya setelah pekerjaan aktif selesai.
Perlu diingat
Gejala
Pratinjau tertinggal dan penggunaan memori meningkat.
Penyebab
Kamera menghasilkan frame lebih cepat daripada inferensi memprosesnya, sehingga antrean tanpa batas terus bertambah.
Penanganan
Browser menangkap bingkai sementara tugas latar belakang menjalankan inferensi. Hanya satu bingkai diproses pada satu waktu, sehingga antrean tidak menumpuk.
Lebih mendalam

Minta penghentian, tolak hasil yang terlambat, dan tunggu pekerjaan aktif selesai sebelum melepaskan sumber dayanya. Pembatalan mesin pemrosesan bergantung pada dukungan platform.

frame-source.tstypescript
import { validateSize } from "./core.ts";

export interface FrameSource {
    next(): Promise<ImageBitmap | null>;
    stop(): void;
}

export class ImageSource implements FrameSource {
    private constructor(private image: ImageBitmap | undefined) {}

    static async open(file: File): Promise<ImageSource> {
        const image = await createImageBitmap(file);

        try {
            validateSize(image);
            return new ImageSource(image);
        } catch (error: unknown) {
            image.close();
            throw error;
        }
    }

    async next(): Promise<ImageBitmap | null> {
        const image = this.image;
        this.image = undefined;
        return image ?? null;
    }

    stop(): void {
        this.image?.close();
        this.image = undefined;
    }
}

export class CameraSource implements FrameSource {
    private stopped = false;

    private constructor(
        private readonly stream: MediaStream,
        private readonly video: HTMLVideoElement,
    ) {}

    static async open(signal: AbortSignal): Promise<CameraSource> {
        if (!navigator.mediaDevices?.getUserMedia) {
            throw new Error(
                "Camera API unavailable. Use HTTPS or localhost in a supported browser.",
            );
        }

        const stream = await navigator.mediaDevices.getUserMedia({
            video: { width: { ideal: 1280 }, height: { ideal: 720 } },
            audio: false,
        });
        const video = document.createElement("video");
        const source = new CameraSource(stream, video);

        if (signal.aborted) {
            source.stop();
            throw new Error("Camera startup stopped.");
        }

        const abort = (): void => source.stop();
        signal.addEventListener("abort", abort, { once: true });
        video.muted = true;
        video.playsInline = true;
        video.srcObject = stream;

        try {
            await video.play();

            if (signal.aborted || source.stopped) {
                throw new Error("Camera startup stopped.");
            }

            validateSize({ width: video.videoWidth, height: video.videoHeight });
            return source;
        } catch (error: unknown) {
            source.stop();
            throw error;
        } finally {
            signal.removeEventListener("abort", abort);
        }
    }

    async next(): Promise<ImageBitmap | null> {
        if (this.stopped) {
            return null;
        }

        if (!this.stream.getVideoTracks().some((track) => track.readyState === "live")) {
            throw new Error("Camera stream ended. Stop, check the device and start again.");
        }

        return createImageBitmap(this.video);
    }

    stop(): void {
        if (this.stopped) {
            return;
        }

        this.stopped = true;

        for (const track of this.stream.getTracks()) {
            track.stop();
        }

        this.video.pause();
        this.video.srcObject = null;
        this.video.removeAttribute("src");
        this.video.load();
    }
}
worker-client.tstypescript
import { isWorkerResponse } from "./protocol.ts";
import type { WorkerRequest, WorkerResponse } from "./protocol.ts";
import type { Detection } from "./types.ts";

export interface PairedResult {
    frame: ImageBitmap;
    detections: Detection[];
}

interface PendingFrame {
    id: number;
    resolve: (result: PairedResult) => void;
    reject: (error: Error) => void;
}

let nextGeneration = 1;

export class WorkerDetector {
    readonly ready: Promise<void>;
    private readonly worker = new Worker(new URL("./worker.ts", import.meta.url), {
        type: "module",
    });
    private readonly generation = nextGeneration++;
    private nextId = 1;
    private pending: PendingFrame | undefined;
    private stopping = false;
    private resolveReady!: () => void;
    private rejectReady!: (error: Error) => void;
    private readonly closed: Promise<void>;
    private resolveClosed!: () => void;
    private shutdownTimer: ReturnType<typeof setTimeout> | undefined;

    constructor(model: ArrayBuffer, labels: string) {
        this.ready = new Promise((resolve, reject) => {
            this.resolveReady = resolve;
            this.rejectReady = reject;
        });
        this.closed = new Promise((resolve) => {
            this.resolveClosed = resolve;
        });
        this.worker.addEventListener("message", (event: MessageEvent<unknown>) => {
            if (!isWorkerResponse(event.data)) {
                this.fail(new Error("Worker sent an invalid response."));
                return;
            }

            this.receive(event.data);
        });
        this.worker.addEventListener("error", (event: ErrorEvent) => {
            event.preventDefault();
            this.fail(new Error(event.message || "Inference worker failed."));
        });
        this.worker.addEventListener("messageerror", () =>
            this.fail(new Error("Worker message could not be decoded.")),
        );
        try {
            this.send({ kind: "initialize", generation: this.generation, model, labels }, [model]);
        } catch (error: unknown) {
            this.fail(error instanceof Error ? error : new Error(String(error)));
        }
    }

    infer(frame: ImageBitmap): Promise<PairedResult> {
        if (this.stopping || this.pending) {
            frame.close();
            return Promise.reject(new Error("Worker is stopping or already processing a frame."));
        }

        return new Promise((resolve, reject) => {
            const id = this.nextId++;
            this.pending = { id, resolve, reject };

            try {
                this.send({ kind: "infer", generation: this.generation, id, frame }, [frame]);
            } catch (error: unknown) {
                frame.close();
                this.pending = undefined;
                reject(error);
            }
        });
    }

    async dispose(): Promise<void> {
        if (this.stopping) {
            return this.closed;
        }

        this.stopping = true;
        const error = new Error("Inference stopped.");
        this.rejectReady(error);
        this.pending?.reject(error);
        this.pending = undefined;
        this.shutdownTimer = setTimeout(() => this.finishShutdown(), 5000);
        this.send({ kind: "stop", generation: this.generation });
        await this.closed;
    }

    private receive(message: WorkerResponse): void {
        if (message.generation !== this.generation) {
            if (message.kind === "result") {
                message.frame.close();
            }

            return;
        }

        if (message.kind === "disposed") {
            this.finishShutdown();
            return;
        }

        if (this.stopping) {
            if (message.kind === "result") {
                message.frame.close();
            }

            return;
        }

        if (message.kind === "ready") {
            this.resolveReady();
            return;
        }

        if (message.kind === "error") {
            const error = new Error(message.message);

            if (message.id === null) {
                this.rejectReady(error);
                return;
            }

            if (message.id === this.pending?.id) {
                this.pending.reject(error);
                this.pending = undefined;
            }

            return;
        }

        if (message.id !== this.pending?.id) {
            message.frame.close();
            return;
        }

        this.pending.resolve({ frame: message.frame, detections: message.detections });
        this.pending = undefined;
    }

    private send(message: WorkerRequest, transfer: Transferable[] = []): void {
        this.worker.postMessage(message, transfer);
    }

    private fail(error: Error): void {
        this.stopping = true;
        this.rejectReady(error);
        this.pending?.reject(error);
        this.pending = undefined;
        this.finishShutdown();
    }

    private finishShutdown(): void {
        clearTimeout(this.shutdownTimer);
        this.worker.terminate();
        this.resolveClosed();
    }
}
05.04

Tempatkan prediksi

Balikkan skala dan padding letterbox, lalu batasi kotak ke gambar sebelum menggambar.

Dasar

letterbox(size) · unproject(...)
Mempertahankan rasio aspek dan memetakan kotak kembali ke gambar asli.
decode(data, dims, classes, transform)
Memvalidasi prediksi dan menghapus kotak tumpang tindih dari kelas yang sama.
Perlu diingat

Perlakukan ukuran gambar, pemotongan, orientasi, dan koordinat sebagai satu kesepakatan. Uji dengan kotak yang diketahui sebelum memercayai penanda.

core.tstypescript
import type { Box, Detection, Letterbox, Size } from "./types.ts";

export const INPUT_EDGE = 640;
const MAX_CANDIDATES = 100_000;

export function parseLabels(text: string): string[] {
    const normalized = text
        .replace(/^\uFEFF/, "")
        .replace(/\r\n/g, "\n")
        .replace(/\n$/, "");
    const labels = normalized.split("\n").map((line) => line.trim());

    if (labels.length > 1024 || labels.some((label) => label.length === 0 || label.length > 256)) {
        throw new Error(
            "Supply 1–1024 nonempty labels, one per line (maximum 256 characters each).",
        );
    }

    if (new Set(labels).size !== labels.length) {
        throw new Error("Labels must be distinct and in the model's exact class order.");
    }

    return labels;
}

export function validateSize(size: Size): void {
    if (
        !Number.isSafeInteger(size.width) ||
        !Number.isSafeInteger(size.height) ||
        size.width <= 0 ||
        size.height <= 0 ||
        size.width > 16384 ||
        size.height > 16384 ||
        size.width * size.height > 64_000_000
    ) {
        throw new Error(
            "Use an image with positive integer dimensions, at most 16384 per side and 64 MP.",
        );
    }
}

export function letterbox(size: Size): Letterbox {
    validateSize(size);
    const ratio = Math.min(INPUT_EDGE / size.width, INPUT_EDGE / size.height);
    const resized = {
        width: Math.max(1, Math.round(size.width * ratio)),
        height: Math.max(1, Math.round(size.height * ratio)),
    };

    return {
        original: { width: size.width, height: size.height },
        resized,
        left: Math.floor((INPUT_EDGE - resized.width) / 2),
        top: Math.floor((INPUT_EDGE - resized.height) / 2),
        edge: INPUT_EDGE,
    };
}

function validateTransform(transform: Letterbox): void {
    const expected = letterbox(transform.original);

    if (
        transform.edge !== expected.edge ||
        transform.left !== expected.left ||
        transform.top !== expected.top ||
        transform.resized.width !== expected.resized.width ||
        transform.resized.height !== expected.resized.height
    ) {
        throw new Error("Invalid letterbox geometry.");
    }
}

export function rgbaToNchw(rgba: Uint8ClampedArray): Float32Array {
    const pixels = INPUT_EDGE * INPUT_EDGE;

    if (rgba.length !== pixels * 4) {
        throw new Error("Expected a letterboxed 640×640 RGBA image.");
    }

    const result = new Float32Array(pixels * 3);

    for (let pixel = 0; pixel < pixels; pixel += 1) {
        result[pixel] = rgba[pixel * 4]! / 255;
        result[pixels + pixel] = rgba[pixel * 4 + 1]! / 255;
        result[pixels * 2 + pixel] = rgba[pixel * 4 + 2]! / 255;
    }

    return result;
}

function unproject(
    centerX: number,
    centerY: number,
    width: number,
    height: number,
    transform: Letterbox,
): Box {
    const { original, resized, left, top } = transform;
    const projectX = (x: number): number =>
        Math.max(0, Math.min(1, (x - left) / resized.width)) * original.width;
    const projectY = (y: number): number =>
        Math.max(0, Math.min(1, (y - top) / resized.height)) * original.height;
    const x = projectX(centerX - width / 2);
    const y = projectY(centerY - height / 2);

    return {
        x,
        y,
        width: projectX(centerX + width / 2) - x,
        height: projectY(centerY + height / 2) - y,
    };
}

function intersectionOverUnion(a: Box, b: Box): number {
    const width = Math.max(0, Math.min(a.x + a.width, b.x + b.width) - Math.max(a.x, b.x));
    const height = Math.max(0, Math.min(a.y + a.height, b.y + b.height) - Math.max(a.y, b.y));
    const intersection = width * height;
    const union = a.width * a.height + b.width * b.height - intersection;

    return union > 0 ? intersection / union : 0;
}

export function validateOutputShape(dims: readonly number[], classes: number): number {
    const count = dims[2];

    if (
        !Number.isSafeInteger(classes) ||
        classes < 1 ||
        classes > 1024 ||
        dims.length !== 3 ||
        dims[0] !== 1 ||
        dims[1] !== 4 + classes ||
        count === undefined ||
        !Number.isSafeInteger(count) ||
        count < 1 ||
        count > MAX_CANDIDATES
    ) {
        throw new Error(
            "Expected float32 [1,4+C,N], C matching labels and 1≤N≤100000. No objectness/NMS.",
        );
    }

    return count;
}

export function decode(
    data: Float32Array,
    dims: readonly number[],
    classes: number,
    transform: Letterbox,
): Detection[] {
    validateTransform(transform);
    const count = validateOutputShape(dims, classes);

    if (data.length !== (4 + classes) * count) {
        throw new Error("Output tensor data does not match its dimensions.");
    }

    const candidates: Detection[] = [];

    for (let index = 0; index < count; index += 1) {
        const centerX = data[index]!;
        const centerY = data[count + index]!;
        const width = data[count * 2 + index]!;
        const height = data[count * 3 + index]!;

        if (![centerX, centerY, width, height].every(Number.isFinite) || width < 0 || height < 0) {
            throw new Error(
                "Detector coordinates must be finite, with nonnegative width and height.",
            );
        }

        const { label, score } = bestClass(data, count, classes, index);

        if (score < 0.25) {
            continue;
        }

        const box = unproject(centerX, centerY, width, height, transform);

        if (box.width > 0 && box.height > 0) {
            candidates.push({ box, label, score });
        }
    }

    return suppressOverlaps(candidates);
}

function bestClass(
    data: Float32Array,
    count: number,
    classes: number,
    index: number,
): { label: number; score: number } {
    let label = 0;
    let score = 0;

    for (let candidate = 0; candidate < classes; candidate += 1) {
        const probability = data[(4 + candidate) * count + index]!;

        if (!Number.isFinite(probability) || probability < 0 || probability > 1) {
            throw new Error("Expected class probabilities in [0,1], not logits or objectness.");
        }

        if (probability > score) {
            score = probability;
            label = candidate;
        }
    }

    return { label, score };
}

function suppressOverlaps(candidates: Detection[]): Detection[] {
    candidates.sort((a, b) => b.score - a.score);
    const selected: Detection[] = [];

    for (const candidate of candidates.slice(0, 3000)) {
        const suppressed = selected.some(
            (previous) =>
                previous.label === candidate.label &&
                intersectionOverUnion(previous.box, candidate.box) > 0.45,
        );

        if (!suppressed) {
            selected.push(candidate);
        }

        if (selected.length === 300) {
            break;
        }
    }

    return selected;
}
05.05

Bangun. Jalankan. Coba model.

Muat model dan label, lalu coba gambar lokal. Mulai kamera hanya saat Anda memilihnya, dan hentikan pemrosesan untuk melepaskan trek kamera serta tugas latar belakang.

Dasar

CameraApp.loadModel()
Membaca berkas lokal yang dipilih dan memvalidasi model serta label sebelum pemrosesan.
CameraApp.start(camera)
Memvalidasi sumber yang dipilih dan meneruskan pemrosesan gambar atau kamera ke bagian yang sesuai.
CameraApp.stop()
Berhenti menerima hasil baru dan melepaskan sumber daya setelah pekerjaan aktif selesai.
Perlu diingat
Gejala
Kotak lama muncul lagi setelah berhenti atau mengganti model.
Penyebab
Operasi asinkron sebelumnya dapat selesai setelah dihentikan atau setelah model baru dipilih.
Penanganan
Minta penghentian, tolak hasil yang terlambat, dan tunggu pekerjaan aktif selesai sebelum melepaskan sumber dayanya. Pembatalan mesin pemrosesan bergantung pada dukungan platform.
Terminalbash
npm run preview -- --port 4176

05.06 / Praktik terbaik

Detail kecil. Contoh yang lebih baik.

  • Jalankan pengambilan gambar dan inferensi di luar thread antarmuka. Batasi antrean alih-alih menumpuk frame lama.
  • Perlakukan ukuran gambar, pemotongan, orientasi, dan koordinat sebagai satu kesepakatan. Uji dengan kotak yang diketahui sebelum memercayai penanda.
  • Minta penghentian, tolak hasil yang terlambat, dan tunggu pekerjaan aktif selesai sebelum melepaskan sumber dayanya. Pembatalan mesin pemrosesan bergantung pada dukungan platform.