BadBob / Playground

Your model.
Out in the wild.

Turn a trained model into something you can use. Small open source examples, clear steps, and code you can make your own.

Open source · MIT · Your model stays yours

FROM MODEL TO APP
Professor Bob smiles over low reading glasses beside a chalkboard covered in complex physics. In an erased corner he has written: MODEL = mc² / 10 PRINT "BAD BOB" / 20 GOTO 10.
  1. 01Camera
  2. 02Model
  3. 03Prediction
  4. 04Your app
Illustration · from calculation to result

01 / CHOOSE YOUR PATH

A small start. Room to build.

Available

Swift

A native Mac camera app with your Core ML detector.

Core ML · macOS
Available

Python

A scripting path for inspecting models and building practical tools.

ONNX Runtime · OpenCV
Available

JavaScript

Browser examples with a supported ONNX model.

Web · ONNX
Available

Rust

A native path with explicit model and runtime requirements.

OpenCV · ONNX
Available

Go

Practical tools with a clear model-serving boundary.

GoCV · ONNX
Available

C++

Camera and local inference with OpenCV and a supported ONNX model.

OpenCV · ONNX

DeviconGo Gopher — Renee FrenchCC BY 3.0

Keep in mind
  1. Symptom
    The interface freezes when capture starts.
    Cause
    Session startup blocks; async alone does not move work off MainActor.
    Remedy
    Run capture and inference on a dedicated serial executor. Update the UI on MainActor.
  2. Symptom
    The preview lags and memory use grows.
    Cause
    The camera produces frames faster than inference handles them, so an unbounded queue builds up.
    Remedy
    Discard late frames, process serially and keep only the newest pending result. Avoid a Task per frame.
  3. Symptom
    Old boxes return after stopping or changing models.
    Cause
    An earlier task can finish late; retained observers may keep resources alive.
    Remedy
    Cancel the old run, reject stale results, stop capture and remove observers. Release temporary models; profile repeated starts and stops with Instruments.
BadBob / Playground
Core ML · Vision · MIT
02.01

Prepare your project

Download the source and run its tests. You need a camera and a Vision-compatible Core ML object detector. No model weights are included.

Basics

swift test
Runs automated checks without opening the camera.
Terminalbash
unzip badbob-playground-swift.zip
cd badbob-playground-swift
swift test
02.02

Connect your model

Load your model and create a Vision request. Check its inputs before starting the camera, and keep any temporary compilation for the lifetime of inference.

Basics

VisionDetector.init(modelURL:)
Loads or compiles the Core ML model and validates its image input.
VisionDetector.predict(_:)
Prepares the image, runs the model and returns detections.

This example requires a Core ML object detector that returns Vision object observations. Other model types and ONNX files need a different adapter.

In depth

Vision reports normalized coordinates from the bottom left. Convert them to your view, preserve the image aspect ratio, and mirror the image and boxes together.

Core ML · Visionyaml
model:
  runtime: Core ML + Vision
  formats: [.mlmodel, .mlpackage, .mlmodelc]
  image_inputs: 1
  results: VNRecognizedObjectObservation
preprocessing:
  crop_and_scale: scaleFill
  orientation: up
coordinates:
  range: [0, 1]
  origin: bottom_left
preview:
  scaling: aspect_fit
  mirroring: paired
VisionDetector.swiftswift
import CoreML
import Foundation
import Vision

/// Owned by one frame processor and used only on its serial capture queue.
final class VisionDetector {
    private let request: VNCoreMLRequest
    private let compiledURL: URL?
    private let scopedURL: URL?

    init(modelURL: URL) throws {
        guard modelURL.isFileURL,
            ["mlmodel", "mlpackage", "mlmodelc"].contains(modelURL.pathExtension.lowercased())
        else {
            throw CaptureFailure.unsupportedModel
        }

        let scoped = modelURL.startAccessingSecurityScopedResource()
        var ownedCompilation: URL?

        do {
            let loadURL: URL

            if modelURL.pathExtension.lowercased() == "mlmodelc" {
                loadURL = modelURL
            } else {
                loadURL = try MLModel.compileModel(at: modelURL)
                ownedCompilation = loadURL
            }

            let model = try MLModel(contentsOf: loadURL)
            guard
                model.modelDescription.inputDescriptionsByName.values.filter({ $0.type == .image })
                    .count == 1
            else {
                throw CaptureFailure.unsupportedModel
            }

            request = VNCoreMLRequest(model: try VNCoreMLModel(for: model))
            request.imageCropAndScaleOption = .scaleFill
            compiledURL = ownedCompilation
            scopedURL = scoped ? modelURL : nil
        } catch {
            if scoped {
                modelURL.stopAccessingSecurityScopedResource()
            }

            if let ownedCompilation {
                try? FileManager.default.removeItem(at: ownedCompilation)
            }

            throw error
        }
    }

    deinit {
        // Predictions have ended; remove only our own temporary compilation.
        if let compiledURL {
            try? FileManager.default.removeItem(at: compiledURL)
        }

        scopedURL?.stopAccessingSecurityScopedResource()
    }

    func predict(_ buffer: CVPixelBuffer) throws -> [Detection] {
        try VNImageRequestHandler(cvPixelBuffer: buffer, orientation: .up).perform([request])

        guard let objects = request.results as? [VNRecognizedObjectObservation] else {
            throw CaptureFailure.unsupportedModel
        }

        return objects.enumerated().compactMap { index, observation in

            guard let label = observation.labels.first, label.confidence.isFinite else {
                return nil
            }

            return Detection(
                id: index, label: label.identifier, confidence: label.confidence,
                bounds: observation.boundingBox)
        }
    }
}
02.03

Bring in the camera

Capture runs on its own serial executor. Late frames are discarded, and only the newest processed frame waits for the interface. Pixels and predictions stay together.

Basics

CameraService.start(runID:modelURL:)
Creates the capture session on its serial executor and returns a stream holding at most one pending frame.
CameraService.stop(runID:)
Stops only the matching run and releases its session, delegate and observers.
Keep in mind
  1. Symptom
    The interface freezes when capture starts.
    Cause
    Session startup blocks; async alone does not move work off MainActor.
    Remedy
    Run capture and inference on a dedicated serial executor. Update the UI on MainActor.
  2. Symptom
    The preview lags and memory use grows.
    Cause
    The camera produces frames faster than inference handles them, so an unbounded queue builds up.
    Remedy
    Discard late frames, process serially and keep only the newest pending result. Avoid a Task per frame.
CameraService.swiftswift
import AVFoundation
import CoreImage
import Dispatch
import Foundation

/// A custom executor keeps blocking AVFoundation work off the main actor.
public actor CameraService: CameraCapturing {
    private nonisolated let queue = DispatchSerialQueue(label: "ai.badbob.playground.camera")
    public nonisolated var unownedExecutor: UnownedSerialExecutor {
        queue.asUnownedSerialExecutor()
    }

    private var session: AVCaptureSession?
    private var output: AVCaptureVideoDataOutput?
    private var processor: FrameProcessor?
    private var monitor: CaptureSessionMonitor?
    private var activeRunID: UUID?

    public init() {}

    public func start(runID: UUID, modelURL: URL) async throws -> FrameStream {
        stopActiveRun()
        let detector = try loadDetector(modelURL: modelURL)

        try Task.checkCancellation()

        guard let camera = AVCaptureDevice.default(for: .video) else {
            throw CaptureFailure.noCamera
        }

        let candidate = AVCaptureSession()
        let frames = AVCaptureVideoDataOutput()
        let (stream, continuation) = FrameStream.makeStream(bufferingPolicy: .bufferingNewest(1))
        let delegate = FrameProcessor(detector: detector, continuation: continuation)
        try configureSession(candidate, output: frames, processor: delegate, camera: camera)

        try Task.checkCancellation()
        session = candidate
        output = frames
        processor = delegate
        activeRunID = runID
        monitor = CaptureSessionMonitor(session: candidate, continuation: continuation)
        continuation.onTermination = { [weak self] _ in
            Task { await self?.stop(runID: runID) }
        }

        candidate.startRunning()

        guard candidate.isRunning else {
            stopActiveRun()
            throw CaptureFailure.configuration
        }

        return stream
    }

    private func loadDetector(modelURL: URL) throws -> VisionDetector {
        do {
            return try VisionDetector(modelURL: modelURL)
        } catch let error as CaptureFailure {
            throw error
        } catch {
            throw CaptureFailure.model(error.localizedDescription)
        }
    }

    private func configureSession(
        _ candidate: AVCaptureSession,
        output frames: AVCaptureVideoDataOutput,
        processor delegate: FrameProcessor,
        camera: AVCaptureDevice
    ) throws {
        candidate.beginConfiguration()

        do {
            let input = try AVCaptureDeviceInput(device: camera)

            guard candidate.canAddInput(input), candidate.canAddOutput(frames) else {
                throw CaptureFailure.configuration
            }

            candidate.addInput(input)
            candidate.addOutput(frames)

            if candidate.canSetSessionPreset(.hd1280x720) {
                candidate.sessionPreset = .hd1280x720
            }

            frames.alwaysDiscardsLateVideoFrames = true
            frames.videoSettings = [
                kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA
            ]
            // The delegate, Vision request and frame renderer share this one serial queue.
            frames.setSampleBufferDelegate(delegate, queue: queue)

            if let connection = frames.connection(with: .video),
                connection.isVideoMirroringSupported
            {
                connection.automaticallyAdjustsVideoMirroring = false
                connection.isVideoMirrored = false
            }

            candidate.commitConfiguration()
        } catch {
            candidate.commitConfiguration()
            frames.setSampleBufferDelegate(nil, queue: nil)
            delegate.finish()
            throw error
        }
    }

    public func stop(runID: UUID) {
        // A cancelled old task must not stop a more recent capture run.
        guard activeRunID == runID else {
            return
        }

        stopActiveRun()
    }

    private func stopActiveRun() {
        // Remove observers first so our deliberate stop cannot become a failure.
        monitor?.invalidate()
        monitor = nil
        session?.stopRunning()
        output?.setSampleBufferDelegate(nil, queue: nil)
        processor?.finish()
        processor = nil
        output = nil
        session = nil
        activeRunID = nil
    }
}

/// AVFoundation invokes this delegate only on CameraService's serial queue.
/// No sample buffer or mutable Vision object crosses into a Task or the UI.
private final class FrameProcessor: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
    private let detector: VisionDetector
    private let continuation: FrameStream.Continuation
    private let renderer = CIContext(options: [.cacheIntermediates: false])
    private var finished = false

    init(detector: VisionDetector, continuation: FrameStream.Continuation) {
        self.detector = detector
        self.continuation = continuation
    }

    func finish() {
        guard !finished else {
            return
        }

        finished = true
        continuation.finish()
    }

    func captureOutput(
        _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer,
        from connection: AVCaptureConnection
    ) {
        guard !finished, let buffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
            return
        }

        autoreleasepool {
            do {
                let detections = try detector.predict(buffer)
                let image = CIImage(cvPixelBuffer: buffer)

                guard let pixels = renderer.createCGImage(image, from: image.extent) else {
                    throw CaptureFailure.imageCreation
                }

                continuation.yield(DetectionFrame(image: pixels, detections: detections))
            } catch {
                finished = true
                continuation.finish(throwing: error)
            }
        }
    }
}
02.04

Put predictions in place

Vision reports normalized coordinates from the bottom left. Convert them to your view, preserve the image aspect ratio, and mirror the image and boxes together.

Basics

OverlayGeometry.project(...)
Projects Vision boxes into an aspect-fit view and mirrors the image and boxes together.
Keep in mind

Treat image size, crop, orientation and coordinates as one contract. Test it with known boxes before trusting an overlay.

OverlayGeometry.swiftswift
import CoreGraphics

/// Convert Vision's normalized, bottom-left coordinates to an aspect-fit view.
public enum OverlayGeometry {
    public static func project(
        _ bounds: CGRect, imageSize: CGSize, viewport: CGSize, mirrored: Bool
    ) -> CGRect? {
        guard
            [imageSize.width, imageSize.height, viewport.width, viewport.height].allSatisfy({
                $0.isFinite && $0 > 0
            }),
            [bounds.origin.x, bounds.origin.y, bounds.width, bounds.height].allSatisfy(\.isFinite),
            bounds.width > 0, bounds.height > 0
        else { return nil }
        let clipped = bounds.intersection(CGRect(x: 0, y: 0, width: 1, height: 1))

        guard !clipped.isNull, !clipped.isEmpty else {
            return nil
        }

        let scale = min(viewport.width / imageSize.width, viewport.height / imageSize.height)
        let width = imageSize.width * scale
        let height = imageSize.height * scale

        return CGRect(
            x: (viewport.width - width) / 2 + (mirrored ? 1 - clipped.maxX : clipped.minX) * width,
            y: (viewport.height - height) / 2 + (1 - clipped.maxY) * height,
            width: clipped.width * width, height: clipped.height * height
        )
    }
}
02.05

Build. Run. Try your model.

Build the local app, choose your detector, and press Start camera. Adjust the confidence threshold and mirror control. Stop capture when you are done.

Basics

CameraController.start()
Requests camera permission, starts capture and updates matching frames on MainActor.
CameraController.stop()
Cancels the active UI task and asks the camera service to stop that same run.
Keep in mind
  1. Symptom
    Old boxes return after stopping or changing models.
    Cause
    An earlier task can finish late; retained observers may keep resources alive.
    Remedy
    Cancel the old run, reject stale results, stop capture and remove observers. Release temporary models; profile repeated starts and stops with Instruments.
Terminalbash
./build-app.sh
open ".build/BadBob Playground.app"

02.06 / Best practices

Small details. Better examples.

  • Keep capture and inference off the UI thread. Bound the queue instead of collecting a backlog of old frames.
  • Treat image size, crop, orientation and coordinates as one contract. Test it with known boxes before trusting an overlay.
  • Handle denied permission, incompatible models and cancellation. Compare a few known predictions on the Mac you will use.