BadBob / Playground
OpenCV · ONNX · MIT
06.01

准备项目

安装列出的依赖项并运行测试。打开摄像头前,先试用一张本地图像。

基础说明

cargo test --locked
运行自动检查,不打开摄像头。
Terminalbash
unzip badbob-playground-rust.zip
cd badbob-playground-rust
cargo build --locked
cargo test --locked
cargo fmt --check
06.02

连接模型

在开启摄像头之前验证模型和类别标签。

基础说明

parse_labels(text)
按模型的类别顺序读取名称;拒绝空标签或重复标签。
OpenCvDetector::load(path, classes)
加载 ONNX 模型,并验证输入、输出和标签数量。
prepare_input(image)
构建 640×640 的 RGB 输入,保持原始比例,并填充剩余区域。
OpenCvDetector::detect(image)
预处理图像,运行模型并返回检测结果。
注意事项
症状
检测框偏移,或模型无法加载。
原因
预处理、输出格式或标签顺序不匹配。
解决方法
此适配器仅支持以下约定。Core ML 文件和其他 ONNX 输出格式需要不同的适配器。

此适配器仅支持以下约定。Core ML 文件和其他 ONNX 输出格式需要不同的适配器。

深入了解

坐标表示检测框的中心点和尺寸,单位为 640×640 输入图像中的像素。标签必须严格按照模型的类别顺序排列。

ONNXyaml
input:
  color: RGB
  dtype: float32
  shape: [1, 3, 640, 640]
  letterbox: 114
  scale: 1/255
output:
  dtype: float32
  shape: [1, 4+C, N]
  boxes: [cx, cy, width, height]
  class_scores: probabilities
  objectness: false
  embedded_nms: false
src/opencv_adapter.rsrust
use crate::{
    Result,
    geometry::{Detection, decode, letterbox},
    runtime::{Detector, FrameSource},
};
use opencv::{
    core::{self, Mat, Rect, Scalar, Size, Vector},
    dnn, imgproc,
    prelude::*,
    videoio,
};
use std::path::Path;

pub struct OpenCvDetector {
    network: dnn::Net,
    classes: usize,
    output_name: String,
}

impl OpenCvDetector {
    pub fn load(path: &Path, classes: usize) -> Result<Self> {
        if path.extension().and_then(|s| s.to_str()) != Some("onnx")
            || !path.is_file()
            || !(1..=1024).contains(&classes)
        {
            return Err("Provide an existing .onnx file and 1-1024 labels".into());
        }
        let mut network =
            dnn::read_net_from_onnx(path.to_str().ok_or("Model path must be UTF-8")?)?;
        network.set_preferable_backend(dnn::DNN_BACKEND_OPENCV)?;
        network.set_preferable_target(dnn::DNN_TARGET_CPU)?;
        let outputs = network.get_unconnected_out_layers_names()?;
        if outputs.len() != 1 {
            return Err("Expected exactly one detector output".into());
        }
        let mut detector = Self {
            network,
            classes,
            output_name: outputs.get(0)?,
        };
        // Validate the actual forward contract before the camera factory runs.
        let sample = Mat::new_rows_cols_with_default(640, 640, core::CV_8UC3, Scalar::all(114.))?;
        detector.detect(&sample)?;
        Ok(detector)
    }
}

impl Detector for OpenCvDetector {
    fn detect(&mut self, image: &Mat) -> Result<Vec<Detection>> {
        let (blob, t) = prepare_input(image)?;
        self.network.set_input(&blob, "", 1., Scalar::default())?;
        let output = self.network.forward_single(&self.output_name)?;
        let shape = output.mat_size();
        let count = validate_output_shape(&shape, output.typ(), self.classes)?;
        let contiguous = output.try_clone()?;
        decode(contiguous.data_typed::<f32>()?, self.classes, count, t)
    }
}

pub struct OpenCvCamera(videoio::VideoCapture);

impl OpenCvCamera {
    pub fn open(index: i32) -> Result<Self> {
        let capture = videoio::VideoCapture::new(index, videoio::CAP_ANY)?;
        if !capture.is_opened()? {
            return Err("Cannot open camera; check index, permissions and backend".into());
        }
        Ok(Self(capture))
    }
}

impl FrameSource for OpenCvCamera {
    fn next_frame(&mut self) -> Result<Mat> {
        let mut image = Mat::default();
        if !self.0.read(&mut image)? || image.empty() {
            return Err("Camera read failed or returned an empty image".into());
        }
        Ok(image)
    }
}

pub fn annotate(image: &mut Mat, detections: &[Detection], labels: &[String]) -> Result<()> {
    for detection in detections {
        let b = detection.bounds;
        let left = (b.x.floor() as i32).clamp(0, image.cols() - 1);
        let top = (b.y.floor() as i32).clamp(0, image.rows() - 1);
        let right = (b.x + b.width).ceil() as i32;
        let bottom = (b.y + b.height).ceil() as i32;
        let bounds = Rect::new(
            left,
            top,
            (right.min(image.cols()) - left).max(1),
            (bottom.min(image.rows()) - top).max(1),
        );
        imgproc::rectangle(
            image,
            bounds,
            Scalar::new(74., 220., 250., 0.),
            2,
            imgproc::LINE_8,
            0,
        )?;
        let text = format!("{} {:.2}", labels[detection.label], detection.score);
        imgproc::put_text(
            image,
            &text,
            core::Point::new(left, (top - 6).max(16)),
            imgproc::FONT_HERSHEY_SIMPLEX,
            0.5,
            Scalar::new(74., 220., 250., 0.),
            1,
            imgproc::LINE_AA,
            false,
        )?;
    }
    Ok(())
}

pub fn encode_png(image: &Mat) -> Result<Vec<u8>> {
    let mut bytes = Vector::<u8>::new();
    if !opencv::imgcodecs::imencode(".png", image, &mut bytes, &Vector::<i32>::new())? {
        return Err("Cannot encode PNG".into());
    }
    Ok(bytes.to_vec())
}

pub fn prepare_input(image: &Mat) -> Result<(Mat, crate::geometry::Letterbox)> {
    if image.empty() || image.typ() != core::CV_8UC3 {
        return Err("Expected a nonempty 8-bit BGR image".into());
    }
    let t = letterbox(image.cols(), image.rows())?;
    let mut resized = Mat::default();
    imgproc::resize(
        image,
        &mut resized,
        Size::new(t.width, t.height),
        0.,
        0.,
        imgproc::INTER_LINEAR,
    )?;
    let mut padded = Mat::new_rows_cols_with_default(640, 640, core::CV_8UC3, Scalar::all(114.))?;
    let mut region = Mat::roi_mut(&mut padded, Rect::new(t.left, t.top, t.width, t.height))?;
    resized.copy_to(&mut region)?;
    let blob = dnn::blob_from_image(
        &padded,
        1. / 255.,
        Size::new(640, 640),
        Scalar::default(),
        true,
        false,
        core::CV_32F,
    )?;
    Ok((blob, t))
}

pub fn validate_output_shape(shape: &[i32], datatype: i32, classes: usize) -> Result<usize> {
    if !(1..=1024).contains(&classes)
        || shape.len() != 3
        || datatype != core::CV_32F
        || shape[0] != 1
        || shape[1] != (4 + classes) as i32
        || !(1..=100_000).contains(&shape[2])
    {
        return Err("Expected float32[1,4+C,N], N<=100000, XYWH + class probabilities; no objectness/embedded NMS".into());
    }
    Ok(shape[2] as usize)
}
06.03

接入摄像头

采集和推理共用一个串行工作线程。界面只接收最新处理完成的图像及其检测结果。

基础说明

run_worker(source, detector, stopped, results)
在同一个工作线程中处理图像采集和推理。
Latest::take()
取出最新处理完成的图像及其检测结果,避免结果在队列中不断积累。
Worker::stop() · Worker::join()
停止接收新结果,并在当前任务结束后释放资源。
注意事项
症状
预览延迟,内存占用不断增加。
原因
摄像头生成图像的速度快于推理处理速度,导致无界队列不断增长。
解决方法
采集和推理共用一个串行工作线程。界面只接收最新处理完成的图像及其检测结果。
深入了解

请求停止,丢弃延迟返回的结果,并等待当前任务结束后再释放其资源。能否取消底层运行取决于平台支持。

src/runtime.rsrust
use crate::{Result, geometry::Detection};
use opencv::{core::Mat, prelude::*};
use std::sync::{
    Arc, Mutex,
    atomic::{AtomicBool, Ordering},
};
use std::thread::{self, JoinHandle};

pub trait FrameSource: Send {
    fn next_frame(&mut self) -> Result<Mat>;
}

pub trait Detector: Send {
    fn detect(&mut self, image: &Mat) -> Result<Vec<Detection>>;
}

pub struct FrameResult {
    pub image: Mat,
    pub detections: Vec<Detection>,
}

struct Slot<T> {
    value: Option<T>,
    closed: bool,
}

pub struct Latest<T>(Mutex<Slot<T>>);

impl<T> Default for Latest<T> {
    fn default() -> Self {
        Self(Mutex::new(Slot {
            value: None,
            closed: false,
        }))
    }
}

impl<T> Latest<T> {
    pub fn publish(&self, value: T) {
        let mut slot = self.0.lock().unwrap_or_else(|e| e.into_inner());
        if !slot.closed {
            slot.value = Some(value);
        }
    }

    pub fn take(&self) -> Option<T> {
        self.0
            .lock()
            .unwrap_or_else(|e| e.into_inner())
            .value
            .take()
    }

    pub fn close(&self) {
        let mut slot = self.0.lock().unwrap_or_else(|e| e.into_inner());
        slot.closed = true;
        slot.value = None;
    }
}

pub struct Worker {
    stop_requested: Arc<AtomicBool>,
    done: Arc<AtomicBool>,
    results: Arc<Latest<Result<FrameResult>>>,
    thread: Option<JoinHandle<()>>,
}

impl Worker {
    pub fn start<S, D>(source: S, detector: D) -> Self
    where
        S: FnOnce() -> Result<Box<dyn FrameSource>> + Send + 'static,
        D: FnOnce() -> Result<Box<dyn Detector>> + Send + 'static,
    {
        let stop_requested = Arc::new(AtomicBool::new(false));
        let done = Arc::new(AtomicBool::new(false));
        let results = Arc::new(Latest::default());
        let stopped = stop_requested.clone();
        let finished = done.clone();
        let slot = results.clone();
        let thread = thread::spawn(move || {
            let _completion = Completion(finished);
            if let Err(error) = run_worker(source, detector, &stopped, &slot)
                && !stopped.load(Ordering::Acquire)
            {
                slot.publish(Err(error));
            }
        });
        Self {
            stop_requested,
            done,
            results,
            thread: Some(thread),
        }
    }

    pub fn take(&self) -> Option<Result<FrameResult>> {
        self.results.take()
    }

    pub fn is_done(&self) -> bool {
        self.done.load(Ordering::Acquire)
    }

    pub fn stop(&self) {
        self.stop_requested.store(true, Ordering::Release);
        self.results.close();
    }

    pub fn join(&mut self) -> Result<()> {
        if let Some(thread) = self.thread.take() {
            thread
                .join()
                .map_err(|_| "Capture/inference worker panicked")?;
        }
        Ok(())
    }
}

impl Drop for Worker {
    fn drop(&mut self) {
        self.stop();
        let _ = self.join();
    }
}

fn run_worker<S, D>(
    source: S,
    detector: D,
    stopped: &AtomicBool,
    results: &Latest<Result<FrameResult>>,
) -> Result<()>
where
    S: FnOnce() -> Result<Box<dyn FrameSource>>,
    D: FnOnce() -> Result<Box<dyn Detector>>,
{
    let mut model = detector()?;
    if stopped.load(Ordering::Acquire) {
        return Ok(());
    }
    let mut frames = source()?;
    while !stopped.load(Ordering::Acquire) {
        let image = frames.next_frame()?;
        if stopped.load(Ordering::Acquire) {
            break;
        }
        if image.empty() {
            return Err("Camera returned an empty frame".into());
        }
        let detections = model.detect(&image)?;
        if stopped.load(Ordering::Acquire) {
            break;
        }
        results.publish(Ok(FrameResult { image, detections }));
    }
    Ok(())
}

struct Completion(Arc<AtomicBool>);

impl Drop for Completion {
    fn drop(&mut self) {
        self.0.store(true, Ordering::Release);
    }
}
06.04

正确定位预测结果

还原 letterbox 缩放和填充,并在绘制前将检测框裁剪到图像边界内。

基础说明

letterbox(width, height) · unproject(box, transform)
保持图像宽高比,并将检测框映射回原始图像。
decode(data, classes, count, transform)
验证预测结果,移除同一类别中重叠的检测框。
注意事项

将图像尺寸、裁剪、方向和坐标作为统一约定。先用已知检测框测试,再信任画面标注。

src/geometry.rsrust
use crate::Result;

#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Box2 {
    pub x: f64,
    pub y: f64,
    pub width: f64,
    pub height: f64,
}

#[derive(Clone, Debug)]
pub struct Detection {
    pub bounds: Box2,
    pub label: usize,
    pub score: f32,
}

#[derive(Clone, Copy, Debug)]
pub struct Letterbox {
    pub original_width: i32,
    pub original_height: i32,
    pub width: i32,
    pub height: i32,
    pub left: i32,
    pub top: i32,
}

pub fn letterbox(width: i32, height: i32) -> Result<Letterbox> {
    if width <= 0 || height <= 0 {
        return Err("Image dimensions must be positive".into());
    }

    let ratio = (640. / f64::from(width)).min(640. / f64::from(height));
    let resized_width = (f64::from(width) * ratio).round().max(1.) as i32;
    let resized_height = (f64::from(height) * ratio).round().max(1.) as i32;
    Ok(Letterbox {
        original_width: width,
        original_height: height,
        width: resized_width,
        height: resized_height,
        left: (640 - resized_width) / 2,
        top: (640 - resized_height) / 2,
    })
}

pub fn unproject(b: Box2, t: Letterbox) -> Result<Box2> {
    if t.original_width <= 0
        || t.original_height <= 0
        || t.width <= 0
        || t.height <= 0
        || t.width > 640
        || t.height > 640
        || t.left < 0
        || t.top < 0
        || t.left > 640 - t.width
        || t.top > 640 - t.height
    {
        return Err("Invalid letterbox geometry".into());
    }
    if ![b.x, b.y, b.width, b.height, b.x + b.width, b.y + b.height]
        .iter()
        .all(|x| x.is_finite())
        || b.width < 0.
        || b.height < 0.
    {
        return Err("Box geometry must be finite and nonnegative".into());
    }

    let x = |v: f64| {
        ((v - f64::from(t.left)) / f64::from(t.width)).clamp(0., 1.) * f64::from(t.original_width)
    };
    let y = |v: f64| {
        ((v - f64::from(t.top)) / f64::from(t.height)).clamp(0., 1.) * f64::from(t.original_height)
    };
    let left = x(b.x);
    let top = y(b.y);
    Ok(Box2 {
        x: left,
        y: top,
        width: x(b.x + b.width) - left,
        height: y(b.y + b.height) - top,
    })
}

fn intersection_over_union(a: Box2, b: Box2) -> f64 {
    let width = (a.x + a.width).min(b.x + b.width) - a.x.max(b.x);
    let height = (a.y + a.height).min(b.y + b.height) - a.y.max(b.y);
    let intersection = width.max(0.) * height.max(0.);
    let area = a.width * a.height + b.width * b.height - intersection;
    if area > 0. { intersection / area } else { 0. }
}

pub fn decode(data: &[f32], classes: usize, count: usize, t: Letterbox) -> Result<Vec<Detection>> {
    if !(1..=1024).contains(&classes)
        || !(1..=100_000).contains(&count)
        || data.len() != (4 + classes) * count
    {
        return Err("Expected float32[1,4+C,N], C<=1024, N<=100000".into());
    }

    let mut candidates = Vec::new();
    for i in 0..count {
        let cx = f64::from(data[i]);
        let cy = f64::from(data[count + i]);
        let width = f64::from(data[2 * count + i]);
        let height = f64::from(data[3 * count + i]);
        // Validate even candidates below the confidence threshold.
        let bounds = unproject(
            Box2 {
                x: cx - width / 2.,
                y: cy - height / 2.,
                width,
                height,
            },
            t,
        )?;
        let mut score = 0.;
        let mut label = 0;
        for c in 0..classes {
            let probability = data[(4 + c) * count + i];
            if !probability.is_finite() || !(0.0..=1.0).contains(&probability) {
                return Err("Expected probabilities in [0,1], not logits/objectness".into());
            }
            if probability > score {
                score = probability;
                label = c;
            }
        }
        if score >= 0.25 && bounds.width > 0. && bounds.height > 0. {
            candidates.push(Detection {
                bounds,
                label,
                score,
            });
        }
    }

    candidates.sort_by(|a, b| b.score.total_cmp(&a.score));
    candidates.truncate(3000);
    let mut selected: Vec<Detection> = Vec::new();
    for candidate in candidates {
        let suppressed = selected.iter().any(|previous| {
            previous.label == candidate.label
                && intersection_over_union(previous.bounds, candidate.bounds) > 0.45
        });
        if suppressed {
            continue;
        }
        selected.push(candidate);
        if selected.len() == 300 {
            break;
        }
    }
    Ok(selected)
}
06.05

构建、运行、测试模型。

明确选择摄像头,或离线处理图像。按 Q 或 Escape 停止。

基础说明

parse_options()
验证参数,并且只选择一种输入:图像或摄像头。
run()
验证所选来源,并将处理任务交给图像或摄像头处理流程。
注意事项
症状
停止所需时间超出预期。
原因
原生图像采集或推理调用仍在运行。
解决方法
请求停止,丢弃延迟返回的结果,并等待当前任务结束后再释放其资源。能否取消底层运行取决于平台支持。
Terminalbash
cargo run --locked -- --model detector.onnx --labels labels.txt --image frame.png --output result.png

cargo run --locked -- --model detector.onnx --labels labels.txt --camera 0

06.06 / 最佳实践

做好细节,写好示例。

  • 让采集和推理远离界面线程。限制队列长度,不要堆积旧帧。
  • 将图像尺寸、裁剪、方向和坐标作为统一约定。先用已知检测框测试,再信任画面标注。
  • 请求停止,丢弃延迟返回的结果,并等待当前任务结束后再释放其资源。能否取消底层运行取决于平台支持。