BadBob / Playground
OpenCV · ONNX · MIT
06.01

準備專案

安裝列出嘅相依套件,再執行測試。開啟相機之前,先試一張本機圖片。

基礎說明

cargo test --locked
執行自動檢查,唔會開啟鏡頭。
Terminalbash
unzip badbob-playground-rust.zip
cd badbob-playground-rust
cargo build --locked
cargo test --locked
cargo fmt --check
06.02

連接模型

開啟鏡頭之前,先驗證模型同類別標籤。

基礎說明

parse_labels(text)
按模型嘅類別次序讀取名稱;拒絕空白或重複標籤。
OpenCvDetector::load(path, classes)
載入 ONNX 模型,並驗證輸入、輸出同標籤數量。
prepare_input(image)
建立 640×640 嘅 RGB 輸入,保留比例,再填補剩低嘅範圍。
OpenCvDetector::detect(image)
預處理影像,執行模型並傳回偵測結果。
要留意
症狀
偵測框移位,或者模型載入失敗。
原因
前處理、輸出格式或者標籤次序唔配合。
處理方法
呢個介接器只支援以下約定。Core ML 檔案同其他 ONNX 輸出格式需要另一個介接器。

呢個介接器只支援以下約定。Core ML 檔案同其他 ONNX 輸出格式需要另一個介接器。

深入了解

座標表示偵測框嘅中心點同尺寸,單位係 640×640 輸入影像嘅像素。標籤必須嚴格跟返模型嘅類別次序排列。

ONNXyaml
input:
  color: RGB
  dtype: float32
  shape: [1, 3, 640, 640]
  letterbox: 114
  scale: 1/255
output:
  dtype: float32
  shape: [1, 4+C, N]
  boxes: [cx, cy, width, height]
  class_scores: probabilities
  objectness: false
  embedded_nms: false
src/opencv_adapter.rsrust
use crate::{
    Result,
    geometry::{Detection, decode, letterbox},
    runtime::{Detector, FrameSource},
};
use opencv::{
    core::{self, Mat, Rect, Scalar, Size, Vector},
    dnn, imgproc,
    prelude::*,
    videoio,
};
use std::path::Path;

pub struct OpenCvDetector {
    network: dnn::Net,
    classes: usize,
    output_name: String,
}

impl OpenCvDetector {
    pub fn load(path: &Path, classes: usize) -> Result<Self> {
        if path.extension().and_then(|s| s.to_str()) != Some("onnx")
            || !path.is_file()
            || !(1..=1024).contains(&classes)
        {
            return Err("Provide an existing .onnx file and 1-1024 labels".into());
        }
        let mut network =
            dnn::read_net_from_onnx(path.to_str().ok_or("Model path must be UTF-8")?)?;
        network.set_preferable_backend(dnn::DNN_BACKEND_OPENCV)?;
        network.set_preferable_target(dnn::DNN_TARGET_CPU)?;
        let outputs = network.get_unconnected_out_layers_names()?;
        if outputs.len() != 1 {
            return Err("Expected exactly one detector output".into());
        }
        let mut detector = Self {
            network,
            classes,
            output_name: outputs.get(0)?,
        };
        // Validate the actual forward contract before the camera factory runs.
        let sample = Mat::new_rows_cols_with_default(640, 640, core::CV_8UC3, Scalar::all(114.))?;
        detector.detect(&sample)?;
        Ok(detector)
    }
}

impl Detector for OpenCvDetector {
    fn detect(&mut self, image: &Mat) -> Result<Vec<Detection>> {
        let (blob, t) = prepare_input(image)?;
        self.network.set_input(&blob, "", 1., Scalar::default())?;
        let output = self.network.forward_single(&self.output_name)?;
        let shape = output.mat_size();
        let count = validate_output_shape(&shape, output.typ(), self.classes)?;
        let contiguous = output.try_clone()?;
        decode(contiguous.data_typed::<f32>()?, self.classes, count, t)
    }
}

pub struct OpenCvCamera(videoio::VideoCapture);

impl OpenCvCamera {
    pub fn open(index: i32) -> Result<Self> {
        let capture = videoio::VideoCapture::new(index, videoio::CAP_ANY)?;
        if !capture.is_opened()? {
            return Err("Cannot open camera; check index, permissions and backend".into());
        }
        Ok(Self(capture))
    }
}

impl FrameSource for OpenCvCamera {
    fn next_frame(&mut self) -> Result<Mat> {
        let mut image = Mat::default();
        if !self.0.read(&mut image)? || image.empty() {
            return Err("Camera read failed or returned an empty image".into());
        }
        Ok(image)
    }
}

pub fn annotate(image: &mut Mat, detections: &[Detection], labels: &[String]) -> Result<()> {
    for detection in detections {
        let b = detection.bounds;
        let left = (b.x.floor() as i32).clamp(0, image.cols() - 1);
        let top = (b.y.floor() as i32).clamp(0, image.rows() - 1);
        let right = (b.x + b.width).ceil() as i32;
        let bottom = (b.y + b.height).ceil() as i32;
        let bounds = Rect::new(
            left,
            top,
            (right.min(image.cols()) - left).max(1),
            (bottom.min(image.rows()) - top).max(1),
        );
        imgproc::rectangle(
            image,
            bounds,
            Scalar::new(74., 220., 250., 0.),
            2,
            imgproc::LINE_8,
            0,
        )?;
        let text = format!("{} {:.2}", labels[detection.label], detection.score);
        imgproc::put_text(
            image,
            &text,
            core::Point::new(left, (top - 6).max(16)),
            imgproc::FONT_HERSHEY_SIMPLEX,
            0.5,
            Scalar::new(74., 220., 250., 0.),
            1,
            imgproc::LINE_AA,
            false,
        )?;
    }
    Ok(())
}

pub fn encode_png(image: &Mat) -> Result<Vec<u8>> {
    let mut bytes = Vector::<u8>::new();
    if !opencv::imgcodecs::imencode(".png", image, &mut bytes, &Vector::<i32>::new())? {
        return Err("Cannot encode PNG".into());
    }
    Ok(bytes.to_vec())
}

pub fn prepare_input(image: &Mat) -> Result<(Mat, crate::geometry::Letterbox)> {
    if image.empty() || image.typ() != core::CV_8UC3 {
        return Err("Expected a nonempty 8-bit BGR image".into());
    }
    let t = letterbox(image.cols(), image.rows())?;
    let mut resized = Mat::default();
    imgproc::resize(
        image,
        &mut resized,
        Size::new(t.width, t.height),
        0.,
        0.,
        imgproc::INTER_LINEAR,
    )?;
    let mut padded = Mat::new_rows_cols_with_default(640, 640, core::CV_8UC3, Scalar::all(114.))?;
    let mut region = Mat::roi_mut(&mut padded, Rect::new(t.left, t.top, t.width, t.height))?;
    resized.copy_to(&mut region)?;
    let blob = dnn::blob_from_image(
        &padded,
        1. / 255.,
        Size::new(640, 640),
        Scalar::default(),
        true,
        false,
        core::CV_32F,
    )?;
    Ok((blob, t))
}

pub fn validate_output_shape(shape: &[i32], datatype: i32, classes: usize) -> Result<usize> {
    if !(1..=1024).contains(&classes)
        || shape.len() != 3
        || datatype != core::CV_32F
        || shape[0] != 1
        || shape[1] != (4 + classes) as i32
        || !(1..=100_000).contains(&shape[2])
    {
        return Err("Expected float32[1,4+C,N], N<=100000, XYWH + class probabilities; no objectness/embedded NMS".into());
    }
    Ok(shape[2] as usize)
}
06.03

接入相機

擷取同推論共用一條循序工作執行緒。介面只會接收最新處理好嘅影像同相應偵測結果。

基礎說明

run_worker(source, detector, stopped, results)
喺同一條工作執行緒處理影像擷取同推論。
Latest::take()
取出最新處理好嘅影像同對應偵測結果,避免結果喺佇列不斷累積。
Worker::stop() · Worker::join()
停止接收新結果,等進行中嘅工作完成之後釋放資源。
要留意
症狀
預覽愈來愈慢,記憶體用量不斷增加。
原因
相機產生影像快過推論處理嘅速度,令冇上限嘅佇列不斷累積。
處理方法
擷取同推論共用一條循序工作執行緒。介面只會接收最新處理好嘅影像同相應偵測結果。
深入了解

要求停止、捨棄遲到嘅結果,等進行中嘅工作完成之後先釋放資源。可唔可以取消底層執行,要睇平台支唔支援。

src/runtime.rsrust
use crate::{Result, geometry::Detection};
use opencv::{core::Mat, prelude::*};
use std::sync::{
    Arc, Mutex,
    atomic::{AtomicBool, Ordering},
};
use std::thread::{self, JoinHandle};

pub trait FrameSource: Send {
    fn next_frame(&mut self) -> Result<Mat>;
}

pub trait Detector: Send {
    fn detect(&mut self, image: &Mat) -> Result<Vec<Detection>>;
}

pub struct FrameResult {
    pub image: Mat,
    pub detections: Vec<Detection>,
}

struct Slot<T> {
    value: Option<T>,
    closed: bool,
}

pub struct Latest<T>(Mutex<Slot<T>>);

impl<T> Default for Latest<T> {
    fn default() -> Self {
        Self(Mutex::new(Slot {
            value: None,
            closed: false,
        }))
    }
}

impl<T> Latest<T> {
    pub fn publish(&self, value: T) {
        let mut slot = self.0.lock().unwrap_or_else(|e| e.into_inner());
        if !slot.closed {
            slot.value = Some(value);
        }
    }

    pub fn take(&self) -> Option<T> {
        self.0
            .lock()
            .unwrap_or_else(|e| e.into_inner())
            .value
            .take()
    }

    pub fn close(&self) {
        let mut slot = self.0.lock().unwrap_or_else(|e| e.into_inner());
        slot.closed = true;
        slot.value = None;
    }
}

pub struct Worker {
    stop_requested: Arc<AtomicBool>,
    done: Arc<AtomicBool>,
    results: Arc<Latest<Result<FrameResult>>>,
    thread: Option<JoinHandle<()>>,
}

impl Worker {
    pub fn start<S, D>(source: S, detector: D) -> Self
    where
        S: FnOnce() -> Result<Box<dyn FrameSource>> + Send + 'static,
        D: FnOnce() -> Result<Box<dyn Detector>> + Send + 'static,
    {
        let stop_requested = Arc::new(AtomicBool::new(false));
        let done = Arc::new(AtomicBool::new(false));
        let results = Arc::new(Latest::default());
        let stopped = stop_requested.clone();
        let finished = done.clone();
        let slot = results.clone();
        let thread = thread::spawn(move || {
            let _completion = Completion(finished);
            if let Err(error) = run_worker(source, detector, &stopped, &slot)
                && !stopped.load(Ordering::Acquire)
            {
                slot.publish(Err(error));
            }
        });
        Self {
            stop_requested,
            done,
            results,
            thread: Some(thread),
        }
    }

    pub fn take(&self) -> Option<Result<FrameResult>> {
        self.results.take()
    }

    pub fn is_done(&self) -> bool {
        self.done.load(Ordering::Acquire)
    }

    pub fn stop(&self) {
        self.stop_requested.store(true, Ordering::Release);
        self.results.close();
    }

    pub fn join(&mut self) -> Result<()> {
        if let Some(thread) = self.thread.take() {
            thread
                .join()
                .map_err(|_| "Capture/inference worker panicked")?;
        }
        Ok(())
    }
}

impl Drop for Worker {
    fn drop(&mut self) {
        self.stop();
        let _ = self.join();
    }
}

fn run_worker<S, D>(
    source: S,
    detector: D,
    stopped: &AtomicBool,
    results: &Latest<Result<FrameResult>>,
) -> Result<()>
where
    S: FnOnce() -> Result<Box<dyn FrameSource>>,
    D: FnOnce() -> Result<Box<dyn Detector>>,
{
    let mut model = detector()?;
    if stopped.load(Ordering::Acquire) {
        return Ok(());
    }
    let mut frames = source()?;
    while !stopped.load(Ordering::Acquire) {
        let image = frames.next_frame()?;
        if stopped.load(Ordering::Acquire) {
            break;
        }
        if image.empty() {
            return Err("Camera returned an empty frame".into());
        }
        let detections = model.detect(&image)?;
        if stopped.load(Ordering::Acquire) {
            break;
        }
        results.publish(Ok(FrameResult { image, detections }));
    }
    Ok(())
}

struct Completion(Arc<AtomicBool>);

impl Drop for Completion {
    fn drop(&mut self) {
        self.0.store(true, Ordering::Release);
    }
}
06.04

放好預測標記

還原 letterbox 縮放同填補,再將偵測框裁切到影像範圍內先繪畫。

基礎說明

letterbox(width, height) · unproject(box, transform)
保持影像長寬比,將偵測框映射返原始影像。
decode(data, classes, count, transform)
驗證預測結果,移除同一類別入面重疊嘅偵測框。
要留意

影像尺寸、裁切、方向同座標要有同一套約定。先用已知偵測框測試,再信任畫面標記。

src/geometry.rsrust
use crate::Result;

#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Box2 {
    pub x: f64,
    pub y: f64,
    pub width: f64,
    pub height: f64,
}

#[derive(Clone, Debug)]
pub struct Detection {
    pub bounds: Box2,
    pub label: usize,
    pub score: f32,
}

#[derive(Clone, Copy, Debug)]
pub struct Letterbox {
    pub original_width: i32,
    pub original_height: i32,
    pub width: i32,
    pub height: i32,
    pub left: i32,
    pub top: i32,
}

pub fn letterbox(width: i32, height: i32) -> Result<Letterbox> {
    if width <= 0 || height <= 0 {
        return Err("Image dimensions must be positive".into());
    }

    let ratio = (640. / f64::from(width)).min(640. / f64::from(height));
    let resized_width = (f64::from(width) * ratio).round().max(1.) as i32;
    let resized_height = (f64::from(height) * ratio).round().max(1.) as i32;
    Ok(Letterbox {
        original_width: width,
        original_height: height,
        width: resized_width,
        height: resized_height,
        left: (640 - resized_width) / 2,
        top: (640 - resized_height) / 2,
    })
}

pub fn unproject(b: Box2, t: Letterbox) -> Result<Box2> {
    if t.original_width <= 0
        || t.original_height <= 0
        || t.width <= 0
        || t.height <= 0
        || t.width > 640
        || t.height > 640
        || t.left < 0
        || t.top < 0
        || t.left > 640 - t.width
        || t.top > 640 - t.height
    {
        return Err("Invalid letterbox geometry".into());
    }
    if ![b.x, b.y, b.width, b.height, b.x + b.width, b.y + b.height]
        .iter()
        .all(|x| x.is_finite())
        || b.width < 0.
        || b.height < 0.
    {
        return Err("Box geometry must be finite and nonnegative".into());
    }

    let x = |v: f64| {
        ((v - f64::from(t.left)) / f64::from(t.width)).clamp(0., 1.) * f64::from(t.original_width)
    };
    let y = |v: f64| {
        ((v - f64::from(t.top)) / f64::from(t.height)).clamp(0., 1.) * f64::from(t.original_height)
    };
    let left = x(b.x);
    let top = y(b.y);
    Ok(Box2 {
        x: left,
        y: top,
        width: x(b.x + b.width) - left,
        height: y(b.y + b.height) - top,
    })
}

fn intersection_over_union(a: Box2, b: Box2) -> f64 {
    let width = (a.x + a.width).min(b.x + b.width) - a.x.max(b.x);
    let height = (a.y + a.height).min(b.y + b.height) - a.y.max(b.y);
    let intersection = width.max(0.) * height.max(0.);
    let area = a.width * a.height + b.width * b.height - intersection;
    if area > 0. { intersection / area } else { 0. }
}

pub fn decode(data: &[f32], classes: usize, count: usize, t: Letterbox) -> Result<Vec<Detection>> {
    if !(1..=1024).contains(&classes)
        || !(1..=100_000).contains(&count)
        || data.len() != (4 + classes) * count
    {
        return Err("Expected float32[1,4+C,N], C<=1024, N<=100000".into());
    }

    let mut candidates = Vec::new();
    for i in 0..count {
        let cx = f64::from(data[i]);
        let cy = f64::from(data[count + i]);
        let width = f64::from(data[2 * count + i]);
        let height = f64::from(data[3 * count + i]);
        // Validate even candidates below the confidence threshold.
        let bounds = unproject(
            Box2 {
                x: cx - width / 2.,
                y: cy - height / 2.,
                width,
                height,
            },
            t,
        )?;
        let mut score = 0.;
        let mut label = 0;
        for c in 0..classes {
            let probability = data[(4 + c) * count + i];
            if !probability.is_finite() || !(0.0..=1.0).contains(&probability) {
                return Err("Expected probabilities in [0,1], not logits/objectness".into());
            }
            if probability > score {
                score = probability;
                label = c;
            }
        }
        if score >= 0.25 && bounds.width > 0. && bounds.height > 0. {
            candidates.push(Detection {
                bounds,
                label,
                score,
            });
        }
    }

    candidates.sort_by(|a, b| b.score.total_cmp(&a.score));
    candidates.truncate(3000);
    let mut selected: Vec<Detection> = Vec::new();
    for candidate in candidates {
        let suppressed = selected.iter().any(|previous| {
            previous.label == candidate.label
                && intersection_over_union(previous.bounds, candidate.bounds) > 0.45
        });
        if suppressed {
            continue;
        }
        selected.push(candidate);
        if selected.len() == 300 {
            break;
        }
    }
    Ok(selected)
}
06.05

建置、執行、試模型。

明確選擇鏡頭,或者離線處理影像。撳 Q 或 Escape 停止。

基礎說明

parse_options()
驗證參數,並且只揀一種輸入:影像或鏡頭。
run()
驗證揀咗嘅來源,再交由圖片或相機流程處理。
要留意
症狀
停止所需時間比預期長。
原因
原生畫面擷取或推論呼叫仲執行緊。
處理方法
要求停止、捨棄遲到嘅結果,等進行中嘅工作完成之後先釋放資源。可唔可以取消底層執行,要睇平台支唔支援。
Terminalbash
cargo run --locked -- --model detector.onnx --labels labels.txt --image frame.png --output result.png

cargo run --locked -- --model detector.onnx --labels labels.txt --camera 0

06.06 / 最佳實踐

做好細節,範例更好。

  • 擷取同推論唔好放喺介面執行緒。限制佇列長度,避免積埋舊影格。
  • 影像尺寸、裁切、方向同座標要有同一套約定。先用已知偵測框測試,再信任畫面標記。
  • 要求停止、捨棄遲到嘅結果,等進行中嘅工作完成之後先釋放資源。可唔可以取消底層執行,要睇平台支唔支援。