06 / Rust
Rust. From model to camera.
Run your ONNX detector locally with live annotations and bounded frame processing.
These examples focus on objects and animals in controlled settings. They do not cover people surveillance, identification or behavioral analysis.
Prepare your project
Install the listed dependencies and run the tests. Try an offline image before opening the camera.
Basics
cargo test --locked- Runs automated checks without opening the camera.
unzip badbob-playground-rust.zip
cd badbob-playground-rust
cargo build --locked
cargo test --locked
cargo fmt --checkConnect your model
Validate the model and labels before opening a camera.
Basics
parse_labels(text)- Reads class names in model order; rejects empty or duplicate labels.
OpenCvDetector::load(path, classes)- Loads the ONNX model and validates its input, output and label count.
prepare_input(image)- Builds a 640×640 RGB input, preserving proportions and padding the remaining area.
OpenCvDetector::detect(image)- Prepares the image, runs the model and returns detections.
Keep in mind
- Symptom
- Boxes drift or the model fails to load.
- Cause
- Preprocessing, output layout or label order does not match.
- Remedy
- This adapter supports only the contract below. Core ML files and other ONNX layouts need a different adapter.
This adapter supports only the contract below. Core ML files and other ONNX layouts need a different adapter.
In depth
Coordinates describe box centers and dimensions in the 640×640 input pixels. Labels follow the exact model class order.
input:
color: RGB
dtype: float32
shape: [1, 3, 640, 640]
letterbox: 114
scale: 1/255
output:
dtype: float32
shape: [1, 4+C, N]
boxes: [cx, cy, width, height]
class_scores: probabilities
objectness: false
embedded_nms: falseuse crate::{
Result,
geometry::{Detection, decode, letterbox},
runtime::{Detector, FrameSource},
};
use opencv::{
core::{self, Mat, Rect, Scalar, Size, Vector},
dnn, imgproc,
prelude::*,
videoio,
};
use std::path::Path;
pub struct OpenCvDetector {
network: dnn::Net,
classes: usize,
output_name: String,
}
impl OpenCvDetector {
pub fn load(path: &Path, classes: usize) -> Result<Self> {
if path.extension().and_then(|s| s.to_str()) != Some("onnx")
|| !path.is_file()
|| !(1..=1024).contains(&classes)
{
return Err("Provide an existing .onnx file and 1-1024 labels".into());
}
let mut network =
dnn::read_net_from_onnx(path.to_str().ok_or("Model path must be UTF-8")?)?;
network.set_preferable_backend(dnn::DNN_BACKEND_OPENCV)?;
network.set_preferable_target(dnn::DNN_TARGET_CPU)?;
let outputs = network.get_unconnected_out_layers_names()?;
if outputs.len() != 1 {
return Err("Expected exactly one detector output".into());
}
let mut detector = Self {
network,
classes,
output_name: outputs.get(0)?,
};
// Validate the actual forward contract before the camera factory runs.
let sample = Mat::new_rows_cols_with_default(640, 640, core::CV_8UC3, Scalar::all(114.))?;
detector.detect(&sample)?;
Ok(detector)
}
}
impl Detector for OpenCvDetector {
fn detect(&mut self, image: &Mat) -> Result<Vec<Detection>> {
let (blob, t) = prepare_input(image)?;
self.network.set_input(&blob, "", 1., Scalar::default())?;
let output = self.network.forward_single(&self.output_name)?;
let shape = output.mat_size();
let count = validate_output_shape(&shape, output.typ(), self.classes)?;
let contiguous = output.try_clone()?;
decode(contiguous.data_typed::<f32>()?, self.classes, count, t)
}
}
pub struct OpenCvCamera(videoio::VideoCapture);
impl OpenCvCamera {
pub fn open(index: i32) -> Result<Self> {
let capture = videoio::VideoCapture::new(index, videoio::CAP_ANY)?;
if !capture.is_opened()? {
return Err("Cannot open camera; check index, permissions and backend".into());
}
Ok(Self(capture))
}
}
impl FrameSource for OpenCvCamera {
fn next_frame(&mut self) -> Result<Mat> {
let mut image = Mat::default();
if !self.0.read(&mut image)? || image.empty() {
return Err("Camera read failed or returned an empty image".into());
}
Ok(image)
}
}
pub fn annotate(image: &mut Mat, detections: &[Detection], labels: &[String]) -> Result<()> {
for detection in detections {
let b = detection.bounds;
let left = (b.x.floor() as i32).clamp(0, image.cols() - 1);
let top = (b.y.floor() as i32).clamp(0, image.rows() - 1);
let right = (b.x + b.width).ceil() as i32;
let bottom = (b.y + b.height).ceil() as i32;
let bounds = Rect::new(
left,
top,
(right.min(image.cols()) - left).max(1),
(bottom.min(image.rows()) - top).max(1),
);
imgproc::rectangle(
image,
bounds,
Scalar::new(74., 220., 250., 0.),
2,
imgproc::LINE_8,
0,
)?;
let text = format!("{} {:.2}", labels[detection.label], detection.score);
imgproc::put_text(
image,
&text,
core::Point::new(left, (top - 6).max(16)),
imgproc::FONT_HERSHEY_SIMPLEX,
0.5,
Scalar::new(74., 220., 250., 0.),
1,
imgproc::LINE_AA,
false,
)?;
}
Ok(())
}
pub fn encode_png(image: &Mat) -> Result<Vec<u8>> {
let mut bytes = Vector::<u8>::new();
if !opencv::imgcodecs::imencode(".png", image, &mut bytes, &Vector::<i32>::new())? {
return Err("Cannot encode PNG".into());
}
Ok(bytes.to_vec())
}
pub fn prepare_input(image: &Mat) -> Result<(Mat, crate::geometry::Letterbox)> {
if image.empty() || image.typ() != core::CV_8UC3 {
return Err("Expected a nonempty 8-bit BGR image".into());
}
let t = letterbox(image.cols(), image.rows())?;
let mut resized = Mat::default();
imgproc::resize(
image,
&mut resized,
Size::new(t.width, t.height),
0.,
0.,
imgproc::INTER_LINEAR,
)?;
let mut padded = Mat::new_rows_cols_with_default(640, 640, core::CV_8UC3, Scalar::all(114.))?;
let mut region = Mat::roi_mut(&mut padded, Rect::new(t.left, t.top, t.width, t.height))?;
resized.copy_to(&mut region)?;
let blob = dnn::blob_from_image(
&padded,
1. / 255.,
Size::new(640, 640),
Scalar::default(),
true,
false,
core::CV_32F,
)?;
Ok((blob, t))
}
pub fn validate_output_shape(shape: &[i32], datatype: i32, classes: usize) -> Result<usize> {
if !(1..=1024).contains(&classes)
|| shape.len() != 3
|| datatype != core::CV_32F
|| shape[0] != 1
|| shape[1] != (4 + classes) as i32
|| !(1..=100_000).contains(&shape[2])
{
return Err("Expected float32[1,4+C,N], N<=100000, XYWH + class probabilities; no objectness/embedded NMS".into());
}
Ok(shape[2] as usize)
}Bring in the camera
Capture and inference share one serial worker. The UI receives only the latest completed image and its detections.
Basics
run_worker(source, detector, stopped, results)- Handles capture and inference on one worker thread.
Latest::take()- Takes the latest completed image and its detections; results cannot build up in a queue.
Worker::stop() · Worker::join()- Stops accepting new results and releases resources after active work finishes.
Keep in mind
- Symptom
- The preview lags and memory use grows.
- Cause
- The camera produces frames faster than inference handles them, so an unbounded queue builds up.
- Remedy
- Capture and inference share one serial worker. The UI receives only the latest completed image and its detections.
In depth
Request stop, reject late results and wait for active work before releasing its resources. Backend cancellation depends on platform support.
use crate::{Result, geometry::Detection};
use opencv::{core::Mat, prelude::*};
use std::sync::{
Arc, Mutex,
atomic::{AtomicBool, Ordering},
};
use std::thread::{self, JoinHandle};
pub trait FrameSource: Send {
fn next_frame(&mut self) -> Result<Mat>;
}
pub trait Detector: Send {
fn detect(&mut self, image: &Mat) -> Result<Vec<Detection>>;
}
pub struct FrameResult {
pub image: Mat,
pub detections: Vec<Detection>,
}
struct Slot<T> {
value: Option<T>,
closed: bool,
}
pub struct Latest<T>(Mutex<Slot<T>>);
impl<T> Default for Latest<T> {
fn default() -> Self {
Self(Mutex::new(Slot {
value: None,
closed: false,
}))
}
}
impl<T> Latest<T> {
pub fn publish(&self, value: T) {
let mut slot = self.0.lock().unwrap_or_else(|e| e.into_inner());
if !slot.closed {
slot.value = Some(value);
}
}
pub fn take(&self) -> Option<T> {
self.0
.lock()
.unwrap_or_else(|e| e.into_inner())
.value
.take()
}
pub fn close(&self) {
let mut slot = self.0.lock().unwrap_or_else(|e| e.into_inner());
slot.closed = true;
slot.value = None;
}
}
pub struct Worker {
stop_requested: Arc<AtomicBool>,
done: Arc<AtomicBool>,
results: Arc<Latest<Result<FrameResult>>>,
thread: Option<JoinHandle<()>>,
}
impl Worker {
pub fn start<S, D>(source: S, detector: D) -> Self
where
S: FnOnce() -> Result<Box<dyn FrameSource>> + Send + 'static,
D: FnOnce() -> Result<Box<dyn Detector>> + Send + 'static,
{
let stop_requested = Arc::new(AtomicBool::new(false));
let done = Arc::new(AtomicBool::new(false));
let results = Arc::new(Latest::default());
let stopped = stop_requested.clone();
let finished = done.clone();
let slot = results.clone();
let thread = thread::spawn(move || {
let _completion = Completion(finished);
if let Err(error) = run_worker(source, detector, &stopped, &slot)
&& !stopped.load(Ordering::Acquire)
{
slot.publish(Err(error));
}
});
Self {
stop_requested,
done,
results,
thread: Some(thread),
}
}
pub fn take(&self) -> Option<Result<FrameResult>> {
self.results.take()
}
pub fn is_done(&self) -> bool {
self.done.load(Ordering::Acquire)
}
pub fn stop(&self) {
self.stop_requested.store(true, Ordering::Release);
self.results.close();
}
pub fn join(&mut self) -> Result<()> {
if let Some(thread) = self.thread.take() {
thread
.join()
.map_err(|_| "Capture/inference worker panicked")?;
}
Ok(())
}
}
impl Drop for Worker {
fn drop(&mut self) {
self.stop();
let _ = self.join();
}
}
fn run_worker<S, D>(
source: S,
detector: D,
stopped: &AtomicBool,
results: &Latest<Result<FrameResult>>,
) -> Result<()>
where
S: FnOnce() -> Result<Box<dyn FrameSource>>,
D: FnOnce() -> Result<Box<dyn Detector>>,
{
let mut model = detector()?;
if stopped.load(Ordering::Acquire) {
return Ok(());
}
let mut frames = source()?;
while !stopped.load(Ordering::Acquire) {
let image = frames.next_frame()?;
if stopped.load(Ordering::Acquire) {
break;
}
if image.empty() {
return Err("Camera returned an empty frame".into());
}
let detections = model.detect(&image)?;
if stopped.load(Ordering::Acquire) {
break;
}
results.publish(Ok(FrameResult { image, detections }));
}
Ok(())
}
struct Completion(Arc<AtomicBool>);
impl Drop for Completion {
fn drop(&mut self) {
self.0.store(true, Ordering::Release);
}
}Put predictions in place
Undo letterbox scaling and padding, then clip boxes before drawing.
Basics
letterbox(width, height) · unproject(box, transform)- Preserves the aspect ratio and maps boxes back to the original image.
decode(data, classes, count, transform)- Validates predictions and removes overlapping boxes of the same class.
Keep in mind
Treat image size, crop, orientation and coordinates as one contract. Test it with known boxes before trusting an overlay.
use crate::Result;
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Box2 {
pub x: f64,
pub y: f64,
pub width: f64,
pub height: f64,
}
#[derive(Clone, Debug)]
pub struct Detection {
pub bounds: Box2,
pub label: usize,
pub score: f32,
}
#[derive(Clone, Copy, Debug)]
pub struct Letterbox {
pub original_width: i32,
pub original_height: i32,
pub width: i32,
pub height: i32,
pub left: i32,
pub top: i32,
}
pub fn letterbox(width: i32, height: i32) -> Result<Letterbox> {
if width <= 0 || height <= 0 {
return Err("Image dimensions must be positive".into());
}
let ratio = (640. / f64::from(width)).min(640. / f64::from(height));
let resized_width = (f64::from(width) * ratio).round().max(1.) as i32;
let resized_height = (f64::from(height) * ratio).round().max(1.) as i32;
Ok(Letterbox {
original_width: width,
original_height: height,
width: resized_width,
height: resized_height,
left: (640 - resized_width) / 2,
top: (640 - resized_height) / 2,
})
}
pub fn unproject(b: Box2, t: Letterbox) -> Result<Box2> {
if t.original_width <= 0
|| t.original_height <= 0
|| t.width <= 0
|| t.height <= 0
|| t.width > 640
|| t.height > 640
|| t.left < 0
|| t.top < 0
|| t.left > 640 - t.width
|| t.top > 640 - t.height
{
return Err("Invalid letterbox geometry".into());
}
if ![b.x, b.y, b.width, b.height, b.x + b.width, b.y + b.height]
.iter()
.all(|x| x.is_finite())
|| b.width < 0.
|| b.height < 0.
{
return Err("Box geometry must be finite and nonnegative".into());
}
let x = |v: f64| {
((v - f64::from(t.left)) / f64::from(t.width)).clamp(0., 1.) * f64::from(t.original_width)
};
let y = |v: f64| {
((v - f64::from(t.top)) / f64::from(t.height)).clamp(0., 1.) * f64::from(t.original_height)
};
let left = x(b.x);
let top = y(b.y);
Ok(Box2 {
x: left,
y: top,
width: x(b.x + b.width) - left,
height: y(b.y + b.height) - top,
})
}
fn intersection_over_union(a: Box2, b: Box2) -> f64 {
let width = (a.x + a.width).min(b.x + b.width) - a.x.max(b.x);
let height = (a.y + a.height).min(b.y + b.height) - a.y.max(b.y);
let intersection = width.max(0.) * height.max(0.);
let area = a.width * a.height + b.width * b.height - intersection;
if area > 0. { intersection / area } else { 0. }
}
pub fn decode(data: &[f32], classes: usize, count: usize, t: Letterbox) -> Result<Vec<Detection>> {
if !(1..=1024).contains(&classes)
|| !(1..=100_000).contains(&count)
|| data.len() != (4 + classes) * count
{
return Err("Expected float32[1,4+C,N], C<=1024, N<=100000".into());
}
let mut candidates = Vec::new();
for i in 0..count {
let cx = f64::from(data[i]);
let cy = f64::from(data[count + i]);
let width = f64::from(data[2 * count + i]);
let height = f64::from(data[3 * count + i]);
// Validate even candidates below the confidence threshold.
let bounds = unproject(
Box2 {
x: cx - width / 2.,
y: cy - height / 2.,
width,
height,
},
t,
)?;
let mut score = 0.;
let mut label = 0;
for c in 0..classes {
let probability = data[(4 + c) * count + i];
if !probability.is_finite() || !(0.0..=1.0).contains(&probability) {
return Err("Expected probabilities in [0,1], not logits/objectness".into());
}
if probability > score {
score = probability;
label = c;
}
}
if score >= 0.25 && bounds.width > 0. && bounds.height > 0. {
candidates.push(Detection {
bounds,
label,
score,
});
}
}
candidates.sort_by(|a, b| b.score.total_cmp(&a.score));
candidates.truncate(3000);
let mut selected: Vec<Detection> = Vec::new();
for candidate in candidates {
let suppressed = selected.iter().any(|previous| {
previous.label == candidate.label
&& intersection_over_union(previous.bounds, candidate.bounds) > 0.45
});
if suppressed {
continue;
}
selected.push(candidate);
if selected.len() == 300 {
break;
}
}
Ok(selected)
}Build. Run. Try your model.
Select a camera explicitly, or process an image offline. Press Q or Escape to stop.
Basics
parse_options()- Validates arguments and selects exactly one input: image or camera.
run()- Validates the selected source and delegates image or camera processing.
Keep in mind
- Symptom
- Stopping takes longer than expected.
- Cause
- A native capture or inference call is still running.
- Remedy
- Request stop, reject late results and wait for active work before releasing its resources. Backend cancellation depends on platform support.
cargo run --locked -- --model detector.onnx --labels labels.txt --image frame.png --output result.png
cargo run --locked -- --model detector.onnx --labels labels.txt --camera 006.06 / Best practices
Small details. Better examples.
- Keep capture and inference off the UI thread. Bound the queue instead of collecting a backlog of old frames.
- Treat image size, crop, orientation and coordinates as one contract. Test it with known boxes before trusting an overlay.
- Request stop, reject late results and wait for active work before releasing its resources. Backend cancellation depends on platform support.