06 / Rust
Rust. Dari model ke kamera.
Jalankan detektor ONNX secara lokal dengan anotasi langsung dan pemrosesan gambar yang dibatasi.
Contoh ini berfokus pada benda dan hewan dalam lingkungan terkendali. Pengawasan manusia, identifikasi, dan analisis perilaku tidak dibahas.
Siapkan proyek
Instal dependensi yang tercantum dan jalankan pengujian. Coba gambar lokal sebelum membuka kamera.
Dasar
cargo test --locked- Menjalankan pemeriksaan otomatis tanpa membuka kamera.
unzip badbob-playground-rust.zip
cd badbob-playground-rust
cargo build --locked
cargo test --locked
cargo fmt --checkHubungkan model
Validasi model dan label sebelum membuka kamera.
Dasar
parse_labels(text)- Membaca nama kelas sesuai urutan model; menolak label kosong atau duplikat.
OpenCvDetector::load(path, classes)- Memuat model ONNX dan memvalidasi masukan, keluaran, serta jumlah label.
prepare_input(image)- Membuat masukan RGB berukuran 640×640, mempertahankan proporsi, dan mengisi area yang tersisa.
OpenCvDetector::detect(image)- Menyiapkan gambar, menjalankan model, dan mengembalikan deteksi.
Perlu diingat
- Gejala
- Kotak bergeser atau model gagal dimuat.
- Penyebab
- Prapemrosesan, tata letak keluaran, atau urutan label tidak sesuai.
- Penanganan
- Adaptor ini hanya mendukung kontrak berikut. Berkas Core ML dan tata letak ONNX lain memerlukan adaptor berbeda.
Adaptor ini hanya mendukung kontrak berikut. Berkas Core ML dan tata letak ONNX lain memerlukan adaptor berbeda.
Lebih mendalam
Koordinat menyatakan pusat dan ukuran kotak dalam piksel masukan 640×640. Label harus mengikuti urutan kelas model secara persis.
input:
color: RGB
dtype: float32
shape: [1, 3, 640, 640]
letterbox: 114
scale: 1/255
output:
dtype: float32
shape: [1, 4+C, N]
boxes: [cx, cy, width, height]
class_scores: probabilities
objectness: false
embedded_nms: falseuse crate::{
Result,
geometry::{Detection, decode, letterbox},
runtime::{Detector, FrameSource},
};
use opencv::{
core::{self, Mat, Rect, Scalar, Size, Vector},
dnn, imgproc,
prelude::*,
videoio,
};
use std::path::Path;
pub struct OpenCvDetector {
network: dnn::Net,
classes: usize,
output_name: String,
}
impl OpenCvDetector {
pub fn load(path: &Path, classes: usize) -> Result<Self> {
if path.extension().and_then(|s| s.to_str()) != Some("onnx")
|| !path.is_file()
|| !(1..=1024).contains(&classes)
{
return Err("Provide an existing .onnx file and 1-1024 labels".into());
}
let mut network =
dnn::read_net_from_onnx(path.to_str().ok_or("Model path must be UTF-8")?)?;
network.set_preferable_backend(dnn::DNN_BACKEND_OPENCV)?;
network.set_preferable_target(dnn::DNN_TARGET_CPU)?;
let outputs = network.get_unconnected_out_layers_names()?;
if outputs.len() != 1 {
return Err("Expected exactly one detector output".into());
}
let mut detector = Self {
network,
classes,
output_name: outputs.get(0)?,
};
// Validate the actual forward contract before the camera factory runs.
let sample = Mat::new_rows_cols_with_default(640, 640, core::CV_8UC3, Scalar::all(114.))?;
detector.detect(&sample)?;
Ok(detector)
}
}
impl Detector for OpenCvDetector {
fn detect(&mut self, image: &Mat) -> Result<Vec<Detection>> {
let (blob, t) = prepare_input(image)?;
self.network.set_input(&blob, "", 1., Scalar::default())?;
let output = self.network.forward_single(&self.output_name)?;
let shape = output.mat_size();
let count = validate_output_shape(&shape, output.typ(), self.classes)?;
let contiguous = output.try_clone()?;
decode(contiguous.data_typed::<f32>()?, self.classes, count, t)
}
}
pub struct OpenCvCamera(videoio::VideoCapture);
impl OpenCvCamera {
pub fn open(index: i32) -> Result<Self> {
let capture = videoio::VideoCapture::new(index, videoio::CAP_ANY)?;
if !capture.is_opened()? {
return Err("Cannot open camera; check index, permissions and backend".into());
}
Ok(Self(capture))
}
}
impl FrameSource for OpenCvCamera {
fn next_frame(&mut self) -> Result<Mat> {
let mut image = Mat::default();
if !self.0.read(&mut image)? || image.empty() {
return Err("Camera read failed or returned an empty image".into());
}
Ok(image)
}
}
pub fn annotate(image: &mut Mat, detections: &[Detection], labels: &[String]) -> Result<()> {
for detection in detections {
let b = detection.bounds;
let left = (b.x.floor() as i32).clamp(0, image.cols() - 1);
let top = (b.y.floor() as i32).clamp(0, image.rows() - 1);
let right = (b.x + b.width).ceil() as i32;
let bottom = (b.y + b.height).ceil() as i32;
let bounds = Rect::new(
left,
top,
(right.min(image.cols()) - left).max(1),
(bottom.min(image.rows()) - top).max(1),
);
imgproc::rectangle(
image,
bounds,
Scalar::new(74., 220., 250., 0.),
2,
imgproc::LINE_8,
0,
)?;
let text = format!("{} {:.2}", labels[detection.label], detection.score);
imgproc::put_text(
image,
&text,
core::Point::new(left, (top - 6).max(16)),
imgproc::FONT_HERSHEY_SIMPLEX,
0.5,
Scalar::new(74., 220., 250., 0.),
1,
imgproc::LINE_AA,
false,
)?;
}
Ok(())
}
pub fn encode_png(image: &Mat) -> Result<Vec<u8>> {
let mut bytes = Vector::<u8>::new();
if !opencv::imgcodecs::imencode(".png", image, &mut bytes, &Vector::<i32>::new())? {
return Err("Cannot encode PNG".into());
}
Ok(bytes.to_vec())
}
pub fn prepare_input(image: &Mat) -> Result<(Mat, crate::geometry::Letterbox)> {
if image.empty() || image.typ() != core::CV_8UC3 {
return Err("Expected a nonempty 8-bit BGR image".into());
}
let t = letterbox(image.cols(), image.rows())?;
let mut resized = Mat::default();
imgproc::resize(
image,
&mut resized,
Size::new(t.width, t.height),
0.,
0.,
imgproc::INTER_LINEAR,
)?;
let mut padded = Mat::new_rows_cols_with_default(640, 640, core::CV_8UC3, Scalar::all(114.))?;
let mut region = Mat::roi_mut(&mut padded, Rect::new(t.left, t.top, t.width, t.height))?;
resized.copy_to(&mut region)?;
let blob = dnn::blob_from_image(
&padded,
1. / 255.,
Size::new(640, 640),
Scalar::default(),
true,
false,
core::CV_32F,
)?;
Ok((blob, t))
}
pub fn validate_output_shape(shape: &[i32], datatype: i32, classes: usize) -> Result<usize> {
if !(1..=1024).contains(&classes)
|| shape.len() != 3
|| datatype != core::CV_32F
|| shape[0] != 1
|| shape[1] != (4 + classes) as i32
|| !(1..=100_000).contains(&shape[2])
{
return Err("Expected float32[1,4+C,N], N<=100000, XYWH + class probabilities; no objectness/embedded NMS".into());
}
Ok(shape[2] as usize)
}Tambahkan kamera
Pengambilan gambar dan inferensi berbagi satu thread pekerja serial. Antarmuka hanya menerima gambar terbaru yang selesai diproses beserta deteksinya.
Dasar
run_worker(source, detector, stopped, results)- Menangani pengambilan gambar dan inferensi pada satu thread pekerja.
Latest::take()- Mengambil gambar terbaru yang selesai diproses beserta deteksinya; hasil tidak dapat menumpuk dalam antrean.
Worker::stop() · Worker::join()- Berhenti menerima hasil baru dan melepaskan sumber daya setelah pekerjaan aktif selesai.
Perlu diingat
- Gejala
- Pratinjau tertinggal dan penggunaan memori meningkat.
- Penyebab
- Kamera menghasilkan frame lebih cepat daripada inferensi memprosesnya, sehingga antrean tanpa batas terus bertambah.
- Penanganan
- Pengambilan gambar dan inferensi berbagi satu thread pekerja serial. Antarmuka hanya menerima gambar terbaru yang selesai diproses beserta deteksinya.
Lebih mendalam
Minta penghentian, tolak hasil yang terlambat, dan tunggu pekerjaan aktif selesai sebelum melepaskan sumber dayanya. Pembatalan mesin pemrosesan bergantung pada dukungan platform.
use crate::{Result, geometry::Detection};
use opencv::{core::Mat, prelude::*};
use std::sync::{
Arc, Mutex,
atomic::{AtomicBool, Ordering},
};
use std::thread::{self, JoinHandle};
pub trait FrameSource: Send {
fn next_frame(&mut self) -> Result<Mat>;
}
pub trait Detector: Send {
fn detect(&mut self, image: &Mat) -> Result<Vec<Detection>>;
}
pub struct FrameResult {
pub image: Mat,
pub detections: Vec<Detection>,
}
struct Slot<T> {
value: Option<T>,
closed: bool,
}
pub struct Latest<T>(Mutex<Slot<T>>);
impl<T> Default for Latest<T> {
fn default() -> Self {
Self(Mutex::new(Slot {
value: None,
closed: false,
}))
}
}
impl<T> Latest<T> {
pub fn publish(&self, value: T) {
let mut slot = self.0.lock().unwrap_or_else(|e| e.into_inner());
if !slot.closed {
slot.value = Some(value);
}
}
pub fn take(&self) -> Option<T> {
self.0
.lock()
.unwrap_or_else(|e| e.into_inner())
.value
.take()
}
pub fn close(&self) {
let mut slot = self.0.lock().unwrap_or_else(|e| e.into_inner());
slot.closed = true;
slot.value = None;
}
}
pub struct Worker {
stop_requested: Arc<AtomicBool>,
done: Arc<AtomicBool>,
results: Arc<Latest<Result<FrameResult>>>,
thread: Option<JoinHandle<()>>,
}
impl Worker {
pub fn start<S, D>(source: S, detector: D) -> Self
where
S: FnOnce() -> Result<Box<dyn FrameSource>> + Send + 'static,
D: FnOnce() -> Result<Box<dyn Detector>> + Send + 'static,
{
let stop_requested = Arc::new(AtomicBool::new(false));
let done = Arc::new(AtomicBool::new(false));
let results = Arc::new(Latest::default());
let stopped = stop_requested.clone();
let finished = done.clone();
let slot = results.clone();
let thread = thread::spawn(move || {
let _completion = Completion(finished);
if let Err(error) = run_worker(source, detector, &stopped, &slot)
&& !stopped.load(Ordering::Acquire)
{
slot.publish(Err(error));
}
});
Self {
stop_requested,
done,
results,
thread: Some(thread),
}
}
pub fn take(&self) -> Option<Result<FrameResult>> {
self.results.take()
}
pub fn is_done(&self) -> bool {
self.done.load(Ordering::Acquire)
}
pub fn stop(&self) {
self.stop_requested.store(true, Ordering::Release);
self.results.close();
}
pub fn join(&mut self) -> Result<()> {
if let Some(thread) = self.thread.take() {
thread
.join()
.map_err(|_| "Capture/inference worker panicked")?;
}
Ok(())
}
}
impl Drop for Worker {
fn drop(&mut self) {
self.stop();
let _ = self.join();
}
}
fn run_worker<S, D>(
source: S,
detector: D,
stopped: &AtomicBool,
results: &Latest<Result<FrameResult>>,
) -> Result<()>
where
S: FnOnce() -> Result<Box<dyn FrameSource>>,
D: FnOnce() -> Result<Box<dyn Detector>>,
{
let mut model = detector()?;
if stopped.load(Ordering::Acquire) {
return Ok(());
}
let mut frames = source()?;
while !stopped.load(Ordering::Acquire) {
let image = frames.next_frame()?;
if stopped.load(Ordering::Acquire) {
break;
}
if image.empty() {
return Err("Camera returned an empty frame".into());
}
let detections = model.detect(&image)?;
if stopped.load(Ordering::Acquire) {
break;
}
results.publish(Ok(FrameResult { image, detections }));
}
Ok(())
}
struct Completion(Arc<AtomicBool>);
impl Drop for Completion {
fn drop(&mut self) {
self.0.store(true, Ordering::Release);
}
}Tempatkan prediksi
Balikkan skala dan padding letterbox, lalu batasi kotak ke gambar sebelum menggambar.
Dasar
letterbox(width, height) · unproject(box, transform)- Mempertahankan rasio aspek dan memetakan kotak kembali ke gambar asli.
decode(data, classes, count, transform)- Memvalidasi prediksi dan menghapus kotak tumpang tindih dari kelas yang sama.
Perlu diingat
Perlakukan ukuran gambar, pemotongan, orientasi, dan koordinat sebagai satu kesepakatan. Uji dengan kotak yang diketahui sebelum memercayai penanda.
use crate::Result;
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Box2 {
pub x: f64,
pub y: f64,
pub width: f64,
pub height: f64,
}
#[derive(Clone, Debug)]
pub struct Detection {
pub bounds: Box2,
pub label: usize,
pub score: f32,
}
#[derive(Clone, Copy, Debug)]
pub struct Letterbox {
pub original_width: i32,
pub original_height: i32,
pub width: i32,
pub height: i32,
pub left: i32,
pub top: i32,
}
pub fn letterbox(width: i32, height: i32) -> Result<Letterbox> {
if width <= 0 || height <= 0 {
return Err("Image dimensions must be positive".into());
}
let ratio = (640. / f64::from(width)).min(640. / f64::from(height));
let resized_width = (f64::from(width) * ratio).round().max(1.) as i32;
let resized_height = (f64::from(height) * ratio).round().max(1.) as i32;
Ok(Letterbox {
original_width: width,
original_height: height,
width: resized_width,
height: resized_height,
left: (640 - resized_width) / 2,
top: (640 - resized_height) / 2,
})
}
pub fn unproject(b: Box2, t: Letterbox) -> Result<Box2> {
if t.original_width <= 0
|| t.original_height <= 0
|| t.width <= 0
|| t.height <= 0
|| t.width > 640
|| t.height > 640
|| t.left < 0
|| t.top < 0
|| t.left > 640 - t.width
|| t.top > 640 - t.height
{
return Err("Invalid letterbox geometry".into());
}
if ![b.x, b.y, b.width, b.height, b.x + b.width, b.y + b.height]
.iter()
.all(|x| x.is_finite())
|| b.width < 0.
|| b.height < 0.
{
return Err("Box geometry must be finite and nonnegative".into());
}
let x = |v: f64| {
((v - f64::from(t.left)) / f64::from(t.width)).clamp(0., 1.) * f64::from(t.original_width)
};
let y = |v: f64| {
((v - f64::from(t.top)) / f64::from(t.height)).clamp(0., 1.) * f64::from(t.original_height)
};
let left = x(b.x);
let top = y(b.y);
Ok(Box2 {
x: left,
y: top,
width: x(b.x + b.width) - left,
height: y(b.y + b.height) - top,
})
}
fn intersection_over_union(a: Box2, b: Box2) -> f64 {
let width = (a.x + a.width).min(b.x + b.width) - a.x.max(b.x);
let height = (a.y + a.height).min(b.y + b.height) - a.y.max(b.y);
let intersection = width.max(0.) * height.max(0.);
let area = a.width * a.height + b.width * b.height - intersection;
if area > 0. { intersection / area } else { 0. }
}
pub fn decode(data: &[f32], classes: usize, count: usize, t: Letterbox) -> Result<Vec<Detection>> {
if !(1..=1024).contains(&classes)
|| !(1..=100_000).contains(&count)
|| data.len() != (4 + classes) * count
{
return Err("Expected float32[1,4+C,N], C<=1024, N<=100000".into());
}
let mut candidates = Vec::new();
for i in 0..count {
let cx = f64::from(data[i]);
let cy = f64::from(data[count + i]);
let width = f64::from(data[2 * count + i]);
let height = f64::from(data[3 * count + i]);
// Validate even candidates below the confidence threshold.
let bounds = unproject(
Box2 {
x: cx - width / 2.,
y: cy - height / 2.,
width,
height,
},
t,
)?;
let mut score = 0.;
let mut label = 0;
for c in 0..classes {
let probability = data[(4 + c) * count + i];
if !probability.is_finite() || !(0.0..=1.0).contains(&probability) {
return Err("Expected probabilities in [0,1], not logits/objectness".into());
}
if probability > score {
score = probability;
label = c;
}
}
if score >= 0.25 && bounds.width > 0. && bounds.height > 0. {
candidates.push(Detection {
bounds,
label,
score,
});
}
}
candidates.sort_by(|a, b| b.score.total_cmp(&a.score));
candidates.truncate(3000);
let mut selected: Vec<Detection> = Vec::new();
for candidate in candidates {
let suppressed = selected.iter().any(|previous| {
previous.label == candidate.label
&& intersection_over_union(previous.bounds, candidate.bounds) > 0.45
});
if suppressed {
continue;
}
selected.push(candidate);
if selected.len() == 300 {
break;
}
}
Ok(selected)
}Bangun. Jalankan. Coba model.
Pilih kamera secara eksplisit atau proses gambar secara offline. Tekan Q atau Escape untuk berhenti.
Dasar
parse_options()- Memvalidasi argumen dan memilih tepat satu masukan: gambar atau kamera.
run()- Memvalidasi sumber yang dipilih dan meneruskan pemrosesan gambar atau kamera ke bagian yang sesuai.
Perlu diingat
- Gejala
- Penghentian memakan waktu lebih lama dari perkiraan.
- Penyebab
- Pemanggilan natif untuk pengambilan gambar atau inferensi masih berjalan.
- Penanganan
- Minta penghentian, tolak hasil yang terlambat, dan tunggu pekerjaan aktif selesai sebelum melepaskan sumber dayanya. Pembatalan mesin pemrosesan bergantung pada dukungan platform.
cargo run --locked -- --model detector.onnx --labels labels.txt --image frame.png --output result.png
cargo run --locked -- --model detector.onnx --labels labels.txt --camera 006.06 / Praktik terbaik
Detail kecil. Contoh yang lebih baik.
- Jalankan pengambilan gambar dan inferensi di luar thread antarmuka. Batasi antrean alih-alih menumpuk frame lama.
- Perlakukan ukuran gambar, pemotongan, orientasi, dan koordinat sebagai satu kesepakatan. Uji dengan kotak yang diketahui sebelum memercayai penanda.
- Minta penghentian, tolak hasil yang terlambat, dan tunggu pekerjaan aktif selesai sebelum melepaskan sumber dayanya. Pembatalan mesin pemrosesan bergantung pada dukungan platform.