When is a model actually good?

BADBOB / IN DEPTH · 10 min

False detections, missed objects, and why one number is not enough.

Reading draft · Svenska and English

We review the writing and illustrations in Swedish and English before adding more languages.

Start by counting what happened

A real object matched by an accepted detection is a true positive, TP. An accepted detection without a correct match is a false positive, FP. A real object left unmatched is a false negative, FN.

For objects and masks, first define a match: the right class, enough overlap and a clear rule preventing duplicate matches. Otherwise counts from different runs mean different things.

Go deeper

Two predictions pointing at the same real object cannot both count as separate correct detections. The examples below start after matching, so you can focus on the threshold. They do not run NMS or mask matching in the browser.

How many detections are right — and how many objects are found?

Precision asks: of the detections shown, how many were right? Recall asks: of the objects actually present, how many were found?

Raise or lower the threshold in the example. Fewer detections do not automatically mean a better model. Choose a balance based on which errors matter for your task.

Precision
75%
Recall
60%
Correctly found
3
False detections
1
Missed
2
CandidateScoreJudged matchShown
A0.96CorrectShown
X0.88IncorrectShown
B0.76CorrectShown
C0.61CorrectShown
Y0.42IncorrectHidden
D0.31CorrectHidden
Constructed example: 5 real objects, 6 candidates after duplicate filtering. Class and overlap have already been judged; each correct candidate matches a different object. E is missed at every threshold.
Go deeper

Precision = TP / (TP + FP). Recall = TP / (TP + FN). When no detections are shown, precision is undefined in this example; it should not automatically be called 100%.

With five real objects and threshold 0.50, TP=3, FP=1 and FN=2: precision 75% and recall 60%. Candidate match judgements are fixed. This is a toy table, not a BadBob quality measurement.

Finding an object is different from following its boundary

In BadBob’s experimental recipe · Recipe reviewed 29 October 2026

Intersection over Union, IoU, divides the intersection of two regions by their union. For masks it counts pixels; for boxes it uses rectangle areas.

Two masks sharing 60 pixels and covering 100 pixels in total have IoU 0.60. A box may have high IoU while the mask still loses a narrow part. Examine the examples as well as averages.

Go deeper

BadBob’s experimental evaluation reports TP, FP, FN and meanBestMaskIoU, among other values. The latter averages each ground-truth instance’s best same-class mask overlap. It is not a replacement for one-to-one matching or an AP/mAP protocol.

An average can hide a weak class or kind of boundary. Break results down by class, size and relevant conditions. AP summarizes a precision–recall curve; its IoU and averaging protocol must be specified.

A high score needs testing on new images

A model can fit training material very well and still fail in new situations. Lighting, background, distance, object size and occlusion are all part of the task.

Failure Bank helps retain concrete mistakes and decide their future role. Protected regression examples should not move into training just to make an evaluation number look better.

Go deeper

Define your comparison protocol before comparing runs: the same data version, class mapping, thresholds and matching rules. Improve the weak part of the system — data, annotations, recipe or postprocessing — then assess again.

Sources and further reading