An ideal object detector should not miss any ground-truth objects (FN = 0) and detect only relevant objects (FP = 0). In general, an object detector can be considered good if when the confidence threshold decreases, its precision stays high while its recall increases (the number of true positives grows, the number of false positives unchanged or increase by a very small amount), which means the precision and recall still be high at varying confidence threshold. As a result, the area under the curve (AUC) is large when both precision and recall are high.

A poor object detector can be:

• High Precision, Low Recall: In the scenario, where an object detector finds only one or a few true positives with high confidence, it results in high Precision because the number of false positives is either zero or very small. However, this also means that the Recall is very low because the model is missing many true positive objects. This situation is common when the model is very conservative and only makes predictions when it is highly confident, often resulting in missed detections.  

• High Recall, Low Precision: Conversely, when a model predicts many detections, it means including a lot of false positives. Leading to high recall because it captures most of the true positive objects. However, the Precision in this case is low because there are many false positives among the detections. This situation occurs when the model is less conservative and makes more detections, but it also makes more mistakes in the form of false positives.

In object detection, the average precision (AP) is the area under the PR curve. One class has its own average precision. The mAP is the average of the AP calculated for all the classes. Assume we have C class, the formula of mAP:

Mean average precision (mAP) is one of the metrics used for evaluating the performance of object detection model by providing an overview of the sensitivity of the model. mAP@0.5 means that is a mAP calculated at IoU threshold 0.5. mAP@[.5:.05:.95] is computed at the 10 IoU threshold from 0.5 to 0.95 with step = 0.05, and then get the average of all results.

A good mAP indicates a model that's stable and consistent across different confidence thresholds. So, to compute mAP, we first find the AP value for each class by calculating the AUC of its PR curve. In general, the PR curve does not exhibit a monotonic behavior; it is zig-zag instead. Therefore, before computing AP, precision-recall pairs have to be interpolated. The interpolated Precision at a Recall R ∈[0,1] is determined by this equation:

An interpolated precision at a given recall R is the maximum precision obtained from the remaining precisions, where their corresponding recall is equal to or greater than R.

After the interpolation process, we have a new set of interpolated precision. The AP is the area under the new PR curve and is calculated by using the Riemann sum approximation.

Assume:

• We have K confidence thresholds

• The confidence threshold at k+1 > confidence threshold at k

There are two approaches to compute this Riemann integral: The N-point interpolation and the all-point interpolation.

• N-point interpolation:

Now, let’s analyze a numerical example.Popular applications of this interpolation method use N = 11 or N = 101  

• All-point interpolation:
  The set values of   Recall(n), Recall(n+1)… Recall(n) used to compute the AP corresponds exactly to the set of original Recall values.

Now, let’s analyze a numerical example.
We've got a model for dog detection, and it's telling us it found 15 dogs (The number of ground truth dogs is 17). Those predicted dogs are in pink boxes, and each box has a letter from 'A' to 'O' on it. In each box, there is a number that shows us how confident the model is about finding the dog. As we can see, the model either successfully detects some dogs, misses some of them, or is also incorrect in detecting them.

First, we sort those detections by their confidence levels in descending. After that, we determine whether they are true or false positives and calculate the precision and recall.

We choose IoU threshold = 0.5 in this example.

Figure 7: Precision-Recall points in class ‘Dog’ with IoU threshold = 0.5

Let’s calculate the average precision (AP)!

• 11-point interpolation:
  Recall = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

  • Recall = 0.0, original Recalls that equal to greater than it: [0. 058824, 0.117647, …, 0.705882] ≥ 0.0; their correspond Precisions = [1.0, 1.0, …, 0.8]; max of those Precisions = 1.0. We have the first pair precision-recall = (1.0, 0.1)

  • Recall = 0.1, original Recalls that equal to greater than it: [0.117647, 0.176471 …, 0.705882] ≥  0.1; their corresponding Precisions = [1.0, 1.0, …, 0.8]; max of those Precisions = 1.0. We have the second pair precision-recall = (1.0, 0.1)

  • Recall = 0.2, original Recalls that equal to greater than it: [0.235294, 0.294118…, 0.705882] ≥  0.2; their correspond Precisions = [1.0, 1.0, …, 0.8]; max of those Precisions = 1.0. We have the third pair precision-recall = (1.0, 0.2)

  • Recall = 0.3, original Recalls that equal to greater than it: [0.352941, 0.411765…, 0.705882] ≥  0.3; their corresponding Precisions = [0.857143, 0.875, …, 0.8]; max of those Precisions = 0.888889. We have the fourth pair precision-recall = (0.888889, 0.3)

  • …

  • Recall = 0.8, original Recalls that equal to greater than it: {∅}; their corresponding Precisions = {∅}; max of those Precisions = 0. Precision-recall = (0.0, 0.0)

  • …

  • Recall = 1.0, original Recalls that equal to greater than it: {∅}; their corresponding Precisions = {∅}; max of those Precisions = 0. Precision-recall = (0.0, 0.0)

• All-point interpolation:
  It is a homework for you. (hint: AP≈0.6527)