In fields like machine learning (ML), natural language processing (NLP), and information retrieval (IR), evaluation is a necessary task, and its metrics often include the following: Accuracy, Precision, Recall, and F1-Measure. (Note: relatively speaking, the ground truth in IR is often an Ordered List rather than a Bool-type Unordered Collection. When everything is found, ranking third versus fourth doesn't matter much, but ranking first versus one hundredth—although both count as "found"—carry very different meanings, so metrics like MAP are more suitable.)

This article briefly introduces a few of these concepts. These metrics are translated into Chinese in various ways, so it's generally recommended to use the English terms.

Aside: I once specifically asked a few language teachers who are outsiders to this field—what's the difference between accuracy and precision? They stared at me: "Are you messing with me? Testing me? What kind of question is this?" So please don't get hung up on it; we'll just stick to the original English terms.

Now let me first assume a concrete scenario as an example.

Suppose a class has 80 boys and 20 girls, 100 people in total. The goal is to find all the girls. Now someone picks out 50 people, of whom 20 are girls, but he also mistakenly picks 30 boys as if they were girls. As the evaluator, you need to evaluate his work.

First, we can calculate the accuracy, which is defined as: for a given test dataset, the ratio of the number of samples correctly classified by the classifier to the total number of samples. In other words, it is the accuracy on the test dataset when the loss function is the 0-1 loss[1].

That sounds a bit abstract, so to put it simply: in the scenario above, the actual situation is that the class has two categories—boys and girls. Someone (the so-called classifier) divides the people in the class into boys and girls. Accuracy asks for the proportion of people he classified correctly out of the total number of people. It's easy to see: he correctly classified 70 people (20 girls + 50 boys), and the total is 100 people, so his accuracy is 70% (70 / 100).

With accuracy, we can indeed, in some contexts and in some sense, determine whether a classifier is effective, but it doesn't always effectively evaluate a classifier's work. Here's an example: Google crawled 100 pages from argcv, while its index contains 10,000,000 pages in total. If we randomly draw one page and classify whether it's an argcv page, and judge my work by accuracy, then I would classify all pages as "not an argcv page," because this is extremely efficient (just return false, one statement), and the accuracy would already reach 99.999% (9,999,900/10,000,000), crushing the values that many other classifiers painstakingly compute. Yet this algorithm is obviously not what the requirements call for. So how do we solve this? This is where precision, recall, and F1-measure come in.

Before talking about precision, recall, and F1-measure, we need to define the four classification cases: TP, FN, FP, TN.

Following the example above, we need to find all the girls from among the people in a class. If we treat this task as a classifier, then girls are what we want, and boys are not. So we call girls the "positive class" or "positive case," and boys the "negative class" or "negative case."

Regarding gender, some people might ask, hey, why are boys "negative"? It's because what we need to screen or test is "what we are looking for." For example, if a classifier is looking for spam emails, then being spam is positive. "Positive" means relevant—"we are looking for it."

Actually Relevant Actually NonRelevant
Retrieved,
judged as yes		 TP: True Positives
Correctly (true) classifying a positive case
In the example: correctly judging "this one is a girl"		 FP: False Positives
Incorrectly (false) classifying a negative case, "accepting the false"
In the example: clearly a boy, yet judged as a girl. With cross-dressers everywhere these days, this mistake is often made
Not Retrieved,
judged as no		 FN: False Negatives
Incorrectly (false) classifying a negative case, "rejecting the true"
Applied to the example: clearly a girl, yet this guy judges her as a boy.
Liang Shanbo says "much obliged, much obliged"		 TN: True Negatives
Correctly (true) classifying a negative case
That is, a boy is judged as a boy. A pure manly man like me would surely fall into this cell

It's easy to see that TRUE/FALSE indicates whether the result is right or wrong, while Positive/Negative indicates whether we considered it a "yes" or a "no."

Plugging into this table, we can easily get these values:

Relevant NonRelevant
Retrieved TP=20 FP=30
Not Retrieved FN=0 TN=50

Then we can simply compute:

The formula for precision is P=TPTP+FPP = \frac{TP}{TP+FP}. It computes the proportion of all "correctly retrieved results (TP)" among all "actually retrieved results (TP+FP)."

In the example, we want to know what proportion of all the people this guy picked are correct people (i.e., girls). So his precision is 40% (20 girls / (20 girls + 30 boys mistakenly judged as girls)).

The formula for recall is R=TPTP+FNR = \frac{TP}{TP+FN}. It computes the proportion of all "correctly retrieved results (TP)" among all "results that should have been retrieved (TP+FN)."

In the example, we want to know what proportion of all the girls in the class this guy managed to get. So his recall is 100% (20 girls / (20 girls + 0 girls mistakenly judged as boys)).

The F1 score is the harmonic mean of precision and recall, which is

2F1=1P+1R\frac{2}{F_1} = \frac{1}{P} + \frac{1}{R}

After adjusting, it is

F1=2PRP+R=2TP2TP+FP+FNF_1 = \frac{2PR}{P+R} = \frac{2TP}{2TP + FP + FN}

In the example, the F1-measure is approximately 57.143% (2∗0.4∗10.4+1\frac{2 * 0.4 * 1}{0.4 + 1}).

It should be noted that someone[3] gave the following formula: for non-negative real numbers β\beta

Fβ=(β2+1)∗PRβ2∗P+RF_\beta = \frac{ (\beta^2 + 1) * PR}{\beta^2*P+R}

generalizes the F-measure.

The F1-measure assumes that precision and recall have equal weight, but in some scenarios we may consider precision more important. By adjusting the parameter β\beta, using Fβ_\beta-measure can help us better evaluate the result.

References