CRF++ is an open-source tool and a powerful asset for natural language processing. It compiles into two binaries, crf_learn and crf_test. The test output compares the predicted results against the ground truth. Unfortunately, it doesn't provide any statistics after the comparison, which is quite annoying.

Fortunately, someone created a Perl tool called conlleval for analyzing test results.

You can find the documentation here.

FIX: If the official link is dead, you can refer to this: conlleval.pl

Rename the downloaded txt file to conlleval.pl or whatever name you prefer. Then chmod +x it to make it executable.

When using it, redirect the crf_test output to a file named output.txt:

Bash
conlleval.pl < output.txt
Then pipe the test results as input to the .pl file, and it will analyze the input and give you statistical results.

Sometimes, an error pops up:

Bash
conlleval: unexpected number of features in line XXX	XX	XX
Why is that?

Take a look at the source code:

Perl
if (@features < 2) { 
    die "conlleval: unexpected number of features in line $line\n"; 
}

Why are there not enough features?

Looking back at the documentation mentioned earlier, it says the default delimiter is a space. But CRF++ actually uses tab as the delimiter.

So those 4 columns are treated as a single column.

The fix is as follows:

Bash
conlleval -d "\t" < output.txt

Pay attention to the parameter after -d here. Tests show that you can also enter an actual tab character here, but typing a tab in the shell is really a hassle, so you can use '\t' instead.

The correct result looks like this:

processed 19172 tokens with 9715 phrases; found: 9308 phrases; correct: 7755.
accuracy:  88.95%; precision:  83.32%; recall:  79.83%; FB1:  81.53
             ADJP: precision:  58.11%; recall:  25.00%; FB1:  34.96  74
             ADVP: precision:  65.44%; recall:  53.29%; FB1:  58.75  272
            CONJP: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
             INTJ: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
               NP: precision:  81.50%; recall:  79.54%; FB1:  80.51  4957
               PP: precision:  88.96%; recall:  95.27%; FB1:  92.01  2129
              PRT: precision:   0.00%; recall:   0.00%; FB1:   0.00  0
             SBAR: precision:  84.62%; recall:  34.38%; FB1:  48.89  78
               VP: precision:  85.32%; recall:  80.61%; FB1:  82.90  1798