F-measure

Revision as of 23:28, 1 April 2013

In the UNL System, the F-measure (or F1-score) is the measure of a grammar's accuracy. It considers both the precision and the recall of the grammar to compute the score, according to the formula

F-measure = 2 x ( (precision x recall) / (precision + recall) )

In the above:

• precision is the number of correct results divided by the number of all returned results
• recall is the number of correct results divided by the number of results that should have been returned

UNLization

RETURNED

A result is considered "RETURNED" when the output is a graph (i.e., all the nodes are interlinked) made only of UW's (i.e., without natural language words)

CORRECT

A result is considered "CORRECT" when:

• The discrepancy of relations between the actual and the expected output is less than 0.3; AND
• The discrepancy of UW's between the actual and the expected output is less than 0.3; AND
• The overall discrepancy is less than 0.3;

WHERE

Discrepancy of relations is calculated by the formula:

(exceeding_relations + missing_relations)/total_relations

Discrepancy of UW's is calculated by the formula:

(exceeding_UW + missing_UW)/total_UW

Overall discrepancy is calculated by the formula:

((3*(exceeding_relations+missing_relations))+(2*(exceding_UW+missing_UW)+(exceding_attribute+missing_attribute))/((3*total_relations)+(2*total_UW)+(total_attribute))

• exceeding_relations is the number of relations present in the actual output but absent from the expected output
• missing_relations is the number of relations absent from the actual output but present in the expected output
• total_relations is the sum of the total number of relations in the actual output and in the expected output
• exceeding_UW is the number of UW's^[1] present in the actual output but absent from the expected output
• missing_UW is the number of UW's^[1] absent from the actual output but present in the expected output
• total_UW is the sum of the total number of UW's^[1] in the actual output and in the expected output
• exceeding_attribute is the number of attributes^[2] present in the actual output but absent from the expected output
• missing_attribute is the number of attributes^[2] absent from the actual output but present in the expected output
• total_attribute is the sum of the total number of attributes^[2] in the actual output and in the expected output

NLization

RETURNED

A result is considered "RETURNED" when the output is a list of natural language words (i.e., without any UW).

CORRECT

A result is considered "CORRECT" when the difference between the actual result and the expected result is less than 30%

The difference between the actual and the expected result is calculated by the formula

(exceeding_words+missing_words)/(total_words)

WHERE

• exceeding_words is the number of words present in the actual output but absent from the expected output
• missing_words is the number of words absent from the actual output but present in the expected output
• total_words is the sum of the total number of words in the actual output and in the expected output

References

1. ↑ ^{1.0 1.1 1.2} For the sake of comparison, UW's appearing in the source position are considered to be different from UW's appearing in the target position of a relation. Scopes are ignored.
2. ↑ ^{2.0 2.1 2.2} For the sake of comparison, attributes appearing in the source position are considered to be different from attributes appearing in the target position of a relation. Optional attributes (such as .@def and .@indef) are ignored.