Coreference Resolution Evaluation Based on Descriptive Specificity
François Trouilleux, Eric Gaussier, Gabriel Bès, Annie Zaenen
This paper introduces a new evaluation method for the coreference resolution task. Considering that
coreference resolution is a matter of linking expressions to discourse referents, we set our evaluation criteron
in terms of an evaluation of the denotations assigned to the expressions. This criterion requires that the
coreference chains identified in one annotation stand in a one-to-one correspondence with the coreference
chains in the other. To determine this correspondence and with a view to keep closer to what human
interpretation of the coreference chains would be, we take into account the fact that, in a coreference chain,
some expressions are more specific to their referent than others. With this observation in mind, we measure
the similarity between the chains in one annotation and the chains in the other, and then compute the optimal
similarity between the two annotations. Evaluation then consists in checking whether the denotations assigned
to the expressions are correct or not. New measures to analyse errors are also introduced. A comparison with
other methods is given at the end of the paper.
Proceedings of the Second International Conference on Language Resources and Evaluation (LREC 2000). Athens, Greece. 2000.
lrec2000.pdf (87.03 kB)