Semantic Change Evaluation

We see an increasing interest in the automatic detection of semantic change in computational linguistics, motivated by expected performance improvements of practical NLP applications, or theoretical interest in language or cultural change. However, a major obstacle in the computational modeling of semantic change is the lack of resources for evaluation.

Besides computational models of semantic change we have developed a number of such evaluation resources. They are freely available for education, research and other non-commercial purposes.

German Diachronic Metaphor Annotation Dataset

Metaphoric change plays a fundamental role in semantic change. We introduce a resource for the evaluation of computational models of metaphoric change and propose a structured annotation process that is generalisable to the creation of gold standards for other types of semantic change.

Two annotators judged for 560 context pairs whether one of the contexts admitted inference of a meaning of the target word which is related metaphorically to the meaning in the other context.

See here on how to obtain the data.


Dominik Schlechtweg, Stefanie Eckmann, Enrico Santus, Sabine Schulte im Walde, Daniel Hole (2017)
German in Flux: Detecting Metaphoric Change via Word Entropy
In: Proceedings of the SiGNLL Conference on Computational Natural Language Learning (CoNNL). Vancouver, Canada.

DURel - Annotation of Diachronic Ursage Relatedness

We extend a framework of synchronic polysemy annotation to the annotation of Diachronic Usage Relatedness (DURel). DURel has a strong theoretical basis and at the same time makes use of established synchronic procedures that rely on the intuitive notion of semantic relatedness.

DURel distinguishes between innovative and reductive meaning changes with high interannotator agreement. The resulting test set for German comprises ratings from five annotators for the relatedness of 1,320 use pairs across 22 target words.

See here on how to obtain the data.


Dominik Schlechtweg, Sabine Schulte im Walde, Stefanie Eckmann (2018)
Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change
In: Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). New Orleans, LA.

Dataset of Grammaticalisation for German Prepositions

We developed a test set containing 206 prepositions with four degrees of grammaticalisation (1: low -- 4: high). The test set distinguishes between
  1. prepositions with the form of a content word (e.g., trotz),
  2. prepositions with the form of a syntactic structure (e.g., am Rande), and
  3. prepositions with the form of a function word (e.g., vor).
Prepositions in 1. and 2. show a low to medium degree of grammaticalisation, while the ones in 3. show a high degree.

See here on how to obtain the data.


Dominik Schlechtweg, Sabine Schulte im Walde (2018)
Distribution-based Prediction of the Degree of Grammaticalization for German Prepositions
In: Proceedings of the Evolution of Language International Conference (EvoLang XII). Torun, Poland.