Deep Semantic Analogies
This resource comprises various evaluation benchmarks regarding
semantic analogies for English and for
German. See here
on how to obtain the data.
de_trans_Google_analogies
The Google semantic/syntactic analogy datasets were introduced in Mikolov et al. (2013). The datasets contain analogy questions of the form A:B::C:D, meaning A is to B as C is to D, where the fourth word (D) is unknown. We constructed German counterparts of the datasets through manual translation and subsequent cross-checking by three human judges. We omitted the relation type "adjective-adverb", because it does not exist in German. The final task set contains 18,552 analogy tasks.
en/de_sem-para_SemRel
The paradigmatic semantic relation dataset also contains analogy tasks. Here, the paradigmatic relation between A and B is the same as between C and D. The dataset was constructed from antonymy, synonymy, and hypernymy relation pairs collected by Lenci & Benotto for English and by Scheible & Schulte im Walde for German, using the methodology described in Scheible and Schulte im Walde (2014). The questions cover the semantic relations adjective antonym, noun hypernym, noun synonym, noun antonym and verb antonym.
Overall, this dataset constitutes a deep semantic challenge, containing very specific, domain-related and potentially low-frequent semantic details that are difficult to solve even for humans. For example, the tasks include antonyms such as biblical:secular::deaf:hearing or screech:whisper::ink:erase. The English dataset contains 7,516 analogies and the German dataset contains 2,462 analogies.
en_sem-para_BLESS
In the same way, we created an analogy dataset with 10,000 unique analogy questions from the hypernymy and meronymy relations in BLESS (Baroni and Lenci, 2011), by randomly picking semantic relation pairs.
Next to analogies, the resource also contains:
de_new-rated_Schm280
Schm280 (Schmidt et al., 2001) translated 280 word pairs from WordSim353 (Finkelstein et al., 2001). As they did not re-rate the German relation pairs after translation, we collected new ratings for the German pairs from 10 subjects, applying the same conditions as the original WordSim353 collection. The dataset contains 280 translated and newly rated word pairs for WordSim350.
References:
Maximilian Köper, Christian Scheible, Sabine Schulte im Walde (2015)
Multilingual Reliability and "Semantic" Structure of Continuous Word Spaces
In: Proceedings of the 11th Conference on Computational Semantics (IWCS). London, UK.
Silke Scheible, Sabine Schulte im Walde (2014)
A Database of Paradigmatic Semantic Relation Pairs for German Nouns, Verbs, and Adjectives
In: Proceedings of the COLING Workshop on Lexical and Grammatical Resources for Language Processing (LG-LP). Dublin, Ireland.