Deep Semantic Analogies

This resource comprises various evaluation benchmarks regarding semantic analogies for English and for German. The resources are freely available for education, research and other non-commercial purposes. See here on how to obtain the data.


The Google semantic/syntactic analogy datasets were introduced in Mikolov et al. (2013). The datasets contain analogy questions of the form A:B::C:D, meaning A is to B as C is to D, where the fourth word (D) is unknown. We constructed German counterparts of the datasets through manual translation and subsequent cross-checking by three human judges. We omitted the relation type "adjective-adverb", because it does not exist in German. The final task set contains 18,552 analogy tasks.


The paradigmatic semantic relation dataset also contains analogy tasks. Here, the paradigmatic relation between A and B is the same as between C and D. The dataset was constructed from antonymy, synonymy, and hypernymy relation pairs collected by Lenci & Benotto for English and by Scheible & Schulte im Walde for German, using the methodology described in Scheible and Schulte im Walde (2014). The questions cover the semantic relations adjective antonym, noun hypernym, noun synonym, noun antonym and verb antonym.

Overall, this dataset constitutes a deep semantic challenge, containing very specific, domain-related and potentially low-frequent semantic details that are difficult to solve even for humans. For example, the tasks include antonyms such as biblical:secular::deaf:hearing or screech:whisper::ink:erase. The English dataset contains 7,516 analogies and the German dataset contains 2,462 analogies.


In the same way, we created an analogy dataset with 10,000 unique analogy questions from the hypernymy and meronymy relations in BLESS (Baroni and Lenci, 2011), by randomly picking semantic relation pairs.

Next to analogies, the resource also contains:


Schm280 (Schmidt et al., 2001) translated 280 word pairs from WordSim353 (Finkelstein et al., 2001). As they did not re-rate the German relation pairs after translation, we collected new ratings for the German pairs from 10 subjects, applying the same conditions as the original WordSim353 collection. The dataset contains 280 translated and newly rated word pairs for WordSim350.


Maximilian Köper, Christian Scheible, Sabine Schulte im Walde
Multilingual Reliability and "Semantic" Structure of Continuous Word Spaces
In: Proceedings of the 11th Conference on Computational Semantics (IWCS). London, UK, April 2015.