Multi-Modal Word Synset Induction (2017)
A word in natural language can be polysemous, having multiple meanings, as well as synonymous, meaning the same thing as other words. Word sense induction attempts to find the senses of polysemous words. Synonymy detection attempts to find when two words are interchangeable. We combine these tasks, first inducing word senses and then detecting similar senses to form word-sense synonym sets (synsets) in an unsupervised fashion. Given pairs of images and text with noun phrase labels, we perform synset induction to produce collections of underlying concepts described by one or more noun phrases. We find that considering multi-modal features from both visual and textual context yields better induced synsets than using either context alone. Human evaluations show that our unsupervised, multi-modally induced synsets are comparable in quality to annotation-assisted ImageNet synsets, achieving about 84% of ImageNet synsets' approval.
In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI-17), pp. 4116--4122, Melbourne, Australia 2017.

Slides (PDF) Poster
Raymond J. Mooney Faculty mooney [at] cs utexas edu
Jesse Thomason Ph.D. Student jesse [at] cs utexas edu