Hindi Visual Genome: A Dataset for Multi-Modal English to Hindi Machine Translation

Abstract

Visual Genome is a dataset connecting structured image information with English language. We present “Hindi Visual Genome”, a multi-modal dataset consisting of text and images suitable for English-Hindi multi-modal machine translation task and multi-modal research. We have selected short English segments (captions) from Visual Genome along with the associated images and automatically translated them to Hindi. A careful manual post-editing followed which took the associated images into account. Overall, we prepared a set of 31,525 segments (which we conveniently split into training, development and testdata), accompanied by a challenge test set of 1,400 segments. This challenge test set was created by searching for particularly ambiguous English words based on the embedding similarity and manually selecting those where the image helps to resolvethe ambiguity. Our dataset is the first manually revised dataset for multi-modal English-Hindi machine translation, freely available for non-commercial research purposes. Our Hindi version of Visual Genome also allows to create Hindi image labelers or other practical tools. Hindi Visual Genome also served in Workshop on Asian Translation (WAT) 2019 Multi-Modal Translation Task.