Informativity in image captions vs. referring expressions
Files
Published version
Date
2020-10-11
DOI
Authors
Coppock, Elizabeth
Dionne, Danielle
Graham, Nathanial
Ganem, Elias
Zhao, Shijie
Lin, Shawn
Liu, Wenxing
Wijaya, Derry
Version
Published version
OA Version
Citation
Elizabeth Coppock, Danielle Dionne, Nathanial Graham, Elias Ganem, Shijie Zhao, Shawn Lin, Wenxing Liu, Derry Wijaya. 2020. "Informativity in Image Captions vs. Referring Expressions." Proceedings of the Conference on Probability and Meaning. Probability and Meaning 2020. University of Gothenburg, 2020-10-14 - 2020-10-15.
Abstract
At the intersection between computer vision and natural language processing, there has been recent progress on two natural language generation tasks: Dense Image Captioning and Referring Expression Generation for objects in complex scenes. The former aims to provide a caption for a specified object in a complex scene for the benefit of an interlocutor who may not be able to see it. The latter aims to produce a referring expression that will serve to identify a given object in a scene that the interlocutor can see. The two tasks are designed for different assumptions about the common ground between the interlocutors, and serve very different purposes, although they both associate a linguistic description with an object in a complex scene. Despite these fundamental differences, the distinction between these two tasks is sometimes overlooked. Here, we undertake a side-by-side comparison between image captioning and reference game human datasets and show that they differ systematically with respect to informativity. We hope that an understanding of the systematic differences among these human datasets will ultimately allow them to be leveraged more effectively in the associated engineering tasks.
Description
License
Copyright 2020 Association for Computational Linguistics