Improving the Detection of Relations Between Objects in an Image Using Textual Semantics

Research output: Chapter in Book/Report/Conference proceedingPaper in conference proceeding

Abstract

In this article, we describe a system that classifies relations between entities extracted from an image. We started from the idea that we could utilize lexical and semantic information from text associated with the image, such as captions or surrounding text, rather than just the geometric and visual characteristics of the entities found in the image. We collected a corpus of images from Wikipedia together with their corresponding articles. In our experimental setup, we extracted two kinds of entities from the images, human beings and horses, and we defined three relations that could exist between them: Ride, Lead,or None. We used geometric features as a baseline to identify the relations between the entities and we describe the improvements brought by the addition of bag-of-word features and predicate–argument structures that we extracted from the text. The best semantic model resulted in a relative error reduction of more than 18 % over the baseline

Details

Authors
Organisations
Research areas and keywords

Subject classification (UKÄ) – MANDATORY

  • Computer and Information Science

Keywords

  • Semantic parsing, Relation extraction from images, Machine learning
Original languageEnglish
Title of host publicationPattern Recognition Applications and Methods /Lecture Notes in Computer Science
EditorsAna Fred, Maria De Marsico, Antoine Tabbone
PublisherSpringer
Pages133-145
Number of pages13
Volume9443
ISBN (Print)978-3-319-25529-3, 978-3-319-25530-9
Publication statusPublished - 2015
Publication categoryResearch
Peer-reviewedYes
Event3rd International Conference on Pattern Recognition Applications an Methods (ICPRAM 2014) - Angers, Angers, France
Duration: 2014 Mar 62014 Mar 8

Publication series

Name
Volume9443

Conference

Conference3rd International Conference on Pattern Recognition Applications an Methods (ICPRAM 2014)
CountryFrance
CityAngers
Period2014/03/062014/03/08