Improving the Detection of Relations Between Objects in an Image Using Textual Semantics

Dennis Medved; Fangyuan Jiang; Peter Exner; Magnus Oskarsson; Pierre Nugues; Karl Åström

doi:10.1007/978-3-319-25530-9_9

Improving the Detection of Relations Between Objects in an Image Using Textual Semantics

Dennis Medved, Fangyuan Jiang, Peter Exner, Magnus Oskarsson, Pierre Nugues, Karl Åström

Research output: Chapter in Book/Report/Conference proceeding › Paper in conference proceeding › peer-review

Abstract

In this article, we describe a system that classifies relations between entities extracted from an image. We started from the idea that we could utilize lexical and semantic information from text associated with the image, such as captions or surrounding text, rather than just the geometric and visual characteristics of the entities found in the image. We collected a corpus of images from Wikipedia together with their corresponding articles. In our experimental setup, we extracted two kinds of entities from the images, human beings and horses, and we defined three relations that could exist between them: Ride, Lead,or None. We used geometric features as a baseline to identify the relations between the entities and we describe the improvements brought by the addition of bag-of-word features and predicate–argument structures that we extracted from the text. The best semantic model resulted in a relative error reduction of more than 18 % over the baseline

Original language	English
Title of host publication	Pattern Recognition Applications and Methods /Lecture Notes in Computer Science
Editors	Ana Fred, Maria De Marsico, Antoine Tabbone
Publisher	Springer
Pages	133-145
Number of pages	13
Volume	9443
ISBN (Print)	978-3-319-25529-3, 978-3-319-25530-9
DOIs	https://doi.org/10.1007/978-3-319-25530-9_9
Publication status	Published - 2015
Event	3rd International Conference on Pattern Recognition Applications an Methods (ICPRAM 2014) - Angers, Angers, France Duration: 2014 Mar 6 → 2014 Mar 8

Publication series

Name
Volume	9443

Conference

Conference	3rd International Conference on Pattern Recognition Applications an Methods (ICPRAM 2014)
Country/Territory	France
City	Angers
Period	2014/03/06 → 2014/03/08

Subject classification (UKÄ)

Computer and Information Science

Free keywords

Semantic parsing
Relation extraction from images
Machine learning

Access to Document

10.1007/978-3-319-25530-9_9

1 Doctoral Thesis (compilation)

Deep Learning Applications for Biomedical Data and Natural Language Processing
Medved, D., 2018, Lund: Department of Computer Science, Lund University. 130 p.
Research output: Thesis › Doctoral Thesis (compilation)

Open Access
File

Cite this

Medved, D., Jiang, F., Exner, P., Oskarsson, M., Nugues, P., & Åström, K. (2015). Improving the Detection of Relations Between Objects in an Image Using Textual Semantics. In A. Fred, M. De Marsico, & A. Tabbone (Eds.), Pattern Recognition Applications and Methods /Lecture Notes in Computer Science (Vol. 9443, pp. 133-145). Springer. https://doi.org/10.1007/978-3-319-25530-9_9

@inproceedings{dfd3070258ac4a52a127e62bd8093251,

title = "Improving the Detection of Relations Between Objects in an Image Using Textual Semantics",

abstract = "In this article, we describe a system that classifies relations between entities extracted from an image. We started from the idea that we could utilize lexical and semantic information from text associated with the image, such as captions or surrounding text, rather than just the geometric and visual characteristics of the entities found in the image. We collected a corpus of images from Wikipedia together with their corresponding articles. In our experimental setup, we extracted two kinds of entities from the images, human beings and horses, and we defined three relations that could exist between them: Ride, Lead,or None. We used geometric features as a baseline to identify the relations between the entities and we describe the improvements brought by the addition of bag-of-word features and predicate–argument structures that we extracted from the text. The best semantic model resulted in a relative error reduction of more than 18 % over the baseline",

keywords = "Semantic parsing, Relation extraction from images, Machine learning",

author = "Dennis Medved and Fangyuan Jiang and Peter Exner and Magnus Oskarsson and Pierre Nugues and Karl {\AA}str{\"o}m",

year = "2015",

doi = "10.1007/978-3-319-25530-9_9",

language = "English",

isbn = "978-3-319-25529-3",

volume = "9443",

publisher = "Springer",

pages = "133--145",

editor = "Ana Fred and {De Marsico}, Maria and Antoine Tabbone",

booktitle = "Pattern Recognition Applications and Methods /Lecture Notes in Computer Science",

address = "Germany",

note = "3rd International Conference on Pattern Recognition Applications an Methods (ICPRAM 2014) ; Conference date: 06-03-2014 Through 08-03-2014",

}

Medved, D, Jiang, F, Exner, P, Oskarsson, M , Nugues, P & Åström, K 2015, Improving the Detection of Relations Between Objects in an Image Using Textual Semantics. in A Fred, M De Marsico & A Tabbone (eds), Pattern Recognition Applications and Methods /Lecture Notes in Computer Science. vol. 9443, Springer, pp. 133-145, 3rd International Conference on Pattern Recognition Applications an Methods (ICPRAM 2014), Angers, France, 2014/03/06. https://doi.org/10.1007/978-3-319-25530-9_9

Improving the Detection of Relations Between Objects in an Image Using Textual Semantics. / Medved, Dennis; Jiang, Fangyuan; Exner, Peter et al.
Pattern Recognition Applications and Methods /Lecture Notes in Computer Science. ed. / Ana Fred; Maria De Marsico; Antoine Tabbone. Vol. 9443 Springer, 2015. p. 133-145.

Research output: Chapter in Book/Report/Conference proceeding › Paper in conference proceeding › peer-review

TY - GEN

T1 - Improving the Detection of Relations Between Objects in an Image Using Textual Semantics

AU - Medved, Dennis

AU - Jiang, Fangyuan

AU - Exner, Peter

AU - Oskarsson, Magnus

AU - Nugues, Pierre

AU - Åström, Karl

PY - 2015

Y1 - 2015

N2 - In this article, we describe a system that classifies relations between entities extracted from an image. We started from the idea that we could utilize lexical and semantic information from text associated with the image, such as captions or surrounding text, rather than just the geometric and visual characteristics of the entities found in the image. We collected a corpus of images from Wikipedia together with their corresponding articles. In our experimental setup, we extracted two kinds of entities from the images, human beings and horses, and we defined three relations that could exist between them: Ride, Lead,or None. We used geometric features as a baseline to identify the relations between the entities and we describe the improvements brought by the addition of bag-of-word features and predicate–argument structures that we extracted from the text. The best semantic model resulted in a relative error reduction of more than 18 % over the baseline

AB - In this article, we describe a system that classifies relations between entities extracted from an image. We started from the idea that we could utilize lexical and semantic information from text associated with the image, such as captions or surrounding text, rather than just the geometric and visual characteristics of the entities found in the image. We collected a corpus of images from Wikipedia together with their corresponding articles. In our experimental setup, we extracted two kinds of entities from the images, human beings and horses, and we defined three relations that could exist between them: Ride, Lead,or None. We used geometric features as a baseline to identify the relations between the entities and we describe the improvements brought by the addition of bag-of-word features and predicate–argument structures that we extracted from the text. The best semantic model resulted in a relative error reduction of more than 18 % over the baseline

KW - Semantic parsing

KW - Relation extraction from images

KW - Machine learning

U2 - 10.1007/978-3-319-25530-9_9

DO - 10.1007/978-3-319-25530-9_9

M3 - Paper in conference proceeding

SN - 978-3-319-25529-3

SN - 978-3-319-25530-9

VL - 9443

SP - 133

EP - 145

BT - Pattern Recognition Applications and Methods /Lecture Notes in Computer Science

A2 - Fred, Ana

A2 - De Marsico, Maria

A2 - Tabbone, Antoine

PB - Springer

T2 - 3rd International Conference on Pattern Recognition Applications an Methods (ICPRAM 2014)

Y2 - 6 March 2014 through 8 March 2014

ER -

Improving the Detection of Relations Between Objects in an Image Using Textual Semantics

Abstract

Publication series

Conference

Subject classification (UKÄ)

Free keywords

Access to Document

Fingerprint

Research output

Deep Learning Applications for Biomedical Data and Natural Language Processing

Cite this