An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper

Kostiantyn Kucher; Nicole Sultanum; Angel Daza; Vasiliki Simaki; Maria Skeppstedt; Barbara Plank; Jean Daniel Fekete; Narges Mahyar

doi:10.1109/BELIV57783.2022.00008

An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper

Kostiantyn Kucher, Nicole Sultanum, Angel Daza, Vasiliki Simaki, Maria Skeppstedt, Barbara Plank, Jean Daniel Fekete, Narges Mahyar

Research output: Chapter in Book/Report/Conference proceeding › Paper in conference proceeding › peer-review

Abstract

Appropriate evaluation and experimental design are fundamental for empirical sciences, particularly in data-driven fields. Due to the successes in computational modeling of languages, for instance, research outcomes are having an increasingly immediate impact on end users. As the gap in adoption by end users decreases, the need increases to ensure that tools and models developed by the research communities and practitioners are reliable, trustworthy, and supportive of the users in their goals. In this position paper, we focus on the issues of evaluating visual text analytics approaches. We take an interdisciplinary perspective from the visualization and natural language processing communities, as we argue that the design and validation of visual text analytics include concerns beyond computational or visual/interactive methods on their own. We identify four key groups of challenges for evaluating visual text analytics approaches (data ambiguity, experimental design, user trust, and "big picture"concerns) and provide suggestions for research opportunities from an interdisciplinary perspective.

Original language	English
Title of host publication	Proceedings - 2022 IEEE 9th Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022
Publisher	IEEE - Institute of Electrical and Electronics Engineers Inc.
Pages	28-37
Number of pages	10
ISBN (Electronic)	9798350396294
DOIs	https://doi.org/10.1109/BELIV57783.2022.00008
Publication status	Published - 2022
Event	9th IEEE Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022 - Virtual, Online, United States Duration: 2022 Oct 17 → …

Conference

Conference	9th IEEE Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022
Country/Territory	United States
City	Virtual, Online
Period	2022/10/17 → …

Subject classification (UKÄ)

Human Computer Interaction
Social Sciences Interdisciplinary

Free keywords

Human-centered computing -
Visualization
Visualization design and evaluation methods

Access to Document

10.1109/BELIV57783.2022.00008

1 Preprint (in preprint archive)

An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper
Kucher, K., Sultanum, N., Daza, A., Simaki, V., Skeppstedt, M., Plank, B., Fekete, J.-D. & Mahyar, N., 2022, arXiv.org, 10 p.
Research output: Working paper/Preprint › Preprint (in preprint archive)

Cite this

Kucher, K., Sultanum, N., Daza, A., Simaki, V., Skeppstedt, M., Plank, B., Fekete, J. D., & Mahyar, N. (2022). An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper. In Proceedings - 2022 IEEE 9th Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022 (pp. 28-37). IEEE - Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BELIV57783.2022.00008

Kucher, Kostiantyn ; Sultanum, Nicole ; Daza, Angel et al. / An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics : Position Paper. Proceedings - 2022 IEEE 9th Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022. IEEE - Institute of Electrical and Electronics Engineers Inc., 2022. pp. 28-37

@inproceedings{cc9d3309aad24ff193edac0d0142de3f,

title = "An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper",

abstract = "Appropriate evaluation and experimental design are fundamental for empirical sciences, particularly in data-driven fields. Due to the successes in computational modeling of languages, for instance, research outcomes are having an increasingly immediate impact on end users. As the gap in adoption by end users decreases, the need increases to ensure that tools and models developed by the research communities and practitioners are reliable, trustworthy, and supportive of the users in their goals. In this position paper, we focus on the issues of evaluating visual text analytics approaches. We take an interdisciplinary perspective from the visualization and natural language processing communities, as we argue that the design and validation of visual text analytics include concerns beyond computational or visual/interactive methods on their own. We identify four key groups of challenges for evaluating visual text analytics approaches (data ambiguity, experimental design, user trust, and {"}big picture{"}concerns) and provide suggestions for research opportunities from an interdisciplinary perspective.",

keywords = "Human-centered computing -, Visualization, Visualization design and evaluation methods",

author = "Kostiantyn Kucher and Nicole Sultanum and Angel Daza and Vasiliki Simaki and Maria Skeppstedt and Barbara Plank and Fekete, {Jean Daniel} and Narges Mahyar",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 9th IEEE Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022 ; Conference date: 17-10-2022",

year = "2022",

doi = "10.1109/BELIV57783.2022.00008",

language = "English",

pages = "28--37",

booktitle = "Proceedings - 2022 IEEE 9th Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022",

publisher = "IEEE - Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

}

Kucher, K, Sultanum, N, Daza, A, Simaki, V, Skeppstedt, M, Plank, B, Fekete, JD & Mahyar, N 2022, An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper. in Proceedings - 2022 IEEE 9th Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022. IEEE - Institute of Electrical and Electronics Engineers Inc., pp. 28-37, 9th IEEE Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022, Virtual, Online, United States, 2022/10/17. https://doi.org/10.1109/BELIV57783.2022.00008

An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper. / Kucher, Kostiantyn; Sultanum, Nicole; Daza, Angel et al.
Proceedings - 2022 IEEE 9th Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022. IEEE - Institute of Electrical and Electronics Engineers Inc., 2022. p. 28-37.

Research output: Chapter in Book/Report/Conference proceeding › Paper in conference proceeding › peer-review

TY - GEN

T1 - An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics

T2 - 9th IEEE Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022

AU - Kucher, Kostiantyn

AU - Sultanum, Nicole

AU - Daza, Angel

AU - Simaki, Vasiliki

AU - Skeppstedt, Maria

AU - Plank, Barbara

AU - Fekete, Jean Daniel

AU - Mahyar, Narges

PY - 2022

Y1 - 2022

N2 - Appropriate evaluation and experimental design are fundamental for empirical sciences, particularly in data-driven fields. Due to the successes in computational modeling of languages, for instance, research outcomes are having an increasingly immediate impact on end users. As the gap in adoption by end users decreases, the need increases to ensure that tools and models developed by the research communities and practitioners are reliable, trustworthy, and supportive of the users in their goals. In this position paper, we focus on the issues of evaluating visual text analytics approaches. We take an interdisciplinary perspective from the visualization and natural language processing communities, as we argue that the design and validation of visual text analytics include concerns beyond computational or visual/interactive methods on their own. We identify four key groups of challenges for evaluating visual text analytics approaches (data ambiguity, experimental design, user trust, and "big picture"concerns) and provide suggestions for research opportunities from an interdisciplinary perspective.

AB - Appropriate evaluation and experimental design are fundamental for empirical sciences, particularly in data-driven fields. Due to the successes in computational modeling of languages, for instance, research outcomes are having an increasingly immediate impact on end users. As the gap in adoption by end users decreases, the need increases to ensure that tools and models developed by the research communities and practitioners are reliable, trustworthy, and supportive of the users in their goals. In this position paper, we focus on the issues of evaluating visual text analytics approaches. We take an interdisciplinary perspective from the visualization and natural language processing communities, as we argue that the design and validation of visual text analytics include concerns beyond computational or visual/interactive methods on their own. We identify four key groups of challenges for evaluating visual text analytics approaches (data ambiguity, experimental design, user trust, and "big picture"concerns) and provide suggestions for research opportunities from an interdisciplinary perspective.

KW - Human-centered computing -

KW - Visualization

KW - Visualization design and evaluation methods

U2 - 10.1109/BELIV57783.2022.00008

DO - 10.1109/BELIV57783.2022.00008

M3 - Paper in conference proceeding

AN - SCOPUS:85145772994

SP - 28

EP - 37

BT - Proceedings - 2022 IEEE 9th Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022

PB - IEEE - Institute of Electrical and Electronics Engineers Inc.

Y2 - 17 October 2022

ER -

Kucher K, Sultanum N, Daza A, Simaki V, Skeppstedt M, Plank B et al. An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper. In Proceedings - 2022 IEEE 9th Workshop on Evaluation and Beyond - Methodological Approaches to Visualization, BELIV 2022. IEEE - Institute of Electrical and Electronics Engineers Inc. 2022. p. 28-37 doi: 10.1109/BELIV57783.2022.00008

An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper

Abstract

Conference

Subject classification (UKÄ)

Free keywords

Access to Document

Fingerprint

Research output

An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper

Cite this