Projects per year
Abstract
Previous research on A/B testing and continuous experimentation has mostly focused on large-scale settings with company-external customers. However, work efficiency and satisfaction for company coworkers may be highly related to internal tools and their fit-for-use. In this study, we therefore explore A/B testing for online services exclusively used by company co-workers. We A/B tested two versions of Customer Admin, a tool that helps 34 500 IKEA co-workers to interact with customer data. The study comprised i) stakeholder interviews to understand objectives and phrase the experimentation goals, and ii) A/B test execution where data was collected and processed from approximately 350 users for 33 days. While the user base is relatively large for this internal system, the primary metric data collected was too scarce to allow distinction between the two versions. However, secondary metrics and a user questionnaire suggest that the users are more efficient in the new menu design and that the users prefer it to the old. We conclude that A/B testing requires lots of data, making it less feasible for internal users, also for large, global organizations. Thus, we propose combining quantitative and qualitative evaluation of internal tools and propose further research on how to adapt A/B testing for smaller-scale settings.
Original language | English |
---|---|
Title of host publication | International Conference on Product-Focused Software Process Improvement |
Editors | Davide Taibi, Marco Kuhrmann, Pekka Abrahamsson |
Publisher | Springer |
Pages | 449–463 |
Number of pages | 15 |
ISBN (Electronic) | 978-3-031-21388-5 |
ISBN (Print) | 978-3-031-21387-8 |
DOIs | |
Publication status | Published - 2022 Nov 22 |
Event | 23:rd International Conference on Product-Focused Software Process Improvement, PROFES - Jyväskylä, Finland Duration: 2022 Nov 21 → 2022 Nov 23 |
Publication series
Name | Lecture Notes in Computer Science |
---|---|
Publisher | Springer |
Volume | 13709 |
ISSN (Print) | 0302-9743 |
ISSN (Electronic) | 1611-3349 |
Conference
Conference | 23:rd International Conference on Product-Focused Software Process Improvement, PROFES |
---|---|
Country/Territory | Finland |
City | Jyväskylä |
Period | 2022/11/21 → 2022/11/23 |
Subject classification (UKÄ)
- Software Engineering
Fingerprint
Dive into the research topics of 'A/B Testing in the Small'. Together they form a unique fingerprint.Projects
- 1 Finished
-
Continuous Experimentation and Optimization
Ros, R. (PI), Runeson, P. (Supervisor) & Bjarnason, E. (Assistant supervisor)
2016/04/01 → 2022/03/04
Project: Dissertation