A/B Testing in the Small

Amalia Paulsson, Per Runeson, Rasmus Ros

Research output: Chapter in Book/Report/Conference proceedingPaper in conference proceedingpeer-review


Previous research on A/B testing and continuous experimentation has mostly focused on large-scale settings with company-external customers. However, work efficiency and satisfaction for company coworkers may be highly related to internal tools and their fit-for-use. In this study, we therefore explore A/B testing for online services exclusively used by company co-workers. We A/B tested two versions of Customer Admin, a tool that helps 34 500 IKEA co-workers to interact with customer data. The study comprised i) stakeholder interviews to understand objectives and phrase the experimentation goals, and ii) A/B test execution where data was collected and processed from approximately 350 users for 33 days. While the user base is relatively large for this internal system, the primary metric data collected was too scarce to allow distinction between the two versions. However, secondary metrics and a user questionnaire suggest that the users are more efficient in the new menu design and that the users prefer it to the old. We conclude that A/B testing requires lots of data, making it less feasible for internal users, also for large, global organizations. Thus, we propose combining quantitative and qualitative evaluation of internal tools and propose further research on how to adapt A/B testing for smaller-scale settings.
Original languageEnglish
Title of host publicationInternational Conference on Product-Focused Software Process Improvement
EditorsDavide Taibi, Marco Kuhrmann, Pekka Abrahamsson
Number of pages15
ISBN (Electronic)978-3-031-21388-5
ISBN (Print)978-3-031-21387-8
Publication statusPublished - 2022 Nov 22
Event23:rd International Conference on Product-Focused Software Process Improvement, PROFES - Jyväskylä, Finland
Duration: 2022 Nov 212022 Nov 23

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference23:rd International Conference on Product-Focused Software Process Improvement, PROFES

Subject classification (UKÄ)

  • Software Engineering


Dive into the research topics of 'A/B Testing in the Small'. Together they form a unique fingerprint.

Cite this