Understanding and Improving Continuous Experimentation: From A/B Testing to Continuous Software Optimization

Rasmus Ros

Research output: ThesisDoctoral Thesis (compilation)

896 Downloads (Pure)

Abstract

Controlled experiments (i.e. A/B tests) are used by many companies with user-intensive products to improve their software with user data. Some companies adopt an experiment-driven approach to software development with continuous experimentation (CE). With CE, every user-affecting software change is evaluated in an experiment and specialized roles seek out opportunities to experiment with functionality.

The goal of the thesis is to describe current practice and support CE in industry. The main contributions are threefold. First, a review of the CE literature on: infrastructure and processes, the problem-solution pairs applied in industry practice, and the benefits and challenges of the practice. Second, a multi-case study with 12 companies to analyze how experimentation is used and why some companies fail to fully realize the benefits of CE. A theory for Factors Affecting Continuous Experimentation (FACE) is constructed to realize this goal. Finally, a toolkit called Constraint Oriented Multi-variate Bandit Optimization (COMBO) is developed for supporting automated experimentation with many variables simultaneously, live in a production environment.

The research in the thesis is conducted under the design science paradigm using empirical research methods, with simulation experiments of tool proposals and a multi-case study on company usage of CE. Other research methods include systematic literature review and theory building.

From FACE we derive three factors that explain CE utility: (1) investments in data infrastructure, (2) user problem complexity, and (3) incentive structures for experimentation. Guidelines are provided on how to strive towards state-of-the-art CE based on company factors. All three factors are relevant for companies wanting to use CE, in particular, for those companies wanting to apply algorithms such as those in COMBO to support personalization of software to users' context in a process of continuous optimization.
Translated title of the contributionFörstå och Förbättra Kontinuerlig Experimentering: Från A/B Testning till Kontinuerlig Mjukvaru Optimering
Original languageEnglish
QualificationDoctor
Supervisors/Advisors
  • Runeson, Per, Supervisor
  • Bjarnason, Elizabeth, Assistant supervisor
Thesis sponsors
Award date2022 Mar 4
Place of PublicationLund
Publisher
ISBN (Print)978-91-8039-177-1
ISBN (electronic) 978-91-8039-178-8
Publication statusPublished - 2022

Bibliographical note

Defence details
Date: 2022-03-04
Time: 13:15
Place: Lecture hall E:A, building E, Ole Römers väg 3, Faculty of Engineering LTH, Lund University, Lund.
External reviewer(s)
Name: Stol, Klaas-Jan
Title: Dr.
Affiliation: University college Cork, Ireland.
---

Subject classification (UKÄ)

  • Software Engineering

Fingerprint

Dive into the research topics of 'Understanding and Improving Continuous Experimentation: From A/B Testing to Continuous Software Optimization'. Together they form a unique fingerprint.

Cite this