Understanding and Improving Continuous Experimentation: From A/B Testing to Continuous Software Optimization

Rasmus Ros

Understanding and Improving Continuous Experimentation: From A/B Testing to Continuous Software Optimization

Bidragets översatta titel : Förstå och Förbättra Kontinuerlig Experimentering: Från A/B Testning till Kontinuerlig Mjukvaru Optimering

Rasmus Ros

Forskningsoutput: Avhandling › Doktorsavhandling (sammanläggning)

878 Nedladdningar (Pure)

Sammanfattning

Kontrollerade experiment (t.ex. A/B test) används av många företag med användarintensiva produkter för att förbättra mjukvara med hjälp av användardata. Några företag anammar ett experimentdrivet tillvägagångssätt för mjukvaruutveckling med kontinuerliga experiment (KE). Med KE så utvärderas alla förändringar som påverkar användare med ett experiment och det finns specialiserade roller som aktivt letar efter möjligheter för att experimentera med funktionalitet.

Målet med avhandlingen är att beskriva och stödja utövandet av KE i industri. Det ingår tre huvudsakliga bidrag. För det första, en recension av publikationer inom KE om: infrastruktur och processer, problem-lösnings par som applicerats i praktiken, och fördelar och nackdelar med KE. För det andra, en multipel fallstudie med 12 företag för att analysera hur experimentering används och varför vissa företag inte till fullo drar nytta av fördelarna med KE. En teori om vilka faktorer som påverkar KE (FACE) konstrueras för att verkställa målet. Slutligen, ett verktyg för bandit optimering med villkor kallat COMBO tas fram för att stödja automatiserade experiment med många variabler samtidigt, i realtid i en produktionsmiljö.

Forskningen i avhandling är utförd inom paradigmen av designvetenskap och med empiriska forskningsmetoder: simuleringsexperiment av verktygsförslag och fallstudier. Andra forskningsmetoder inkluderar systematisk recension och teoribyggnad.

Från FACE tas tre faktorer fram som förklarar nyttan med KE: (1) investeringar i datainfrastruktur, (2) komplexiteten i problemområdet, och (3) incitamentsstrukturer för experimentering. Riktlinjer förses om hur företag kan sträva mot att nå bättre KE baserat på dess faktorer. Alla tre faktorer är relevanta för företag som vill tillämpa KE, speciellt för de företag som vill tillämpa algoritmer så som de i COMBO för att stödja personalisering av mjukvara till användares kontext i en process av kontinuerlig optimering.

Bidragets översatta titel	Förstå och Förbättra Kontinuerlig Experimentering: Från A/B Testning till Kontinuerlig Mjukvaru Optimering
Originalspråk	engelska
Kvalifikation	Doktor
Handledare	Runeson, Per, handledare Bjarnason, Elizabeth, Biträdande handledare
Sponsorer för avhandling	Wallenberg AI, Autonomous Systems and Software Program (WASP)
Tilldelningsdatum	2022 mars 4
Utgivningsort	Lund
Förlag	Department of Computer Science, Lund University
ISBN (tryckt)	978-91-8039-177-1
ISBN (elektroniskt)	978-91-8039-178-8
Status	Published - 2022

Bibliografisk information

Defence details
Date: 2022-03-04
Time: 13:15
Place: Lecture hall E:A, building E, Ole Römers väg 3, Faculty of Engineering LTH, Lund University, Lund.
External reviewer(s)
Name: Stol, Klaas-Jan
Title: Dr.
Affiliation: University college Cork, Ireland.
---

Ämnesklassifikation (UKÄ)

Programvaruteknik

Tillgång till dokument

Thesis_Rasmus_RosPublicerad version, 1,14 MB

2 Artikel i vetenskaplig tidskrift
1 Konferenspaper i proceeding

Controlled experimentation in continuous experimentation: Knowledge and challenges
Auer, F., Ros, R., Kaltenbrunner, L., Runeson, P. & Felderer, M., 2021, I: Information and Software Technology. 134, 106551.
Forskningsoutput: Tidskriftsbidrag › Artikel i vetenskaplig tidskrift › Peer review

Öppen tillgång
Data-driven software design with Constraint Oriented Multi-variate Bandit Optimization (COMBO)
Ros, R. & Hammar, M., 2020 aug. 18, I: Empirical Software Engineering.
Forskningsoutput: Tidskriftsbidrag › Artikel i vetenskaplig tidskrift › Peer review

Öppen tillgång
Continuous experimentation scenarios: A case study in e-commerce
Ros, R. & Bjarnason, E., 2018 okt. 22, Proceedings - 44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018. Bures, T. & Angelis, L. (red.). IEEE - Institute of Electrical and Electronics Engineers Inc., s. 353-356 4 s. 8498231
Forskningsoutput: Kapitel i bok/rapport/Conference proceeding › Konferenspaper i proceeding › Peer review

Citera det här

@phdthesis{ab3676ccbb8a41c698f08b5c8380b404,

title = "Understanding and Improving Continuous Experimentation: From A/B Testing to Continuous Software Optimization",

abstract = "Controlled experiments (i.e. A/B tests) are used by many companies with user-intensive products to improve their software with user data. Some companies adopt an experiment-driven approach to software development with continuous experimentation (CE). With CE, every user-affecting software change is evaluated in an experiment and specialized roles seek out opportunities to experiment with functionality. The goal of the thesis is to describe current practice and support CE in industry. The main contributions are threefold. First, a review of the CE literature on: infrastructure and processes, the problem-solution pairs applied in industry practice, and the benefits and challenges of the practice. Second, a multi-case study with 12 companies to analyze how experimentation is used and why some companies fail to fully realize the benefits of CE. A theory for Factors Affecting Continuous Experimentation (FACE) is constructed to realize this goal. Finally, a toolkit called Constraint Oriented Multi-variate Bandit Optimization (COMBO) is developed for supporting automated experimentation with many variables simultaneously, live in a production environment.The research in the thesis is conducted under the design science paradigm using empirical research methods, with simulation experiments of tool proposals and a multi-case study on company usage of CE. Other research methods include systematic literature review and theory building.From FACE we derive three factors that explain CE utility: (1) investments in data infrastructure, (2) user problem complexity, and (3) incentive structures for experimentation. Guidelines are provided on how to strive towards state-of-the-art CE based on company factors. All three factors are relevant for companies wanting to use CE, in particular, for those companies wanting to apply algorithms such as those in COMBO to support personalization of software to users' context in a process of continuous optimization. ",

author = "Rasmus Ros",

note = "Defence details Date: 2022-03-04 Time: 13:15 Place: Lecture hall E:A, building E, Ole R{\"o}mers v{\"a}g 3, Faculty of Engineering LTH, Lund University, Lund. External reviewer(s) Name: Stol, Klaas-Jan Title: Dr. Affiliation: University college Cork, Ireland. --- ",

year = "2022",

language = "English",

isbn = "978-91-8039-177-1",

series = "LU-CS-DISS 2022-22",

publisher = "Department of Computer Science, Lund University",

type = "Doctoral Thesis (compilation)",

}

TY - THES

T1 - Understanding and Improving Continuous Experimentation

T2 - From A/B Testing to Continuous Software Optimization

AU - Ros, Rasmus

N1 - Defence details Date: 2022-03-04 Time: 13:15 Place: Lecture hall E:A, building E, Ole Römers väg 3, Faculty of Engineering LTH, Lund University, Lund. External reviewer(s) Name: Stol, Klaas-Jan Title: Dr. Affiliation: University college Cork, Ireland. ---

PY - 2022

Y1 - 2022

N2 - Controlled experiments (i.e. A/B tests) are used by many companies with user-intensive products to improve their software with user data. Some companies adopt an experiment-driven approach to software development with continuous experimentation (CE). With CE, every user-affecting software change is evaluated in an experiment and specialized roles seek out opportunities to experiment with functionality. The goal of the thesis is to describe current practice and support CE in industry. The main contributions are threefold. First, a review of the CE literature on: infrastructure and processes, the problem-solution pairs applied in industry practice, and the benefits and challenges of the practice. Second, a multi-case study with 12 companies to analyze how experimentation is used and why some companies fail to fully realize the benefits of CE. A theory for Factors Affecting Continuous Experimentation (FACE) is constructed to realize this goal. Finally, a toolkit called Constraint Oriented Multi-variate Bandit Optimization (COMBO) is developed for supporting automated experimentation with many variables simultaneously, live in a production environment.The research in the thesis is conducted under the design science paradigm using empirical research methods, with simulation experiments of tool proposals and a multi-case study on company usage of CE. Other research methods include systematic literature review and theory building.From FACE we derive three factors that explain CE utility: (1) investments in data infrastructure, (2) user problem complexity, and (3) incentive structures for experimentation. Guidelines are provided on how to strive towards state-of-the-art CE based on company factors. All three factors are relevant for companies wanting to use CE, in particular, for those companies wanting to apply algorithms such as those in COMBO to support personalization of software to users' context in a process of continuous optimization.

AB - Controlled experiments (i.e. A/B tests) are used by many companies with user-intensive products to improve their software with user data. Some companies adopt an experiment-driven approach to software development with continuous experimentation (CE). With CE, every user-affecting software change is evaluated in an experiment and specialized roles seek out opportunities to experiment with functionality. The goal of the thesis is to describe current practice and support CE in industry. The main contributions are threefold. First, a review of the CE literature on: infrastructure and processes, the problem-solution pairs applied in industry practice, and the benefits and challenges of the practice. Second, a multi-case study with 12 companies to analyze how experimentation is used and why some companies fail to fully realize the benefits of CE. A theory for Factors Affecting Continuous Experimentation (FACE) is constructed to realize this goal. Finally, a toolkit called Constraint Oriented Multi-variate Bandit Optimization (COMBO) is developed for supporting automated experimentation with many variables simultaneously, live in a production environment.The research in the thesis is conducted under the design science paradigm using empirical research methods, with simulation experiments of tool proposals and a multi-case study on company usage of CE. Other research methods include systematic literature review and theory building.From FACE we derive three factors that explain CE utility: (1) investments in data infrastructure, (2) user problem complexity, and (3) incentive structures for experimentation. Guidelines are provided on how to strive towards state-of-the-art CE based on company factors. All three factors are relevant for companies wanting to use CE, in particular, for those companies wanting to apply algorithms such as those in COMBO to support personalization of software to users' context in a process of continuous optimization.

M3 - Doctoral Thesis (compilation)

SN - 978-91-8039-177-1

T3 - LU-CS-DISS 2022-22

PB - Department of Computer Science, Lund University

CY - Lund

ER -

Understanding and Improving Continuous Experimentation: From A/B Testing to Continuous Software Optimization

Sammanfattning

Bibliografisk information

Ämnesklassifikation (UKÄ)

Tillgång till dokument

Fingeravtryck

Forskningsoutput

Controlled experimentation in continuous experimentation: Knowledge and challenges

Data-driven software design with Constraint Oriented Multi-variate Bandit Optimization (COMBO)

Continuous experimentation scenarios: A case study in e-commerce

Citera det här