Evidence-Based Guidelines for Advancing Continuous Experimentation

Continuous experimentation (CE) is used by many internet-facing companies to improve the value of their products based on user feedback gathered, e.g. through on-line experiments using A/B testing. Frameworks and theories for CE have been derived through academic research from applications in large internet facing companies. To assist practitioners in a broader range of companies, we herein present guidelines for CE, based on empirical data from interviews with 27 practitioners at 12 companies of varying sizes and CE maturity. The guidelines include advise derived from our (previously published) theory of factors that affect CE (FACE). These practitioner guidelines may assist companies in making informed decisions concerning their CE practices and contribute to efficient and effective experimentation. Our advice includes building processes and infrastructure gradually, combining quantitative and qualitative methods, focusing on goals and incentives for CE, and being mindful of ethics.