TY - GEN
T1 - On Rapid Releases and Software Testing
AU - Mäntylä, Mika
AU - Khomh, Foutse
AU - Adams, Bram
AU - Engström, Emelie
AU - Kai, Petersen
PY - 2013
Y1 - 2013
N2 - Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in software testing effort after moving to rapid releases. We analyze the results of 312,502 execution runs of the 1,547 mostly manual system level test cases of Mozilla Fire fox from 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulated our findings with a Mozilla QA engineer. In rapid releases, testing has a narrower scope that enables deeper investigation of the features and regressions with the highest risk, while traditional releases run the whole test suite. Furthermore, rapid releases make it more difficult to build a large testing community, forcing Mozilla to increase contractor resources in order to sustain testing for rapid releases.
AB - Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in software testing effort after moving to rapid releases. We analyze the results of 312,502 execution runs of the 1,547 mostly manual system level test cases of Mozilla Fire fox from 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulated our findings with a Mozilla QA engineer. In rapid releases, testing has a narrower scope that enables deeper investigation of the features and regressions with the highest risk, while traditional releases run the whole test suite. Furthermore, rapid releases make it more difficult to build a large testing community, forcing Mozilla to increase contractor resources in order to sustain testing for rapid releases.
KW - Mozilla Software testing agile releases bugs builds opensource release model
U2 - 10.1109/ICSM.2013.13
DO - 10.1109/ICSM.2013.13
M3 - Paper in conference proceeding
SP - 20
EP - 29
BT - [Host publication title missing]
A2 - Lisa, O'Conner
PB - IEEE - Institute of Electrical and Electronics Engineers Inc.
T2 - 29th IEEE International Conference on Maintenance (ICSM), 2013
Y2 - 22 September 2013 through 28 September 2013
ER -