Mutation Testing: Fewer, Faster, and Smarter

The growing reliance on automated software tests raises a fundamental question: How trustworthy are these automated tests? Today, mutation testing is acknowledged within academic circles as the most promising technique for assessing the fault-detection capability of a test suite. The technique deliberately injects faults (called mutants) into the production code and counts how many of them are caught by the test suite.

Mutation testing shines in systems with high statement coverage because uncaught mutants reveal weaknesses in code which is supposedly covered by tests. Safety-critical systems –where safety standards dictate high statement coverage - are therefore a prime candidate for mutation testing. In safety-critical software, C and C++ dominate the technology stack. Yet in the mutation testing community, the C language family is somehow neglected: a systematic literature review on mutation testing from 2019 reports that less than 25% of the primary studies target source code from the C language family. Despite the apparent potential, mutation testing is difficult to adopt in industrial settings, because the technique -in its basic form- requires a tremendous amount of computing power. Without optimisations, the entire code base must be compiled and tested separately for each injected mutant. Hence for medium to large test suites, mutation testing without optimisations becomes prohibitively expensive.

To make mutation testing effective in an industrial setting, we set three objectives: (1) generate fewer mutants, (2) process them smarter and (3) execute them faster. To meet our objectives, we investigate the most promising techniques from the current state-of-the-art. This ranges from leveraging cloud technology to compiler integrated techniques using the Clang front-end. These optimisation strategies allow to eliminate the compilation and execution overhead in order to support efficient mutation testing for the C language family.

As a final step, we perform an empirical study on the perception of mutation testing in industry. The aim is to investigate whether the advances are sufficient to allow industrial adoption and to identify any remaining barriers preventing industrial adoption.

In this thesis, we show that a combination of mutation testing optimisation techniques from the do fewer, do faster, and do smarter are needed to perform mutation testing in a continuous integration setting. Furthermore, the industrial perception of mutation testing is evolving as additional organisations recognise its potential.