How to measure test quality

Use mutation testing to benchmark unit test quality

Traditional test coverage (i.e line, statement, branch, etc.) measures what code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code.

Mutation testing fills this gap. Faults (or mutations) are seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived. The quality of your tests can be gauged from the percentage of mutations killed.

Source code is changed or mutated in three ways – value, decision and statement.

Value mutations

Change a small value to a larger value or vice versa.

/* Initial code */

int mod = 1000000007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;

/* Changed code */

int mod = 1007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;

Decision mutations

Change logical or arithmetic operators to detect errors.

/* Initial code */

if (a < b) {
  c = 10;
} else {
  c = 20;
}

/* Changed code */

if (a > b) {
  c = 10;
} else {
  c = 20;
}

Statement mutations

Delete a statement or replace it with some other statement

/* Initial code */

if (a < b) {
  c = 10;
} else {
  c = 20;
}

/* Changed code */

if (a < b) {
  d = 10;
} else {
  d = 20;
}

Advantages of mutation testing

• Brings a good level of error detection in the program

• Discovers ambiguities in the source code

• Helps testers write better tests and measure effectiveness of tests written

• Makes source code more efficient

Mutation Testing Plugins

• PIT (recommended)

• Judy

• Jester

• Jumble

---

Setting up PIT on your project

Maven

Documentation | Plugin

Add this plugin to your pom.xml:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.17.4</version>
</plugin>

Then run: mvn test-compile org.pitest:pitest-maven:mutationCoverage, which will output an HTML report to target/pit-reports/<YYYYMMDDHHMI>.

For multi-module projects, please see Aggregating tests across modules.

Gradle

Documentation | Plugin

Add this plugin to your build.gradle:

plugins {
    id 'info.solidsoft.pitest' version '1.15.0'
}

// To aggregate Pitest results for a multi-module projects
apply plugin: 'info.solidsoft.pitest.aggregator'

// For multi-module projects
subprojects {
    apply plugin: 'java'
    apply plugin: 'info.solidsoft.pitest'

    pitest {
        threads = 4
        // Export mutations.xml and line coverage for aggregation
        outputFormats = ["XML"]
        exportLineCoverage = true
        timestampedReports = false

        if (project.name in ['module-without-any-test']) {
            failWhenNoMutations = false
        }

        reportAggregator {
            // aggregation configuration (optional)
        }
    }
}

Run gradle pitest pitestReportAggregate task to have an aggregated report be produced at ${PROJECT_DIR}/build/reports/pitest.