LogoLogo
diffblue.comFree trial
  • Discover Diffblue Cover
  • Get Started
    • What is Diffblue Cover?
    • Get started
      • Free trial
      • Get started - Cover Plugin
      • Get started - Cover CLI
      • Get started - Cover Pipeline
      • Get started - Cover Reports
    • Specs & Reqs
    • Reference Deployments
    • Licensing
      • Online license activation
      • Offline license activation
      • Diffblue License Manager
    • Update Cover
    • FAQs
    • Diffblue Learning
      • Get started
        • Just the basics
        • Free trial
        • Cover Plugin (IDE)
        • Cover CLI (Command Line)
      • Developer
        • Unit tests (IDE)
        • Unit tests (CLI)
        • Test coverage
      • DevOps
        • GitHub
        • GitLab
        • Other CI
      • Administrator
        • Admin - IntelliJ
        • Admin - CLI
        • Admin - Reports
      • Test coverage
        • Developer
        • Senior developer
        • Cover Reports Administrator
  • EVALUATION & ONBOARDING
    • Proof of Value
    • Jumpstart
      • Prerequisites for onboarding
      • Phase 1: Up and running
        • Module 1: Create your Cover unit test baseline
        • Module 2: Cover Pipeline CI integration
      • Phase 2: Developer productivity
        • Module 3: Getting started using Cover
        • Module 4: Introduction to Cover Reports
      • Phase 3: Advanced topics
        • Module 5: Speed up your test execution
        • Module 6: Getting more from Cover
  • Features
    • Cover Plugin
      • Writing tests
        • Gutter icons
        • Menu options
        • Run configurations
        • Cover Plugin tool window
        • Test Review
        • Test examples
        • Creating partial tests
        • Creating skeleton tests
        • Covering all enum values
        • Test insertion order
        • Diffblue Sandbox
        • Environment Check Cache
      • Project configuration
        • General dependencies
        • Test framework dependencies
      • Cover Plugin settings
        • Test Naming
        • Test Formatting
        • Spring configuration options
        • Method Annotations
        • Test Directory
        • Reset Cover Plugin settings
      • Cover Plugin admin
        • Core Maintenance
        • Cover Plugin toolbar menu
        • Cover Plugin status bar widget
        • Telemetry
        • Memory management
        • Using SonarQube with Cover Plugin
        • Log files
        • Troubleshooting
    • Cover CLI
      • Writing tests
        • Command summary
        • Test examples
        • Creating partial tests
        • Customizing test inputs
        • Customizing test setup
        • Test naming
        • Test formatting
        • Test insertion order
        • Patch files
        • Diffblue Sandbox
        • Operational behaviors
        • Test validation
      • Project configuration
        • Preflight checks
        • General dependencies
        • Test framework dependencies
        • Compiling your project successfully
          • Building a Maven project
          • Building a Gradle project
        • Configuring Cover to work with your project's build system
          • Quick guide
          • Configuration file in detail
          • Getting Started with Ant Projects
          • Default configuration
        • Mocking using Mockito
        • Spring profiles
        • Runtime environment
      • Commands & Arguments
        • Environment configuration for CI
        • Packages, classes, and methods
        • Quoting command line arguments
        • Argument files
      • Cover CLI admin
        • Core Maintenance
        • Telemetry
        • Memory management
        • Using Cover CLI in Eclipse
        • Using SonarQube with Cover CLI
        • Log files
        • Troubleshooting
      • Environment Configuration
    • Cover Pipeline
      • Cover Pipeline for CI
        • Quick Start - General
        • Quick Start - Jenkins
        • Quick Start - Azure Pipelines
        • Quick Start - AWS Codebuild
        • Integrating Diffblue Cover into CI on pull requests
        • Installation - VMs or CI run
      • Cover Pipeline for GitLab
        • Introduction
        • Installation and initial setup
        • GitLab workflow
        • Configuration
        • Troubleshooting
        • Related topics
      • Cover Pipeline for GitHub
        • Introduction
        • Installation and initial setup
        • GitHub workflow
        • Configuration
        • Troubleshooting
        • Related topics
    • Cover Reports
      • Cover Reports Contributor
        • Java project config (JaCoCo)
        • Generate and upload reports bundles
        • Authenticated uploads
      • Cover Reports User
        • Navigation
        • Dashboards
        • Telemetry data
        • Export API
        • Considerations
      • Cover Reports Administrator
        • Install and update Cover Reports
        • Configuration options
        • Database backup
        • SSO with Cover Reports
        • Uninstall Cover Reports
    • Cover Optimize
      • Get started - Cover Optimize
      • Cover Optimize & Gradle
      • Cover Optimize & Maven
      • Patch files
    • Cover Refactor
      • Get started - Cover Refactor
    • Cover Annotations
      • Mocking Annotations
      • Custom Input Annotations
      • Interesting Value Annotations
    • Output Codes
      • E - Environment Codes
      • L - License Codes
      • R - Reason Codes
      • T - Testability Codes
      • V - Validation Codes
      • Working with output codes
        • Working with code E020
        • Working with codes E057 to E065
        • Working with code E085
        • Working with code R005
        • Working with code R006
        • Working with code R008
        • Working with code R011
        • Working with code R012
        • Working with code R013
        • Working with code R026
        • Working with code R031
        • Working with code V003
    • Tutorials
      • How to measure test quality
      • How to improve code coverage
      • How to test a new feature
      • How to find regressions
      • How to use Diffblue Cover in test driven development (TDD)
      • How to write tests for Kotlin projects
      • Examples of tests created by Diffblue Cover
      • Best practices for testing private methods
  • Updates & Upgrades
    • Update Cover
    • Cover Editions
    • What's new
    • Release archive
      • 2025-06-01
      • 2025-05-02
      • 2025-05-01
      • 2025-04-02
      • 2025-04-01
      • 2025-03-02
      • 2025-03-01
      • 2025-02-02
      • 2025-02-01
      • 2025-01-02
      • 2025-01-01
      • 2024-12-02
      • 2024-12-01
      • 2024-11-02
      • 2024-11-01
      • 2024-10-02
      • 2024-10-01
      • 2024-09-02
      • 2024-09-01
      • 2024-08-02
      • 2024-08-01
      • 2024-07-04
      • 2024-07-03
      • 2024-07-01
      • 2024-06-02
      • 2024-06-01
      • 2024-05-02
      • 2024-05-01
      • 2024-04-02
      • 2024-04-01
      • 2024-03-02
      • 2024-03-01
      • 2024-02-02
      • 2024-02-01
      • 2024-01-02
      • 2024-01-01
      • 2023-12-02
      • 2023-12-01
      • 2023-11-03
      • 2023-11-02
      • 2023-11-01
      • 2023-10-02
      • 2023-10-01
      • 2023-09-02
      • 2023-09-01
      • 2023-08-02
      • 2023-08-01
      • 2023-07-03
      • 2023-07-02
      • 2023-07-01
      • 2023-06-02
      • 2023-06-01
      • 2023-05-02
      • 2023-05-01
      • 2023-04-02
      • 2023-04-01
      • 2023-03-02
      • 2023-03-01
      • 2023-02-02
      • 2023-02-01
      • 2023-01-02
      • 2023-01-01
      • 2022-12-02
      • 2022-12-01
      • 2022-11-02
      • 2022-11-01
      • 2022-10-03
      • 2022-10-02
      • 2022-10-01
      • 2022-09-02
      • 2022-09-01
      • 2022-08-05
      • 2022-08-03
      • 2022-08-02
      • 2022-08-01
      • 2022-07-02
      • 2022-07-01
      • 2022-06-02
      • 2022-06-01
      • 2022-05-02
      • 2022-05-01
      • 2022-04-02
      • 2022-04-01
      • 2022-03-02
      • 2022-03-01
      • 2022-02-02
      • 2022-02-01
      • 2022-01-02
      • 2022-01-01
      • 2021-06-02
      • 2021-02-01
  • Legal
    • Diffblue Legal
      • Diffblue End User License Agreement (EULA)
      • Cover Third Party Notices and/or Licenses
      • Privacy Notice
Powered by GitBook
On this page
  • Advantages of mutation testing
  • Mutation Testing Plugins
  • Setting up PIT on your project

Was this helpful?

Export as PDF
  1. Features
  2. Tutorials

How to measure test quality

Use mutation testing to benchmark unit test quality

PreviousTutorialsNextHow to improve code coverage

Last updated 2 months ago

Was this helpful?

Traditional test coverage (i.e line, statement, branch, etc.) measures what code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code.

Mutation testing fills this gap. Faults (or mutations) are seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived. The quality of your tests can be gauged from the percentage of mutations killed.

Source code is changed or mutated in three ways – , and .

Value mutations

Change a small value to a larger value or vice versa.

/* Initial code */

int mod = 1000000007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;

/* Changed code */

int mod = 1007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;

Decision mutations

Change logical or arithmetic operators to detect errors.

/* Initial code */

if (a < b) {
  c = 10;
} else {
  c = 20;
}

/* Changed code */

if (a > b) {
  c = 10;
} else {
  c = 20;
}

Statement mutations

Delete a statement or replace it with some other statement

/* Initial code */

if (a < b) {
  c = 10;
} else {
  c = 20;
}

/* Changed code */

if (a < b) {
  d = 10;
} else {
  d = 20;
}

Advantages of mutation testing

  • Brings a good level of error detection in the program

  • Discovers ambiguities in the source code

  • Helps testers write better tests and measure effectiveness of tests written

  • Makes source code more efficient

Mutation Testing Plugins


Setting up PIT on your project

Maven

Add this plugin to your pom.xml:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.17.4</version>
</plugin>

Then run: mvn test-compile org.pitest:pitest-maven:mutationCoverage, which will output an HTML report to target/pit-reports/<YYYYMMDDHHMI>.

Gradle

Add this plugin to your build.gradle:

plugins {
    id 'info.solidsoft.pitest' version '1.15.0'
}

// To aggregate Pitest results for a multi-module projects
apply plugin: 'info.solidsoft.pitest.aggregator'

// For multi-module projects
subprojects {
    apply plugin: 'java'
    apply plugin: 'info.solidsoft.pitest'

    pitest {
        threads = 4
        // Export mutations.xml and line coverage for aggregation
        outputFormats = ["XML"]
        exportLineCoverage = true
        timestampedReports = false

        if (project.name in ['module-without-any-test']) {
            failWhenNoMutations = false
        }

        reportAggregator {
            // aggregation configuration (optional)
        }
    }
}

Run gradle pitest pitestReportAggregate task to have an aggregated report be produced at ${PROJECT_DIR}/build/reports/pitest.

(recommended)

|

For multi-module projects, please see .

|

PIT
Judy
Jester
Jumble
Documentation
Plugin
Aggregating tests across modules
Documentation
Plugin
value
decision
statement