diffblue.comFree trial

How to improve code coverage

Introduction

Diffblue Cover is designed to create tests that provide good coverage for your code “out of the box” without needing any configuration. That said, the amount of coverage you get is also dependent on the testability of your code, the build configuration of the project, and also the specific domain or environment that your code operates within. In this section we’ll give you several ways to increase your code coverage.

Be aware that improving coverage is often not resolved with a single fix, and while incremental is typically achieved in jumps and plateaus. Improving testability of the code may take multiple changes to come into full effect.

The following categories of advice below can help guide where to look depending on where you are in your code coverage journey.

  1. Ensuring your project is Cover ready

  2. Best practices

  3. General improvements

  4. Special situation handling

Ensuring your project is Cover ready

This section should be followed before trying to create tests for your project to ensure Cover is able to perform best out of the box.

Completely compile your project

This might sound obvious, but sometimes getting a complete compilation of a project can be a high bar! If the project is not completely compiled, Cover won’t be able to write a full set of tests because it works on compiled bytecode.

Run the command to build the entire project, which is for example something like mvn install -DskipTests for Maven projects - the docs for the project should tell you. See Compiling your project successfully for further troubleshooting.

IntelliJ-specific compilation advice

If you're using IntelliJ, note that sometimes it isn’t obvious that the project didn’t completely compile due to the interaction between IntelliJ and the underlying build system. Compiling the project externally by running a Maven command from the command line or a run configuration is not sufficient.

IntelliJ needs to import the build configuration which can be achieved by selecting Maven or Gradle from the right side bar and clicking the Reload All Projects button (two circular arrows symbol). The project must compile correctly using the Build Project button (green hammer symbol).

Fix missing/broken dependencies

Diffblue Cover works by running your code, and so it needs all the dependencies to be correct. Crucially, it requires the dependencies listed in the build system to match the actual dependencies in the code. In our experience, it's common to see discrepancies between the POM file (Maven) dependencies and the code due to the complexity of dependency management and Java’s dynamic class loading.

If your project is fully compiled and you are seeing class loading errors in the output of Cover, it’s usually a sign there’s a missing dependency, or an incorrect version of a dependency.

  • Code that is deployed on an application server such as Tomcat may make use of provided-scope dependencies. Such dependencies are assumed to be provided by the application server, but Cover does not have access to these dependencies.

  • Dependencies-of-dependencies can also be a problem: they are not required for building the project because none of their classes are directly referenced, but they are required for running your code because they are referenced from a class in your dependencies. For example, interfaces and implementations are often packaged separately; whereas your code compiles with just the interface (often called api) dependency, it does not execute without having the dependency containing the interface implementations on the classpath too. Similarly, mock implementations of the API required for unit testing may be packaged separately and need to be added to test scope.

In either situation, you have to make sure that these dependencies are indeed part of the compile and test classpaths. Cover outputs the classpath that it receives from Maven or Gradle in its log file. If required dependencies are missing there, then change the scope of these dependencies in the build system configuration.

Best practices

This section describes best practices to follow when working with Cover on your code. This ensures not only the best performance from Cover itself, but the easier way to ensure progress and not lose progress along the way.

Commit Progress

Commit code and configuration changes to a branch in small, tidy commits immediately after making them. This will maintain a clear and comprehensible record of the changes made, why they were made, and make it easy to revisit or revert to a known good state of progress. This will also help to understand the progress made and facilitate later merging into mainline.

Refactor and Update

Be prepared to change production code to improve testability and resolve discovered issues. Sometimes this is unavoidable when the code is not reasonably unit-testable or found to have conflicting or unusable configuration in practice.

Allocate more memory / CPU

Cover requires at least 8GB of memory and 2 cores to function, but the more memory and CPU you can give it, the faster it goes. When Cover conducts a search for the best test for your code, a slow machine with insufficient memory can result in the search timing out, producing fewer tests. Ensure that sufficient memory is available to Cover by following the advice in Memory management.

You may also see warnings when your machine is heavily loaded, e.g. because you have multiple applications running that take up most of the cores of your machines (browser, video conferencing, IDE, compilation). In such cases, there is not enough CPU available to run Cover and it will slow down too much to operate properly. You have to reduce the load on your machine or make additional cores/memory available to Cover. If you are running Cover inside a VM or a container, change the settings to allocate more vCPUs and/or more memory to the VM and restart.

Create a Diffblue Arguments File

Add a .diffblue/create.args file to your project to ensure you maintain the arguments given to Cover. This will ensure you do not forget important arguments, and that you will not change anything unexpectedly. If you have version control for your project then commit the Diffblue arguments file into the repository. This will help manage complex Cover command lines, increase reproducibility and avoid surprises about lost coverage by forgetting crucial command line arguments. Full details on a arguments file can be found here.

Effective Debugging

The following techniques can make debugging much more effective.

  • Use the --keep-partial-tests option to obtain a partial test for debugging. This will help you understand how far Cover is able to progress and where to target further effort.

  • Run a partial test with coverage in the IDE to understand how far Cover progresses in creating a test for the method considered.

  • Run a partial test with debugging in the IDE to understand why it doesn’t take the right branch or throws an exception.

  • Drive coverage forward by a combination of configuring Cover, refactoring your code (if necessary) and changing the test manually. In the best case, you’ll get a test that can be fully generated by Cover (and thus maintained and re-generated); in the worst case you end up with a manually written test (and thus efforts are not lost and coverage is increased). Note that this last option is a catch-all for many of the later details on this page.

General Improvements

This section describes ways to improve coverage that apply to many projects and situations.

Uncovered branches

This occurs when one or more tests have been successfully written, but these tests don’t cover all the branches of the method under test. In this case proceed as follows:

  1. Run the tests with coverage to see which branches are not covered.

  2. Figure out which inputs are necessary for triggering the uncovered branches.

  3. Likely, you will need to use the techniques explained in #special-inputs-or-initializations to make Cover exercise these branches.

Disable sandboxing

By default, Cover runs with the Diffblue sandbox turned on because we do not want to damage your system environment by wiping your filesystem, sending sensitive data on the network, or deleting your database. Also, if code under test writes to the file system or does network I/O then we run the risk of non-deterministic behavior. We try to mock out these kind of I/O operations, but there are situations where they are beyond our control.

Disabling the sandbox when you're running in a safe environment could allow Diffblue Cover to get more coverage for this code, and then you can look at refining mocking controls and other behavior.

Use the --disable-sandbox argument to disable the Diffblue sandbox - see Diffblue Sandbox and Commands & Arguments for details.

Customized inputs or test set-up

The need for specific inputs or initializations is typically indicated by output code R013. The message of this output code contains a stacktrace that is the starting point for resolving the issue. To understand the issue, proceed as follows:

  1. Go to the place in your code where the exception is thrown according to the stacktrace.

  2. Try to understand why the exception is thrown and which inputs to the method under test or other initializations would be required to make the exception go away.

  3. If necessary, take the partial test that Cover produces with the --keep-partial-tests option, remove the @Ignore/@Disabled annotation and debug the test execution to understand how far the test progresses and what may be required to complete the test.

  4. Figure out how the input can be supplied to the code under test, among others this could be:

    1. An argument to the method under test or an argument to the class under test’s constructor/factory. In this provide custom inputs for primitives, String, or Enums using:

      • Annotations on the argument or

      • Custom inputs for the argument type, e.g. use parameter condition to make specific to the parameter.

      Provide custom inputs for other types using:

    2. Another method called on an object involved in the test’s Arrange section or a field in object involved in the test’s Arrange section. Use the techniques 4.1. above on the object involed.

    3. Call to a static method not involved in the test’s Arrange section or a static field in a class not involved in the test’s Arrange section. Set up a custom base class to call the required static methods or set the static field in a @BeforeAll or a @BeforeEach method.

Provide hints on test inputs

The hardest part about writing unit tests is creating the right inputs for your method under test. Cover is pretty smart about constructing inputs "out of the box", using a variety of techniques and heuristics. But when there are highly specific requirements on test inputs, for example specifically formatted strings, Cover may not be able to create the right input. We provide a couple of ways how you can provide hints for Cover about how to create the most useful inputs when writing tests for your project.

Annotations

Java Annotations are a powerful method for adding metadata to Java code. We provide annotations that enable you to tell Cover which specific values you want to see in your tests for a particular method or which objects to mock.

For example the following method is annotated with some genuine examples of song titles that can be used to achieve coverage:

public static boolean isDayRelatedSongTitle(@InTestsUseStrings({"I Don't Like Mondays", "Here Comes The Weekend"}) String title) {
    return Stream.of(DayOfWeek.values())
            .map(DayOfWeek::name)
            .map(String::toLowerCase)
            .anyMatch(title.toLowerCase()::contains);
}

See Cover Annotations to find out more.

Rules

More complex rules for creating test inputs can by codified via a file. For example, you can tell Diffblue Cover to use a particular factory method whenever creating an object of a particular type as shown below.

Consider that you are working with a library project and need to deal with International Standard Book Numbers in the form of an org.example.ISBN class objects, created by parsing ISBN formatted strings. On it's own, Cover may not be able to discover valid ISBN string formats. To tell Cover what to do in this specific situation, you can create a DiffblueRules.yml file in the root of the module containing the following factory rules that specify three valid ISBN strings that can be used to create test objects of type org.example.ISBN:

com.example.ISBN:
  - factory:
      method: com.example.ISBN.parse:(Ljava/lang/String;)Lcom/example/ISBN;
      params: [ "1-56619-909-3" ] # An 'old' style ISBN-10 formatted identifier
  - factory:
      method: com.example.ISBN.parse:(Ljava/lang/String;)Lcom/example/ISBN;
      params:
        - "978-1-56619-909-4" # A 'new' style ISBN-13 formatted identifier
  - factory:
      method: com.example.Platform.getBean:(Ljava/lang/Class;)Ljava/lang/Object;
      params: [ com.example.ISBN ]

See Customizing test inputs to find out more.

Test factory pattern

Let’s consider the following method AuthorInfo.getAge to compute the age of an author.

package org.example.authors;
import java.time.LocalDate;

public class AuthorInfo {
  private LocalDate currentDate;

  public AuthorInfo(LocalDate currentDate) {
    this.currentDate = currentDate;
  }

  public int getAge(Author author) {
    if (author.getDied() == null) {
      return currentDate.getYear() - author.getBorn().getYear();
    }
    return author.getDied().getYear() - author.getBorn().getYear();
  }
}

with the entity class for the author:

package org.example.authors;
import jakarta.annotation.Nullable;
import lombok.Data;
import java.time.LocalDate;

@Data
public class Author {
  private String firstName;
  private String lastName;
  private LocalDate born;
  @Nullable private LocalDate died;
}

In order to test this getAge method we need Author instances that are initialized in particular ways, Diffblue Cover may not be smart enough to come up with.

Using Cover Annotations we can help Diffblue Cover by providing appropriate Author instances in a test factory. We put a class AuthorTestUtil into the src/test/java tree:

package org.example.authors;
import com.diffblue.cover.annotations.InterestingTestFactory;
import java.time.LocalDate;

public class AuthorTestUtil {
  private AuthorTestUtil() {}

  @InterestingTestFactory
  public static final Author authorAlive() {
    return new Author("J. K.", "Rowling", LocalDate.of(1965, 7, 31), null);
  }

  @InterestingTestFactory
  public static final Author authorDead() {
    return new Author("Franz", "Kafka", LocalDate.of(1883, 7, 3), LocalDate.of(1924, 6, 3));
  }
}

This utility class provides factory methods to return suitably initialized Author instances.

We also annotate the factory methods with @InterestingTestFactory in order to give Cover a hint that these factory methods should be preferred.

Instead of using Cover annotations, alternatively Customizing test inputs can be used to modify a DiffblueRules.yaml file:

org.example.authors.Author:
  - factory:
   method: org.example.authors.AuthorTestUtil.authorAlive:()Lorg/example/authors/Author;
  - factory:
   method: org.example.authors.AuthorTestUtil.authorDead:()Lorg/example/authors/Author;

Cover then generates the following tests:

@Test
void testGetAge_authorAlive() {
     // Arrange
     AuthorInfo authorInfo = new AuthorInfo(LocalDate.of(1970, 1, 1));

     // Act and Assert
     assertEquals(5, authorInfo.getAge(AuthorTestUtil.authorAlive()));
}

@Test
void testGetAge_authorDead() {
     // Arrange
     AuthorInfo authorInfo = new AuthorInfo(LocalDate.of(1970, 1, 1));

     // Act and Assert
     assertEquals(41, authorInfo.getAge(AuthorTestUtil.authorDead()));
}

Test Factory from Resource Pattern

We can also load test data from resource files, e.g. we could have example Author data in a JSON file src/test/resources/authors.json:

[
  {
    "firstName": "J. K.",
    "lastName": "Rowling",
    "born": "1965-07-31",
    "died": null
  },
  {
    "firstName": "Franz",
    "lastName": "Kafka",
    "born": "1883-07-03",
    "died": "1924-06-03"
  }
]

We put then a utility class AuthorTestUtil for loading Authors from the resource into the src/test/java tree:

package org.example.authors;
import com.diffblue.cover.annotations.InterestingTestFactory;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jsr310.JSR310Module;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;

public class AuthorTestUtil {
  private AuthorTestUtil() {}
  private static final List<Author> AUTHORS;
  static {
    ObjectMapper objectMapper = new ObjectMapper();
    objectMapper.registerModule(new JSR310Module());
    try(InputStream authorInputStream = AuthorTestUtil.class.getResourceAsStream("/authors.json")) {
      AUTHORS = objectMapper.readValue(authorInputStream, new TypeReference<List<Author>>(){});
    } catch(IOException e) {
      throw new RuntimeException("Failed to load authors.json", e);
    }
  }

  @InterestingTestFactory
  public static final Author authorAlive() {
    return AUTHORS.get(0);
  }

  @InterestingTestFactory
  public static final Author authorDead() {
    return AUTHORS.get(1);
  }
}

The generated tests will look like those above in Test factory pattern.

Special situation handling

This section describes approaches that help only in specific situations.

Spring application configuration

The following details how to improve coverage when you have spring configurations. THese are broken down based on how your spring configurations are configured.

Spring configuration files

These are assumed to be in src/main/resources. How to help Diffblue Cover understand these configurations depends on how you distinguish production from test configurations.

  • If you have no distinction between production and test configurations, the best approach is to make a copy of your configurations file from your main source tree to your test source tree (e.g. src/main/resources/applications.properties to src/test/resources/application-test.properties) and pass in the active profiles open as above.

Configuration server

Another (not recommended) approach is to create a profile on your configuration server (if such a feature exists) and then use such a configuration for creating and running tests. However, this also means that your tests will constantly call to the configuration server, making them slow and flaky. Also, a servers-side configuration test can make all your tests fail.

Home grown configuration system

You need to configure your configuration system so that it provides configuration values during unit test creation and execution. Your configuration system may, for example, provide you with a mechanism to configure this through system properties, e.g. let’s assume this is -DconfigProfile=test . Note that if there is no distinction between production and test configurations, they should be distinguished before performing the following.

Spring dependency injection

There are various approaches to improve Diffblue Cover's test generation when spring dependency injection is used. This section discusses how to work with some of these scenarios.

Programmatic spring dependency injection

This should just work out-of-the-box. If you see R026 and R027, you may resolve them by providing appropriate beans in a Configuration class that you put into the src/test/java tree, e.g. my.SpringTestConfiguration, and then pass the --spring-configuration my.SpringTestConfiguration to Diffblue Cover. Cover will then use this additional configuration class to construct the test context.

XML files for spring dependency injection

This often works out-of-the-box. You may need to repartition your XML configuration to avoid certain beans (e.g. database connectors, external systems clients) to be loaded in the test profile and provide test versions of these beans instead. You may see R026 or R027 output codes otherwise when the loading of the production beans fails.

Handling getBean() calls

You will likely see R013, R026 or R027 output codes. You have to get rid of the getBean() calls by refactoring your code to inject these beans through standard mechanisms (e.g. Autowired).

Singletons for external calls

Static methods for external calls

Database interfaces

The approach to use here depends on the kind of database interface technology used in your project.

JPA-compatible / Spring repositories

These database interfaces should just work out-of-the-box. To test repository classes you’ll need to have a test-scope dependency to an in-memory database, e.g. H2.

Other Database Interfaces (Hibernate, MyBatis, JDBC)

File operations

For read-only file status queries, e.g. Files.exists add the --mock-static java.nio.file.Files option to Cover.

For reading files only add a test file with appropriate content into src/test/resources and pass or inject the filename as necessary (e.g. using custom inputs or annotations).

For creating or writing to files:

  • Pass or inject an appropriate directory where to create the file (e.g. System.getProperty("java.io.tmpdir")) as necessary (e.g. using custom inputs or annotations).

  • Use an @AfterAll method in a custom base class for cleaning up the file.

  • Also note, that each test method must write to a different file, otherwise there will be undesired interactions during test execution.

  • If the only side effect is writing the file then Cover will not be able to write a complete test. In that case it is recommended to move the test into the user’s realm and add a assertions manually, e.g. that the file exists.

For deleting files:

  • Use a @BeforeAll method in a custom base class or creating a file to be deleted.

  • Pass or inject an appropriate directory where to create the file (e.g. System.getProperty("java.io.tmpdir")) as necessary (e.g. using custom inputs or annotations).

Note: Never try to override the user.dir system property. It is not the user’s home directory, but the directory where the JVM is run from. Changing that will crash the JVM.

PreviousTutorialsNextHow to test a new feature

Last updated