Customizing test inputs

By default, Diffblue Cover will automatically choose input values to use when writing tests, but it's possible to provide custom inputs that may unlock additional coverage.

Background

A key feature of Diffblue Cover is automatically identifying appropriate inputs necessary to write tests for a given method under test.

However, sometimes Diffblue Cover is unable to find appropriate inputs and thus does not produce useful tests. One such example is with the open source XXL-JOB project where Cover fails to produce tests for the CronExpression class, resulting in an R013 output code:

com.xxl.job.admin.core.cron.CronExpression.<init>
  R013: No inputs found that don't throw a trivial exception
    Diffblue Cover tried to run the arrange/act section, but the method under
    test threw
    java.lang.NullPointerException
        at com.xxl.job.admin.core.cron.CronExpression.<init>(CronExpression.java:295)
    In order to prevent <init>(CronExpression)
    from throwing NullPointerException, add constructors or factory
    methods that make it easier to construct fully initialized objects used in
    <init>(CronExpression).
    See https://diff.blue/R013 to resolve this issue.

Also, because the R013 occurs in the primary constructor for the class under test, Diffblue Cover is then unable to produce CronExpression instances for a further 28 methods and R008 output codes:

com.xxl.job.admin.core.cron.CronExpression.addToSet
  R008: Failed to instantiate class under test
    Diffblue Cover was unable to construct an instance of CronExpression.
    Add a package-visible constructor or a factory method for testing which
    (ideally) takes no arguments, and does not throw, return null or return
    a subtype.
    See https://diff.blue/R008

These tests need input strings of a very specific format in order to successfully construct a CronExpression instance. In fact the javadoc in CronExpression.java devotes over 100 lines of comments to explaining the requirements for this format - helpfully providing an example of a valid cron expression string: "0 0 14-6 ? * FRI-MON".

java.lang.String:
  - immediate: "0 0 14-6 ? * FRI-MON"
    parameter: cronExpr

With this configuration file in place, when java.lang.String values are required, Diffblue Cover will try to use the constant cron expression value (immediate: "0 0 14-6 ? * FRI-MON") if the parameter begins with cronExpr (parameter: cronExpr). Using just this one custom input allows this class to go from 54 tests with 21% line coverage to 194 tests with 59% line coverage.

eLearning

Custom Inputs

You can specify custom inputs in a DiffblueRules.yaml or DiffblueRules.yml file. If both files exist in the same directory then both are read, with the .yaml variant taking precedence.

Diffblue Cover will stop with an E068 error if the file is malformed due to wrong indentation or invalid rule syntax. This behavior is intended to give quick feedback so that the identified problem can be addressed, rather than waste time and resources attempting to create tests without all the custom rules.

If a rule is read successfully but the class can't be found, an E143 warning will be given. This may be due to the class not being available (e.g. in a different module) or due to an incorrect class name. Test generation will not be stopped in this case.

Custom Inputs for Strings

When writing unit tests, by default Diffblue uses predefined / context-based values for String objects. However, the user can come up with a specific String such as a name or a password. For example if the user needs to use “Peter” wherever in the codebase an argument called clientName appears and “rabbit” for an argument called password, the syntax will be:

java.lang.String:
  - immediate: "Peter"
    parameter: clientName
  - immediate: "rabbit"
    parameter: password

This is shown in the image below:

Custom Inputs for Primitives

When writing unit tests, by default Diffblue uses predefined / context-based values for primitives such as int, long, etc. However, the user might want to push a certain value when tests are being written. For example, if the user wants a value of 11 for all primitives of type long to be used in the whole codebase, the following lines should be added in the DiffblueRules.yaml:

long:
   - immediate: 11

If a particular primitive of long type such as 0987654321 is needed to match a certain method argument such as accountNumber in the whole codebase, the syntax should be:

long:
 -  immediate : 987654321
    parameter: accountNumber

Finally, the user might decide to target a given method and one of its arguments. For example if the user wants the argument called amount in the method addBalance() to take the value 16, a syntax like this should be used:

long:
  - immediate: 16
    parameter: amount
    method: addBalance

This is shown in the image below:

Custom Inputs for .properties files

Sometimes the user wants to use specific arguments such as URIs from a given file such as a .properties file.

For example:

production-env.properties contains baseUri=https://app.example.com/

and

staging-env.properties contains baseUri=http://app.staging.example.com:8080/

To have these URIs used in unit tests, there two options depending on the accessibility of the .properties files:

1. Using files from the resources directory

In order to use http://app.staging.example.com:8080/ from the staging-env.properties file in your unit tests, add the following lines to the DiffblueRules.yaml file:

java.util.Properties:
  properties:
	resource: /staging-env.properties

2. Using files at the root of the project

In order to use http://app.staging.example.com:8080/ from the staging-env.properties file in your unit tests, add the following lines to the DiffblueRules.yaml file:

java.util.Properties:
  properties:
    file: /staging-env.properties

Rules Location

Custom rules are loaded from the project working directory and any of its ancestor directories. Assuming you have your project checked out into ~/Projects/MyProject and are working with a module SomeModule within that, then rules will be read from each of the following locations in order, if they exist:

• ~/Projects/MyProject/SomeModule/DiffblueRules.yaml

• ~/Projects/MyProject/SomeModule/DiffblueRules.yml

• ~/Projects/MyProject/DiffblueRules.yaml

• ~/Projects/DiffblueRules.yaml

• ~/DiffblueRules.yaml

• …

• /DiffblueRules.yaml

The intention here is that module and project level rules can be checked into the project source control system, but additional fallbacks into the user home directory are also possible. Rules are loaded from each discovered file in order, and so applicable rules local to the module will take precedence over rules defined at a wider scope.

Rules Format

Custom rules are defined in a YAML format with the structure shown here. Rules are grouped by the type of input produced, and so the top level keys are the type being produced, each with a list of rules at the next level down. For example the following would mean that the constant values "Diffblue", "Cover", "Custom", "Inputs" are all considered as candidate values to use whenever a String is required. Note that this does not guarantee that all of these constant values will be used, in practice the first candidate constant value may produce full coverage and so further candidate values are not needed.

java.lang.String:
  - immediate: "Diffblue"
  - immediate: "Cover"
  - immediate: "Custom"
  - immediate: "Inputs"

immediate: Rule

The first supported mechanism for producing values is via immediate constant values. This mechanism supports primitives, java.lang.String, and single- or multi-dimensional arrays of those types. Rules using this mechanism will have an immediate: <constant> component. Arrays of immediate values can be specified by adding one or more [] at the end of the normal type declaration, then using one of the YAML syntaxes for a list of values. For example, the following configuration specifies a single unconditional default rule for each supported type:

boolean:
  - immediate: true

byte:
  - immediate: 65 # 0x41

char:
  - immediate: '*'

float:
  - immediate: 888.888f

double:
  - immediate: 12345.6789

int:
  - immediate: 7 # low values are recommended in case used for resource allocation

long:
  - immediate: -23

short:
  - immediate: 291 # 0x0123

java.lang.String:
  - immediate: "Lorem ipsum dolor sit amet"

java.lang.String[]:
  - immediate: [ "Hello", "World!" ]
  - immediate: # this rule uses different syntax for the same rule as above
    - "Hello"
    - "World!"

java.lang.String[][]:
  - immediate: [ [ "Multidimension", "arrays" ], [ "are", "also", "supported" ] ]
  - immediate: # this rule uses different syntax for the same rule as above
    - [ "Multidimension", "arrays" ]
    - [ "are", "also", "supported" ]
  - immediate: # this rule uses another syntax variation for the same rule
    - - "Multidimension"
      - "arrays"
    - - "are"
      - "also"
      - "supported"

factory: Rule

The second supported mechanism for producing values is via factory methods. This mechanism supports producing class and interface instances and their single- or multi-dimensional arrays, but not enumerations, java.lang.String or primitives.

Factory methods must meet the following criteria in order to be used:

• Must be static methods or constructors.

• Must have public visibility.

• Any parameters must be able to take primitives, java.lang.String, java.lang.Class, or single- or multi-dimensional arrays of those types.

• Must have a return type assignable to the requested type.

Any factory methods that don't fit these criteria, or cannot be loaded with the user's classpath will be silently ignored.

The factory rule syntax will have a factory: key, with sub-keys of method: and params:. The method sub-key must be filled with a method identifier, starting with a class name, followed by a dot and then the method name (or <init> for constructors). The identifier is completed with a colon and the javap style method descriptor. The params sub-key must be filled with a list of values to be used as the parameters.

For example, if you are working with a library then you might need to deal with International Standard Book Numbers in the form of an ISBN class, created by parsing ISBN formatted strings. On it's own, Cover may not be able to discover valid ISBN string formats, in which case custom rules could be used to specify a couple of valid ISBN values that can be used to test methods that require an ISBN instance:

com.example.ISBN:
  - factory:
      method: com.example.ISBN.parse:(Ljava/lang/String;)Lcom/example/ISBN;
      params: [ "1-56619-909-3" ] # An 'old' style ISBN-10 formatted identifier
  - factory:
      method: com.example.ISBN.parse:(Ljava/lang/String;)Lcom/example/ISBN;
      params:
        - "978-1-56619-909-4" # A 'new' style ISBN-13 formatted identifier
  - factory:
      method: com.example.Platform.getBean:(Ljava/lang/Class;)Ljava/lang/Object;
      params: [ com.example.ISBN ]

In addition to general purpose factory methods, it's also common to provide classes in the src/test/java source tree with factory methods for use specifically in tests. In the following example a factory method is used to provide an array of ISBN instances for use where an array of ISBN objects is required:

com.example.ISBN[]:
  - factory:
      method: com.example.ISBNTestSupport.isbnArray:()[Lcom/example/ISBN;

properties: Rule

The properties rule is a special case rule specifically for creating and populating a java.util.Properties instance.

The properties rule can specify a file using an absolute or relative path to a .properties file. When an absolute path is provided then that absolute path will be used directly from the test. When a relative path is provided then the properties file will be located relative to the DiffblueRules.yaml, and will be re-resolved against the project's working directory.

For example, given the following rule supplied in the project root:

java.util.Properties:
  - properties:
      file: relative-file.properties

Then tests will attempt to populate a java.util.Properties instance as follows:

BufferedReader newBufferedReaderResult =
  Files.newBufferedReader(Paths.get("relative-file.properties"));

Properties properties = new Properties();
properties.load(newBufferedReaderResult);
newBufferedReaderResult.close();

Alternatively the properties rule can specify a classpath resource to load. Resources are loaded relative to the class under test, so typically an absolute resource path should be used.

For example, given the following rule:

java.util.Properties:
  - properties:
      resource: /resource.properties

Then tests of ExampleApp will attempt to populate a java.util.Properties instance as follows:

InputStream resourceAsStream =
  ExampleApp.class.getResourceAsStream("/resource.properties");

Properties properties = new Properties();
properties.load(resourceAsStream);
resourceAsStream.close();

Conditions

Typically a custom constant value is associated with one or more conditions for when this custom value should be used. When producing inputs for a particular method, supported conditions allow matching against the fully qualified class name, method name, or parameter name. For ease of use, matching is typically performed using case-insensitive substrings, but if finer control is required then the pattern can be wrapped in a ^ and $ and the pattern will be treated as a regular expression instead.

Consider the following example code under test and the numbered contexts where string inputs need to be produced:

package com.example.myapp;

public class StringProcessor {

  public static int parse(String text /* Context #1 */) {
    try {
      return Integer.parseInt(text);
    } catch (NumberFormatException e) {
      throw new IllegalArgumentException("Invalid input: " + text);
    }
  }

  public static String concatenateStrings(String str1 /* Context #2 */, String str2 /* Context #3 */) {
    return str1 + str2;
  }

  public static int getStringLength(String str1 /* Context #4 */) {
    return str1.length();
  }
}

package com.example.myapp;

public class NumberProcessor {

    public static int processNumber(String str1 /* Context #5 */) {
        try {
            return Integer.parseInt(str1);
        } catch (NumberFormatException e) {
            throw new IllegalArgumentException("Invalid input: " + str1);
        }
    } 
}

class: Condition

Class name conditions match against the fully qualified class name, com.example.myapp.StringProcessor in context #1-4 above, or com.example.myapp.NumberProcessor in context #5. For example class: "example.myapp" and class: "^.*Processor$" could both be used to match all contexts above.

method: Condition

Method name conditions match against the name of the method, parse in context #1 above, concatenateStrings in #2-3. For example method: "parse" would match context #1, whereas method: "^.*String.*$" would match contexts #2-4.

parameter: Condition

Parameter name conditions match against the name of the parameter, text in context #1 above, str1 in contexts #2,4,5. For example parameter: "text" would match context #1 above, parameter: "^str.*$" would match contexts #2-5.

Condition Combination

Up to one of each condition can be combined in a single rule, so for example the following rule can be used to offer the input "Hello World!" only in context #4 above:

java.lang.String:
  - immediate: "Hello World!"
    method: "concatenateStrings"
    parameter: "str1"

When combining conditions with rules be careful about using the correct indentation, i.e., the condition should be indented to the same level as the rule itself. For example, when using a factory: rule with a class: condition the correct indentation is:

java.io.File:
  - factory:
      method: java.io.File.<init>:(Ljava/lang/String;)V
      params: [ src/test/resources/testJPEG.jpg ]
    class: "com.example.SomeClass"

Note that the class: key is aligned with the factory: key, while method: and params: sub-keys of factory: are indented by 2 extra spaces.

Full example

java.lang.String:
  # Matches the first argument of the `concatenateStrings` method
  - immediate: "Hello "
    method: "concatenateStrings"
    parameter: "str1"
  # Matches the second argument of the `concatenateStrings` method
  - immediate: "World!"
    method: "concatenateStrings"
    parameter: "str2"
  # Matches arguments of methods whose name contains `String`
  - immediate: "Hello"
    method: "^.*String.*$"
  # Matches all arguments of methods in the `NumberProcessor` class
  - immediate: "800"
    class: com.example.myapp.NumberProcessor
  # Matches all arguments whose name starts with `str`
  - immediate: "Hello World!"
    parameter: "^str.*$"