Knowledge Base > DCover CLI > Cover Replay

Cover Replay

Introduction

Cover Replay is an optional, advanced feature for the DCover CLI. It allows you to record an arbitrary execution of your application and then extract unit tests that exercise your application in the same way that your execution did. This feature is available for Diffblue Cover Enterprise Edition users only.

For instance, if you have a web app, you can record its execution while you interact with it on a web browser. You can then extract unit tests that call your methods in exactly the same fashion as your recorded execution. You could also record the execution of some end-to-end tests and extract unit tests that “shift left” your testing.

A diagram of Cover Replay is shown below:

Cover Replay has two significant advantages:

  • The extracted tests exercise your code using real world scenarios.
  • No matter how complex your code is, if you can record an execution that covers a method, then you can extract a test for that method.

When your recorded execution doesn’t cover a method you want to test, you can still get tests for it (we will use the default test-creation engine). Replay does not require any changes to your code. We record your application using a Java agent attached to your Java Virtual Machine (JVM). The agent is configurable. You can use Cover Replay locally on your machine or in a CI pipeline.

Getting Started

This section shows how to use Cover Replay to record and extract tests for a demo project.

1. Unzip the Cover CLI

Please first get the Cover CLI and then:

cd ~
mkdir cover # the name matters for step 4
cd cover
unzip path/to/diffblue-cover-cli-<version>-<os>.zip

2. Get the project under test

Download and compile the source code of the CoreBanking demo project:

cd ..
wget https://github.com/diffblue/CoreBanking/archive/refs/heads/replay-demo.zip
unzip replay-demo.zip
cd CoreBanking-replay-demo/
mvn compile
mvn dependency:copy-dependencies # necessary for steps 3 & 4

3. Run the application

The demo project has a batch application that processes bank transactions stored in .txt files. Run it like this:

java -cp 'target/classes:target/dependency/*' \
   io.diffblue.corebanking.ui.batch.BatchProcessor src/test/resources/batch/*.txt

This is an example from one of the input files (src/test/resources/batch/batch1.txt):

CLIENT|James Brown
ACCOUNT|1234|James Brown|10
...
CASH|120000|2020-02-04|1234
CASH|-125|2020-02-05|1234
CASH|120000|2020-02-28|1234
...

4. Record the application

We now record the execution of this application using the Cover Replay agent. To do this, just execute the following script (if you unzipped Cover into a different directory in step 1, please amend line 3 of the script):

./record.sh

The script will execute the following commands, which you could instead type yourself if you prefer:

AGENT=$HOME/cover/cover-replay-agent.jar
export COVER_TARGET=io.diffblue.
java -javaagent:$AGENT -cp 'target/classes:target/dependency/*' \
   io.diffblue.corebanking.ui.batch.BatchProcessor src/test/resources/batch/*.txt

Note that:

  • Running the java command now takes a bit longer.
  • The $AGENT variable points to the Cover Replay agent that you unzipped in step 1.
  • The $COVER_TARGET environment variable instructs the agent to record just enough to extract tests for classes that live in the package io.diffblue. Learn more about Agent configuration.

The agent will save the recording to a new folder within $HOME/.diffblue/rec/:

drwxrwxr-x 2 user group 4096 Nov  8 20:13 2021-11-08_20-13-21.573.pid29414
-rw-rw-r-- 1 user group   78 Nov  8 20:13 settings.properties

5. Extract tests from the recording

To extract tests from the recording, do this:

$HOME/cover/dcover create --trace --verbose

Option --trace tells Cover to use the last recording to extract tests. The output on your terminal will look like this:

[...]
Found 106 callable methods in 22 classes

Creating tests:
---------------
Loading the trace file '/home/user/.diffblue/rec/2021-11-08_20-13-21.573.pid29414'...
Trace loaded: 1 threads active, 233 methods invoked, 0 fields set, 4415/1157 known methods...
      [1/106] io.diffblue.corebanking.CoreBanking.&lt;init&gt;
      [2/106] io.diffblue.corebanking.CoreBanking.getAccounts
      [3/106] io.diffblue.corebanking.CoreBanking.getClients
      [3/106]   Complete tests created: 1
[...]

Note that Cover indicates exactly which recording it uses to extract tests, see the line:

Loading the trace file '/home/user/.diffblue/rec/2021-11-08_20-13-21.573.pid29414'...

It also provides information about that recording, which might help you identify what you recorded:

Trace loaded: 1 threads active, 233 methods invoked, 0 fields set, 4415/1157 known methods...

6. Inspect the created tests

Search for files named *DiffblueTest.java in src/test/java/. For instance, we see the following test in ComplianceRuleLargeCashDepositsDiffblueTest.java, which was extracted from your recording. Note the references to James Brown and the CashTransaction with the amount 120000L, which we saw in step 3:

@Test
public void testValidateAccountCompliance2() throws AccountException {
  // Arrange
  Account account = new Account(1234L, new Client("James Brown"), 10L);
  account.addTransaction(new CashTransaction(15L, new Date(1577836800000L), account));
  account.addTransaction(new CashTransaction(120000L, new Date(1580774400000L), account));
  account.addTransaction(new CashTransaction(-125L, new Date(1580860800000L), account));
  account.addTransaction(new CashTransaction(120000L, new Date(1582848000000L), account));
  ComplianceRuleLargeCashDeposits complianceRuleLargeCashDeposits =
     new ComplianceRuleLargeCashDeposits();

  // Act
  complianceRuleLargeCashDeposits.validateAccountCompliance(account);

  // Assert
  assertEquals(1, complianceRuleLargeCashDeposits.getCompliantAccounts().size());
  assertFalse(complianceRuleLargeCashDeposits.hasNonCompliantAccounts());
}

Recording agent

Cover Replay uses a Java agent to record your application.

A Java agent is a .jar file that you attach to your JVM using the -javaagent option of thenjava command. Java agents can modify the bytecode of classes loaded by the JVM they attach to. Our agent uses this to intercept method calls and get access to their arguments.

By default the Replay agent saves new recordings to the directory ~/diffblue/rec/ in Linux/macOS and C:\Users\<user>\.diffblue\rec\ in Windows (but see Agent configuration). All recordings you make, regardless of which application or project you record, are saved as sub-folders of this directory. The names of the sub-folders are dates of the form 2021-11-08_20-13-21.573.

The size of a recording can vary from a few Kbs to up to 200-300Mb. For instance, the recording we made in the Getting started guide is around 600Kb. Various factors affect the size of the recording, but the two most important ones are:

  • How many different methods are called in the execution you record.
  • How many methods match the COVER_TARGET agent setting (see below).

Your application will run slower when you record it. With default agent settings you should expect it to run 2-5x slower, but higher slowdowns are possible. It’s often possible to tweak the agent settings to reduce the overhead. See Agent Configuration or contact us for details on how to do this.

Agent configuration

The Cover Replay agent can be configured via environment variables or Java system properties. The following shell snippet provides example values using environment variables:

export COVER_TARGET='io.diffblue.'
export COVER_MAX_IPM='5'
export COVER_MOCKING_ENABLED='y'
export COVER_EXCLUDE='com.example.Person{getName; set*;}, com.example.*Provider}'
export COVER_INCLUDE='com.example.PersonProvider'
export COVER_REC_DIR='path/to/dir'
export COVER_LOG_LEVEL='none'

The table below explains the meaning and accepted values of every option:

Environment variable System property Valid values Default Description
COVER_TARGET cover.target Prefix of a Java package name "" Defines what methods you want to extract tests for, i.e. the methods that may be called in the “act” section of an extracted test. The fewer methods that match this prefix, the smaller the agent overhead and recording size will be.
COVER_MAX_IPM cover.max-ipm Non-negative integer 5 Defines the maximum number of tests that you may extract for any method that lives in the COVER_TARGET. The smaller the number, the smaller the agent overhead and recording size will be.
COVER_MOCKING_ENABLED cover.mocking-enabled y or n y When enabled, the agent will save extra information to construct mocks in the extracted tests. Disable this if you believe that no extracted test will need to mock any object. Disabling reduces agent overhead and recording size.
COVER_EXCLUDE cover.exclude Method pattern "" The agent will not track any method that matches this pattern (and consequently it will run faster). Use this option to provide a list of classes/methods that you would never want to see in the ‘arrange’ or ‘act’ section of a test. See Including and excluding methods in the recording.
COVER_INCLUDE cover.include Method pattern "" The agent will track methods that match this pattern, even when they were excluded via COVER_EXCLUDE, or internally excluded by the agent. See Including and excluding methods in the recording.
COVER_REC_DIR cover.rec-dir Path to a directory ~/.diffblue/rec in Linux/macOS, C:\Users\<user>\.diffblue\rec\ in Windows The directory where the agent saves new recordings. New recordings become subdirectories of this directory.
COVER_LOG_LEVEL cover.log-level none, debug, trace none Defines how much information the agent writes in the agent log. No log is created when none. Log location: /tmp/agent/log in Linux, $TMPDIR/agent/log in macOS, %TEMP%\agent\log in Windows.

Including and excluding methods in the recording

The Cover Replay agent intercepts calls to methods to inspect their arguments. In general, tracking calls to all methods in the JVM is not necessary. And the fewer methods the agent tracks, the faster your application runs.

By default the agent will exclude tracking invocations to methods in certain well-known libraries, such as parts of the Spring framework, or the Hibernate library. But you can configure exactly which methods are tracked via the agent settings COVER_INCLUDE and COVER_EXCLUDE.

For every method, the agent will decide whether to track calls to it using the following steps:

  1. If the method matches the COVER_INCLUDE settings, then do track invocations.
  2. Otherwise, if the method matches the COVER_EXCLUDE setting, then do not track invocations.
  3. Otherwise, if the method matches the internal exclusion patterns, then do not track invocations.
  4. Otherwise, do track invocations.

The internal exclusion patterns match methods of the following classes:

  • Most classes in org.springframework.
  • Classes in org.aspectj.
  • Classes in sun.security.
  • Classes in org.hibernate.

Method patterns

Agent settings COVER_INCLUDE and COVER_EXCLUDE accept a method pattern. A method pattern is a succinct expression that represents a set of Java methods, those that match the pattern. Here are some examples, and the methods they match:

  • Matches the method asList and any other method of the class java.util.Arrays which starts by s:

    java.util.Arrays { asList; s*; }
    
  • Matches all methods of all classes that live in the package com.example (but not its subpackages) and whose name starts by Ma and finishes by n:

    com.example.Ma*n
    
  • Matches method getName in any class that lives in com.example or its subpackages, and whose name finishes by n:

    com.example.**n { getName; }
    
  • Matches all methods matched by method patterns P1 or P2:

    P1, P2
    

Extracting tests from the recording

To extract tests from a recording please use DCover’s option --trace. For instance, in the Getting Started guide we used:

dcover create --trace --verbose
  • If you don’t provide --trace, Cover will write tests using its default test-creation engine.
  • If you provide --trace, Cover will write tests using both its default engine and the execution recording.
  • If you provide --trace --fuzzing-iterations=0, Cover will only use the execution recording to extract tests (no default engine). This can be useful to understand what coverage or test inputs you get from the recording alone.

Currently, the --trace option only extracts tests from your last recording.

Using Cover Replay in your project

You can use Cover Replay on your local machine, during manual, interactive testing, or in a CI pipeline, to automatically create and update tests.

Using Cover Replay on your local machine

You might want to use Cover Replay on your machine:

  • During a one-off testing campaign, to increase the overall coverage of your project.
  • While you evaluate Replay on a new project.
  • While preparing to use Replay in your CI pipeline.

To use Replay on your machine we strongly recommend that you write a small script which:

  1. Runs the application that you want to record, using the same inputs every time.
  2. Calls the dcover command to extract tests.

Automating the execution of the application you want to record is necessary because you want to easily and deterministically run the same application, with the same inputs, every time you create tests. Remember that the Replay-extracted unit tests will mimic how methods were called during the recorded execution. Recording with different inputs may potentially cause methods to be called with different inputs, and consequently Replay would extract different tests.

Here is how to do it for the CoreBanking demo project using the Getting started guide. We created a new script named cover.sh that essentially looks like this:

#!/bin/bash

DCOVER=path/to/dcover
./record.sh
$DCOVER create --trace --verbose "$@"

That calls record.sh (see Getting Started) to record the application, and then calls dcover to extract the tests. If you want to run this example in your local machine, do this:

  1. Get the Cover CLI and unzip it:

    cd ~
    mkdir cover
    cd cover
    unzip path/to/diffblue-cover-cli-<version>-<os>.zip
    
  2. Get the source code of the demo project, compile it, and run Replay on it:

    cd ..
    wget https://github.com/diffblue/CoreBanking/archive/refs/heads/replay-demo.zip
    unzip replay-demo.zip
    cd CoreBanking-replay-demo/
    mvn compile
    mvn dependency:copy-dependencies
    ./cover.sh # record the application and extract tests
    

Using Cover Replay in your CI pipeline

Cover Replay, much like DCover, can be integrated to a CI pipeline so that tests are created and updated by CI. The CI pipeline that you run on every pull request will need to:

  1. Check out the code of the ‘feature’ branch associated to the pull request.
  2. Compile the code, including the application that you are going to record.
  3. Run and record the application.
  4. Call dcover to extract tests from the recording.

Steps 1 and 2 are standard and most likely your CI pipeline already does those. To implement steps 3 and 4, please follow the instructions to run Cover Replay locally.