diffblue.comFree trial

Example - using Jenkins, GitHub, and Maven

Introduction

https://github.com/diffblue/cover-pipelines/tree/master/jenkins-github-maven/create

This topic, along with the .jenkins directory above containing bash scripts, xml files and a Jenkinsfile, is intended to serve as an example for how you might integrate Diffblue into your CI pipeline. It assumes starting from nothing but a repository on a remote host and an existing Jenkins server, so walks through setting up credentials and webhooks, as well as getting the jobs onto the Jenkins server.

The Jenkinsfile is for a Multibranch pipeline and it calls three freestyle Jenkins projects - two of which are Diffblue specific and one which is intended to represent where your existing CI might fit.

Note that this example assumes git for SCM, GitHub as a remote host, Maven as a build system and the Jenkins server running on Linux machines. You will need admin access to the remote host and Jenkins server.

Configuring build system: Maven

Estimated time: 5-15 min

Diffblue tests will be stored separately to your tests - in the folder src/diffbluetest/java. This profile will change the test source and test build directories so that Diffblue tests can be run and compiled separately from user tests.

Maven example

In the root pom of the project, add the following:

<project>
    <profiles>
        <profile>
        <id>DiffblueTests</id>
        <properties>
            <test.source.dir>src/diffbluetest/java</test.source.dir>
            <test.build.dir>target/diffblue-test-classes</test.build.dir>
        </properties>
        </profile>
    </profiles>

    <build>
        <testSourceDirectory>${test.source.dir}</testSourceDirectory>
        <testOutputDirectory>${test.build.dir}</testOutputDirectory>
    </build>

    <properties>
        <test.source.dir>src/test/java</test.source.dir>
        <test.build.dir>target/test-classes</test.build.dir>
    </properties> <!--or whatever the defaults are -->
</project>

With this setup, the command mvn test -P DiffblueTests will only run Diffblue tests, while the command mvn test will remain unchanged.

For multi-module projects, ensure the above configuration is added in the correct parent pom.

If the pom.xml is already modifying the testSourceDirectory or testOutputDirectory, the equivalent will need to be carefully integrated.

For Gradle or other build systems, the equivalent will need to be implemented.

Configuring remote host

Remote host: setup a dedicated Diffblue bot user

Estimated time: 5 min

It is necessary that you create a dedicated "bot" user on your remote host that will only be used by the Diffblue CI pipeline. Note the name and email for later in the setup.

The Diffblue CI pipeline will add commits and comments to pull requests. Adding commits to a pull request may re-trigger a CI run, but the pipeline is set up to prevent this for the bot user.

Remote host: setup a webhook

Estimated time: 5-10 min

The Diffblue CI Pipeline will be triggered by a webhook, so it is necessary to set up a webhook on the remote host.

Setting up a webhook is well-documented on Github: https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks

  • The payload URL should be JENKINS_URL/multibranch-webhook-trigger/invoke?token=github-webhook

  • The type is json/application

  • Let me select individual events - pull request only

The Jenkins pipeline will use the Multibranch Scan Webhook Trigger plugin in Jenkins which is triggered by the payload URL above. Since only pull requests trigger the webhook, the job will run when a pull request is made or updated.

Remote host: setup credentials

Estimated time: 15-20 min

The Diffblue CI pipeline will need two types of credentials set up on the remote host. You may already have these set up as part of your CI system that you can reuse, but these instructions assume there is nothing set up.

SSH for fetching and pushing on remote host

On GitHub, this is done by a deploy key with write access.

Username and password for commenting on the pull request for the Diffblue bot

On GitHub, this is best done by a personal access token and is well documented.

Make note of the private key and token to store in Jenkins credentials manager next.

Installing Diffblue Cover in your CI environment

Before you begin the installation, please obtain the link to download Cover (from your product update email, or contact Diffblue). Please note you need version 2022.03.02 or above, installed and activated with an appropriate license.

For full details, please see Installation.

Jenkins server: setup credentials and secrets

Estimated time: 15-20 min

The credentials from above need to be stored in Jenkins.

Read/write credentials for checking out repo

In Manage Jenkins -> Manage Credentials, add credentials of type SSH Username with private key and add directly the private key from above. Add an ID such as SSH_KEY to make this available in the Jenkins environment.

Username and password for commenting on the pull request

In Manage Jenkins -> Manage Credentials, add credentials of type username / password and the username of the bot plus the personal access token from above. Add an ID such as TOKEN to make this available in the Jenkins environment.

Cover jars

In this example, the Cover jars are stored in the cloud in a zip file. In Manage Jenkins -> Manage Credentials, add credentials of type secret text and put the url into the secret text box. Add an ID such as RELEASE_URL to make this available in the Jenkins environment.

Jenkins server: install required plugins

Estimated time: 5-15 min

This example requires the following plugins. Make sure they are installed on your Jenkins instance.

  • GitHub plugin or other remote host plugin that can checkout the project

  • Jenkins Git plugin

  • Multibranch Scan Webhook Trigger

  • Credentials Binding plugin

  • JUnit plugin

  • GitHub Branch Source plugin or other remote host branch source plugin

  • GitHub Custom Notification Context SCM Behaviour or equivalent.

Adding required files

Estimated time: 30-60 min

Add the .jenkins folder to the root of your repository. This contains a Jenkinsfile, scripts and other files that the Diffblue CI pipeline will use.

Configure scripts

You will need to configure the values in .jenkins/common.sh to match your repository's needs. Notably

  • Build commands for project: Update these functions to echo the correct build commands for the listed scenarios

  • COMMIT_BOT_NAME/COMMIT_BOT_EMAIL: Change these to your dedicated Diffblue bot user

  • GITHUB_ORG/GITHUB_REPO: Change to your org/repo. If you're not using GitHub, this will be different and should be updated as required where the function are called.

  • remoteHostAuthenticaion: This needs to set up the SSH authentication for the remote host bot user so that it can pull/fetch from and push to remote. This is called in most scripts.

  • getDcover: This is an example where the zipped Cover jars live at a url. If the Cover jars are already installed, for example on the VM hosting your CI system, this can do nothing and see getDcoverScriptLocation.

  • activateDcover: This is an example of activating dcover with an enterprise license. How you activate dcover depends on your particular license. See the separate topic for licensing.

  • getDcoverScriptLocation: This should return the command to run the dcover script included with the dcover jars, relative to the workspace.

For GitHub projects, that should hopefully be all the configuration you need for the scripts.

The pipeline/updateComment.sh will need the curl commands required by your remote host's API for getting, deleting and adding a comment, and the corresponding jq command updated. The example shown is for GitHub. This is not essential for the Diffblue CI Pipeline, so calls to this script could be commented out.

The other scripts make calls to dcover, git and other scripts and should hopefully not need much, if any, additional changes.

Configure Jenkinsfile

This assumes that you will take the provided .jenkins/Jenkinsfile and modify it to fit your project.

In the provided .jenkins/Jenkinsfile, in the environment section, modify the SSH_KEY and TOKEN variables to use the credentials IDs stored in Jenkins.

Jenkins server: import Jenkins projects

Estimated time: 1-3 hr, perhaps more if you are not using GitHub and are unfamiliar with Jenkins/your remote host plugin.

The Jenkins structure of the Diffblue CI Pipeline consists of four Jenkins projects.

main-job

This is a Multibranch Pipeline configured by .jenkins/Jenkinsfile, set up to be triggered by a pull request.

main-job has several stages, the most notable are the calls to build the following jobs in parallel:

  1. run-user-tests -> This is a freestyle Jenkins project that represents where your existing CI tests or run might go.

  2. existing-diffblue-tests -> This is a freestyle Jenkins project that runs the Diffblue tests in the base branch of the pull request against the code changes in the pull request.

  3. update-diffblue-tests -> This is a freestyle Jenkins project that updates the Diffblue tests for the code changes in the pull request.

Use the Jenkins CLI to import the projects

The XML configuration for each Jenkins project is contained in .jenkins/jobs. These can be imported using the Jenkins CLI and configured, as shown below.

java -jar jenkins-cli.jar -s http://JENKINS_SERVER -auth USER:TOKEN create-job main-job < path/to/.jenkins/jobs/main-job.xml
java -jar jenkins-cli.jar -s http://JENKINS_SERVER -auth USER:TOKEN create-job run-user-tests < path/to/.jenkins/jobs/run-user-tests.xml
java -jar jenkins-cli.jar -s http://JENKINS_SERVER -auth USER:TOKEN create-job existing-diffblue-tests < path/to/.jenkins/jobs/existing-diffblue-tests.xml
java -jar jenkins-cli.jar -s http://JENKINS_SERVER -auth USER:TOKEN create-job update-diffblue-tests < path/to/.jenkins/jobs/update-diffblue-tests.xml

Configure the imported projects

main-job

This is the main pipeline job configured by .jenkins/Jenkinsfile. It orchestrates the other jobs and manages the commenting and new commits on the remote host.

Under Configure modify the repository, remote host and credentials as required.

  • Branch sources: Update URL and credentials, and checkout over SSH credentials

When you make a pull request with the .jenkins folder and this project in place, this job should now trigger and show up on the remote host (but will likely fail at this point).

run-user-tests

This is the project where you should put any non-Diffblue steps, i.e., your existing CI.

Navigate to the project on the Jenkins server and go to Configure.

  • General - Modify the Github project to be your own GitHub project or another remote host project.

  • Source code management - Modify the repository URL and SSH credentials

  • Build - Add the build and test command for your projects tests, for example. Your existing CI might be more complicated than this, but this will show you how your CI can work alongside Diffblue Cover.

  • Post build actions, junit - Check that this makes sense for your project.

  • Post build actions, set GitHub commit status - This may need to be changed if you are not using GitHub.

existing-diffblue-tests

This project runs the Diffblue tests in the base branch against the PR branch. It creates a test report.

Navigate to the project on the Jenkins server and go to Configure.

  • General - Modify the GitHub project to be your own GitHub project or another remote host project.

  • Source code management - Modify the repository URL and SSH credentials

  • Build environment, bindings - Make the variable RELEASE_URL point to the URL of the zipped Cover files, and LICENSE_KEY point to the secret activation key for the dcover license.

  • Post build actions, junit - Check that this makes sense for your project.

  • Post build actions, set GitHub commit status - This may need to be changed if you are not using GitHub.

update-diffblue-tests

This is the project that analyzes and updates Diffblue tests. It will push test changes in commits to a temporary branch, ready for main-job to commit them to the PR branch.

Navigate to the project on the Jenkins server and go to Configure.

  • General - Modify the GitHub project to be your own GitHub project or another remote host project.

  • Source code management - Modify the repository URL and SSH credentials

  • Build environment, bindings - Make the variable RELEASE_URL point to the secret URL of the zipped dcover files, LICENSE_KEY point to the secret activation key for the dcover license, SSH_KEY to the secret private SSH key, and TOKEN to the secret for the password/token for remote host API calls

  • Post build actions, set GitHub commit status - This may need to be changed if you are not using GitHub.

Running for the first time

Create a pull request that adds the .jenkins folder and the changes above to your repository. This will trigger the jobs to run. A comment should appear on the pull request UI if that has been set up. The updating tests jobs may take a while because the whole project is analysed if there are no existing Diffblue tests. Once this is done and the pull request is merged, then the tests will be available on the main branch and subsequent runs will be faster.

Forwarding Errors and Warnings

All warnings and errors created by the CLI are collected into the file: .diffblue/reports/advisories.txt. (The path is relative to the working directory.)

The content of this file can be forwarded to the pipeline user however you choose (e.g. email notification, a pull request comment, etc.). The file is created at each execution of Diffblue Cover and overwrites any previous file. If there are no messages to report, the file will be empty.

Troubleshooting

Scripts

Note that for the cancelPreviousBuilds() script, you may need to allow some scripts to run on the server by clicking the link in the log. This function is experimental and of course optional, but you should not be able to have two jobs running on the same PR this way or another way that you may have already implemented. The script is inspired by this stackoverflow post.

Updating the comment on the remote host is optional and the calls to .jenkins/scripts/pipeline/updateComment.sh can be removed while setting up the pipeline if you have not yet worked out the curl commands for your remote host's API.

If you are using a remote host that is not GitHub, some important notes are:

  • When checking out the repository, each job should merge the pull request branch with the current target branch revision

  • The freestyle project jobs should only be triggered by the main job - not by an SCM scan

  • You will have to set up post-build communication with the remote host on job status.

Conflicts on PRs

As with any source-controlled code, merge conflicts may arise with Diffblue tests, particularly when pull requests are large or long-running. While you work on your feature branch, other developers are merging code with updated Diffblue tests into the base branch. Diffblue tests are stored separately both in the source code and in the commit history in this example pipeline, so conflicts with Diffblue tests can be addressed easily with an interactive rebase to remove the commits made by Diffblue, then push the branch and let the Diffblue CI Pipeline regenerate the tests.

How to do an interactive rebase to remove Diffblue test commits

This assumes that you are comfortable performing advanced git commands such as rebase and dealing with merge conflicts. If not, this tutorial on rebasing and this tutorial on merge conflicts may help you get started. Note that rebasing is usually only performed on feature branches, not on the main, develop or any other primary branch, as data can be lost.

First, start by checking out a local copy of the remote branch with conflicts. Use git fetch <remote-name> then git checkout -b <feature-branch> <remote-name>/<feature-branch> to check out a fresh copy if you do not already have a local copy, or git checkout <feature-branch> then git pull if you do already have a local copy. If your local copy is long out of date, git pull may try to create a merge commit. In this case, it is recommended that you checkout a fresh copy or otherwise reset your branch to the state on remote without a merge commit.

Then do git rebase -i <remote-name>/<base-branch>. A dialog with a list of commits between the base branch and your local copy will appear. Change pick to drop to remove only the Diffblue commits as illustrated below.

pick 7441bca798 A lot of refactoring that touches a lot of code.
drop ad2ab5ecb5 Update tests from Cover for module1
drop 6beadb14c4 Update tests from Cover for module2
pick d3bd9c54b0 A change that happened much later.

# Rebase 76edc7187a..6beadb14c2 onto 76edc7187a (5 commands)
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out

When this completes, git log should show the commit history without the Diffblue commits. You can then git push --force-with-lease or git push --force to trigger the Diffblue CI Pipeline to regenerate tests for your code, which should resolve the merge conflicts.

What to do if you still have conflicts

It's likely that if there are conflicts in the Diffblue tests that there are also conflicts in your project code. In this case, the rebase will stop on the offending commits and you will have to manually fix the conflicts in the code as the rebase goes along and run git rebase --continue. git rebase --abort can stop the rebase entirely and reset the branch to the state before the rebase began.

It's possible that when doing automatic refactoring using an IDE, Diffblue tests may have been modified as part of a commit that was intended to refactor project code, and merge conflicts with Diffblue tests may still arise. In this case, you can tell git to take changes from the base branch only for Diffblue tests so that merge conflicts are automatically resolved and the tests will still be updated when pushed. This can be done in two ways:

  1. By running git checkout --ours -- */src/diffbluetest/java/ then git add */src/diffbluetest/java/ when you encounter merge conflicts on rebase with Diffblue tests. This will resolve conflicts by replacing the Diffblue test files with what is in the base branch. 2) By setting up a .gitattributes file and modifying the git config as follows:

Add the following to the .gitattributes file

# Automatically address merge conflicts in Diffblue tests by taking what is in the base branch
**/src/diffbluetest/java/**/*.java merge=ours

then run

git config merge.ours.driver true

When you rebase, there should be no conflicts on Diffblue tests because conflicts are automatically resolved by replacing Diffblue test files with what is in the base branch. Once this approach is set up, it should work without having to repeat for each rebase.

Updating the version of Cover

As newer versions of Diffblue Cover are released with improvements, you may want to update your Cover executables to the latest ones. This will keep your test quality to a high standard, and you will benefit from the latest features. Be sure to look at the release notes to check if there are any command-line changes.

Should I regenerate the whole testbase when updating?

On active projects, we recommend taking no particular action when updating. New test features will be organically brought into the test base by successive code changes. Regenerating tests for the whole codebase would take time (depending on the size of your project) and could introduce merge conflicts for active developers.

Of course, you can always decide to regenerate all the tests. The simplest way is to create a pull request containing a commit to delete the whole Diffblue test folder (src/diffbluetest) for each module. The pipeline (as implemented on the provided scripts) will then rewrite tests for the modules where you deleted the folders.

Last updated