Example - using Jenkins, GitHub, and Maven
Introduction
https://github.com/diffblue/cover-pipelines/tree/master/jenkins-github-maven/create
This topic, along with the .jenkins directory above containing bash scripts, xml files and a Jenkinsfile, is intended to serve as an example for how you might integrate Diffblue into your CI pipeline. It assumes starting from nothing but a repository on a remote host and an existing Jenkins server, so walks through setting up credentials and webhooks, as well as getting the jobs onto the Jenkins server.
The Jenkinsfile is for a Multibranch pipeline and it calls three freestyle Jenkins projects - two of which are Diffblue specific and one which is intended to represent where your existing CI might fit.
Note that this example assumes git for SCM, GitHub as a remote host, Maven as a build system and the Jenkins server running on Linux machines. You will need admin access to the remote host and Jenkins server.
Configuring build system: Maven
Estimated time: 5-15 min
Diffblue tests will be stored separately to your tests - in the folder src/diffbluetest/java. This profile will change the test source and test build directories so that Diffblue tests can be run and compiled separately from user tests.
Maven example
In the root pom of the project, add the following:
With this setup, the command mvn test -P DiffblueTests will only run Diffblue tests, while the command mvn test will remain unchanged.
For multi-module projects, ensure the above configuration is added in the correct parent pom.
If the pom.xml is already modifying the testSourceDirectory or testOutputDirectory, the equivalent will need to be carefully integrated.
For Gradle or other build systems, the equivalent will need to be implemented.
Configuring remote host
Remote host: setup a dedicated Diffblue bot user
Estimated time: 5 min
It is necessary that you create a dedicated "bot" user on your remote host that will only be used by the Diffblue CI pipeline. Note the name and email for later in the setup.
The Diffblue CI pipeline will add commits and comments to pull requests. Adding commits to a pull request may re-trigger a CI run, but the pipeline is set up to prevent this for the bot user.
Remote host: setup a webhook
Estimated time: 5-10 min
The Diffblue CI Pipeline will be triggered by a webhook, so it is necessary to set up a webhook on the remote host.
Setting up a webhook is well-documented on Github: https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks
The payload URL should be
JENKINS_URL/multibranch-webhook-trigger/invoke?token=github-webhookThe type is
json/applicationLet me select individual events - pull request only
The Jenkins pipeline will use the Multibranch Scan Webhook Trigger plugin in Jenkins which is triggered by the payload URL above. Since only pull requests trigger the webhook, the job will run when a pull request is made or updated.
Remote host: setup credentials
Estimated time: 15-20 min
The Diffblue CI pipeline will need two types of credentials set up on the remote host. You may already have these set up as part of your CI system that you can reuse, but these instructions assume there is nothing set up.
SSH for fetching and pushing on remote host
On GitHub, this is done by a deploy key with write access.
In the GitHub repo under Settings, add the public SSH key as a deploy key on GitHub https://docs.github.com/en/developers/overview/managing-deploy-keys#deploy-keys
Username and password for commenting on the pull request for the Diffblue bot
On GitHub, this is best done by a personal access token and is well documented.
Make note of the private key and token to store in Jenkins credentials manager next.
Installing Diffblue Cover in your CI environment
Before you begin the installation, please obtain the link to download Cover (from your product update email, or contact Diffblue). Please note you need version 2022.03.02 or above, installed and activated with an appropriate license.
For full details, please see Installation.
Jenkins server: setup credentials and secrets
Estimated time: 15-20 min
The credentials from above need to be stored in Jenkins.
Read/write credentials for checking out repo
In Manage Jenkins -> Manage Credentials, add credentials of type SSH Username with private key and add directly the private key from above. Add an ID such as SSH_KEY to make this available in the Jenkins environment.
Username and password for commenting on the pull request
In Manage Jenkins -> Manage Credentials, add credentials of type username / password and the username of the bot plus the personal access token from above. Add an ID such as TOKEN to make this available in the Jenkins environment.
Cover jars
In this example, the Cover jars are stored in the cloud in a zip file. In Manage Jenkins -> Manage Credentials, add credentials of type secret text and put the url into the secret text box. Add an ID such as RELEASE_URL to make this available in the Jenkins environment.
Jenkins server: install required plugins
Estimated time: 5-15 min
This example requires the following plugins. Make sure they are installed on your Jenkins instance.
GitHub plugin or other remote host plugin that can checkout the project
Jenkins Git plugin
Multibranch Scan Webhook Trigger
Credentials Binding plugin
JUnit plugin
GitHub Branch Source plugin or other remote host branch source plugin
GitHub Custom Notification Context SCM Behaviour or equivalent.
Adding required files
Estimated time: 30-60 min
Add the .jenkins folder to the root of your repository. This contains a Jenkinsfile, scripts and other files that the Diffblue CI pipeline will use.
Configure scripts
You will need to configure the values in .jenkins/common.sh to match your repository's needs. Notably
Build commands for project: Update these functions to echo the correct build commands for the listed scenarios
COMMIT_BOT_NAME/COMMIT_BOT_EMAIL: Change these to your dedicated Diffblue bot user
GITHUB_ORG/GITHUB_REPO: Change to your org/repo. If you're not using GitHub, this will be different and should be updated as required where the function are called.
remoteHostAuthenticaion: This needs to set up the SSH authentication for the remote host bot user so that it can pull/fetch from and push to remote. This is called in most scripts.
getDcover: This is an example where the zipped Cover jars live at a url. If the Cover jars are already installed, for example on the VM hosting your CI system, this can do nothing and see getDcoverScriptLocation.
activateDcover: This is an example of activating
dcoverwith an enterprise license. How you activatedcoverdepends on your particular license. See the separate topic for licensing.getDcoverScriptLocation: This should return the command to run the
dcoverscript included with thedcoverjars, relative to the workspace.
For GitHub projects, that should hopefully be all the configuration you need for the scripts.
The pipeline/updateComment.sh will need the curl commands required by your remote host's API for getting, deleting and adding a comment, and the corresponding jq command updated. The example shown is for GitHub. This is not essential for the Diffblue CI Pipeline, so calls to this script could be commented out.
The other scripts make calls to dcover, git and other scripts and should hopefully not need much, if any, additional changes.
Configure Jenkinsfile
This assumes that you will take the provided .jenkins/Jenkinsfile and modify it to fit your project.
In the provided .jenkins/Jenkinsfile, in the environment section, modify the SSH_KEY and TOKEN variables to use the credentials IDs stored in Jenkins.
Jenkins server: import Jenkins projects
Estimated time: 1-3 hr, perhaps more if you are not using GitHub and are unfamiliar with Jenkins/your remote host plugin.
The Jenkins structure of the Diffblue CI Pipeline consists of four Jenkins projects.
main-job
This is a Multibranch Pipeline configured by .jenkins/Jenkinsfile, set up to be triggered by a pull request.
main-job has several stages, the most notable are the calls to build the following jobs in parallel:
run-user-tests -> This is a freestyle Jenkins project that represents where your existing CI tests or run might go.
existing-diffblue-tests -> This is a freestyle Jenkins project that runs the Diffblue tests in the base branch of the pull request against the code changes in the pull request.
update-diffblue-tests -> This is a freestyle Jenkins project that updates the Diffblue tests for the code changes in the pull request.
Use the Jenkins CLI to import the projects
The XML configuration for each Jenkins project is contained in .jenkins/jobs. These can be imported using the Jenkins CLI and configured, as shown below.
Configure the imported projects
main-job
This is the main pipeline job configured by .jenkins/Jenkinsfile. It orchestrates the other jobs and manages the commenting and new commits on the remote host.
Under Configure modify the repository, remote host and credentials as required.
Branch sources: Update URL and credentials, and checkout over SSH credentials
When you make a pull request with the .jenkins folder and this project in place, this job should now trigger and show up on the remote host (but will likely fail at this point).
run-user-tests
This is the project where you should put any non-Diffblue steps, i.e., your existing CI.
Navigate to the project on the Jenkins server and go to Configure.
General - Modify the Github project to be your own GitHub project or another remote host project.
Source code management - Modify the repository URL and SSH credentials
Build - Add the build and test command for your projects tests, for example. Your existing CI might be more complicated than this, but this will show you how your CI can work alongside Diffblue Cover.
Post build actions, junit - Check that this makes sense for your project.
Post build actions, set GitHub commit status - This may need to be changed if you are not using GitHub.
existing-diffblue-tests
This project runs the Diffblue tests in the base branch against the PR branch. It creates a test report.
Navigate to the project on the Jenkins server and go to Configure.
General - Modify the GitHub project to be your own GitHub project or another remote host project.
Source code management - Modify the repository URL and SSH credentials
Build environment, bindings - Make the variable RELEASE_URL point to the URL of the zipped Cover files, and LICENSE_KEY point to the secret activation key for the
dcoverlicense.Post build actions, junit - Check that this makes sense for your project.
Post build actions, set GitHub commit status - This may need to be changed if you are not using GitHub.
update-diffblue-tests
This is the project that analyzes and updates Diffblue tests. It will push test changes in commits to a temporary branch, ready for main-job to commit them to the PR branch.
Navigate to the project on the Jenkins server and go to Configure.
General - Modify the GitHub project to be your own GitHub project or another remote host project.
Source code management - Modify the repository URL and SSH credentials
Build environment, bindings - Make the variable RELEASE_URL point to the secret URL of the zipped dcover files, LICENSE_KEY point to the secret activation key for the
dcoverlicense, SSH_KEY to the secret private SSH key, and TOKEN to the secret for the password/token for remote host API callsPost build actions, set GitHub commit status - This may need to be changed if you are not using GitHub.
Running for the first time
Create a pull request that adds the .jenkins folder and the changes above to your repository. This will trigger the jobs to run. A comment should appear on the pull request UI if that has been set up. The updating tests jobs may take a while because the whole project is analysed if there are no existing Diffblue tests. Once this is done and the pull request is merged, then the tests will be available on the main branch and subsequent runs will be faster.
Forwarding Errors and Warnings
All warnings and errors created by the CLI are collected into the file: .diffblue/reports/advisories.txt. (The path is relative to the working directory.)
The content of this file can be forwarded to the pipeline user however you choose (e.g. email notification, a pull request comment, etc.). The file is created at each execution of Diffblue Cover and overwrites any previous file. If there are no messages to report, the file will be empty.
Troubleshooting
Scripts
Note that for the cancelPreviousBuilds() script, you may need to allow some scripts to run on the server by clicking the link in the log. This function is experimental and of course optional, but you should not be able to have two jobs running on the same PR this way or another way that you may have already implemented. The script is inspired by this stackoverflow post.
Updating the comment on the remote host is optional and the calls to .jenkins/scripts/pipeline/updateComment.sh can be removed while setting up the pipeline if you have not yet worked out the curl commands for your remote host's API.
If you are using a remote host that is not GitHub, some important notes are:
When checking out the repository, each job should merge the pull request branch with the current target branch revision
The freestyle project jobs should only be triggered by the main job - not by an SCM scan
You will have to set up post-build communication with the remote host on job status.
Conflicts on PRs
As with any source-controlled code, merge conflicts may arise with Diffblue tests, particularly when pull requests are large or long-running. While you work on your feature branch, other developers are merging code with updated Diffblue tests into the base branch. Diffblue tests are stored separately both in the source code and in the commit history in this example pipeline, so conflicts with Diffblue tests can be addressed easily with an interactive rebase to remove the commits made by Diffblue, then push the branch and let the Diffblue CI Pipeline regenerate the tests.
How to do an interactive rebase to remove Diffblue test commits
This assumes that you are comfortable performing advanced git commands such as rebase and dealing with merge conflicts. If not, this tutorial on rebasing and this tutorial on merge conflicts may help you get started. Note that rebasing is usually only performed on feature branches, not on the main, develop or any other primary branch, as data can be lost.
First, start by checking out a local copy of the remote branch with conflicts. Use git fetch <remote-name> then git checkout -b <feature-branch> <remote-name>/<feature-branch> to check out a fresh copy if you do not already have a local copy, or git checkout <feature-branch> then git pull if you do already have a local copy. If your local copy is long out of date, git pull may try to create a merge commit. In this case, it is recommended that you checkout a fresh copy or otherwise reset your branch to the state on remote without a merge commit.
Then do git rebase -i <remote-name>/<base-branch>. A dialog with a list of commits between the base branch and your local copy will appear. Change pick to drop to remove only the Diffblue commits as illustrated below.
When this completes, git log should show the commit history without the Diffblue commits. You can then git push --force-with-lease or git push --force to trigger the Diffblue CI Pipeline to regenerate tests for your code, which should resolve the merge conflicts.
What to do if you still have conflicts
It's likely that if there are conflicts in the Diffblue tests that there are also conflicts in your project code. In this case, the rebase will stop on the offending commits and you will have to manually fix the conflicts in the code as the rebase goes along and run git rebase --continue. git rebase --abort can stop the rebase entirely and reset the branch to the state before the rebase began.
It's possible that when doing automatic refactoring using an IDE, Diffblue tests may have been modified as part of a commit that was intended to refactor project code, and merge conflicts with Diffblue tests may still arise. In this case, you can tell git to take changes from the base branch only for Diffblue tests so that merge conflicts are automatically resolved and the tests will still be updated when pushed. This can be done in two ways:
By running
git checkout --ours -- */src/diffbluetest/java/thengit add */src/diffbluetest/java/when you encounter merge conflicts on rebase with Diffblue tests. This will resolve conflicts by replacing the Diffblue test files with what is in the base branch. 2) By setting up a.gitattributesfile and modifying the git config as follows:
Add the following to the .gitattributes file
then run
When you rebase, there should be no conflicts on Diffblue tests because conflicts are automatically resolved by replacing Diffblue test files with what is in the base branch. Once this approach is set up, it should work without having to repeat for each rebase.
Updating the version of Cover
As newer versions of Diffblue Cover are released with improvements, you may want to update your Cover executables to the latest ones. This will keep your test quality to a high standard, and you will benefit from the latest features. Be sure to look at the release notes to check if there are any command-line changes.
Should I regenerate the whole testbase when updating?
On active projects, we recommend taking no particular action when updating. New test features will be organically brought into the test base by successive code changes. Regenerating tests for the whole codebase would take time (depending on the size of your project) and could introduce merge conflicts for active developers.
Of course, you can always decide to regenerate all the tests. The simplest way is to create a pull request containing a commit to delete the whole Diffblue test folder (src/diffbluetest) for each module. The pipeline (as implemented on the provided scripts) will then rewrite tests for the modules where you deleted the folders.
Last updated