Diffblue Sandbox

Diffblue Cover writes unit tests by running your code thousands of times as it searches for the best tests that achieve maximum coverage and regression detection. By default, your code is run inside a sandbox which blocks calls with potentially disruptive side-effects such as network, file system, or system changes. However, this can also limit the number of tests that Diffblue Cover can write. Disabling the Diffblue sandbox will allow Cover to run all of your code regardless of side-effects, thereby maximizing coverage. Disabling the sandbox must be done with caution as your code may behave in ways you don't expect (e.g. file system modifications).

Note that if you receive the output code R011 when using Diffblue Cover, this means the method tested performs operations that violate Diffblue Cover’s sandbox policy.

Sandbox Policy

Sandbox Policy Enabled (Default)	Sandbox Policy Disabled (Changes Highlighted)
Allow access to reflective Java operations (e.g. getting the list of methods of a class).	Allow access to reflective Java operations (e.g. getting the list of methods of a class).
Allow setting properties (JUnit needs this during test execution).	Allow setting properties (JUnit needs this during test execution).
Allow creating classloaders.	Allow creating classloaders.
Allow reads of all files.	Allow reads of all files.
Allow writes to files created in this session (including deletion).	Allow writes of all files (including deletion).
Allow writes of files in java.io.tmpdir (including deletion).	Allow writes of all files (including deletion).
Deny writes of files, not created in this session and outside of java.io.tmpdir (including deletion).	Allow writes of all files (including deletion).
Deny URL accesses (unless it's a file:// url with a GET request, in which case it's treated like a regular file operation).	Allow URL accesses.
Deny retrieving network information.	Allow retrieving network information.
Deny creating Sockets.	Allow creating Sockets.
Deny GUI access.	Allow GUI access.
Deny sun.misc.Unsafe/sun.misc.*	Allow sun.misc.Unsafe/sun.misc.*
Deny external process accesses.	Allow external process accesses.
Deny Java Management Extensions (JMX) access.	Allow Java Management Extensions (JMX) access.
Deny Java Native Interface (JNI) access.	Allow Java Native Interface (JNI) access.
Deny Smartcards (java.smartcardio.*).	Allow Smartcards (java.smartcardio.*).
Deny Printers (javax.print.*).	Allow Printers (javax.print.*).
Deny setting the security manager.	Allow setting the security manager.
Deny access control context.	Allow access control context.
Deny Kerberos Authentication access.	Allow Kerberos Authentication access.
Deny System.exit calls.	Deny System.exit calls.

Disabling the Diffblue Sandbox

The sandbox policy can be disabled to allow almost all of the operations outlined above (except calling System.exit). However, we recommend that you fully familiarize yourself with the information contained in this topic in order to avoid any undesirable effects by allowing these previously denied operations.

To disable the sandbox policy, use the --disable-sandbox option.

Note that if you just want to allow a small set of JNI libraries, use the JNI allow list (a comma separated list of JNI library name prefixes) - specified using the --allow-jni=<value>[,<value>...] argument.

Also, you may want to consider simply refactoring parts of your code. For example, if you have a method that executes a denied operation but also contains business logic that needs to be verified, you could refactor your code so that any logic that can be executed within the sandbox is contained in a separate, unit-testable method - this ensures the business logic is tested and consequently increases code coverage.

Harmful Example

Let's look at an example of a denied operation and what could happen if it were allowed. One of the operations prevented by default is accessing files outside of the temp directory (specified by java.io.tmpdir). We further restrict this by allowing only writing to newly created files. The root parameter may end up pointing to a real directory on your system (it could be given any value, such as /, C:\, /home/User/), and then be executed, removing all the files contained within that directory along with the directory itself. With the default policy in place, trying to write tests for this method would result in an R011 output code.

public class Harmful {
    public void deleteRecursively(final Path root) {
        final File[] contents = directoryToBeDeleted.listFiles();
        if (contents != null) {
            for (final File file : contents) {
                deleteRecursively(file);
            }
        }
        return directoryToBeDeleted.delete();
    }
}

Operations

The tables below outline the denied operations (Diffblue sandbox enabled) and explains what can happen (in the worst case) if they are allowed (Diffblue sandbox disabled). Note that:

• The exact nature of the risk is heavily dependent on the application being analyzed and can't be predicted ahead of time. If the application being analyzed is a web-based application in an automated pipeline that has no access to production systems, the risk of experiencing many of the adverse effects mentioned below is relatively low.

• The impact of the Diffblue sandbox policy is not restricted to just the method, but methods that call these methods, and so on.

I/O Operation

Operation	Allowed by Default?	Change with "Disable Sandbox"	Possible Effect if Allowed
Networking	No	Allowed	Unexpected network traffic, data loss on remote machines, instability in unit tests.
FileIO	Read all files, write new files.	Read all files, write all files.	Data loss on the machine where Cover is running, instability in unit tests.
TmpFileIO	Read all files, write new files.	Read all files, write all files.	Unnecessary disk space is consumed, instability in unit tests.

• Data loss is likely to occur if methods (like a hypothetical delete method in a typical web application) are invoked and happen to have access to live credentials (e.g. production or test environments).

• Likewise, unexpected network traffic can occur if the application makes raw network requests to servers.

• Disk space may be consumed because the application may not clean up temporary files.

Java Libraries

Library	Allowed by Default?	Change with "Disable Sandbox"	Possible Effect if Allowed
GUI (Swing/AWT)	No, java.awt.headless is set to true	Allowed, java.awt.headless is set to false	N/A
sun.misc.*	No	Allowed	Instability of tests across JVM vendors and operating systems.
sun.misc.Unsafe	No	Allowed	Unintended side effects across unit tests in a class, or test suite.

• Trying to write unit tests for methods that invoke GUIs is difficult. In those cases, the java.awt.headless property is usually set to true to disable those GUI components. If you require tests for the logic behind the GUIs, it's better to refactor the code to allow that logic to be tested independently of the GUI.

• The sun.misc.* packages provide very low level access to fundamental system operations and do not form a part of the public interface. Because of the very low level access, we deem these operations unsafe due to the risk of unintended side effects.

• Furthermore, the following is taken from the published Oracle FAQ regarding the use of these packages.

The sun.* packages are not part of the supported, public interface.

A Java program that directly calls into sun.* packages is not guaranteed to work on all Java-compatible platforms. In fact, such a program is not guaranteed to work even in future versions on the same platform.

Each company that implements the Java platform will do so in their own private way. The classes in sun.* are present in the JDK to support Oracle's implementation of the Java platform: the sun.* classes are what make the Java platform classes work "under the covers" for Oracle's JDK. These classes will not in general be present on another vendor's Java platform. If your Java program asks for a class "sun.package.Foo" by name, it may fail with ClassNotFoundError, and you will have lost a major advantage of developing in Java.

Technically, nothing prevents your program from calling into sun.* by name. From one release to another, these classes may be removed, or they may be moved from one package to another, and it's fairly likely that their interface (method names and signatures) will change. (From Oracle's point of view, since we are committed to maintaining the Java platform, we need to be able to change sun.* to refine and enhance the platform.) In this case, even if you are willing to run only on Oracle's implementation, you run the risk of a new version of the implementation breaking your program.

In general, writing java programs that rely on sun.* is risky: those classes are not portable, and are not supported.

System Operations

Operation	Allowed by Default?	Change with "Disable Sandbox"	Possible Effect if Allowed
External Processes	No	Allowed	Instability of unit tests, random behavior of other processes.
JMX	No	Allowed	Instability of unit tests, random behavior of JMX enabled beans/services/etc.
JNI	No, except JNI allow list - see below.	Allowed	Tests won't generate coverage for the native code.
System.exit	No	No	Causes the analysis service JVM to exit and would increase in the occurrence of R024/R025 output codes (from Diffblue Cover).

* Note that if the Diffblue sandbox policy is blocking access to a JNI library that you believe to be safe to use, then you can add it to the JNI allow list (a comma separated list of JNI library name prefixes can be used):

• Cover CLI: Use the --allow-jni option - see allow-jni and/or Working with code R012 for more information.

• Cover Plugin: Go to Diffblue > Change Settings > Sandboxed Environment > Allowed JNI prefixes.

Peripherals

Generally, these operations require access to specific hardware which may or may not be present. Because this can't be determined beforehand, unit tests for methods using these APIs would be inherently unstable.

Peripheral	Allowed by Default?	Change with "Disable Sandbox"	Possible Effect if Allowed
Smartcards	No	Allowed	Corruption of data on the smartcards.
Printing	No	Allowed	Print jobs are submitted to configured printers.

Security

Security	Allowed by Default?	Change with "Disable Sandbox"	Possible Effect if Allowed
Setting security manager	No	Allowed	Unit tests won't be created or verified.
Create access control context	No	Allowed	Attempts to bypass sandbox will be allowed.
Kerberos Authentication	No	Allowed	Unintended side-effects with Kerberos systems.