LogoLogo
diffblue.comFree trial
  • Discover Diffblue Cover
  • Get Started
    • What is Diffblue Cover?
    • Get started
      • Free trial
      • Get started - Cover Plugin
      • Get started - Cover CLI
      • Get started - Cover Pipeline
      • Get started - Cover Reports
    • Specs & Reqs
    • Reference Deployments
    • Licensing
      • Online license activation
      • Offline license activation
      • Diffblue License Manager
    • Update Cover
    • FAQs
    • Diffblue Learning
      • Get started
        • Just the basics
        • Free trial
        • Cover Plugin (IDE)
        • Cover CLI (Command Line)
      • Developer
        • Unit tests (IDE)
        • Unit tests (CLI)
        • Test coverage
      • DevOps
        • GitHub
        • GitLab
        • Other CI
      • Administrator
        • Admin - IntelliJ
        • Admin - CLI
        • Admin - Reports
      • Test coverage
        • Developer
        • Senior developer
        • Cover Reports Administrator
  • EVALUATION & ONBOARDING
    • Proof of Value
    • Jumpstart
      • Prerequisites for onboarding
      • Phase 1: Up and running
        • Module 1: Create your Cover unit test baseline
        • Module 2: Cover Pipeline CI integration
      • Phase 2: Developer productivity
        • Module 3: Getting started using Cover
        • Module 4: Introduction to Cover Reports
      • Phase 3: Advanced topics
        • Module 5: Speed up your test execution
        • Module 6: Getting more from Cover
  • Features
    • Cover Plugin
      • Writing tests
        • Gutter icons
        • Menu options
        • Run configurations
        • Cover Plugin tool window
        • Test Review
        • Test examples
        • Creating partial tests
        • Creating skeleton tests
        • Covering all enum values
        • Test insertion order
        • Diffblue Sandbox
        • Environment Check Cache
      • Project configuration
        • General dependencies
        • Test framework dependencies
      • Cover Plugin settings
        • Test Naming
        • Test Formatting
        • Spring configuration options
        • Method Annotations
        • Test Directory
        • Reset Cover Plugin settings
      • Cover Plugin admin
        • Core Maintenance
        • Cover Plugin toolbar menu
        • Cover Plugin status bar widget
        • Telemetry
        • Memory management
        • Using SonarQube with Cover Plugin
        • Log files
        • Troubleshooting
    • Cover CLI
      • Writing tests
        • Command summary
        • Test examples
        • Creating partial tests
        • Customizing test inputs
        • Customizing test setup
        • Test naming
        • Test formatting
        • Test insertion order
        • Patch files
        • Diffblue Sandbox
        • Operational behaviors
        • Test validation
      • Project configuration
        • Preflight checks
        • General dependencies
        • Test framework dependencies
        • Compiling your project successfully
          • Building a Maven project
          • Building a Gradle project
        • Configuring Cover to work with your project's build system
          • Quick guide
          • Configuration file in detail
          • Getting Started with Ant Projects
          • Default configuration
        • Mocking using Mockito
        • Spring profiles
        • Runtime environment
      • Commands & Arguments
        • Environment configuration for CI
        • Packages, classes, and methods
        • Quoting command line arguments
        • Argument files
      • Cover CLI admin
        • Core Maintenance
        • Telemetry
        • Memory management
        • Using Cover CLI in Eclipse
        • Using SonarQube with Cover CLI
        • Log files
        • Troubleshooting
      • Environment Configuration
    • Cover Pipeline
      • Cover Pipeline for CI
        • Quick Start - General
        • Quick Start - Jenkins
        • Quick Start - Azure Pipelines
        • Quick Start - AWS Codebuild
        • Integrating Diffblue Cover into CI on pull requests
        • Installation - VMs or CI run
      • Cover Pipeline for GitLab
        • Introduction
        • Installation and initial setup
        • GitLab workflow
        • Configuration
        • Troubleshooting
        • Related topics
      • Cover Pipeline for GitHub
        • Introduction
        • Installation and initial setup
        • GitHub workflow
        • Configuration
        • Troubleshooting
        • Related topics
    • Cover Reports
      • Cover Reports Contributor
        • Java project config (JaCoCo)
        • Generate and upload reports bundles
        • Authenticated uploads
      • Cover Reports User
        • Navigation
        • Dashboards
        • Telemetry data
        • Export API
        • Considerations
      • Cover Reports Administrator
        • Install and update Cover Reports
        • Configuration options
        • Database backup
        • SSO with Cover Reports
        • Uninstall Cover Reports
    • Cover Optimize
      • Get started - Cover Optimize
      • Cover Optimize & Gradle
      • Cover Optimize & Maven
      • Patch files
    • Cover Refactor
      • Get started - Cover Refactor
    • Cover Annotations
      • Mocking Annotations
      • Custom Input Annotations
      • Interesting Value Annotations
    • Output Codes
      • E - Environment Codes
      • L - License Codes
      • R - Reason Codes
      • T - Testability Codes
      • V - Validation Codes
      • Working with output codes
        • Working with code E020
        • Working with codes E057 to E065
        • Working with code E085
        • Working with code R005
        • Working with code R006
        • Working with code R008
        • Working with code R011
        • Working with code R012
        • Working with code R013
        • Working with code R026
        • Working with code R031
        • Working with code V003
    • Tutorials
      • How to measure test quality
      • How to improve code coverage
      • How to test a new feature
      • How to find regressions
      • How to use Diffblue Cover in test driven development (TDD)
      • How to write tests for Kotlin projects
      • Examples of tests created by Diffblue Cover
      • Best practices for testing private methods
  • Updates & Upgrades
    • Update Cover
    • Cover Editions
    • What's new
    • Release archive
      • 2025-05-01
      • 2025-04-02
      • 2025-04-01
      • 2025-03-02
      • 2025-03-01
      • 2025-02-02
      • 2025-02-01
      • 2025-01-02
      • 2025-01-01
      • 2024-12-02
      • 2024-12-01
      • 2024-11-02
      • 2024-11-01
      • 2024-10-02
      • 2024-10-01
      • 2024-09-02
      • 2024-09-01
      • 2024-08-02
      • 2024-08-01
      • 2024-07-04
      • 2024-07-03
      • 2024-07-01
      • 2024-06-02
      • 2024-06-01
      • 2024-05-02
      • 2024-05-01
      • 2024-04-02
      • 2024-04-01
      • 2024-03-02
      • 2024-03-01
      • 2024-02-02
      • 2024-02-01
      • 2024-01-02
      • 2024-01-01
      • 2023-12-02
      • 2023-12-01
      • 2023-11-03
      • 2023-11-02
      • 2023-11-01
      • 2023-10-02
      • 2023-10-01
      • 2023-09-02
      • 2023-09-01
      • 2023-08-02
      • 2023-08-01
      • 2023-07-03
      • 2023-07-02
      • 2023-07-01
      • 2023-06-02
      • 2023-06-01
      • 2023-05-02
      • 2023-05-01
      • 2023-04-02
      • 2023-04-01
      • 2023-03-02
      • 2023-03-01
      • 2023-02-02
      • 2023-02-01
      • 2023-01-02
      • 2023-01-01
      • 2022-12-02
      • 2022-12-01
      • 2022-11-02
      • 2022-11-01
      • 2022-10-03
      • 2022-10-02
      • 2022-10-01
      • 2022-09-02
      • 2022-09-01
      • 2022-08-05
      • 2022-08-03
      • 2022-08-02
      • 2022-08-01
      • 2022-07-02
      • 2022-07-01
      • 2022-06-02
      • 2022-06-01
      • 2022-05-02
      • 2022-05-01
      • 2022-04-02
      • 2022-04-01
      • 2022-03-02
      • 2022-03-01
      • 2022-02-02
      • 2022-02-01
      • 2022-01-02
      • 2022-01-01
      • 2021-06-02
      • 2021-02-01
  • Legal
    • Diffblue Legal
      • Diffblue End User License Agreement (EULA)
      • Cover Third Party Notices and/or Licenses
      • Privacy Notice
Powered by GitBook
On this page
  • Diffblue Sandbox
  • Sandbox Policy
  • Disabling the Diffblue Sandbox
  • Harmful Example
  • Operations
  • I/O Operation
  • Java Libraries
  • System Operations
  • Peripherals
  • Security

Was this helpful?

Export as PDF
  1. Features
  2. Output Codes
  3. Working with output codes

Working with code R011

R011 - Sandboxing policy violation

If you receive the output code R011, this means the method tested performs operations that violate Diffblue Cover’s sandbox policy.

Diffblue Sandbox

Diffblue Cover writes unit tests by running your code thousands of times as it searches for the best tests that achieve maximum coverage and regression detection. By default, your code is run inside a sandbox which blocks calls with potentially disruptive side-effects such as network, file system, or system changes. However, this can also limit the number of tests that Diffblue Cover can write. Disabling the Diffblue sandbox will allow Cover to run all of your code regardless of side-effects, thereby maximizing coverage. Disabling the sandbox must be done with caution as your code may behave in ways you don't expect (e.g. file system modifications).

Sandbox Policy

Sandbox Policy Enabled (Default)
Sandbox Policy Disabled (Changes Highlighted)

Allow access to reflective Java operations (e.g. getting the list of methods of a class).

Allow access to reflective Java operations (e.g. getting the list of methods of a class).

Allow setting properties (JUnit needs this during test execution).

Allow setting properties (JUnit needs this during test execution).

Allow creating classloaders.

Allow creating classloaders.

Allow reads of all files.

Allow reads of all files.

Allow writes to files created in this session (including deletion).

Allow writes of all files (including deletion).

Allow writes of files in java.io.tmpdir (including deletion).

Allow writes of all files (including deletion).

Deny writes of files, not created in this session and outside of java.io.tmpdir (including deletion).

Allow writes of all files (including deletion).

Deny URL accesses (unless it's a file:// url with a GET request, in which case it's treated like a regular file operation).

Allow URL accesses.

Deny retrieving network information.

Allow retrieving network information.

Deny creating Sockets.

Allow creating Sockets.

Deny GUI access.

Allow GUI access.

Deny sun.misc.Unsafe/sun.misc.*

Allow sun.misc.Unsafe/sun.misc.*

Deny external process accesses.

Allow external process accesses.

Deny Java Management Extensions (JMX) access.

Allow Java Management Extensions (JMX) access.

Deny Java Native Interface (JNI) access.

Allow Java Native Interface (JNI) access.

Deny Smartcards (java.smartcardio.*).

Allow Smartcards (java.smartcardio.*).

Deny Printers (javax.print.*).

Allow Printers (javax.print.*).

Deny setting the security manager.

Allow setting the security manager.

Deny access control context.

Allow access control context.

Deny Kerberos Authentication access.

Allow Kerberos Authentication access.

Deny System.exit calls.

Deny System.exit calls.

Disabling the Diffblue Sandbox

The sandbox policy can be disabled to allow almost all of the operations outlined above (except calling System.exit). However, we recommend that you fully familiarize yourself with the information contained in this topic in order to avoid any undesirable effects by allowing these previously denied operations.

To disable the sandbox policy:

  • Cover CLI: Use the --disable-sandbox option.

  • Cover Plugin: Go to Diffblue > Change Settings > Sandboxed Environment and untick Enable sandboxing.

Note that if you just want to allow a small set of JNI libraries, use the JNI allow list (a comma separated list of JNI library name prefixes):

  • Cover CLI: Use the --allow-jni option - see allow-jni and/or Working with code R012 for more information.

  • Cover Plugin: Go to Diffblue > Change Settings > Sandboxed Environment > Allowed JNI prefixes.

Also, you may want to consider simply refactoring parts of your code. For example, if you have a method that executes a denied operation but also contains business logic that needs to be verified, you could refactor your code so that any logic that can be executed within the sandbox is contained in a separate, unit-testable method - this ensures the business logic is tested and consequently increases code coverage.

Harmful Example

Let's look at an example of a denied operation and what could happen if it were allowed. One of the operations prevented by default is accessing files outside of the temp directory (specified by java.io.tmpdir). We further restrict this by allowing only writing to newly created files. The root parameter may end up pointing to a real directory on your system (it could be given any value, such as /, C:\, /home/User/), and then be executed, removing all the files contained within that directory along with the directory itself. With the default policy in place, trying to write tests for this method would result in an R011 output code.

public class Harmful {
    public void deleteRecursively(final Path root) {
        final File[] contents = directoryToBeDeleted.listFiles();
        if (contents != null) {
            for (final File file : contents) {
                deleteRecursively(file);
            }
        }
        return directoryToBeDeleted.delete();
    }
}

Operations

The tables below outline the denied operations (Diffblue sandbox enabled) and explains what can happen (in the worst case) if they are allowed (Diffblue sandbox disabled). Note that:

  • The exact nature of the risk is heavily dependent on the application being analyzed and can't be predicted ahead of time. If the application being analyzed is a web-based application in an automated pipeline that has no access to production systems, the risk of experiencing many of the adverse effects mentioned below is relatively low.

  • The impact of the Diffblue sandbox policy is not restricted to just the method, but methods that call these methods, and so on.

I/O Operation

Operation
Allowed by Default?
Change with "Disable Sandbox"
Possible Effect if Allowed

Networking

No

Allowed

Unexpected network traffic, data loss on remote machines, instability in unit tests.

FileIO

Read all files, write new files.

Read all files, write all files.

Data loss on the machine where Cover is running, instability in unit tests.

TmpFileIO

Read all files, write new files.

Read all files, write all files.

Unnecessary disk space is consumed, instability in unit tests.

  • Data loss is likely to occur if methods (like a hypothetical delete method in a typical web application) are invoked and happen to have access to live credentials (e.g. production or test environments).

  • Likewise, unexpected network traffic can occur if the application makes raw network requests to servers.

  • Disk space may be consumed because the application may not clean up temporary files.

Java Libraries

Library
Allowed by Default?
Change with "Disable Sandbox"
Possible Effect if Allowed

GUI (Swing/AWT)

No, java.awt.headless is set to true

Allowed, java.awt.headless is set to false

N/A

sun.misc.*

No

Allowed

Instability of tests across JVM vendors and operating systems.

sun.misc.Unsafe

No

Allowed

Unintended side effects across unit tests in a class, or test suite.

  • Trying to write unit tests for methods that invoke GUIs is difficult. In those cases, the java.awt.headless property is usually set to true to disable those GUI components. If you require tests for the logic behind the GUIs, it's better to refactor the code to allow that logic to be tested independently of the GUI.

  • The sun.misc.* packages provide very low level access to fundamental system operations and do not form a part of the public interface. Because of the very low level access, we deem these operations unsafe due to the risk of unintended side effects.

The sun.* packages are not part of the supported, public interface.

A Java program that directly calls into sun.* packages is not guaranteed to work on all Java-compatible platforms. In fact, such a program is not guaranteed to work even in future versions on the same platform.

Each company that implements the Java platform will do so in their own private way. The classes in sun.* are present in the JDK to support Oracle's implementation of the Java platform: the sun.* classes are what make the Java platform classes work "under the covers" for Oracle's JDK. These classes will not in general be present on another vendor's Java platform. If your Java program asks for a class "sun.package.Foo" by name, it may fail with ClassNotFoundError, and you will have lost a major advantage of developing in Java.

Technically, nothing prevents your program from calling into sun.* by name. From one release to another, these classes may be removed, or they may be moved from one package to another, and it's fairly likely that their interface (method names and signatures) will change. (From Oracle's point of view, since we are committed to maintaining the Java platform, we need to be able to change sun.* to refine and enhance the platform.) In this case, even if you are willing to run only on Oracle's implementation, you run the risk of a new version of the implementation breaking your program.

In general, writing java programs that rely on sun.* is risky: those classes are not portable, and are not supported.

System Operations

Operation
Allowed by Default?
Change with "Disable Sandbox"
Possible Effect if Allowed

External Processes

No

Allowed

Instability of unit tests, random behavior of other processes.

JMX

No

Allowed

Instability of unit tests, random behavior of JMX enabled beans/services/etc.

JNI

No, except JNI allow list - see below.

Allowed

Tests won't generate coverage for the native code.

System.exit

No

No

Causes the analysis service JVM to exit and would increase in the occurrence of R024/R025 output codes (from Diffblue Cover).

* Note that if the Diffblue sandbox policy is blocking access to a JNI library that you believe to be safe to use, then you can add it to the JNI allow list (a comma separated list of JNI library name prefixes can be used):

  • Cover CLI: Use the --allow-jni option - see allow-jni and/or Working with code R012 for more information.

  • Cover Plugin: Go to Diffblue > Change Settings > Sandboxed Environment > Allowed JNI prefixes.

Peripherals

Generally, these operations require access to specific hardware which may or may not be present. Because this can't be determined beforehand, unit tests for methods using these APIs would be inherently unstable.

Peripheral
Allowed by Default?
Change with "Disable Sandbox"
Possible Effect if Allowed

Smartcards

No

Allowed

Corruption of data on the smartcards.

Printing

No

Allowed

Print jobs are submitted to configured printers.

Security

Security
Allowed by Default?
Change with "Disable Sandbox"
Possible Effect if Allowed

Setting security manager

No

Allowed

Unit tests won't be created or verified.

Create access control context

No

Allowed

Attempts to bypass sandbox will be allowed.

Kerberos Authentication

No

Allowed

Unintended side-effects with Kerberos systems.

PreviousWorking with code R008NextWorking with code R012

Last updated 1 year ago

Was this helpful?

Furthermore, the following is taken from the published Oracle regarding the use of these packages.

FAQ