Testing in Java & JVM projects

All JVM testing revolves around a single task type: Test. This runs a collection of test cases using any supported test library — JUnit, JUnit Platform or TestNG — and collates the results. You can then turn those results into a report via an instance of the TestReport task type.

In order to operate, the Test task type requires just two pieces of information:

When you’re using a JVM language plugin — such as the Java Plugin — you will automatically get the following:

The JVM language plugins use the source set to configure the task with the appropriate execution classpath and the directory containing the compiled test classes. In addition, they attach the test task to the check lifecycle task.

It’s also worth bearing in mind that the test source set automatically creates corresponding dependency configurations — of which the most useful are testImplementation and testRuntimeOnly — that the plugins tie into the test task’s classpath.

All you need to do in most cases is configure the appropriate compilation and runtime dependencies and add any necessary configuration to the test task. The following example shows a simple setup that uses JUnit Platform and changes the maximum heap size for the tests' JVM to 1 gigabyte:

The Test task has many generic configuration options as well as several framework-specific ones that you can find described in JUnitOptions, JUnitPlatformOptions and TestNGOptions. We cover a significant number of them in the rest of the chapter.

If you want to set up your own Test task with its own set of test classes, then the easiest approach is to create your own source set and Test task instance, as shown in Configuring integration tests.

Gradle executes tests in a separate ('forked') JVM, isolated from the main build process. This prevents classpath pollution and excessive memory consumption for the build process. It also allows you to run the tests with different JVM arguments than the build is using.

You can control how the test process is launched via several properties on the Test task, including the following:

See Test for details on all the available configuration options.

It’s a common requirement to run subsets of a test suite, such as when you’re fixing a bug or developing a new test case. Gradle provides two mechanisms to do this:

Filtering supersedes the inclusion/exclusion mechanism, but you may still come across the latter in the wild.

With Gradle’s test filtering you can select tests to run based on:

You can enable filtering either in the build script or via the --tests command-line option. Here’s an example of some filters that are applied every time the build runs:

For more details and examples of declaring filters in the build script, please see the TestFilter reference.

The command-line option is especially useful to execute a single test method. When you use --tests, be aware that the inclusions declared in the build script are still honored. It is also possible to supply multiple --tests options, all of whose patterns will take effect. The following sections have several examples of using the command-line option.

The following two sections look at the specific cases of simple class/method names and fully-qualified names.

The Test task generates the following results by default:

In most cases, you’ll work with the standard HTML report, which automatically includes the results from all your Test tasks, even the ones you explicitly add to the build yourself. For example, if you add a Test task for integration tests, the report will include the results of both the unit tests and the integration tests if both tasks are run.

Unlike with many of the testing configuration options, there are several project-level convention properties that affect the test reports. For example, you can change the destination of the test results and reports like so:

Follow the link to the convention properties for more details.

There is also a standalone TestReport task type that you can use to generate a custom HTML test report. All it requires are a value for destinationDir and the test results you want included in the report. Here is a sample which generates a combined report for the unit tests from all subprojects:

In this example, we use a convention plugin myproject.java-conventions to expose the test results from a project to Gradle’s variant aware dependency management engine.

The plugin declares a consumable binaryTestResultsElements configuration that represents the binary test results of the test task. In the aggregation project’s build file, we declare the testReportData configuration and depend on all of the projects that we want to aggregate the results from. Gradle will automatically select the binary test result variant from each of the subprojects instead of the project’s jar file. Lastly, we add a testReport task that aggregates the test results from the testResultsDirs property, which contains all of the binary test results resolved from the testReportData configuration.

You should note that the TestReport type combines the results from multiple test tasks and needs to aggregate the results of individual test classes. This means that if a given test class is executed by multiple test tasks, then the test report will include executions of that class, but it can be hard to distinguish individual executions of that class and their output.

By default, Gradle will run all tests that it detects, which it does by inspecting the compiled test classes. This detection uses different criteria depending on the test framework used.

For JUnit, Gradle scans for both JUnit 3 and 4 test classes. A class is considered to be a JUnit test if it:

For TestNG, Gradle scans for methods annotated with @Test.

Note that abstract classes are not executed. In addition, be aware that Gradle scans up the inheritance tree into jar files on the test classpath. So if those JARs contain test classes, they will also be run.

If you don’t want to use test class detection, you can disable it by setting the scanForTestClasses property on Test to false. When you do that, the test task uses only the includes and excludes properties to find test classes.

If scanForTestClasses is false and no include or exclude patterns are specified, Gradle defaults to running any class that matches the patterns **/*Tests.class and **/*Test.class, excluding those that match **/Abstract*.class.

Gradle allows fine-tuned control over events that are logged to the console. Logging is configurable on a per-log-level basis and by default, the following events are logged:

Test logging can be modified on a per-log-level basis by adjusting the appropriate TestLogging instances in the testLogging property of the test task. For example, to adjust the INFO level test logging configuration, modify the TestLoggingContainer.getInfo() property.

JUnit, JUnit Platform and TestNG allow sophisticated groupings of test methods.

JUnit 4.8 introduced the concept of categories for grouping JUnit 4 tests classes and methods.^[1] Test.useJUnit(org.gradle.api.Action) allows you to specify the JUnit categories you want to include and exclude. For example, the following configuration includes tests in CategoryA and excludes those in CategoryB for the test task:

JUnit Platform introduced tagging to replace categories. You can specify the included/excluded tags via Test.useJUnitPlatform(org.gradle.api.Action), as follows:

The TestNG framework uses the concept of test groups for a similar effect.^[2] You can configure which test groups to include or exclude during the test execution via the Test.useTestNG(org.gradle.api.Action) setting, as seen here:

JUnit 5 is the latest version of the well-known JUnit test framework. Unlike its predecessor, JUnit 5 is modularized and composed of several modules:

The JUnit Platform serves as a foundation for launching testing frameworks on the JVM. JUnit Jupiter is the combination of the new programming model and extension model for writing tests and extensions in JUnit 5. JUnit Vintage provides a TestEngine for running JUnit 3 and JUnit 4 based tests on the platform.

The following code enables JUnit Platform support in build.gradle:

See Test.useJUnitPlatform() for more details.

TestNG allows explicit control of the execution order of tests when you use a testng.xml file. Without such a file — or an equivalent one configured by TestNGOptions.getSuiteXmlBuilder() — you can’t specify the test execution order. However, what you can do is control whether all aspects of a test — including its associated @BeforeXXX and @AfterXXX methods, such as those annotated with @Before/AfterClass and @Before/AfterMethod — are executed before the next test starts. You do this by setting the TestNGOptions.getPreserveOrder() property to true. If you set it to false, you may encounter scenarios in which the execution order is something like: TestA.doBeforeClass() → TestB.doBeforeClass() → TestA tests.

While preserving the order of tests is the default behavior when directly working with testng.xml files, the TestNG API that is used by Gradle’s TestNG integration executes tests in unpredictable order by default.^[3] The ability to preserve test execution order was introduced with TestNG version 5.14.5. Setting the preserveOrder property to true for an older TestNG version will cause the build to fail.

The groupByInstance property controls whether tests should be grouped by instance rather than by class. The TestNG documentation explains the difference in more detail, but essentially, if you have a test method A() that depends on B(), grouping by instance ensures that each A-B pairing, e.g. B(1)-A(1), is executed before the next pairing. With group by class, all B() methods are run and then all A() ones.

Note that you typically only have more than one instance of a test if you’re using a data provider to parameterize it. Also, grouping tests by instances was introduced with TestNG version 6.1. Setting the groupByInstances property to true for an older TestNG version will cause the build to fail.

A common requirement for projects is to incorporate integration tests in one form or another. Their aim is to verify that the various parts of the project are working together properly. This often means that they require special execution setup and dependencies compared to unit tests.

The simplest way to add integration tests to your build is by leveraging the incubating JVM Test Suite plugin. If an incubating solution is not something for you, here are the steps you need to take in your build:

You may also need to perform some additional configuration depending on what form the integration tests take. We will discuss those as we go.

Let’s start with a practical example that implements the first three steps in a build script, centered around a new source set intTest:

This will set up a new source set called intTest that automatically creates:

The example also does the following, not all of which you may need for your specific integration tests:

In most cases, you want your integration tests to have access to the classes under test, which is why we ensure that those are included on the compilation and runtime classpaths in this example. But some types of test interact with the production code in a different way. For example, you may have tests that run your application as an executable and verify the output. In the case of web applications, the tests may interact with your application via HTTP. Since the tests don’t need direct access to the classes under test in such cases, you don’t need to add the production classes to the test classpath.

Another common step is to attach all the unit test dependencies to the integration tests as well — via intTestImplementation.extendsFrom testImplementation — but that only makes sense if the integration tests require all or nearly all the same dependencies that the unit tests have.

There are a couple of other facets of the example you should take note of:

Creating and configuring a source set automatically sets up the compilation stage, but it does nothing with respect to running the integration tests. So the last piece of the puzzle is a custom test task that uses the information from the new source set to configure its runtime classpath and the test classes:

Again, we’re accessing a source set to get the relevant information, i.e. where the compiled test classes are — the testClassesDirs property — and what needs to be on the classpath when running them — classpath.

Users commonly want to run integration tests after the unit tests, because they are often slower to run and you want the build to fail early on the unit tests rather than later on the integration tests. That’s why the above example adds a shouldRunAfter() declaration. This is preferred over mustRunAfter() so that Gradle has more flexibility in executing the build in parallel.

For information on how to determine code coverage for tests in additional source sets, see the JaCoCo Plugin and the JaCoCo Report Aggregation Plugin chapters.

If you are developing Java Modules, everything described in this chapter still applies and any of the supported test frameworks can be used. However, there are some things to consider depending on whether you need module information to be available, and module boundaries to be enforced, during test execution. In this context, the terms whitebox testing (module boundaries are deactivated or relaxed) and blackbox testing (module boundaries are in place) are often used. Whitebox testing is used/needed for unit testing and blackbox testing fits functional or integration test requirements.

Sample: Java Modules multi-project with integration tests

If you want to skip the tests when running a build, you have a few options. You can either do it via command line arguments or in the build script. To do it on the command line, you can use the -x or --exclude-task option like so:

This excludes the test task and any other task that it exclusively depends on, i.e. no other task depends on the same task. Those tasks will not be marked "SKIPPED" by Gradle, but will simply not appear in the list of tasks executed.

Skipping a test via the build script can be done a few ways. One common approach is to make test execution conditional via the Task.onlyIf(String, org.gradle.api.specs.Spec) method. The following sample skips the test task if the project has a property called mySkipTests:

In this case, Gradle will mark the skipped tests as "SKIPPED" rather than exclude them from the build.

In well-defined builds, you can rely on Gradle to only run tests if the tests themselves or the production code change. However, you may encounter situations where the tests rely on a third-party service or something else that might change but can’t be modeled in the build.

You can always use the --rerun built-in task option to force a task to rerun.

Alternatively, if build caching is not enabled, you can also force tests to run by cleaning the output of the relevant Test task — say test — and running the tests again, like so:

cleanTest is based on a task rule provided by the Base Plugin. You can use it for any task.

On the few occasions that you want to debug your code while the tests are running, it can be helpful if you can attach a debugger at that point. You can either set the Test.getDebug() property to true or use the --debug-jvm command line option, or use --no-debug-jvm to set it to false.

When debugging for tests is enabled, Gradle will start the test process suspended and listening on port 5005.

You can also enable debugging in the DSL, where you can also configure other properties:

With this configuration the test JVM will behave just like when passing the --debug-jvm argument but it will listen on port 4455.

To debug the test process remotely via network, the host needs to be set to the machine’s IP address or "*" (listen on all interfaces).