# The JVM Test Suite Plugin

version 7.6.3

The JVM Test Suite plugin (plugin id: `jvm-test-suite`) provides a DSL and API to model multiple groups of automated tests into test suites in JVM-based projects. Tests suites are intended to grouped by their purpose and can have separate dependencies and use different testing frameworks.

For instance, this plugin can be used to define a group of Integration Tests, which might run much longer than unit tests and have different environmental requirements.

## Usage

This plugin is applied automatically by the `java` plugin but can be additionally applied explicitly if desired. The plugin cannot be used without a JVM language plugin applied as it relies on several conventions of the `java` plugin.

Example 1. Applying the JVM Test Suite plugin
build.gradle

    plugins {
        id 'java'
        id 'jvm-test-suite'
    }

build.gradle.kts

    plugins {
        java
        `jvm-test-suite`
    }

The plugins adds the following objects to the project:

- A `testing` extension (type: TestingExtension) to the project used to configure test suites.

When used with the Java Plugin:

- A test suite named `test` (type: JvmTestSuite).

- A `test` SourceSet.

- Several configurations derived from the `test` SourceSet name: `testImplementation`, `testCompileOnly`, `testRuntimeOnly`

- A single test suite target backed by a task named `test`.

The `test` task, SourceSet and derived configurations are identical in name and function to those used in prior Gradle releases.

## Tasks

The JVM Test Suite plugin adds the following task to the project:

`test` — Test

  Depends on: `testClasses` from the `java` plugin, and all tasks which produce the test runtime classpath

  Runs the tests using the framework configured for the default test suite.

Additional instances of Test tasks will be automatically created for each test suite added via the `testing` extension.

## Configuration

See the TestingExtension class in the API documentation.

### Terminology and Modeling

The JVM Test Suite Plugin introduces some modeling concepts backed by new APIs. Here are their definitions.

#### Test Suite

A test suite is a collection of JVM-based tests.

#### Test Suite Target

For the initial release of this plugin, each test suite has a single target. This results in a 1:1:1 relationship between test suite, test suite target and a matching Test task. The name of the `Test` task is derived from the suite name. Future iterations of the plugin will allow defining multiple targets based other attributes, such as a particular JDK/JRE runtime.

Each test suite has some configuration that is common across for all tests contained in the suite:

- Testing framework

- Sources

- Dependencies

In the future, other properties may be specified in the test suite which may influence the toolchains selected to compile and run tests.

The `Test` task associated with the target inherits its name from the suite. Other properties of the `Test` task are configurable.

#### Test Suite Type

Each test suite must be assigned a type. Types can be used to group related test suites across multiple Gradle projects within a build.

The type of a test suite can be configured using the suites’s test type property. The type must be unique across all test suites in the same Gradle project. By convention, the type is set to the name of the test suite, converted to dash-case - with the exception of the built-in test suite, which uses the value 'unit-test'.

Common values are available as constants in TestSuiteType.

## Configuration Examples

Here are several examples to illustrate the configurability of test suites.

## Declare an additional test suite

build.gradle

    testing {
        suites { (1)
            test { (2)
                useJUnitJupiter() (3)
            }

            integrationTest(JvmTestSuite) { (4)
                dependencies {
                    implementation project() (5)
                }

                targets { (6)
                    all {
                        testTask.configure {
                            shouldRunAfter(test)
                        }
                    }
                }
            }
        }
    }

    tasks.named('check') { (7)
        dependsOn(testing.suites.integrationTest)
    }

build.gradle.kts

    testing {
        suites { (1)
            val test by getting(JvmTestSuite::class) { (2)
                useJUnitJupiter() (3)
            }

            val integrationTest by registering(JvmTestSuite::class) { (4)
                dependencies {
                    implementation(project()) (5)
                }

                targets { (6)
                    all {
                        testTask.configure {
                            shouldRunAfter(test)
                        }
                    }
                }
            }
        }
    }

    tasks.named("check") { (7)
        dependsOn(testing.suites.named("integrationTest"))
    }

1 | Configure all test suites for this project.
2 | Configure the built-in `test` suite. This suite is automatically created for backwards compatibility. You must specify a testing framework to use in order to run these tests (e.g. JUnit 4, JUnit Jupiter). This suite is the only suite that will automatically have access to the production source’s `implementation` dependencies, all other suites must explicitly declare these.
3 | Declare this test suite uses JUnit Jupiter. The framework’s dependencies are automatically included. It is always necessary to explicitly configure the built-in `test` suite even if JUnit4 is desired.
4 | Define a new suite called `integrationTest`. Note that all suites other than the built-in `test` suite will by convention work as if `useJUnitJupiter()` was called. You do not have to explicitly configure the testing framework on these additional suites, unless you wish to change to another framework.
5 | Add a dependency on the production code of the project to the `integrationTest` suite targets. By convention, only the built-in `test` suite will automatically have a dependency on the production code of the project.
6 | Configure all targets of this suite. By convention, test suite targets have no relationship to other `Test` tasks. This example shows that the slower integration test suite targets should run after all targets of the `test` suite are complete.
7 | Configure the `check` task to depend on all `integrationTest` targets. By convention, test suite targets are not associated with the `check` task.

Invoking the `check` task on the above configured build should show output similar to:

    > Task :compileJava
    > Task :processResources NO-SOURCE
    > Task :classes
    > Task :jar
    > Task :compileTestJava
    > Task :processTestResources NO-SOURCE
    > Task :testClasses
    > Task :test
    > Task :compileIntegrationTestJava
    > Task :processIntegrationTestResources NO-SOURCE
    > Task :integrationTestClasses
    > Task :integrationTest
    > Task :check

    BUILD SUCCESSFUL in 0s
    6 actionable tasks: 6 executed

Note that the `integrationTest` test suite does not run until after the `test` test suite completes.

## Configure the built-in `test` suite

build.gradle

    testing { (1)
        suites {
            test {
                useTestNG() (1)

                targets {
                    all {
                        testTask.configure { (2)
                            // set a system property for the test JVM(s)
                            systemProperty 'some.prop', 'value'
                            options { (3)
                                preserveOrder = true
                            }
                        }
                    }
                }
            }
        }
    }

build.gradle.kts

    testing {
        suites {
            val test by getting(JvmTestSuite::class) {
                useTestNG() (1)

                targets {
                    all {
                        testTask.configure { (2)
                            // set a system property for the test JVM(s)
                            systemProperty("some.prop", "value")
                            options { (3)
                                val options = this as TestNGOptions
                                options.preserveOrder = true
                            }
                        }
                    }
                }
            }
        }
    }

1 | Declare the `test` test suite uses the TestNG test framework.
2 | Lazily configure the test task of all targets of the suite; note the return type of `testTask` is `TaskProvider<Test>`.
3 | Configure more detailed test framework options. The `options` will be a subclass of `org.gradle.api.tasks.testing.TestFrameworkOptions`, in this case it is `org.gradle.api.tasks.testing.testng.TestNGOptions`.

## Configure dependencies of a test suite

build.gradle

    testing {
        suites {
            test { (1)
                dependencies {
                    // Note that this is equivalent to adding dependencies to testImplementation in the top-level dependencies block
                    implementation 'org.assertj:assertj-core:3.21.0' (2)
                    annotationProcessor 'com.google.auto.value:auto-value:1.9' (3)
                }
            }
        }
    }

build.gradle.kts

    testing {
        suites {
            val test by getting(JvmTestSuite::class) { (1)
                dependencies {
                    // Note that this is equivalent to adding dependencies to testImplementation in the top-level dependencies block
                    implementation("org.assertj:assertj-core:3.21.0") (2)
                    annotationProcessor("com.google.auto.value:auto-value:1.9") (3)
                }
            }
        }
    }

1 | Configure the built-in `test` test suite.
2 | Add the assertj library to the test’s compile and runtime classpaths. The `dependencies` block within a test suite is already scoped for that test suite. Instead of having to know the global name of the configuration, test suites have a consistent name that you use in this block for declaring `implementation`, `compileOnly`, `runtimeOnly` and `annotationProcessor` dependencies.
3 | Add the Auto Value annotation processor to the suite’s annotation processor classpath so that it will be run when compiling the tests.

## Configure source directories of a test suite

build.gradle

    testing {
        suites {
            integrationTest(JvmTestSuite) { (1)
                sources { (2)
                    java { (3)
                        srcDirs = ['src/it/java'] (4)
                    }
                }
            }
        }
    }

build.gradle.kts

    testing {
        suites {
            val integrationTest by registering(JvmTestSuite::class) { (1)
                sources { (2)
                    java { (3)
                        setSrcDirs(listOf("src/it/java")) (4)
                    }
                }
            }
        }
    }

1 | Declare and configure a suite named `integrationTest`. The `SourceSet` and synthesized `Test` tasks will be based on this name.
2 | Configure the `sources` of the test suite.
3 | Configure the `java` SourceDirectorySet of the test suite.
4 | Overwrite the `srcDirs` property, replacing the conventional `src/integrationTest/java` location with `src/it/java`.

## Configure the type of a test suite

build.gradle

    testing {
        suites {
            secondaryTest(JvmTestSuite) {
                testType = TestSuiteType.INTEGRATION_TEST (1)
            }
        }
    }

build.gradle.kts

    testing {
        suites {
            val secondaryTest by registering(JvmTestSuite::class) {
                testType.set(TestSuiteType.INTEGRATION_TEST) (1)
            }
        }
    }

1 | This suite would use a type value of 'secondary-test' by default. This explicitly sets the type to 'integration-test'.

## Configure the `Test` task for a test suite

build.gradle

    testing {
        suites {
            integrationTest {
                targets {
                    all { (1)
                        testTask.configure {
                            maxHeapSize = '512m' (2)
                        }
                    }
                }
            }
        }
    }

build.gradle.kts

    testing {
        suites {
            val integrationTest by getting(JvmTestSuite::class) {
                targets {
                    all { (1)
                        testTask.configure {
                            setMaxHeapSize("512m") (2)
                        }
                    }
                }
            }
        }
    }

1 | Configure the `integrationTest` task created by declaring a suite of the same name.
2 | Configure the `Test` task properties.

`Test` tasks associated with a test suite target can also be configured directly, by name. It is not necessary to configure via the test suite DSL.

## Sharing configuration between multiple test suites

There are several ways to share configuration between multiple test suites to avoid the duplication of dependencies or other configuration. Each method is a tradeoff between flexibility and the use of a declarative vs. imperative configuration style.

1. Use the `configureEach` method on the `suites` container to configure every test suite the same way.

2. Use `withType` and `matching` with `configureEach` to filter test suites and configure a subset of them.

3. Extract the configuration block to a local variable and apply it only to the desired test suites.

### Method 1: Use `configureEach`

This is the most straightforward way to share configuration across every test suite. The configuration is applied to each test suite.

build.gradle

    testing {
        suites {
            configureEach { (1)
                useJUnitJupiter()
                dependencies { (2)
                    implementation('org.mockito:mockito-junit-jupiter:4.6.1')
                }
            }
            (3)
            integrationTest(JvmTestSuite)
            functionalTest(JvmTestSuite) {
                dependencies { (4)
                    implementation('org.apache.commons:commons-lang3:3.11')
                }
            }
        }
    }

build.gradle.kts

    testing {
        suites {
            configureEach { (1)
                if (this is JvmTestSuite) {
                    useJUnitJupiter()
                    dependencies { (2)
                        implementation("org.mockito:mockito-junit-jupiter:4.6.1")
                    }
                }
            }
            (3)
            val integrationTest by registering(JvmTestSuite::class)
            val functionalTest by registering(JvmTestSuite::class) {
                dependencies { (4)
                    implementation("org.apache.commons:commons-lang3:3.11")
                }
            }
        }
    }

1 | Configure every JVM test suite
2 | Provide the dependencies to be shared by all test suites
3 | Define the additional test suites which will be configured upon creation
4 | Add dependencies specific to the `functionalTest` test suite

### Method 2: Use `withType`, `matching` and `configureEach`

This method adds filtering commands to only apply the configuration to a subset of the test suites, as identified by name.

build.gradle

    testing {
        suites {
            withType(JvmTestSuite).matching { it.name in ['test', 'integrationTest'] }.configureEach { (1)
                useJUnitJupiter()
                dependencies {
                    implementation('org.mockito:mockito-junit-jupiter:4.6.1')
                }
            }

            integrationTest(JvmTestSuite)
            functionalTest(JvmTestSuite) {
                useJUnit() (2)
                dependencies { (3)
                    implementation('org.apache.commons:commons-lang3:3.11')
                }
            }
        }
    }

build.gradle.kts

    testing {
        suites {
            withType(JvmTestSuite::class).matching { it.name in listOf("test", "integrationTest") }.configureEach { (1)
                useJUnitJupiter()
                dependencies {
                    implementation("org.mockito:mockito-junit-jupiter:4.6.1")
                }
            }
            val integrationTest by registering(JvmTestSuite::class)
            val functionalTest by registering(JvmTestSuite::class) {
                useJUnit() (2)
                dependencies { (3)
                    implementation("org.apache.commons:commons-lang3:3.11")
                }
            }
        }
    }

1 | Configure every JVM test suite matching the given criteria
2 | Use a different testing framework for the `functionalTest` test suite
3 | Add dependencies specific to the `functionalTest` test suite

### Method 3: Extract a custom configuration block

This method is the most flexible, but also the most imperative.

build.gradle

    testing {
        suites {
            def applyMockito = { suite -> (1)
                suite.useJUnitJupiter()
                suite.dependencies {
                    implementation('org.mockito:mockito-junit-jupiter:4.6.1')
                }
            }

            /* This is the equivalent of:
                test {
                    applyMockito(this)
                }
             */
            test(applyMockito) (2)

            /* This is the equivalent of:
                integrationTest(JvmTestSuite)
                applyMockito(integrationTest)
             */
            integrationTest(JvmTestSuite, applyMockito) (3)

            functionalTest(JvmTestSuite) {
                useJUnit()
                dependencies {
                    implementation('org.apache.commons:commons-lang3:3.11')
                }
            }
        }
    }

build.gradle.kts

    testing {
        suites {
            val applyMockito = { suite: JvmTestSuite -> (1)
                suite.useJUnitJupiter()
                suite.dependencies {
                    implementation("org.mockito:mockito-junit-jupiter:4.6.1")
                }
            }

            /* This is the equivalent of:
                val test by getting(JvmTestSuite::class) {
                    applyMockito(this)
                }
             */
            val test by getting(JvmTestSuite::class, applyMockito)  (2)

            /* This is the equivalent of:
                val integrationTest by registering(JvmTestSuite::class)
                applyMockito(integrationTest.get())
             */
            val integrationTest by registering(JvmTestSuite::class, applyMockito) (3)

            val functionalTest by registering(JvmTestSuite::class) {
                useJUnit()
                dependencies {
                    implementation("org.apache.commons:commons-lang3:3.11")
                }
            }
        }
    }

1 | Define a closure which takes a single `JvmTestSuite` parameter and configures it
2 | Apply the closure to a test suite, using the default (`test`) test suite
3 | Alternate means of applying a configuration closure to a test suite outside of its declaration, using the `integrationTest` test suite

## Differences between the test suite `dependencies` and the top-level `dependencies` blocks

Gradle 7.6 changed the API of the test suite’s `dependencies` block to be more strongly-typed.

Each dependency bucket returns a `DependencyAdder` that provides strongly-typed methods to add and configure dependencies.

There is also a `DependencyFactory` with factory methods to create new dependencies from different notations. Dependencies can be created lazily using these factory methods, as shown below.

This API differs from the top-level `dependencies` block in the following ways:

- Dependencies must be declared using a `String`, an instance of `Dependency`, a `FileCollection`, a `Provider` of `Dependency`, or a `ProviderConvertible` of `MinimalExternalModuleDependency`.

- Outside of Gradle build scripts, you must explicitly call a getter for the `DependencyAdder` and `add`.

  - `dependencies.add("implementation", x)` becomes `getImplementation().add(x)`

- You cannot declare dependencies with the `Map` notation from Kotlin and Java. Use multi-argument methods instead in Kotlin and Java.

  - Kotlin: `compileOnly(mapOf("group" to "foo", "name" to "bar"))` becomes `compileOnly(module(group = "foo", name = "bar"))`

  - Java: `compileOnly(Map.of("group", "foo", "name", "bar"))` becomes `getCompileOnly().add(module("foo", "bar", null))`

- You cannot add a dependency with an instance of `Project`. You must turn it into a `ProjectDependency` first.

- You cannot add version catalog bundles directly. Instead, use the `bundle` method on each configuration.

  - Kotlin and Groovy: `implementation(libs.bundles.testing)` becomes `implementation.bundle(libs.bundles.testing)`

- You cannot use providers for non-`Dependency` types directly. Instead, map them to a `Dependency` using the `DependencyFactory`.

  - Kotlin and Groovy: `implementation(myStringProvider)` becomes `implementation(myStringProvider.map { dependencyFactory.create(it) })`

  - Java: `implementation(myStringProvider)` becomes `getImplementation().add(myStringProvider.map(getDependencyFactory()::create)`

The `dependencies` block within a test suite may not provide access to the same methods as the top-level `dependencies` block. To access these methods, you need to use the top-level `dependencies` block.

| Adding dependencies via the test suite’s own `dependencies` block is the preferred and recommended way. You can also access the configurations of a test suite in a top-level `dependencies` block after creating the test suite, but the exact names of the configurations used by test suites should be considered an implementation detail subject to change.

## Differences between similar methods on JvmTestSuite and Test task types

Instances of JvmTestSuite have methods useJUnit() and useJUnitJupiter(), which are similar in name to methods on the Test task type: useJUnit() and useJUnitPlatform(). However, there are important differences.

Unlike the methods on the Test task, JvmTestSuite methods perform two additional pieces of configuration:

1. JvmTestSuite#useJUnit(), #useJUnitJupiter() and other framework-specific methods automatically apply the relevant testing framework libraries to the compile and runtime classpaths of the suite targets. Note the overloaded methods like JvmTestSuite#useJUnit(String) and #useJUnitJupiter(String) allow you to provide specific versions of the framework dependencies.

2. JvmTestSuite#useJUnit() and #useJUnitJupiter() automatically configure the suite targets' Test tasks to execute using the specified framework.

An example of this behavior is shown in the first configuration example above, Configuring the built-in `test` suite.

Note that unlike the Test task, the aforementioned methods on JvmTestSuite do not accept a closure or action for configuring the framework. These framework configuration options can be set on the individual targets.

## Outgoing Variants

Each test suite creates an outgoing variant containing its test execution results. These variants are designed for consumption by the Test Report Aggregation Plugin.

The attributes will resemble the following. User-configurable attributes are highlighted below the sample.

outgoingVariants task output

    --------------------------------------------------
    Variant testResultsElementsForTest (i)
    --------------------------------------------------
    Description = Directory containing binary results of running tests for the test Test Suite's test target.

    Capabilities
        - org.gradle.sample:list:1.0.2 (default capability)
    Attributes
        - org.gradle.category              = verification
        - org.gradle.testsuite.name        = test           (1)
        - org.gradle.testsuite.target.name = test           (2)
        - org.gradle.testsuite.type        = unit-test      (3)
        - org.gradle.verificationtype      = test-results

    Artifacts
        - build/test-results/test/binary (artifactType = directory)

1 | TestSuiteName attribute; value is derived from JvmTestSuite#getName().
2 | TestSuiteTargetName attribute; value is derived from JvmTestSuiteTarget#getName().
3 | TestSuiteType attribute; value is derived from JvmTestSuite#getTestType().