Task Configuration Avoidance - Gradle User Manual

# Task Configuration Avoidance

We are asking early adopters to try out a new API for creating and configuring tasks to iron out any issues and gather feedback. This chapter provides a quick introduction to the feature and explains some guidelines for migrating your build to use the new API. We intend for this new API to eventually replace the existing API through our usual deprecation process over several major releases.

The new Task configuration API is currently incubating. Please be aware that the DSL or API may change in minor Gradle versions.

## What is the new API good for?

In a nutshell, the new API allows builds to avoid the cost of creating and configuring tasks during Gradle’s configuration phase when those tasks will never be executed. We identified task configuration as a contributor to overall configuration time. Configuration time affects every build, so making it faster is good for everyone. For example, if you’re only compiling, Gradle doesn’t need to also configure code quality, testing and publishing tasks. For tasks that aren’t used often by developers, configuration can be expensive (e.g., running `git status` or querying a webservice) and can slow down builds that do not need to execute them. This new API avoids configuring those tasks if they are not needed.

This feature is designed to be backwards compatible when mixed with the old and new APIs, so you can gradually migrate your plugins and build scripts to the new API. Instead of immediately creating a task and running all configuration actions, Gradle exposes a `Provider` to a task. A task will only be created and configured when the task is required. If at any point, a task registered with the new API is accessed through the old API, the task will be created and configured just as if it had been created with the old API.

Many built-in Gradle plugins have been converted to use this new API already, so you may see some benefits without changing anything.

## What do we want you to do?

We’ve been using this feature in many of the built-in Gradle plugins, internal Gradle projects and the Gradle build itself. Now we would appreciate your help in evaluating it with your builds as well.

We’d like for you to take a few steps:

- Try your build with the latest nightly (see the version number at the top of this page).

- How many tasks do you see registered with the new API? See the section called “How do I know if it’s working?” for how to find this.

- Which task types are most numerous? Which ones are created with the old API? Which ones are registered with the new API, but have been created/configured anyways?

- Following the section called “Migration Guide”, change your build scripts to use the new API to avoid creating/configuring tasks that are from the built-in Gradle plugins. Is there an impact to the number of tasks that are created?

- Convert some of your custom plugins/tasks to use the new API. Is there an impact to the number of tasks using the new API and the number of new API tasks that are created?

- Run the gradle-profiler to measure the difference between a build with less task configuration and a build where all tasks are configured. What do you see? Use the scenario file below.

## Guidelines

### How do I defer task creation?

This feature requires build authors to opt-in by migrating task creation from the `TaskContainer.create(java.lang.String)` APIs to the `TaskContainer.register(java.lang.String)` APIs. The `register(…​)` API registers a task to be created at a later time if and only if the task is needed. The `create(…​)` API continues to eagerly create and configure tasks when it is called.

Using the new creation API may not be enough to avoid all task configuration completely. You may need to change other code that configures tasks by name or by type, as explained in the following sections.

### How do I defer task configuration?

Existing APIs like `DomainObjectCollection.all(org.gradle.api.Action)` and `DomainObjectCollection.withType(java.lang.Class, org.gradle.api.Action)` will immediately create and configure any registered tasks. To defer task configuration, you will need to migrate to a new API equivalent. See the table below to identify the new API alternative.

### How do I reference a task without creating/configuring it?

Calling `Provider.get()` or looking up a task by name with `TaskCollection.getByName(java.lang.String)` will cause the task to be created and configured. Methods like `Task.dependsOn(java.lang.Object[])` and `ConfigurableFileCollection.builtBy(java.lang.Object[])` work with `TaskProvider` in the same way as `Task`, so you do not need to unwrap a `Provider` for explicit dependencies to continue to work.

If you are configuring a task by name, you will need to use the new API equivalent. See the table below to identify the new API alternative.

### How to get an instance of a Task?

In the event you still need to get access to a Task instance, you can use `TaskCollection.named(java.lang.String)` and call `Provider.get()`. This will cause the task to be created/configured, but everything should work as it has with the eager APIs.

Please contact us if you find tricky build logic that requires you to get a Task instance.

### How do I know if it’s working?

We provide a couple of internal flags to print out information after a build. This will eventually be captured and displayed in an easier to digest form.

gradle -Dorg.gradle.internal.tasks.stats help.

    Task counts: Old API 188, New API 587, total 775 # 1
    Task counts: created 613, avoided 162, %-lazy 21 # 2

    Task types that were created with the old API # 3
    class org.gradle.api.DefaultTask 31
    class org.jetbrains.kotlin.gradle.tasks.KotlinCompile 28
    class org.gradle.api.tasks.JavaExec 24
    class org.jlleitschuh.gradle.ktlint.KtlintCheck 24
    class org.gradle.api.tasks.bundling.Jar 16
    class org.gradle.kotlin.dsl.accessors.tasks.PrintAccessors 16
    class org.gradle.plugin.devel.tasks.ValidateTaskProperties 15
    class org.gradle.another.CoolTask 13
    class org.gradle.plugin.devel.tasks.GeneratePluginDescriptors 10
    class org.gradle.plugin.devel.tasks.PluginUnderTestMetadata 10
    class org.gradle.kotlin.dsl.accessors.tasks.UpdateProjectSchema 1

    Task types that were registered with the new API but were created anyways # 4
    class org.gradle.api.DefaultTask 142
    class org.gradle.api.tasks.Delete 106
    class org.gradle.api.tasks.compile.JavaCompile 32
    class org.gradle.language.jvm.tasks.ProcessResources 32
    class org.gradle.api.tasks.testing.Test 16
    class org.gradle.api.tasks.javadoc.Javadoc 16
    class org.gradle.plugins.ide.idea.GenerateIdeaModule 15
    class org.gradle.plugins.ide.eclipse.GenerateEclipseClasspath 15
    class org.gradle.plugins.ide.eclipse.GenerateEclipseJdt 15
    class org.gradle.plugins.ide.eclipse.GenerateEclipseProject 15
    class org.gradle.api.tasks.compile.GroovyCompile 10
    class org.gradle.api.tasks.javadoc.Groovydoc 5
    class org.gradle.api.plugins.quality.Checkstyle 4
    class org.gradle.api.plugins.quality.CodeNarc 2

1. 188 tasks are created with the old API and 587 tasks are registered with the new API for a total of 775 tasks.

2. 613 tasks were created and configured and only 162 tasks were avoided (never created or configured). 21% of all tasks were avoided (higher is better).

3. Lists of the type of tasks that were created with the old API. This is a good list to work down to increase the amount of possible avoidable task configuration.

4. Lists of the type of tasks that were created with the new API but were created/configured anyways. This is a good list to work down to increase the amount of task configuration that is avoided.

These statistics are printed out once per build. Projects with buildSrc and composite builds will display this information multiple times. In a build that uses the new APIs perfectly, we should see 0 tasks created with the old API and only 1 created/configured task because we are only executing the `help` task. If you run other tasks (like `build`), you should expect many more tasks to be created and configured.

You can use the list of task types to guide which tasks would provide the biggest bang for your buck when you migrate to the new API.

To approximate the time it takes to configure a build without executing tasks, you can run `gradle help`. Please use the Gradle Profiler to measure your build as described in the section called “Profiling with a Gradle Profiler Scenario File”.

## Migration Guide

The following sections will go through some general guidelines to adhere when migrating the build logic as well as the steps we recommend following. We also cover some troubleshooting and pitfalls to help you work around some issues you may encountered during the migration.

### General

1. Use `help` task as a benchmark during the migration. The `help` task is the perfect candidate to benchmark your migration process. In a completely lazy build, a build scan should show no task created immediately or created during configuration. Only a single task, the `help` task, should be created during task graph calculation. Be mindful of the pitfall around the lazy API usage in the build scan plugin.

2. Only mutate the current task inside configuration action. Because the task configuration action can now run immediately, later or never, mutating anything other than the current task will expose flaky-like behavior in your build. In the future, Gradle will consider these flaky-mutations as failures. It also means the configuration needs to flow from the extensions to the tasks. Thus, the task should be configured by the extension and not the other way around.

3. Prefer small incremental changes. Smaller changes are easier to sanity check. If you ever break your build logic, it will be easier to analyze the changelog since the last successful verification.

4. Ensure a good plan is established for validating the build logic. Usually, a simple `build` task invocation should do the trick to validate all corner of your build logic. However, if all automation aspect is centralized around Gradle, you may need to validate other task execution path.

5. Prefer automatic testing to manual testing. It’s good practice to write integration test for your build logic using TestKit.

6. Avoid referencing a task by name. In the majority of the case, referencing a task by name is a fragile pattern and should be avoided. Although task name is available on the `TaskProvider`, effort should be made to use references from a strongly typed model instead.

7. Use the lazy task API as much as possible. Although realizing a task within a lazy configuration action may not have any immediate impact, it makes misuse of the lazy API more costly. The lazy task API is an opt-in feature, meaning someone may be realizing task eagerly which would cause the realization to realize other tasks transitively. Using `TaskProvider` helps create an indirection that protects transitive realization.

### Migration Steps

The first part of the migration process is to go through the code and manually migrate eager task creation and configuration to use lazy APIs. The following explores the recommended steps for a successful migration. While going through those steps, keep in mind the guidelines.

1. Migrate task configuration that affects all tasks (`tasks.all {}`) or subsets by type (`tasks.withType(…​) {}`). This will cause your build to create fewer built-in Gradle task types. Note that despite some built-in Gradle task aren’t using the lazy APIs, the configuration affecting those task type should still be migrated. We recommend using regex code search capability offered by modern IDE. For example, the Intellij regex `\.all\s*((\(?\s*\{)|\(\s*new)` will identify every call to `.all` of a `DomainObjectCollection` and subtype. We will need to go through those result and identify and migrate only the method calls affecting the `TaskContainer` or `TaskCollection`. Be aware of the common pitfall around deferred configuration.

2. Migrate tasks configured by name. Just like the previous point, it will cause your build to create fewer built-in Gradle tasks. For example, logic that uses `TaskContainer#getByName(String, Closure/Action)` would be converted to `TaskContainer#named(String).configure(Closure/Action)`. It also include task configuration as DSL blocks. Be aware of the common pitfall around deferred configuration.

3. Migrate tasks creation to `register(…​)`. It will cause your build to create fewer tasks in general. Be aware of the common pitfall around deferred configuration.

At this point, we should see a decent improvement regarding build configuration. We recommend profiling your build and note the improvement. Under some circumstance, you may notice no improvement. It’s unfortunate, and we feel your frustration. Keep reading onto the next section.

### Troubleshooting

- The build became unstable after migrating to lazy task configuration. It’s a common issue that happens during the migration. Follow these next steps to find out the issue:

  1. Is the build succeed when all tasks are eagerly resolved on creation using the command line flag (`-Dorg.gradle.internal.tasks.eager=true`)? Verify that each task configuration only mutating the task.

  2. Was there any build logic accidentally deleted? Verify your change log for disparity in the configuration that may have been deleted.

  3. It may be a real issue. For example, the issue of `FileCollection` using `TaskProvider` was a real discrepancy found while migrating a build to lazy task API. As a rule of thumb, a `TaskProvider` should be accepted everywhere a `Task` instance is accepted.

- Where a task was resolved? As we keep developing the feature, more reporting, and troubleshooting information will be made available to answer this question. In the meantime, build scan is the best way to answer this question. Follow these steps:

  1. Create a build scan. Execute the Gradle command using the `--scan` flag.

  2. Navigate to the configuration performance tab.

     Figure: Navigate to configuration performance tab in build scan

     1. Navigate to the performance card from the left side menu.

     2. Navigate to the configuration tab from the top of the performance card.

  3. All the information requires will be presented.

     Figure: Configuration performance tab in build scan annotated

     1. Total tasks present when each task are created or not.

        - Created immediately represent all the task created using the eager task APIs.

        - Created during configuration represent all the tasks that were created using the lazy task APIs, but was realized explicitly (via `TaskProvider#get()`) or implicitly using the eager task query APIs.

        - Both "Created immediately" and "Created during configuration" numbers are considered the "bad" numbers that should be minimized as much as possible.

        - Created during task graph calculation number represent all the task created while building the execution task graph. Typically, this number should be equal to the number of tasks that is executed.

        - Not created number represents all the tasks that were avoided in this build session.

     2. The next section helps answer the question of where a task was realized. For each scripts, plugins and lifecycle callbacks, the last column represents the tasks that were created either immediately or during configuration. Ideally, this column should be empty. The migration should be focusing on the build logic under your control.

     3. Extending any scripts, plugins, or lifecycle callbacks subsection shows a break down in which project it was applied.

- The build performance haven’t improved after migrating to lazy API. It is important to note that lazy task configuration is an opt-in feature. It means the entire configuration chain needs to be lazy to get the benefit. If at any point, an eager API is used for a specific named task, task type or, the worst, the entire task container, all benefits are mitigated. The first step is to have a look at the statistics to see how much more the build can be improved. If the statistic shows the build is completely lazy (work-avoided) please contact us so we can discuss where this feature falls short and improve it for the next Gradle version.

### Pitfalls

- Lazy task configuration is an opt-in feature. Any eager realization of a task will mitigate the benefit of this new API. In the far future, eager realizations will become an error, until them, it’s up to the build author to avoid eager realization.

- Beware of the hidden eager task realization. Gradle has lots of ways to configure a task eagerly. `TaskContainer#getByName(String, Action)` is an explicit configuration by name. Configuring a task as a DSL block is an alias to the previous explicit configuration, meaning a lazy task will be prematurely resolved:

      // Given a task lazily created with
      tasks.register "someTask"

      // Some time later, the task is configured using a DSL block like
      someTask {
          // The task is resolved immediately, and this closure is executed immediately
      }

  You will have to migrate the DSL task configuration to the following:

      tasks.named("someTask").configure {
          // ...
          // Beware of the pitfalls here
      }

  While migrating to the lazy task configuration, you may still have eager task configuration that references a task by name using Groovy’s property to task reference syntactic sugar. If this lookup is done from anything other than a lazy configuration action, the lazy task reference by the name will be prematurely realized:

      tasks.register "someTask"

      // Sometime later, an eager task is configured like
      task aEagerTask {
          // The task is resolved immediately as it will be treated as a property of the project and will result in an eager task lookup by name
          dependsOn someTask
      }

  You can solve the following in three ways:

  - Use `TaskProvider` variable. Useful when the task is referenced multiple time in the same build script.

        def someTask = tasks.register "someTask"

        task aEagerTask {
            dependsOn someTask
        }

  - Migrate consumer task. It will only work under specific circumstances. If the task gets resolve later or the realization may happen in a configuration action with no ways of actually knowing where the task is created or configured, this solution won’t be helpful.

        tasks.register "someTask"

        tasks.register("aEagerTask") {
            dependsOn someTask
        }

  - Lookup the task lazily. Useful when the task registration isn’t co-located with its eager usage.

        tasks.register "someTask"

        task aEagerTask {
            dependsOn tasks.named("someTask")
        }

- The `jar` task is eagerly realized, always. The following issue is tracking this pitfall. Even though this issue wasn’t released with Gradle 4.9, you should reference the `jar` task lazily.

- The build scan plugin `buildScanPublishPrevious` task is eager until version 1.15. Upgrade the build scan plugin in your build to use the latest version.

## Old vs New API overview

- Methods that take a `groovy.lang.Closure` are covered in the new API with methods taking `org.gradle.api.Action`.

- More convenience methods may be added in the future based on user feedback.

- Some old API methods may never have a direct replacement in the new API.

Old vs New API | Description
--- | ---
Instead of: `task myTask(type: MyTask) {}` | There is not a shorthand Groovy DSL for using the new API.
Use: `tasks.register("myTask", MyTask) {}`
Instead of: `TaskContainer.create(java.util.Map)` | Use one of the alternatives below.
Use: No direct equivalent.
Instead of: `TaskContainer.create(java.util.Map, groovy.lang.Closure)` | Use one of the alternatives below.
Use: No direct equivalent.
Instead of: `TaskContainer.create(java.lang.String)` | This returns a `TaskProvider` instead of a `Task`.
Use: `TaskContainer.register(java.lang.String)`
Instead of: `TaskContainer.create(java.lang.String, groovy.lang.Closure)` | This returns a `TaskProvider` instead of a `Task`.
Use: `TaskContainer.register(java.lang.String, org.gradle.api.Action)`
Instead of: `TaskContainer.create(java.lang.String, java.lang.Class)` | This returns a `TaskProvider` instead of a `Task`.
Use: `TaskContainer.register(java.lang.String, java.lang.Class)`
Instead of: `TaskContainer.create(java.lang.String, java.lang.Class, org.gradle.api.Action)` | This returns a `TaskProvider` instead of a `Task`.
Use: `TaskContainer.register(java.lang.String, java.lang.Class, org.gradle.api.Action)`
Instead of: `TaskContainer.create(java.lang.String, java.lang.Class, java.lang.Object[])` | This returns a `TaskProvider` instead of a `Task`.
Use: `TaskContainer.register(java.lang.String, java.lang.Class, java.lang.Object[])`
Instead of: `TaskCollection.getByName(java.lang.String)` | This returns a `TaskProvider` instead of a `Task`.
Use: `TaskCollection.named(java.lang.String)`
Instead of: `TaskCollection.getByName(java.lang.String, groovy.lang.Closure)` | This returns a `TaskProvider` instead of a `Task`.
Use: `named(java.lang.String).configure(Action)`
Instead of: `TaskContainer.getByPath(java.lang.String)` | Accessing tasks from another project requires a specific ordering of project evaluation.
Use: No direct equivalent.
Instead of: `NamedDomainObjectCollection.findByName(java.lang.String)` | `named(String)` is the closest equivalent, but will fail if the task does not exist. Using `findByName(String)` will cause tasks registered with the new API to be created/configured.
Use: No direct equivalent.
Instead of: `TaskContainer.findByPath(java.lang.String)` | See `getByPath(String)` above.
Use: No direct equivalent.
Instead of: `TaskCollection.withType(java.lang.Class)` | This is OK to use because it does not require tasks to be created immediately.
Use: OK
Instead of: `withType(java.lang.Class).getByName(java.lang.String)` | This returns a `TaskProvider` instead of a `Task`.
Use: `withType(java.lang.Class).named(java.lang.String)`
Instead of: `DomainObjectCollection.withType(java.lang.Class, org.gradle.api.Action)` | This returns `void`, so it cannot be chained.
Use: `withType(java.lang.Class).configureEach(org.gradle.api.Action)`
Instead of: `DomainObjectCollection.all(org.gradle.api.Action)` | This returns `void`, so it cannot be chained.
Use: `DomainObjectCollection.configureEach(org.gradle.api.Action)`
Instead of: `NamedDomainObjectSet.findAll(groovy.lang.Closure)` | Avoid calling this method. `matching(Spec)` and `configureEach(Action)` are more appropriate in most cases.
Use: OK, with issues.
Instead of: `TaskCollection.matching(groovy.lang.Closure)` | This is OK to use because it does not require tasks to be created immediately.
Use: OK
Instead of: `TaskCollection.getAt(java.lang.String)` | Avoid calling this directly as it’s a Groovy convenience method. The alternative returns a `TaskProvider` instead of a `Task`.
Use: `TaskCollection.named(java.lang.String)`
Instead of: `iterator()` or implicit iteration over the `Task` collection | Avoid doing this as it requires creating and configuring all tasks. See `findAll(Closure)` above.
Use: OK, with issues.
Instead of: `remove(org.gradle.api.Task)` | Avoid calling this. The behavior of `remove` with the new API may change in the future.
Use: OK, with issues.
Instead of: `TaskContainer.replace(java.lang.String)` | Avoid calling this. The behavior of `replace` with the new API may change in the future.
Use: OK, with issues.
Instead of: `TaskContainer.replace(java.lang.String, java.lang.Class)` | Avoid calling this. The behavior of `replace` with the new API may change in the future.
Use: OK, with issues.

## Profiling with a Gradle Profiler Scenario File

The Gradle Profiler is a tool to measure build times for Gradle builds in a predictable and reproducible manner. The tool automates collecting profiling and benchmark information from a Gradle build and mitigates environmental impacts to measuring build time (like JIT warmups and cached dependencies). Clone and build `gradle-profiler` locally.

To measure the impact of the new API on your build, we’ve included a sample scenario file you can use. This scenario runs `gradle help` on your build with a special flag to enable/disable the new API to make it easier to measure improvements. `gradle help` approximates the time it takes Gradle to configure your build by running only a single, simple task.

Save as help.scenario.

    defaults {
        tasks = ["help"]
        warm-ups = 20
    }
    eagerHelp = ${defaults} {
        gradle-args = ["-Dorg.gradle.internal.tasks.eager=true"]
    }
    lazyHelp = ${defaults} {
        gradle-args = ["-Dorg.gradle.internal.tasks.eager=false"]
    }

Run `gradle-profiler` in the root of your build. The results will go into a file called `profile-out-N` where `N` is unique for each invocation.

- When measuring your build with `gradle-profiler`, you should make sure the machine running the benchmark is not also busy doing other things. You may get false positives/negatives if resources are spent doing other things.

- Get a baseline for how long your build takes before making any changes. Run

    gradle-profiler --benchmark
        --iterations 20
        --gradle-version [some Gradle version]
        --scenario-file help.scenario
        eagerHelp lazyHelp

- In the `profile-out-N` directory, `gradle-profiler` will generate a CSV and a HTML file to display the results of the benchmarking.

- After making some changes to decrease the number of tasks that are created and configured, re-run the command above.

- For the Gradle build itself, we saw improvements after 50% of the tasks were no longer configured each time. Your mileage may vary depending on how expensive particular tasks are to create and configure.

- Please provide feedback on this issue. Or send us an email at performance@gradle.com.