Optimize and enhance CI pipelines
You can optimize your CI pipelines to make them faster, more efficient, and more versatile.
Background services, caches, and shared volumes
For information about service dependencies, caches, and shared volumes, go to:
Optimize test times
For information about optimizing test times, go to Run tests in CI pipelines.
Optimize Docker images
The following practices can reduce your build times.
Pre-build images that include all required dependencies
If most of the build time is spent downloading dependencies, you should pre-build an image with all required dependencies in a separate pipeline. Then, set up a periodic pipeline that builds the image with all the latest dependencies and pushes it to your Docker registry. Use this image in all of your build pipelines.
Pre-building images with all required dependencies is more efficient than downloading them to a baseline image as part of the Build setup. This is especially true if you update your images often to ensure that they include all the latest updates.
Exclude unnecessary files and packages from your images
In addition to reducing build times, excluding unnecessary files and packages makes the resulting images smaller, simpler, and more portable. You can use dockerignore files to exclude unnecessary files and folders from your images.
Optimize Dockerfiles and enable Docker layer caching
- Sort multi-line arguments in your Dockerfile alphabetically. This makes it easier to update and avoid duplicate packages.
- Review Docker's best practices for writing Dockerfiles.
- Enable Docker layer caching.
Group, loop, and reuse
These strategies help you streamline complex pipelines and reuse pipelines and pipeline components.
You can use step groups to organize complex or busy stages and improve build times.
Looping strategies enable you to run a stage or step multiple times with different inputs. This eliminates the need to copy the same stage or step for each variation you need. It also makes the pipeline more organized, clean, and easy to maintain. Looping strategies enable use cases such as:
- You want to test a UI feature in multiple browsers and platforms. You can define a matrix that specifies the browsers and platforms to test.
- You want to build artifacts for multiple JDK versions in the same Build Stage.
- You have a Build Pipeline with 20 unit tests. To speed up execution, you want to run the tests in parallel across 4 jobs that run 5 tests each.
Expressions and runtime inputs make your pipelines more dynamic.
- Variables and expressions
- Fixed values, runtime inputs, and expressions
- Built-in and custom Harness variables reference
- Built-in CI codebase variables reference
- Secrets management
- Input sets and overlays
Use templates to share and utilize pre-build pipelines, stages, and steps.
Integrate and automate
Use these strategies to automate and manage build sequences.
- Queue Intelligence
- Pipeline chaining
- Run specific stages in a pipeline
- Define failure strategies
- Automated triggers
There are many ways you can incorporate third party tools and services in your CI pipelines, such as GitHub Actions, Slack notifications, and Jira issue updates.
- Explore plugins
- Bring-your-own Secrets Manager
- Use a GitHub App in a GitHub connector
- Send notifications
Increase step resources
Check your infrastructure monitoring tools for potential bottlenecks during the time windows when your builds are running. Increasing memory or CPU capacity in your Build steps might help speed up your builds.
You can also enforce pipeline concurrency limits.