Performance test reports
This topic includes a series of test reports focusing on the performance and scalability of Harness Self-Managed Enterprise Edition. The reports provide detailed insights into the test environment, resource configurations, test scenarios, and their outcomes.
The test reports offer valuable insights into the performance, scalability, and reliability of Harness Self-Managed Enterprise Edition. They demonstrate the platform's capability to handle concurrent CI/CD executions effectively across different scenarios and workload conditions. The detailed results and observations serve as a reference for evaluating the platform's suitability for various deployment environments and project requirements.
Test environment
The test reports document the environment in which the performance tests were conducted. This includes details such as the Kubernetes version used (GKE), the database configuration (Mongo Atlas M60), and the Redis setup (GCP Memory Store).
Harness services
The reports outline the various services utilized within the Harness Self-Managed Enterprise Edition, deployed using Helm charts. Each service's configuration, including the number of replicas and resource allocations per replica (CPU and memory), is provided for a comprehensive understanding of the setup.
Test scenarios
Concurrent CI executions
Concurrent CI executions are initiated with specific steps, including initializing Kubernetes pods, cloning repositories, and executing parallel steps with sleep intervals. These tests evaluate the platform's ability to handle high loads of CI executions efficiently.
Concurrent CD executions
Concurrent CD executions involve fetching Docker artifacts from external repositories and executing sequential steps, including Canary deploy, Canary delete, Rolling deploy, and Kubernetes Delete. These tests focus on the platform's ability to orchestrate complex deployment workflows under heavy loads.
Performance test reports
Harness publishes performance test reports with each release. Select a report below to view report details.
2024 reports
April 1, 2024
This document details information about the following:
- Test Environment and resource configuration
- Test scenario and results
Environment
- GKE (Kubernetes Version): 1.26.x
Database
- Mongo Atlas M60
Redis
- GCP Memory Store (11 GB)
Harness Services
| Service Name | Replicas | CPU (per replica) | Memory (per replica) | Version |
|---|---|---|---|---|
| access-control | 4 | 1 | 5 | harness-0.14.6 |
| ci-manager | 4 | 3 | 6 | harness-0.14.6 |
| pipeline-service | 7 | 4 | 10 | harness-0.14.6 |
| manager | 7 | 3 | 12 | harness-0.14.6 |
| log-service | 3 | 3 | 12 | harness-0.14.6 |
| ng-manager | 5 | 2 | 6 | harness-0.14.6 |
| scm | 2 | 0.5 | 1 | harness-0.14.6 |
| gateway | 5 | 1 | 4 | harness-0.14.6 |
| default-backend | 1 | 0.1 | 0.2 | harness-0.14.6 |
| nginx-ingress-controller | 1 | 5 | 10 | harness-0.14.6 |
| change-data-capture | 1 | 4 | 6 | harness-0.14.6 |
| next-gen-ui | 2 | 0.5 | 0.5 | harness-0.14.6 |
| ng-auth-ui | 2 | 0.1 | 0.1 | harness-0.14.6 |
| platform-service | 2 | 0.5 | 3 | harness-0.14.6 |
| template-service | 2 | 1 | 8 | harness-0.14.6 |
| ti-service | 2 | 1 | 6 | harness-0.14.6 |
| sto-core | 4 | 0.5 | 1.5 | harness-0.14.6 |
| sto-manager | 2 | 3 | 6 | harness-0.14.6 |
| gitops | 2 | 2 | 2 | harness-0.14.6 |
| ui | 3 | 0.1 | 0.5 | harness-0.14.6 |
| policy-mgmt | 3 | 0.3 | 1 | harness-0.14.6 |
| timescaledb | 2 | 1 | 2 | harness-0.14.6 |
| verification-svc | 2 | 0.3 | 4 | harness-0.14.6 |
| ng-dashboard-aggregator | 2 | 0.25 | 2 | harness-0.14.6 |
Override file
Test scenarios
> 2000 concurrent CI Executions INLINE
Each CI pipeline would:
- initialize a K8s pod and Git clone repo
- run 5 parallel steps (100 sec sleep)
- run template with 2 parallel steps (140sec sleep)
Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 15 (1cpu/4gi) Trigger type: webhook Test class: CI_PIPELINE_WEBHOOK_RUN
Result: PASS Avg Execution Time: 6.45min
> 2000 concurrent CD Executions INLINE
Each CD pipeline would:
- fetch docker artifact from AWS ECR repo
- run following steps in order:
- Canary deploy
- Canary delete
- Rolling deploy
- K8s Delete
Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 47 (1cpu/4gi) Test class: CD_PIPELINE_RUN
Result: PASS Avg Execution Time: 5.20min
February 29, 2024
This document details information about the following:
- Test Environment and resource configuration
- Test scenario and results
Environment
- GKE (Kubernetes Version): 1.26.x
Database
- Mongo Atlas M60
Redis
- GCP Memory Store (11 GB)
Harness services
Helm chart : https://github.com/harness/helm-charts/releases/tag/harness-0.13.4
| Service Name | Replicas | CPU (per replica) | Memory (per replica) | Version |
|---|---|---|---|---|
| access-control | 4 | 1 | 5 | harness-0.13.4 |
| ci-manager | 4 | 3 | 6 | harness-0.13.4 |
| pipeline-service | 7 | 4 | 10 | harness-0.13.4 |
| manager | 7 | 3 | 12 | harness-0.13.4 |
| log-service | 3 | 3 | 12 | harness-0.13.4 |
| ng-manager | 5 | 2 | 6 | harness-0.13.4 |
| scm | 2 | 0.5 | 1 | harness-0.13.4 |
| gateway | 5 | 1 | 4 | harness-0.13.4 |
| default-backend | 1 | 0.1 | 0.2 | harness-0.13.4 |
| nginx-ingress-controller | 1 | 5 | 10 | harness-0.13.4 |
| change-data-capture | 1 | 4 | 6 | harness-0.13.4 |
| next-gen-ui | 2 | 0.5 | 0.5 | harness-0.13.4 |
| ng-auth-ui | 2 | 0.1 | 0.1 | harness-0.13.4 |
| platform-service | 2 | 0.5 | 3 | harness-0.13.4 |
| template-service | 2 | 1 | 8 | harness-0.13.4 |
| ti-service | 2 | 1 | 6 | harness-0.13.4 |
| sto-core | 4 | 0.5 | 1.5 | harness-0.13.4 |
| sto-manager | 2 | 3 | 6 | harness-0.13.4 |
| gitops | 2 | 2 | 2 | harness-0.13.4 |
| ui | 3 | 0.1 | 0.5 | harness-0.13.4 |
| policy-mgmt | 3 | 0.3 | 1 | harness-0.13.4 |
| timescaledb | 2 | 1 | 2 | harness-0.13.4 |
| verification-svc | 2 | 0.3 | 4 | harness-0.13.4 |
| ng-dashboard-aggregator | 2 | 0.25 | 2 | harness-0.13.4 |
Override file
Test scenarios
> 2000 concurrent CI Executions INLINE
Each CI pipeline would:
- initialize a k8s pod and git clone repo
- run 5 parallel steps (100 sec sleep)
- run template with 2 parallel steps (140sec sleep)
Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 15 (1cpu/4gi) Trigger type: webhook Test class: CI_PIPELINE_WEBHOOK_RUN
Result : PASS Avg Execution Time: 6.5min
> 1500 concurrent CD Executions INLINE
Each CD pipeline would
- fetch docker artifact from AWS ECR repo
- run following steps in order:
- Canary deploy
- Canary delete
- Rolling deploy
- K8s Delete
Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 40 (1cpu/4gi) Test class: CD_PIPELINE_RUN
Result: PASS Avg Execution Time: 5.1min
January 31, 2024
This document details information about the following:
- Test Environment and resource configuration
- Test scenario and results
Environment
- GKE (Kubernetes Version): 1.25.x
Database
- Mongo Atlas M60
Redis
- GCP Memory Store (11 GB)
Harness services
Helm chart : https://github.com/harness/helm-charts/releases/tag/harness-0.13.0
| Service Name | Replicas | CPU (per replica) | Memory (per replica) | Version |
|---|---|---|---|---|
| access-control | 4 | 1 | 5 | harness-0.13.0 |
| ci-manager | 4 | 3 | 6 | harness-0.13.0 |
| pipeline-service | 7 | 4 | 10 | harness-0.13.0 |
| manager | 7 | 3 | 12 | harness-0.13.0 |
| log-service | 3 | 3 | 12 | harness-0.13.0 |
| ng-manager | 5 | 2 | 6 | harness-0.13.0 |
| scm | 2 | 0.5 | 1 | harness-0.13.0 |
| gateway | 5 | 1 | 4 | harness-0.13.0 |
| default-backend | 1 | 0.1 | 0.2 | harness-0.13.0 |
| nginx-ingress-controller | 1 | 5 | 10 | harness-0.13.0 |
| change-data-capture | 1 | 4 | 6 | harness-0.13.0 |
| next-gen-ui | 2 | 0.5 | 0.5 | harness-0.13.0 |
| ng-auth-ui | 2 | 0.1 | 0.1 | harness-0.13.0 |
| platform-service | 2 | 0.5 | 3 | harness-0.13.0 |
| template-service | 2 | 1 | 8 | harness-0.13.0 |
| ti-service | 2 | 1 | 6 | harness-0.13.0 |
| sto-core | 4 | 0.5 | 1.5 | harness-0.13.0 |
| sto-manager | 2 | 3 | 6 | harness-0.13.0 |
| gitops | 2 | 2 | 2 | harness-0.13.0 |
| ui | 3 | 0.1 | 0.5 | harness-0.13.0 |
| policy-mgmt | 3 | 0.3 | 1 | harness-0.13.0 |
| timescaledb | 2 | 1 | 2 | harness-0.13.0 |
| verification-svc | 2 | 0.3 | 4 | harness-0.13.0 |
| ng-dashboard-aggregator | 2 | 0.25 | 2 | harness-0.13.0 |
Override file
Test scenarios
> 2000 concurrent CI Executions INLINE
Each CI pipeline would
- initialize a k8s pod and Git clone repo
- run 5 parallel steps (100 sec sleep)
- run template with 2 parallel steps (140sec sleep)
Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 15 (1cpu/4gi) Trigger type: webhook Test class: CI_PIPELINE_WEBHOOK_RUN
Result : PASS Avg Execution Time: 6.5min
> 1500 concurrent CI Executions GitX
Each CI pipeline would
- initialize a k8s pod and git clone repo
- run 5 parallel steps (360 sec sleep) and echo statements
Projects : 1 Pipelines : 1500 Stages per pipeline : 1 Delegates : 15 (1cpu/2gi) Trigger type : webhook Test class : CI_PIPELINE_REMOTE_RUN
Result : PASS Avg Execution Time: 8.5min
> 1000 concurrent CD Executions INLINE
Each CD pipeline would:
- fetch docker artifact from AWS ECR repo
- run following steps in order:
- Canary deploy
- Canary delete
- Rolling deploy
- K8s Delete
Projects: 1 Pipelines: 1000 Stages per pipeline: 1 Delegates: 26 (1cpu/4gi) Test class: CD_PIPELINE_RUN
Result: PASS Avg Execution Time: 4.5min
2023 reports
December 28, 2023
This document details information about the following:
- Test Environment and resource configuration
- Test scenario and results
Environment
- GKE (Kubernetes Version): 1.25.x
Database
- Mongo Atlas M60
Redis
- GCP Memory Store (5GB)
Harness services
| Service Name | Replicas | CPU (per replica) | Memory (per replica) | Version |
|---|---|---|---|---|
| access-control | 5 | 1 | 5 | harness-0.11.2 |
| ci-manager | 6 | 3 | 6 | harness-0.11.2 |
| pipeline-service | 10 | 4 | 12 | harness-0.11.2 |
| manager | 6 | 3 | 12 | harness-0.11.2 |
| log-service | 3 | 3 | 12 | harness-0.11.2 |
| ng-manager | 4 | 3 | 8 | harness-0.11.2 |
| scm | 2 | 0.5 | 1 | harness-0.11.2 |
| gateway | 2 | 2 | 6 | harness-0.11.2 |
| default-backend | 1 | 0.1 | 0.2 | harness-0.11.2 |
| nginx-ingress-controller | 2 | 5 | 10 | harness-0.11.2 |
| change-data-capture | 1 | 4 | 5 | harness-0.11.2 |
| next-gen-ui | 2 | 0.5 | 0.5 | harness-0.11.2 |
| ng-auth-ui | 2 | 0.1 | 0.1 | harness-0.11.2 |
| platform-service | 2 | 0.5 | 3 | harness-0.11.2 |
| template-service | 2 | 1 | 8 | harness-0.11.2 |
| ti-service | 1 | 3 | 6 | harness-0.11.2 |
| sto-core | 1 | 0.5 | 0.75 | harness-0.11.2 |
| sto-manager | 1 | 3 | 6 | harness-0.11.2 |
| gitops | 1 | 2 | 2 | harness-0.11.2 |
| ui | 1 | 0.5 | 0.5 | harness-0.11.2 |
| policy-mgmt | 1 | 0.5 | 0.5 | harness-0.11.2 |
| timescaledb | 2 | 1 | 2 | harness-0.11.2 |
Override file
Manager config : update LOG_STREAMING_SERVICE_EXTERNAL_URL = <smp host url>/log-service/
Test scenarios
> 1800 concurrent CI Executions INLINE
Each CI pipeline would:
- initialize a K8s pod and Git clone repo
- run 5 parallel steps (100 sec sleep)
- run template with 2 parallel steps (140sec sleep)
Projects: 1 Pipelines: 1800 Stages per pipeline: 1 Delegates: 15 (1cpu/2gi) trigger: webhook
Result: PASS Total Execution Time: 7.2min
> 1500 concurrent CI Executions GitX
Each CI pipeline would
- initialize a K8s pod and Git clone repo
- run 5 parallel steps (360 sec sleep) and echo statements
Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 15 (1cpu/2gi) trigger: webhook
Result: PASS Total Execution Time: 10.3min
> 1000 concurrent CD Executions INLINE
Each CD pipeline would
- fetch Docker artifact from AWS ECR repo
- run following steps in order:
- Canary deploy
- Canary delete
- Rolling deploy
- K8s Delete
Projects: 1 Pipelines: 1000 Stages per pipeline: 1 Delegates: 27 (1cpu/4gi)
Result: PASS Total Execution Time: 4.5min
October 27, 2023
This document details information about the following:
- Test environment and resource configuration
- Test scenario and results
Environment
- GKE (Kubernetes Version): 1.26.x
Database
- Mongo Atlas M60
Redis
- GCP Memory Store (5GB)
Harness services
| Service Name | Replicas | CPU (per replica) | Memory (per replica) | Version |
|---|---|---|---|---|
| access-control | 5 | 1 | 5 | harness-0.9.2 |
| ci-manager | 6 | 3 | 6 | harness-0.9.2 |
| pipeline-service | 10 | 4 | 12 | harness-0.9.2 + fixes (to be released in 0.11) |
| manager | 6 | 3 | 12 | harness-0.9.2 + fixes (to be released in 0.10) |
| log-service | 1 | 10 | 24 | harness-0.9.2 |
| ng-manager | 4 | 3 | 8 | harness-0.9.2 + fixes (to be released in 0.10) |
| scm | 2 | 0.5 | 1 | harness-0.9.2 |
| gateway | 2 | 2 | 6 | harness-0.9.2 |
| default-backend | 1 | 0.1 | 0.2 | harness-0.9.2 |
| nginx-ingress-controller | 2 | 5 | 10 | harness-0.9.2 |
| change-data-capture | 1 | 4 | 5 | harness-0.9.2 |
| next-gen-ui | 2 | 0.5 | 0.5 | harness-0.9.2 |
| ng-auth-ui | 2 | 0.1 | 0.1 | harness-0.9.2 |
| platform-service | 2 | 2.5 | 3 | harness-0.9.2 |
| template-service | 2 | 1 | 8 | harness-0.9.2 + fixes (to be released in 0.11) |
| ti-service | 1 | 3 | 6 | harness-0.9.2 |
| sto-core | 1 | 0.5 | 0.75 | harness-0.9.2 |
| sto-manager | 1 | 3 | 6 | harness-0.9.2 |
| gitops | 1 | 2 | 2 | harness-0.9.2 |
| ui | 1 | 0.5 | 0.5 | harness-0.9.2 |
| policy-mgmt | 1 | 0.5 | 0.5 | harness-0.9.2 |
| timescaledb | 2 | 1 | 2 | harness-0.9.2 |
Override file
Manager config
update LOG_STREAMING_SERVICE_EXTERNAL_URL = <smp host url>/log-service/
Test scenarios
> 1500 concurrent CI executions INLINE
Each CI pipeline would:
- initialize a K8s pod and Git clone repo
- run 5 parallel steps (70 sec sleep)
- run template with 2 parallel steps (140sec sleep)
Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 10 (1cpu/2gi)
Result: PASS Total Execution Time: 6min
> 1500 concurrent CI executions GitX
Each CI pipeline would:
- initialize a K8s pod and Git clone repo
- run 5 parallel steps (140 sec sleep) and echo statements
Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 10 (1cpu/2gi)
Result: PASS Total Execution Time: 5min
> 500 concurrent CD executions INLINE
Each CD pipeline would:
- fetch Docker artifact from AWS ECR repo
- run the following steps in order:
- Canary deploy
- Canary delete
- Rolling deploy
- K8s Delete
Projects: 1 Pipelines: 500 Stages per pipeline: 1 Delegates: 18 (1cpu/4gi)
Result: PASS Total Execution Time: 4.5min
October 6, 2023
This document details information about the following:
- Test environment and resource configuration
- Test scenario and results
Environment
- GKE (Kubernetes Version): 1.26.x
Database
- Mongo Atlas M60
Redis
- GCP Memory Store (5GB)
Harness services
| Service Name | Replicas | CPU (per replica) | Memory (per replica) | Version |
|---|---|---|---|---|
| access-control | 5 | 1 | 5 | harness-0.8.2 |
| ci-manager | 6 | 3 | 6 | harness-0.8.2 |
| pipeline-service | 10 | 4 | 12 | harness-0.8.2 + fixes (to be released in 0.11) |
| manager | 6 | 3 | 12 | harness-0.8.2 |
| log-service | 1 | 10 | 24 | harness-0.8.2 |
| ng-manager | 4 | 3 | 8 | harness-0.8.2 |
| scm | 2 | 0.5 | 1 | harness-0.8.2 |
| gateway | 2 | 2 | 6 | harness-0.8.2 |
| default-backend | 1 | 0.1 | 0.2 | harness-0.8.2 |
| nginx-ingress-controller | 2 | 5 | 10 | harness-0.8.2 |
| change-data-capture | 1 | 4 | 5 | harness-0.8.2 |
| next-gen-ui | 2 | 0.5 | 0.5 | harness-0.8.2 |
| ng-auth-ui | 2 | 0.1 | 0.1 | harness-0.8.2 |
| platform-service | 2 | 2.5 | 3 | harness-0.8.2 |
| template-service | 2 | 1 | 8 | harness-0.8.2 + fixes (to be released in 0.11) |
| ti-service | 1 | 3 | 6 | harness-0.8.2 |
| sto-core | 1 | 0.5 | 0.75 | harness-0.8.2 |
| sto-manager | 1 | 3 | 6 | harness-0.8.2 |
| gitops | 1 | 2 | 2 | harness-0.8.2 |
| ui | 1 | 0.5 | 0.5 | harness-0.8.2 |
| policy-mgmt | 1 | 0.5 | 0.5 | harness-0.8.2 |
| timescaledb | 2 | 1 | 2 | harness-0.8.2 |
Override file
Test scenarios
> 1500 concurrent CI executions INLINE
Each CI pipeline would:
- initialize a K8s pod and Git clone repo
- run 4 parallel steps (70 sec sleep)
- run template with 2 parallel steps (140sec sleep)
Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 10 (1cpu/2gi)
Result: PASS Total Execution Time: 6min
> 1500 concurrent CI executions GitX
Each CI pipeline would
- initialize a K8s pod and Git clone repo
- run 5 parallel steps (140 sec sleep) and echo statements
Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 10 (1cpu/2gi)
Result: PASS Total Execution Time: 4.2min
> 500 concurrent CD executions INLINE
Each CD pipeline would:
- fetch Docker artifact from AWS ECR repo
- run the following steps in order:
- Canary deploy
- Canary delete
- Rolling deploy
- K8s Delete
Projects: 1 Pipelines: 500 Stages per pipeline: 1 Delegates: 18 (1cpu/4gi)
Result: PASS Total Execution Time: 4.2min