> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/harness-cloud-operations/computing-uptime.md). # Compute Uptime for Harness Modules This operational reference lists the service-level indicators (SLIs) for Harness modules. Harness calculates each service-level objective (SLO) from these user-centric SLIs. ### Weighting factor Harness operations apply a weighting factor to the SLIs post any incidents. * Major outage = 100% of the downtime hit * Partial = 30% of the downtime hit * Degraded performance = None. A degraded performance affects the user experience, but it is not technically downtime. A production incident is an unexpected event in a live production environment. It causes a complete or partial service disruption. A partial incident makes one or more module functions unavailable. Production incidents appear on the [Harness status page](https://status.harness.io), where you can subscribe to updates. ### Recent incidents and availability calculations **Oct 4th - Impacted Continuous Integration Enterprise(CIE) - Self Hosted Runners** Incident: Issue with sending Git Status for PR URL: * Component: Platform/Delegate * SLI: API Error Rate * Availability - Partial Outage of 28 Minutes * Threshold: More than 1% over 5 min rolling window * SLA Impact - Partial Outage of 28 Minutes/1680 seconds, taking 30% of the downtime hit comes to 504 seconds. During the incident, the error rate for the Platform/Delegate component exceeded 1% over a 5-minute rolling window due to a missing dependency and we had a partial outage for CIE - Self Hosted Runners in Prod-2. **Oct 16th - Impacted all the components in Prod-2** Incident: Failed to retrieve license information seen for some customers URL: * Component: Platform/Login * SLI: API Error Rate * Availability - Partial Outage of 8 Minutes * Threshold: More than 1% over 5 min rolling window * SLA Impact - Partial Outage of 8 Minutes/480 seconds, taking 30% of the downtime hit comes to 144 seconds. During the incident, the error rate for the Platform/Login component exceeded 1% over a 5-minute rolling window and we had a partial outage across all of our components in Prod-2. ### Service-level indicators for Harness modules #### Pipelines A pipeline is a core Harness Platform construct. These SLIs apply to Continuous Delivery (CD), Continuous Integration (CI), Security Test Orchestration (STO), and other modules that use pipelines. | **Component** | **SLI** | **Threshold** | **Availability** | | ------------------------------------------------------ | --------------------- | --------------------------------------------------- | -------------------- | | Pipeline/Triggers/Input Sets | APIs Error rate | More than 1% over 5 min rolling window | Major Outage | | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded Performance | | Pipeline Executions failure caused by Harness platform | Failure rate Increase | More than 50% over 5 min rolling window | Major Outage | | | Failure rate Increase | More than 1% over 5 min rolling window | Partial Outage | | | Slow Executions | 2x of average latency in a rolling window of 5 mins | Degraded Performance | | Triggers | Trigger Activations | More than 1% over 5 min rolling window | Degraded Performance | #### Platform Core platform constructs and services support Harness modules. A breach of these SLIs affects all Harness modules. | **Component** | **SLI** | **Threshold** | **Availability** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------- | | Access Control | Permissions Change Processing Time | New permissions (additions/removals) should take effect within 5 minutes | Degraded Performance | | Platform resources (All APIs) - Account, Login, Project/Org, Connectors, Secrets, Delegate, Settings, Notifications, Audits, Templates, Services, Environments , Policies, File Store, Log Uploads | API Error rate | More than 1% over 5 min rolling window | Partial Outage | | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded Performance | | Notifications | Notification Delivery Latency | 99% of notifications are dispatched within 1 minute from the moment they are sent to the notification service | Degraded Performance | #### CD and GitOps (NextGen) All the Pipeline and Platform SLIs are applicable here. | **Component** | **SLI** | **Threshold** | **Availability** | | ------------- | ----------------- | ----------------------------------------------- | -------------------- | | GitOps | APIs Error rate | More than 1% over 5 min rolling window | Partial Outage | | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded Performance | #### CI Test Intelligence All the Pipeline and Platform SLIs are applicable here. | **SLI** | **Threshold** | **Availability** | | ----------------- | ----------------------------------------------- | -------------------- | | APIs Error rate | More than 1% over 5 min rolling window | Degraded performance | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded Performance | #### Feature Flags All the Platform SLIs are applicable here. Pipeline relevant if the FF use case is tied to a pipeline. | **SLI** | **Threshold** | **Availability** | | -------------------------------- | -------------------------------------------------------------------------------- | -------------------- | | Admin UI response time | 95th percentile: > 30s over a 5 minute rolling window | Degraded Performance | | Admin UI error rate | 5% of requests over 5 min rolling window fails to respond or returns a 5xx error | Partial Outage | | Authentication response time | 95th percentile: > 30s over a 5 minute rolling window | Degraded Performance | | Authentication error rate | 5% of requests over 5 min rolling window fails to respond or returns a 5xx error | Major Outage | | SDK evaluation response time | 95th percentile: > 30s over a 5 minute rolling window | Degraded Performance | | SDK evaluation error rate | 5% of requests over 5 min rolling window fails to respond or returns a 5xx error | Major Outage | | SDK metrics response time | 95th percentile: > 30s over a 5 minute rolling window | Degraded Performance | | SDK metrics error rate | 5% of requests over 5 min rolling window fails to respond or returns a 5xx error | Partial Outage | | SDK events request response time | 95th percentile: > 30s over a 5 minute rolling window | Degraded Performance | | SDK events request error rate | 5% of requests over 5 min rolling window fails to respond or returns a 5xx error | Major Outage | #### Dashboards | **SLI** | **Threshold** | **Availability** | | ----------------------------- | --------------------------------------------------- | -------------------- | | Dashboards not Loading | For a duration of 60 secs | Major Outage | | Latency in Loading dashboards | 2x of average latency in a rolling window of 5 mins | Degraded performance | | CRUD/Actions not working | For a duration of 60 secs | Partial Outage | #### Cloud Cost Management All the Pipeline and Platform SLIs are applicable here. | **SLI** | **Threshold** | **Availability** | | ----------------------------------- | ---------------------------------------------------------- | -------------------- | | APIs Error rate | More than 1% over 5 min rolling window | Major Outage | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded performance | | CCM UI is down (ping failure) | For a consecutive duration of 30secs | Major Outage | | Perspective load times | Greater than 2 mins for a consecutive duration of 10 mins | Partial Outage | | Max AutoStopping rule warmup time | Greater than 10 mins for a consecutive duration of 30 mins | Partial Outage | | Max asset gov policy evaluation | Greater than 15 mins for a consecutive duration of 30 mins | Partial Outage | | Cloud provider data ingestion delay | Greater than 48hrs of no data received | Partial Outage | | K8s data at hourly granularity | No events received for more than 6 hrs | Partial Outage | | K8s data at daily granularity | No events received for more than 48 hrs | Partial Outage | #### Chaos Engineering All the Platform SLIs are applicable here. Pipeline relevant if the chaos use case is tied to a pipeline. | **SLI** | **Threshold** | **Availability** | | ----------------------------------- | -------------------------------------------------------------------------------------------------------- | -------------------- | | APIs Error rate | More than 1% over 5 min rolling window | Major Outage | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded performance | | Load times on UI | Data load time > 10s consecutively over a 5 min period | Degraded performance | | ChaosGuard Rule Evaluation Duration | The ChaosGuard rule evaluation stage takes >10s consecutively over a 5 min period across experiment runs | Degraded performance | #### Service Reliability Management All the Platform SLIs are applicable here. | **Component** | **SLI** | **Threshold** | **Availability** | | ------------------------------ | --------------- | -------------------------------------- | ---------------- | | SLO Creation API | APIs Error rate | More than 5% over 5 min rolling window | Major outage | | | | More than 1% over 5 min rolling window | Partial outage | | SLO Update API | APIs Error rate | More than 5% over 5 min rolling window | Major outage | | | | More than 1% over 5 min rolling window | Partial outage | | SLO List API | APIs Error rate | More than 5% over 5 min rolling window | Major outage | | | | More than 1% over 5 min rolling window | Partial outage | | Monitored service creation API | APIs Error rate | More than 5% over 5 min rolling window | Major outage | | | | More than 1% over 5 min rolling window | Partial outage | | Monitored Service update API | APIs Error rate | More than 5% over 5 min rolling window | Major outage | | | | More than 1% over 5 min rolling window | Partial outage | | Monitored Service List API | APIs Error rate | More than 5% over 5 min rolling window | Major outage | | | | More than 1% over 5 min rolling window | Partial outage | #### Security Test Orchestration All the Platform SLIs are applicable here. Pipeline relevant if the STO use case is tied to a pipeline. | **Component** | **SLI** | **Threshold** | **Availability** | | -------------- | -------------------------------- | ------------------------------------------------------------------------------- | ----------------------- | | STO APIs | 4xx Error Rate | More than 5% over 5 min rolling window | Possible Partial Outage | | STO APIs | 5xx Error Rate | More than 1% over 5 min rolling window | Partial Outage | | STO APIs | 5xx Error Rate | More than 5% over 5 min rolling window | Major Outage | | STO APIs | Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded performance | | Pipeline Steps | Security Step Execution Failures | 25% increase in security stage execution failures in a rolling window of 5 mins | Partial Outage | #### Continuous Error Tracking All the Platform SLIs are applicable here. | **SLI** | **Threshold** | **Availability** | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------- | | APIs Error rate | More than 1% over 5 min rolling window | Major Outage | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded performance | | Agent cannot connect to CET collector | For a consecutive duration of 60 secs | Major outage | | Agent not being shown as connected in the UI | For a consecutive duration of 60 secs | Partial outage | | | Latency greater than 30 seconds for a consecutive duration of 10 mins (95th percentile) | Degraded performance | | UI is down | For a consecutive duration of 30 secs | Major outage | | ARC screen is down | No hit is openable | Major outage | | | Some hits aren’t openable - at least 20% of the hits in a total of at least 20 unique events | Degraded performance | | Tiny links not working | Tiny link doesn’t direct to a viewable ARC screen | Partial outage | | | Tiny link should be clickable after no more than 90s after it was logged | Degraded performance | | New events/Metrics don’t show up on the summary or event list | For a consecutive duration of 180 secs | Major outage | | | Latency greater than 125 seconds in metrics since happened in the agent until shown in the UI | Degraded performance | | Notifications | Expected notification doesn’t arrive for a consecutive duration of 60 secs after the ETA | Major outage | | | Latency greater than 30 seconds | Degraded performance | | | Links in notifications don’t work | Degraded performance | | Admin operations not working (Including: Tokens, Critical events, hide & resolve events, Jira integration, Notifications, Saved Search) | For a consecutive duration of 30 secs | Major outage | #### Internal Developer Portal All the Platform SLIs are applicable here. | **SLI** | **Threshold** | **Availability** | | --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------- | | IDP UI is down(Included: Catalog, Self service Hub, Scorecards Excluded: Non-Harness owned plugins) | For a consecutive duration of 30secs | Major Outage | | IDP admin UI is down | For a consecutive duration of 30secs | Partial Outage | | Unable to access Service Catalog APIs | 5XX Errors for a consecutive duration of 30secs (95th percentile) | Major outage | | | Latency greater than 30 seconds for a consecutive duration of 10 mins (95th percentile) | Partial outage | | Scorecards not functional | 5XX Errors for a consecutive duration of 30secs (95th percentile) | Partial outage | | | Latency greater than 60 seconds for a consecutive duration of 10 mins (95th percentile) | Degraded Performance | | Issue with IDP admin operations | 5XX Errors for a consecutive duration of 30secs (95th percentile) | Partial outage | | | Latency greater than 10 seconds for a consecutive duration of 10 mins (95th percentile) | Degraded Performance | | Open Source Plugins functionality | 5XX Errors for a consecutive duration of 30secs (95th percentile) | Degraded Performance | | | Latency greater than 30 seconds for a consecutive duration of 10 mins (95th percentile) | Degraded Performance | #### Code Repository All the Platform SLIs are applicable here. | **SLI** | **Threshold** | **Availability** | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------- | -------------------- | | Git Operations success rate(Clone, Pull, Push and associated operations like Merge, Blame ) | > 99.9% over a rolling 5 min window | Major Outage | | Git Operations execution time(Clone, Pull, Push and associated operations like Merge, Blame ) | 2X increase of time for git operations | Degraded Performance | | CODE Reviews Error Rate Increase | 5% increase in 5xx errors in a rolling window of 5 mins | Partial Outage | | CODE Reviews Latency Increase | 2x of average latency in a rolling window of 5 mins | Degraded Performance | | PR Checks & Webhooks - Error Rate Increase is PR Checks | 5% increase in 5xx errors in a rolling window of 5 mins | Degraded Performance | | PR Checks & Webhooks - Webhooks are not triggered | 5% increase in 5xx errors in a rolling window of 5 mins | Degraded Performance | | UI unable to render page | For a consecutive duration of 2 min | Major Outage | #### Infrastructure as Code Management | **SLI** | **Threshold** | **Availability** | | --------------------------------------------- | ---------------------------------------------------------------- | -------------------- | | APIs Error rate | More than 1% over 5 min rolling window | Major Outage | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded performance | | Unable to run IaC Stage & Steps in a Pipeline | API’s are down for a consecutive duration of 60 seconds | Major Outage | | | 10% of traffic generates 5xx error in a rolling window of 5 mins | Partial Outage | | | 2x of average latency in a rolling window of 5 mins | Degraded Performance | #### Supply Chain Security All the Platform and Pipeline SLIs are applicable here. | **SLI** | **Threshold** | **Availability** | | ----------------- | ----------------------------------------------- | -------------------- | | APIs Error rate | More than 1% over 5 min rolling window | Major Outage | | API Response Time | 95th percentile: > 1s over 5 min rolling window | Degraded performance | #### Software Engineering Insights | **SLI** | **Threshold** | **Availability** | | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | APIs Error rate | More than 5% over 5 min rolling window | Major outage | | API Response Time | 95th percentile: > 2s over 5 min rolling window | Degraded performance | | Ingestion & data processing delay |

Delay of more than 72 hours for the latest data to appear on the dashboard.
This threshold excludes delays caused by pending customer actions. In the event of failures, processing historical data may take additional time depending on the volume of data that needs to be backfilled.

| Degraded performance |