> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/cloud-cost-management/cost-optimization/recommendations/1-home-recommendations/kubernetes-1-home-recommendations.md).

# Kubernetes

View and act on Kubernetes cost recommendations including node pool rightsizing and workload optimization suggestions

{% @harness-package-selector/package-selector platforms="%5B%7B%22label%22%3A%22Kubernetes%22%2C%22slug%22%3A%22kubernetes%22%2C%22path%22%3A%22cloud-cost-management%2Fcost-optimization%2Frecommendations%2F1-home-recommendations%2Fkubernetes-1-home-recommendations%22%7D%2C%7B%22label%22%3A%22AWS%22%2C%22slug%22%3A%22aws%22%2C%22path%22%3A%22cloud-cost-management%2Fcost-optimization%2Frecommendations%2F1-home-recommendations%2Faws-1-home-recommendations%22%7D%2C%7B%22label%22%3A%22GCP%22%2C%22slug%22%3A%22gcp%22%2C%22path%22%3A%22cloud-cost-management%2Fcost-optimization%2Frecommendations%2F1-home-recommendations%2Fgcp-1-home-recommendations%22%7D%2C%7B%22label%22%3A%22Azure%22%2C%22slug%22%3A%22azure%22%2C%22path%22%3A%22cloud-cost-management%2Fcost-optimization%2Frecommendations%2F1-home-recommendations%2Fazure-1-home-recommendations%22%7D%5D" selectedPlatform="kubernetes" %}

## Kubernetes recommendations

{% tabs %}
{% tab title="Nodepool Recommendations" %}
{% hint style="warning" %}

### Autoscaled node pools

Harness skips node pool recommendations for a cluster that has a running pod whose workload name contains `cluster-autoscaler`. The AKS autoscaler runs in the control plane and usually has no such pod, so most autoscaled AKS node pools still get recommendations.

The recommendations should be treated as a nudge, not a source of accurate cost data, in such cases. The cost on an autoscaled pool can differ from the cloud bill. To hide these recommendations, contact [Harness Support](mailto:support@harness.io).
{% endhint %}

## Before you begin

Harness CACM uses labels to process node pool recommendations. Make sure to add one of the labels listed to the Kubernetes nodes in your cluster. The following table shows the labels for the respective cloud providers:

| **Cloud provider**          | **Labels**                                                                                                                                                                                                                           |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Amazon Web Services (AWS)   | <ul><li><code>eks.amazonaws.com/nodegroup​</code></li><li><code>alpha.eksctl.io/nodegroup-name​</code></li><li><code>node-pool-name​</code></li><li><code>kops.k8s.io/instancegroup</code></li><li><code>nodegroup​</code></li></ul> |
| Google Cloud Platform (GCP) | <ul><li><code>cloud.google.com/gke-nodepool</code></li><li><code>nodegroup​</code></li><li><code>kops.k8s.io/instancegroup</code></li></ul>                                                                                          |
| Microsoft Azure             | <ul><li><code>Agentpool​</code></li><li><code>node-pool-name​</code></li><li><code>nodegroup​</code></li><li><code>kops.k8s.io/instancegroup</code></li></ul>                                                                        |

## Types of nodepool recommendations

* **Node Pool Rightsizing**: Node Pool Rightsizing is a strategy that analyzes your Kubernetes infrastructure to identify opportunities for more efficient resource allocation.

## How are nodepool recommendations computed?

The node pool recommendations are computed by analyzing historical utilization data and requests metrics of Pods. CACM recommends the optimal resource configurations for the Spot and On-demand instances. It uses the following parameters to determine the maximum node counts:

* Total CPUs
* Total memory
* Max CPUs
* Max Memory

The recommendations are calculated by aggregating the resource utilization and request across all pods running across nodes in the node pool. You can select the number of days to compute the recommended size from the utilization data. The available options are last 7, 30, 60, and 90 days. The range does not change the current node count.

## Recommendation calculation formula

```
Nodepool Recommendation = Max at a given instant [Aggregated Max(Resource Utilization, Resource Requests)]
```

## Detailed example

To illustrate how nodepool recommendations are computed, let us walk through a practical example with two nodes in a nodepool, each running two pods. We will analyze CPU utilization and requests at two different time points.

**Time Instant 1**

| Node       | Pod   | CPU Utilization | CPU Request |
| ---------- | ----- | --------------- | ----------- |
| **Node 1** | Pod 1 | 2.1 vCPU        | 4.0 vCPU    |
|            | Pod 2 | 1.4 vCPU        | 1.0 vCPU    |
| **Node 2** | Pod 1 | 1.1 vCPU        | 4.0 vCPU    |
|            | Pod 2 | 0.9 vCPU        | 1.0 vCPU    |

**Time Instant 2**

| Node       | Pod   | CPU Utilization | CPU Request |
| ---------- | ----- | --------------- | ----------- |
| **Node 1** | Pod 1 | 2.2 vCPU        | 4.0 vCPU    |
|            | Pod 2 | 1.5 vCPU        | 1.0 vCPU    |
| **Node 2** | Pod 1 | 2.0 vCPU        | 4.0 vCPU    |
|            | Pod 2 | 0.3 vCPU        | 1.0 vCPU    |

## Calculation process

For each pod, we take the maximum of its utilization and request, then sum these values for all pods at each time instant:

**Time Instant 1:**

```
Total = Max(2.1, 4.0) + Max(1.4, 1.0) + Max(1.1, 4.0) + Max(0.9, 1.0)
Total = 4.0 + 1.4 + 4.0 + 1.0 = 10.4 vCPU
```

**Time Instant 2:**

```
Total = Max(2.2, 4.0) + Max(1.5, 1.0) + Max(2.0, 4.0) + Max(0.3, 1.0)
Total = 4.0 + 1.5 + 4.0 + 1.0 = 10.5 vCPU
```

## Final recommendation

The final CPU recommendation is the maximum value across all time instants:

```
Recommended CPU = Max(10.4, 10.5) = 10.5 vCPU
```

## Current node count

**Current node count** is the highest number of nodes the pool had in the last 7 days.

Select 7, 30, 60, or 90 days to change the utilization charts. The current node count stays the 7-day number, including in the comparison table.

{% hint style="warning" %}

### Node count can exceed the AKS maximum

Harness does not read the AKS maximum or target. Nothing caps the node count to that maximum. The 7-day number can be higher than that maximum when nodes overlap during an upgrade or a blue/green rollout. A brief spike in those 7 days raises **Potential Monthly Spend**, because spend uses this number.
{% endhint %}

## Monthly spend and savings

Harness calculates **Potential Monthly Spend** and **Potential Monthly Savings** from the public on-demand price:

```
Potential Monthly Spend = current node count * on-demand price per node per hour * 720
Potential Monthly Savings = (current pool hourly cost - recommended pool hourly cost) * 720
```

### Formula variables

* **Potential Monthly Spend**: the estimated cost of the current node pool for a month.
* **Potential Monthly Savings**: the difference between that cost and the recommended node pool.
* **Current node count**: the highest number of nodes the pool had in the last 7 days. Navigate to [Current node count](#current-node-count) to review how Harness counts nodes.
* **On-demand price per node per hour**: the public on-demand price of one node of that SKU.
* **Current pool hourly cost**: the current node count multiplied by the on-demand price per node per hour.
* **Recommended pool hourly cost**: the recommended node count multiplied by the recommended price per node per hour.
* **720**: 24 hours multiplied by 30 days.

**Potential Monthly Spend** from this formula is not the cloud bill. It assumes the 7-day peak runs for 720 hours at the public on-demand price.

The date in the drilldown header, such as **(by Oct 30)**, is only a label. The monthly figure is always calculated with 720 hours, so it does not change with the number of days in the month.

The hourly price is the public on-demand price of that SKU in the node pool region, in US dollars. When **Currency Preferences** is set, Harness converts that price to the destination currency. Navigate to [Currency preferences](https://developer.harness.io/cloud-cost-management/cost-reporting/5-currency-preferences.md) to set the destination currency. The drilldown and the CSV do not show a currency label. This price is not a negotiated rate, and it does not match the cloud portal price in the local currency. Public pricing data may differ from actual price. Harness does not use a negotiated rate card to pick the SKU.

{% hint style="success" %}

### Price when Currency Preferences is not set

When **Currency Preferences** is not set, the price stays in US dollars.
{% endhint %}

The recommended node count can be higher than the current node count. Savings remain positive when the recommended pool hourly cost is lower, for example when the recommendation uses more, smaller virtual machines.

### Cost from billing data

Public on-demand prices are used in every case. When there is a successful billing match for the current pool, Harness also uses that bill, and scales the recommended pool's public price by the same ratio.

The default cost type configured for your account decides which part of the bill counts, for example **Amortized** on Azure. **Potential Monthly Spend** is the billed cost of the current pool for the last 30 days.

Harness does not have a bill for the recommended pool. Harness estimates that cost by applying the current pool's ratio of billed monthly cost to public monthly cost to the recommended pool's public monthly cost. This assumes the same ratio applies to the recommended pool. A ratio below 1 means the current pool has a discount.

Savings keep the same ratio as the public-price calculation:

```
Savings = billed monthly cost * (1 - recommended pool hourly cost / current pool hourly cost)
```

### Formula variables for billed figures

* **Billed monthly cost**: the billed cost of the current pool for the last 30 days.
* **Recommended pool hourly cost**: the recommended node count multiplied by the public on-demand price per node per hour.
* **Current pool hourly cost**: the current node count multiplied by the public on-demand price per node per hour.

The savings ratio uses the public prices, so the savings percentage stays the same as in the public-price calculation. Only the base amount changes to the billed cost.

The SKU still comes from utilization and the public catalog. The current node count stays the 7-day peak. **Cost per node per hour** in the drilldown can still show the public on-demand price when the monthly figures come from billing. The billed amount is already in the billing currency, so **Currency Preferences** is not required for that amount.

{% hint style="info" %}

### Missing billing data

When billing data for the current pool is missing or unusable, Harness uses public on-demand prices for the current pool and the recommended pool. The cost type changes to public on-demand.
{% endhint %}

## Recommendation drilldown

* **Potential Monthly Spend**: Projected cost with and without recommendations
* **Potential Monthly Savings**: Expected monthly cost reduction
* **Current and Recommended**:
  * Estimated Savings
  * Instance Family
  * Node Count
  * CPU (vCPU)
  * Memory (GiB)
  * Cost per node per hour
  * Region
  * Potential monthly cost (current and after recommendation) The recommended values show Spot and On-demand configurations, estimated savings and potential monthly cost.
* **Cluster details and Resource Utilization** (CPU, Memory, Node Count) in the last X days where X is the number of days selected for the recommendation (7,30,60 or 90).
* **Tune recommendations**: You can either set a preset or tune your recommendations by modifying:
  * Minimum Node count
  * Percentage Buffer to be considered for CPU/Memory values
  * Preferred Instance families: **All** or **Specify families**. A specified list is stored per region. If the node pool's region has no list, the drilldown shows **All**.
  * Minimum Workload Requirements (CPU and RAM): These are calculated using the peak historical CPU and memory usage observed on the nodes. You can adjust the values if needed.
  * Preferred Resource Requirements(CPU and RAM): These are calculated by aggregating CPU and memory usage and requests from all pods running across nodes in the node pool. You can adjust the values if needed. As and when these values are adjusted, the recommendationed values are computed based on these values.

To download a daily snapshot of the recommendations, select **Export CSV** on the recommendations list. Harness creates the snapshot once a day, and it already includes any preset you applied. The **Potential Savings** value in the CSV comes from that snapshot.

The drilldown can recalculate the SKU, the node count, and the savings when you apply a preset, change **Tune recommendations**, or select a range other than the default. Drilldown values can therefore differ from the CSV on the same day.

**Total Resource Utilisation Max Node Count** in the CSV is a capacity ceiling, not the current node count. The count shows how many nodes the stored CPU and memory utilization could fill, so it can be higher or lower than the current node count.

### Drilldown fields and CSV columns

| Drilldown field                    | CSV column                  |
| ---------------------------------- | --------------------------- |
| Potential Monthly Spend            | Potential Monthly Spend     |
| Potential Monthly Savings          | Potential Savings           |
| Instance family (current pool)     | Current Instance Family     |
| Instance family (recommended pool) | Recommended Instance Family |
| Node count (current pool)          | Current Node Count          |
| Node count (recommended pool)      | Recommended Node Count      |
| CPU (current pool)                 | Current CPU                 |
| CPU (recommended pool)             | Recommended CPU             |
| Memory (current pool)              | Current Memory              |
| Memory (recommended pool)          | Recommended Memory          |
| Region                             | Region                      |

Cost per node per hour is shown in the drilldown but not in the CSV.

The CSV also has these columns, which are not shown in the drilldown:

* Total Resource Utilisation Max CPU
* Total Resource Utilisation Max Memory
* Total Resource Utilisation Max Node Count
  {% endtab %}

{% tab title="Workload Recommendations" %}

## Before you begin

* Connect your Kubernetes cluster in Harness and [set up CACM for cost management](https://developer.harness.io/cloud-cost-management/integrations/kubernetes.md).
* Cost Visibility and the Inventory Management features should be enabled on your Kubernetes CACM connector.
* Enabling the **Visibility** feature allows retrieving recommendations from the Azure Advisor. The **Inventory Management** feature allows you to fetch the CPU utilization data and display the corresponding recommendations. If the Inventory Management feature is not enabled, the graph and table may show a null state.

## How are workload recommendations computed?

In Harness CACM, the workload recommendations are computed by analyzing the past utilization of CPU and memory of your workload. The implementation uses a histogram method to compute the recommendations.

The computation adds a 15% buffer to the recommended resources by default. CACM also allows you to add any additional buffer using the Tune recommendations option.

When you enable Cost Visibility for your Kubernetes Cluster, the Delegate associated with your Connector starts collecting CPU and memory resource utilization metrics for every node and pod (including individual containers) present in the cluster every minute using a metrics server. CACM relies on the Metrics Server and initializes recommendations after an initial data collection of 24-48 hours. The Metrics Server is queried by the controller every minute for utilization data.

The utilization data collected every minute is then aggregated in the Delegate for a 20-minute window. The 20-minute aggregated data is then sent to Harness:

* **CPU**: For CPU values, a partial histogram for the last 20 minutes is sent.
* **Memory**: The maximum value of the memory is sent as a single data point.

We use this data for further processing and to compute the complete histogram displayed in the UI (partial histograms are merged for the CPU and memory histogram is computed using the data points).

Each of these daily histograms has an equal weightage for a given workload. As a result, if you select the last 30 days of data to aggregate, we will assign equal weightage to each of the 30 days.

You can choose to tune the recommendations by changing the Quality of Service (QoS) and the percentage of recommended CPU and memory requests/limits buffer. See Tune Recommendations.

You can also customize your recommendations by increasing or decreasing the request and limit coverage of the CPU and memory samples. Drag the slider of the number of samples to adjust the percentile. The slider indicates the percentile of all the CPU and memory samples that are covered to compute the resource recommendations. Based on your selection, the recommendations for your workload are made.

{% hint style="info" %}
**WHY HISTOGRAM?**

A histogram is used to account for the seasonality of high resource utilization on certain days of the week. Assume your application receives a lot of traffic (and thus a lot of resource utilization) on weekends and we are using a decaying histogram. In that case:

* If you view workload recommendation on Friday and selected the last seven days of utilization data, then Saturday will be given the least weightage, followed by Sunday, hence the recommended resources will be low.
* If you view the workload recommendation on Monday, Sunday will be given the most weightage, hence your recommended resources may be high.

To avoid this, we use the histogram method and give equal weight to all previous days.
{% endhint %}

## Categorization of recommendations

The recommendations are categorized as the following:

* Cost Optimized
* Performance Optimized

## Cost optimized

The cost-optimized recommendations are computed as the following:

* The lower bound is determined by analyzing the 50th percentiles of CPU samples and memory peaks. This means that the recommendations will consider the average or median usage levels of CPU and memory resources over a given period.
* There is no upper bound for CPU samples and memory peaks are based on the 95th percentile. It is recommended not to set the upper bound for the CPU samples, because the CPU is considered a compressible resource. If your application starts hitting the CPU limits, Kubernetes starts throttling the pods.
* There is no limit set for the CPU. This is to avoid the throttling of the workloads when there is a spike in CPU usage. The usage of the CPU may go up in certain conditions, for example during the pod start. In such a scenario, the CPU can burst and get the required additional resources.
* The potential monthly spend and savings are calculated based on the 90th percentiles of CPU samples and memory peaks.

Since the recommendations are computed using the 50th percentile of the CPU samples and memory peaks, this may potentially lead to system performance issues. Before using cost-optimized recommendations, ensure that you evaluate the recommendation's impact thoroughly.

## Performance optimized

The performance-optimized recommendations are computed using the 95th percentile of CPU samples and memory peaks. Because of this, the probability of having any effect on the performance is minimum. However, the cost may go high for the resources that are optimized using this method.

The potential monthly spend and savings are calculated based on the 90th percentiles of CPU samples and memory peaks.

**Example**

Let us try to understand how the recommendations are computed using the following example. The following example illustrates how the resources can be optimized using the performance-optimized recommendations:

```yaml
Current Resources:  
  limits:  
    memory: 8Gi  
    cpu: 1  
   requests:  
     memory: 8Gi  
     cpu: 1  
   
Recommended Resources based on utilization data for the last 7 days (QoS: Guaranteed, %of buffer: 0)  
  
 limits:  
  memory: 3.5Gi  
  cpu: 1.1  
requests:  
  memory: 3.5Gi  
  cpu: 1.1
```

The current resources are provisioned using `8Gi memory` and `1 CPU`, the recommended resources require only `3.5Gi memory` and `1.1m CPU` for limits and requests both.

***

## Recommendations drilldown

{% hint style="info" %}
These workload recommendations are computed based on number of samples received which shows low resource utilisation. The recommended numbers might not be sufficient for starting the workload as resource consumption at start up might be higher. Take this into account while applying recommendation.
{% endhint %}

When you click on an individual recommendation, the following details are shown:

* Potential Monthly Workload Spend
* Potential Monthly Workload Savings
* Resource Changes to be made to apply the recommendation
* Cost Optimized Histogram of samples analysed over the last X days where X is the number of days selected for the recommendation (7,30,60 or 90).
* Workload Details: Including Cluster, Namespace, Workload and more details as [shown here.](https://developer.harness.io/cloud-cost-management/cost-optimization/recommendations/1-home-recommendations.md#faq)
* Tune Recommendations: You can either apply a preset or tune the recommendations by changing:
  * Quality Of Service(QoS): You can set burstable or guaranteed QoS for recommended requests/limits. For more information, go to [Configure Quality of Service for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/).
    * Burstable: A pod is assigned to a QoS class of burstable if:

      * The pod does not meet the criteria for QoS class Guaranteed.
      * At least one container in the pod has a memory or CPU request.

      In this QoS class, resources can go beyond requests up to the limits. This recommended action does not affect other workloads and stays within bounds of usage. The scheduler uses the request to place the pod on a node, but the pod can use more resources up to the limit before it’s killed or throttled. For more information, go to [QoS class of Burstable](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-burstable).

      We do not recommend CPU limits in the case of burstable QoS.
    * Guaranteed: In this Quality of Service (QoS) class, resource requests and limits are set to the same values. Setting them to the same values guarantees that the resources requested by the container will be available to it when it is scheduled. This is considered as the ideal QoS class for the most stable Kubernetes clusters.

      For a pod to be given a QoS class of guaranteed:

      * Every container in the pod must have a memory limit and a memory request, and they must be the same.
      * Every container in the pod must have a CPU limit and a CPU request, and they must be the same.

      For more information, go to [QoS class of Guaranteed](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed).
  * Buffer for workloads: You can set the percentage of buffer for recommended requests/limits.

## FAQ

<details>

<summary>What additional details are displayed for a recommendation? And what are the aggregation types?</summary>

* **Workload details**: The following information is displayed:
  * Workload
  * Namespace
  * Workload Type
  * Cluster
* **Cost details**: The following cost details are displayed:
  * **Total Cost**: For Kubernetes clusters, the total cost is the sum of all the node costs. For ECS clusters, the sum of all container instances.
  * **Idle Cost**: Idle cost is the cost of idle resources (CPU and memory) allocated to a Kubernetes pod or an Amazon ECS task but is not utilized. For more information, go to [Idle Cost](https://developer.harness.io/cloud-cost-management/use-cacm/key-concepts.md#idle-cost).
  * **Utilized Cost**: Utilized cost is the cost estimate for the utilized node or pod resources.
* **Aggregation**: The aggregated CPU and memory utilization value of your workload. The values are displayed as the following:
  * **Time-weighted**: This considers the active duration of pods when calculating the CPU and memory utilization.
  * **Absolute**: This aggregates the CPU and memory utilization values.

Go to [Kubernetes workload utilization aggregations](https://developer.harness.io/cloud-cost-management/cost-optimization/recommendations/1-home-recommendations.md#kubernetes-workload-utilization-aggregations) to review worked examples of time-weighted and absolute aggregation.

</details>
{% endtab %}
{% endtabs %}

{% @harness-feedback/feedback %}
