> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/cloud-cost-management/3.0/connect-your-data/data-jobs.md).

# Data Jobs

Monitor data ingestion jobs for your connectors and review run details and history

Data Jobs lists the data ingestion and processing jobs that bring billing data into Harness Cloud & AI Costs. In the left navigation, select **Cloud & AI Costs**, then select **Data Jobs**.

***

## What you will learn from this topic

The page covers the jobs list and the details panel.

* How to read the status, phase, and progress of a job
* How to filter the jobs list by connector, type, and status
* How to open a job and read the current run and its log
* Why execution month and billing month differ in run history

***

## Before you begin

Confirm the license and a connector.

* Verify you have an Enterprise Edition license. Data Jobs is available only for Enterprise Edition licenses. Navigate to [Licensing](https://developer.harness.io/cloud-cost-management/3.0/administer/licensing.md) to confirm your edition.
* Verify you have at least one connector. The Data Jobs page lists active connectors and their latest job runs.

{% hint style="info" %}

### Supported providers

Cloud connectors on this page use these billing exports:

* AWS (Cost and Usage Reports)
* Azure (Cost Exports, both **Actual Costs** and **Amortized Costs**)
* GCP (BigQuery billing exports)

The jobs list also includes AI connectors, for example, Anthropic and OpenAI.

Navigate to [Connect AWS (CUR 2.0)](https://developer.harness.io/cloud-cost-management/3.0/connect-your-data/aws/connect-aws-cur-2.md), [Azure](https://developer.harness.io/cloud-cost-management/3.0/connect-your-data/azure.md), or [GCP](https://developer.harness.io/cloud-cost-management/3.0/connect-your-data/gcp.md) to connect a cloud account. Navigate to [Anthropic and OpenAI](https://developer.harness.io/cloud-cost-management/3.0/connect-your-data/ai-providers/anthropic-and-openai.md) to connect an AI provider.
{% endhint %}

{% hint style="warning" %}

### Self-Managed Platform Not Supported

Data Jobs is not supported in Self-Managed Platform deployments (ClickHouse-enabled environments). Navigate to [Self-Managed Enterprise Edition](https://developer.harness.io/cloud-cost-management/3.0/administer/self-managed-enterprise-edition.md) to review that deployment.
{% endhint %}

***

## Jobs list

When you connect cloud accounts (AWS, Azure, GCP) to Harness Cloud & AI Costs, background jobs run to ingest billing data, process it, and make it available for cost analysis. The page shows active data ingestion jobs, jobs that require attention (failed or stuck), completed jobs, job details and logs, and historical job runs by billing period.

The Data job status page shows the cards and the jobs list.

![The Data job status page, with Active jobs, Requires attention, and Completed jobs above the jobs list](https://3362046280-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLgkNK1xq1Uj6rzIm2kum%2Fuploads%2Fgit-blob-af71b12579a3800b637dd64aea7d0d0cfc4bc996%2Fdata-jobs-list.png?alt=media)

{% hint style="success" %}

### Refresh the job status

The page shows when the list was last updated. Click **Refresh** to load the latest job status.
{% endhint %}

### Summary cards

The page displays four cards at the top:

* **Active jobs:** Jobs that are running or queued across all connectors. This includes ingestion, backfills, and other processing tasks.
* The breakdown card next to **Active jobs** splits the active count into **Active ingestion**, **Active PDT refresh**, and **Active backfills**.
* **Requires attention:** Jobs that have failed or are taking longer than expected.
* **Completed jobs:** Jobs that finished successfully.

{% hint style="info" %}

### Initial data population

After enabling this feature for your account, it takes approximately 24 hours to populate the job statuses for all connectors. During this period, you may see incomplete or missing data.
{% endhint %}

### Filter jobs

To filter the jobs list:

* **Search connector:** Enter a connector name.
* **Type:** Select **All** or **Ingestion**. Ingestion ingests raw billing data from cloud providers.
* **Status:** Select a status.
  * **All:** Shows all jobs regardless of status.
  * **Queued:** Jobs waiting in queue to be processed.
  * **Running:** Jobs that are executing.
  * **Success:** Jobs that completed successfully.
  * **Failed:** Jobs that encountered an error and stopped.
  * **Partial Success:** Jobs that completed with some errors or warnings.
  * **Cancelled:** Jobs that a user cancelled manually.
  * **Timed Out:** Jobs that exceeded the maximum allowed execution time.
  * **Unknown:** Jobs with an undetermined or unrecognized status.

### What each column shows

The table is titled **Active connectors and their latest runs** and lists each active connector and its latest job run:

* **Connector:** The cloud connector name and ID, with the cloud provider icon.
* **Job type:** Type of job.
* **Progress:** Visual progress bar showing job completion status.
* **Time elapsed:** Duration since the job started.
* **Status:** Current job status with phase and message details. The phase names are:
  * **Data Sync:** Transfers billing data from your cloud provider (AWS, GCP, or Azure) to Cloud & AI Costs storage.
  * **Data Enrichment:** Applies post-processing such as monthly rollups, cost category stamping, label processing, currency conversion, and Managed Service Providers (MSP) adjustments. Navigate to [Cost Categories and shared costs](https://developer.harness.io/cloud-cost-management/3.0/concepts/cost-categories-and-shared-costs.md) to read about cost categories. Navigate to [MSP markup and margin obfuscation](https://developer.harness.io/cloud-cost-management/3.0/administer/msp-markup-and-margin-obfuscation.md) to read about MSP adjustments.
  * **Data Load:** Completes core ingestion and final load, after which your data is fresh and available for analysis.
* **Last successful run:** Timestamp of the most recent successful job completion for this connector.

{% hint style="success" %}

### Azure connector cost type

An Azure connector shows the cost type in the **Connector** cell, for example **Actual**.
{% endhint %}

{% hint style="info" %}

### Azure cost types

For Azure connectors, jobs are tracked separately for:

* ACTUAL costs (pay-as-you-go)
* AMORTIZED costs (reserved instance amortization)

If you are using the API, specify the `azureCostType` parameter to filter history. Navigate to [REST API](https://developer.harness.io/cloud-cost-management/3.0/reference/rest-api.md) to call the API. Navigate to [Cost types](https://developer.harness.io/cloud-cost-management/3.0/concepts/cost-types.md) to compare Actual and Amortized cost.
{% endhint %}

{% hint style="info" %}

### Connector name and connector ID

Connector identity follows these rules:

* Jobs are stored with `connectorId` (immutable identifier).
* On connector rename, the latest name is displayed on the next sync.
* Always use `connectorId` for programmatic access.
  {% endhint %}

***

## Job details panel

Select a row to open the details panel. The panel has two tabs, **Job run details** and **Run history**.

### Job run details

The **Job run details** tab displays the latest run for the connector.

![The job details panel on Job run details, with the run fields, the progress bar, and the log](https://3362046280-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLgkNK1xq1Uj6rzIm2kum%2Fuploads%2Fgit-blob-937bffc4d19c1612385de843e814287d2809679b%2Fdata-jobs-run-details.png?alt=media)

The tab displays the current run:

| Field              | Description                                                      |
| ------------------ | ---------------------------------------------------------------- |
| Connector          | Connector ID. The panel heading shows the connector name.        |
| Job ID             | Unique identifier for this job run                               |
| Billing period     | **Start** and **End** for the billing period being processed     |
| Started            | Date the job started                                             |
| Cloud provider     | Provider for the run, such as AWS, Azure, GCP, or an AI provider |
| Status             | Current status with a visual indicator                           |
| Time elapsed       | How long the job has been running                                |
| Average time taken | Average time taken based on last 30 days ingestion               |

A progress bar visualizes the job's progress relative to its average completion time.

The tab also shows logs. Each log line includes a time and a status, such as INF or OK.

* The log shows the phase and the status. Navigate to [What each column shows](#what-each-column-shows) to read what **Data Sync**, **Data Enrichment**, and **Data Load** do.
* If multiple ingestions occur in a single day, the log shows multiple entries for that execution day.
* The log shows any errors or warnings encountered.
* The log shows processing details based on the billing month.

The billing period follows these rules:

* **Start** and **End** are the billing period being processed.
* Billing periods do not overlap. Multiple billing months can be ingested on the same day, and each job processes one billing period.

### Run history

The **Run history** tab lists previous runs for the connector.

![The Run history tab, with Execution month, Billing month, and one row expanded to its log](https://3362046280-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLgkNK1xq1Uj6rzIm2kum%2Fuploads%2Fgit-blob-d89393a426a50927fac82d6d27880f84b2882366%2Fdata-jobs-run-history.png?alt=media)

Filter historical job runs for the selected connector by:

* **Execution month:** The month when the job was actually executed and processed. The default is the current month.
* **Billing month:** The billing period the job was processing data for. For example, January billing data may be processed in February. The default is **All**.

The history table displays these columns:

* **Execution date:** The date when the job was executed.
* **Billing month:** The billing period that was processed by the job.
* **Status:** The final status of the job run.
* **Time taken:** Total duration from start to completion.

Click the expand icon to view detailed logs for any historical run.

{% hint style="info" %}

### Run history retention

The job runs listed in **Run history** for a connector are automatically deleted after six months.
{% endhint %}

***

## Troubleshooting

<details>

<summary>No cloud connectors are configured</summary>

If no cloud connectors are configured, the page prompts you to add a connector. Navigate to [Choose a provider](https://developer.harness.io/cloud-cost-management/3.0/connect-your-data/choose-a-provider.md) to add a connector.

</details>

<details>

<summary>No jobs match your filters</summary>

The page shows **No jobs match your filters**. The list title count is 0. Clear the filters to see all jobs.

Once the feature is enabled, it takes approximately 24 hours to display all connector statuses. Navigate to [Initial data population](#initial-data-population) to read what to expect during that wait.

</details>

<details>

<summary>A job needs attention</summary>

**Requires attention** counts jobs that have failed and jobs that are taking longer than expected.

Select **Failed**, **Partial Success**, or **Timed Out** in **Status** to list jobs that stopped with a problem. A job that is taking longer than expected can still be **Running**, so those three statuses do not list it. Compare **Time elapsed** on that row with other runs for the connector.

</details>

***

## Next steps

Confirm that ingestion is running for a connector, or connect a provider that is missing from the list.

* Navigate to [Check connection health](https://developer.harness.io/cloud-cost-management/3.0/connect-your-data/check-connection-health.md) to confirm a connector is connected.
* Navigate to [Choose a provider](https://developer.harness.io/cloud-cost-management/3.0/connect-your-data/choose-a-provider.md) to connect a cloud, Kubernetes, or AI provider.

{% @harness-feedback/feedback %}
