> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/web-application-and-api-protection-waap/data-collection/gateways-load-balancers/azure/data-collection-with-log-analytics.md).

# Data collection with log analytics

**Log Analytics** in Azure APIM is a tool for gathering and examining detailed logs and metrics from your APIs. It helps you monitor, troubleshoot, and optimize your APIs effectively. Traceable uses Log Analytics to collect data from your APIs, which is sent to the Traceable Platform for further analysis.

This topic outlines the steps for configuring data capture for Azure API Management (APIM) using Log Analytics. The process involves creating a Storage Account, enabling diagnostic logs on APIM, setting up a Log Analytics export rule, and creating an Azure Function to handle the log data.

The following is a high-level data flow diagram for data collection using log analytics.

<img src="/files/Wsa748necNQhrBqevokr" alt="" width="900">

* **APIM Gateway**: Collects and sends logs.
* **Diagnostic Logs**: Stores logs before forwarding them to a Storage Account.
* **Storage Account**: Receives logs from Diagnostic Logs and triggers the Azure Function.
* **Azure Function**: Processes logs and sends data to the Traceable Platform agent.
* **Traceable Platform agent**: Consumes the data.
* **Log Analytics Workspace**: Stores and allows analysis of the processed data.

***

## Before you begin

Make a note of the following points before you start with the configuration:

* Make sure that the Traceable Platform agent is deployed. Note the platform agent's IP address. For more information, see [Platform agent](https://docs.traceable.ai/docs/tpa).
* An Azure API Management gateway is created.
* A Log Analytics workspace is created.

***

## Configuration

### Step 1 — Create a storage account

The Storage Account receives application insight logs, which are processed by an Azure App Function and exported to the Traceable Platform agent. Complete the following steps to create a Storage account:

1. Log into your Azure account and click **Storage accounts**.

   <img src="/files/wv1hlvOAnhPZ1Rh6Gmio" alt="" width="900">
2. Click **Create** on the Storage accounts page.

   <img src="/files/tzZrTxwVhla3IGaFvO0k" alt="" width="650">
3. On the **Create a storage account** page, navigate to the **Basics** tab and fill in the values. In the **Redundancy** field, you may choose Locally-redundant storage (LRS).

   <img src="/files/6LQ2Kik0ZLaEr9nkzd0Y" alt="" width="650">
4. Navigate to the **Networking** tab and configure the following. These configurations limit access to this Storage account so that it is accessible from the Log Analytics workspace and the App Function created in the later steps.
   1. **Network access** — Enable public access from selected virtual networks and IP addresses.
   2. **Virtual networks** — Configure your Virtual network subscription, Virtual network, and Subnets.
   3. **Network routing** — Select Microsoft network routing.

      <img src="/files/g7yWnk03TTOYCNkfbZs9" alt="" width="600">
5. Once the Storage account is configured, configure the retention limits. The integration is event-based, so a short retention period is acceptable. In the Storage account you created above, navigate to Lifecycle Management.

   <img src="/files/l6vyk7cjo5inXJWRTCsa" alt="" width="650">
6. In the **Details** tab of **Add a rule** page, provide the following:
   1. Rule name
   2. Rule scope — Select **Apply rule to all blob in your storage account**.
   3. Blob type — Select **Block blobs**.
   4. Blob subtype — Select **Base blob**.

      <img src="/files/NnIlOXAnsFzEPnzRczNe" alt="" width="600">
7. In the **Base** blobs tab, configure the rule conditions. You may select **Delete the blob** in the `then` operation.

   <img src="/files/NcUAicB8G2g65QL45t4Q" alt="" width="500">
8. Click on **Add**.

***

### Step 2 — Enable API management diagnostic logs

Complete the following steps:

1. In Azure, navigate to your APIM instance.
2. Go to **Monitoring** → **Diagnostic Settings**.
3. Create a new Diagnostic Setting:
   * Select the **Send to Log Analytics Workspace** checkbox.
   * Ensure the **Destination table** is set to **Resource Specific**.

<img src="/files/N6Td2lIETyfL8qTSFN8Y" alt="" width="900">

***

### Step 3 — Configure diagnostic logging on target APIs

Complete the following steps:

1. Navigate to **APIs** → \<select an API> → **Settings**.
2. Enable the **Azure Monitor** setting.
3. Set sampling to 100% to capture all requests.
4. Ensure the **Log Client IP address** option is selected.
5. Under **Additional settings**, enable:
   * Frontend Request
   * Frontend Response
6. Specify any headers to be captured for both the Frontend Request and Response. For body capture to work correctly, ensure you capture content type and Content length.
7. Configure the maximum body bytes to capture. The maximum value is 8192 bytes.
8. Click on **Save**.

<img src="/files/2kgmDXGhMLAUHQSCvrKh" alt="" width="900">

***

### Step 4 — Create log analytics export rule

Complete the following steps:

1. Navigate to the Log Analytics workspace.
2. Set up an export rule to send data to the Storage account you created earlier. Navigate to **All** **services** → **Analytics** → **Log Analytics workspace**.

   <img src="/files/97SJ5NBEvr3mJWypDzVU" alt="" width="900">
3. Select your workspace and navigate to **Settings** → **Data export**.

   <img src="/files/mBybOShkijoDIBBVLmur" alt="" width="900">
4. Click **+New export rule** on the Data export page.

   <img src="/files/j63SqBwmSGnyKXL12e4e" alt="" width="600">
5. Click **Next** and select the **Source**.

   <img src="/files/ncdg5Cqt3IFRWv7qYVC2" alt="" width="900">
6. Select Storage account in the **Destination** tab. Click on **Next** and **Review + Create**.

***

### Step 5 — Create a Function App and a Function

Create an Azure App Function to be triggered when APIM logs are sent to the Storage account. The App Function that you create will need network access to the Storage accounts and Traceable Platform agent. If the Traceable Platform agent is hosted privately, you must use Functions Premium to enable vNet connectivity between the App Function and Traceable Platform agent. Complete the following steps to configure the App Function.

1. Navigate to Function App and click **Create**.

   <img src="/files/6ACsqjLHMEBBAuKkJA7K" alt="" width="600">
2. Choose **Functions Premium** as the hosting option.
3. Choose **Node.js** as the Runtime stack and configure the other options as shown. Click **Next**

   <img src="/files/GVuKontFDbkuD1qTZXoW" alt="" width="450">
4. In the **Storage** tab, select the storage account that you created earlier.
5. In the **Networking** tab:
   1. Select **Off** in the Enable public access option
   2. Select **On** in the Enable network injection option.
   3. Select the Virtual network.
   4. Inbound access — Select **Off** in the Enable private endpoints option.
   5. Outbound access — Select **On** in the Enable Vnet integration and select the **Outbound subnet** from the drop-down list.

      <img src="/files/VbIW20eLGhxbVYBqMEQe" alt="" width="500">
6. Navigate through the rest of the Monitoring, Deployment, and Tags tabs. Finally, Review and Create the App Function.
7. Navigate to your newly created Function App and click **Create** to create a new function, as shown below.
8. In the **Select template** tab, select **Azure Blob Storage trigger** and click **Next**.

   <img src="/files/qxuFXq1sCM7VcuRNtgHn" alt="" width="900">
9. In the **Template details** tab, provide the following:
   1. Function name
   2. Path — This must be `am-apimanagementgatewaylogs`.
   3. Storage account connection — Select the previously created Storage account.

      <img src="/files/wPxqZ4B0KKXTkUWvOs59" alt="" width="700">
10. Click **Create** to create the Function, and navigate to the **Code** + **Test** tab to configure the function.
11. Navigate to Traceable’s [download](https://docs.traceable.ai/docs/downloads.traceable.ai) site and copy the function.js from **agent** → **azure-logging** → **latest** → **function.js**.
12. Paste the function you copied in Step 10 in the **Code + Test tab,** as shown in Step 9. Configure the Traceable Platform agent IP address and service name. A default service is already provided.

    <img src="/files/w8QAF5RFl32brxvHlgTl" alt="" width="900">