> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/feature-management-experimentation/use-fme/warehouse-native-experimentation/integrations/databricks.md).

# Connect Databricks

Learn how to integrate Databricks with Harness FME to enable Warehouse Native Experimentation.

{% hint style="info" %}
This feature is in Beta[^1] and requires enablement. [View all early access features.](/release-notes/features.md)

{% @harness-feature-flag-access/request-feature-flag-access module="feature-management-experimentation" pagePath="feature-management-experimentation/use-fme/warehouse-native-experimentation/integrations/databricks" featureFlags="%5B%7B%22name%22%3A%22Databricks%20for%20Warehouse%20Native%20Experimentation%20(Classic%20Experience)%22%2C%22key%22%3A%22whn\_databricks\_enabled%22%2C%22status%22%3A%22BETA%22%2C%22description%22%3A%22Enables%20Databricks%20as%20a%20data%20warehouse%20for%20Warehouse%20Native%20Experimentation%20in%20the%20Harness%20FME%20Classic%20Experience.%22%7D%2C%7B%22name%22%3A%22Databricks%20for%20Warehouse%20Native%20Experimentation%20(New%20Experience)%22%2C%22key%22%3A%22FME\_ENABLE\_WHN\_DATABRICKS%22%2C%22status%22%3A%22BETA%22%2C%22description%22%3A%22Enables%20Databricks%20as%20a%20data%20warehouse%20for%20Warehouse%20Native%20Experimentation%20in%20the%20Harness%20FME%20New%20Experience.%22%7D%5D" %}
{% endhint %}

Warehouse Native Experimentation allows you to run experiments on data that already lives in your data warehouse. By connecting Harness FME directly to your Databricks SQL warehouse, you can securely query and analyze experiment data from your source of truth.

To begin, connect your Databricks SQL warehouse as a data source using either OAuth machine-to-machine (M2M) authentication with a service principal or a personal access token.

### Prerequisites

Ensure that you have the following before getting started:

* Access to a Databricks SQL warehouse, including its server hostname and HTTP path
* A Unity Catalog catalog and schema containing your experiment data
* A designated results table where experiment results are stored
* A Databricks service principal with an OAuth client ID and secret, or a personal access token

### Setup

Harness recommends the following best practices:

* Use a service principal with OAuth M2M authentication rather than a personal access token tied to an individual user.
* Grant read-only access to the catalog and schema containing the assignment and event data.
* Grant write access only to the experiment results table.

To integrate Databricks as a data warehouse for Warehouse Native Experimentation:

1. From the Harness FME navigation menu, click **FME Settings** and click **View** on a project on the **Projects** page. Then, navigate to the **Data Source** tab.
2. Select Databricks as your data warehouse. In the **Data Sources** tab of your Harness FME project, select **Databricks** from the list of supported data warehouses.

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p><strong>PROJECT EXPERIMENTATION TYPE</strong></p><p>A project uses a single experimentation type based on the metric source used.</p><p>When you add a data source to a project, the project's experimentation type is set to <code>Warehouse Native</code>. All metrics in the project must then use Warehouse Native metric sources.</p><p>If a project instead uses metrics created from an <a href="/feature-management-experimentation/new-to-fme/get-started/send-event-data.md">ingested event source</a>, the project's experimentation type is set to <code>Cloud</code>.</p></div>
3. Enter the following connection details:

   | Field           | Description                                                                    | Example                                  |
   | --------------- | ------------------------------------------------------------------------------ | ---------------------------------------- |
   | Server Hostname | The server hostname of your Databricks SQL warehouse.                          | `dbc-1a2b3c4d-5e6f.cloud.databricks.com` |
   | Port            | The port of your Databricks SQL warehouse. Defaults to `443` if not specified. | `443`                                    |
   | HTTP Path       | The HTTP path of your Databricks SQL warehouse.                                | `/sql/1.0/warehouses/a1b2c3d4e5f67890`   |

   Azure Databricks workspaces use a server hostname in the form `adb-<workspace-id>.<n>.azuredatabricks.net`.

   ![](https://1473625385-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FrT2RGHAjtBTaObzchfpb%2Fuploads%2Fgit-blob-7f8883942dd46a7205f09eddefaff9e19e0d83c2%2Fdatabricks-data-source.png?alt=media)
4. Select an authentication method from the **Authentication** list. Harness FME supports two authentication methods for connecting to Databricks:
   * **OAuth M2M** (recommended): Harness FME authenticates as a Databricks service principal using an OAuth client ID and client secret.

     | Field         | Description                                                   |
     | ------------- | ------------------------------------------------------------- |
     | Client ID     | The OAuth2 client ID of the Databricks service principal.     |
     | Client Secret | The OAuth2 client secret of the Databricks service principal. |
   * **Personal access token**: Provide a Databricks personal access token with read access to the catalog and schema required for experiment metrics, and write access to the results table.
5. Configure network access for private or IP-restricted workspaces. If your Databricks workspace restricts inbound traffic using an [IP access list](https://docs.databricks.com/aws/en/security/network/front-end/ip-access-list) or private connectivity, Harness must be allowed network access to run experiment calculations against your warehouse.

   To enable connectivity:

   * Add the Harness outbound IP addresses to your workspace IP access list.
   * Allow traffic on the Databricks SQL warehouse port (default: `443`).

   Harness FME maintains a list of [static outbound IP addresses](https://our-ips.split.io/) used for warehouse connections. You must allow these addresses for the connection to succeed.
6. Test the connection by clicking **Test Connection**.
7. Select a catalog and schema. After authentication, Harness FME retrieves the catalogs and schemas accessible to the service principal or token. Select the one containing your assignment and metric source tables.
8. Specify a results table. Create a results table where Harness FME will write experiment analysis results, and ensure that:

   * The table exists in your catalog and schema.
   * The schema matches the expected format for experiment results below.

   | Field                 | Type        | Description                                                                                         |
   | --------------------- | ----------- | --------------------------------------------------------------------------------------------------- |
   | `METRICID`            | `STRING`    | Unique identifier for the metric being calculated.                                                  |
   | `METRICNAME`          | `STRING`    | Human-readable name of the metric being calculated.                                                 |
   | `METRICRESULTID`      | `STRING`    | Unique identifier representing a specific calculation per metric, per experiment, per analysis run. |
   | `EXPID`               | `STRING`    | Unique identifier for the experiment associated with this metric calculation.                       |
   | `EXPNAME`             | `STRING`    | Human-readable name of the experiment associated with this metric calculation.                      |
   | `TREATMENT`           | `STRING`    | The experiment variant (e.g., Control or Treatment) associated with the metric results.             |
   | `DIMENSIONNAME`       | `STRING`    | The name of the dimension being analyzed (e.g., country, platform).                                 |
   | `DIMENSIONVALUE`      | `STRING`    | The corresponding value of the analyzed dimension.                                                  |
   | `ATTRIBUTEDKEYSCOUNT` | `BIGINT`    | Count of unique keys (users, sessions, etc.) attributed to this metric result.                      |
   | `REQUESTTIMESTAMP`    | `TIMESTAMP` | Timestamp when the metric computation request occurred.                                             |
   | `MIN`                 | `DOUBLE`    | Minimum observed value for the metric.                                                              |
   | `MAX`                 | `DOUBLE`    | Maximum observed value for the metric.                                                              |
   | `COUNT`               | `BIGINT`    | Total number of observations included in the metric calculation.                                    |
   | `SUM`                 | `DOUBLE`    | Sum of all observed metric values.                                                                  |
   | `MEAN`                | `DOUBLE`    | Average (mean) of the metric values.                                                                |
   | `P50`                 | `DOUBLE`    | 50th percentile (median) metric value.                                                              |
   | `P95`                 | `DOUBLE`    | 95th percentile metric value.                                                                       |
   | `P99`                 | `DOUBLE`    | 99th percentile metric value.                                                                       |
   | `VARIANCE`            | `DOUBLE`    | Variance of the metric values.                                                                      |
   | `EXCLUDEDUSERCOUNT`   | `BIGINT`    | Number of users excluded from the analysis (due to filters, SRM, etc.).                             |
   | `ASOFTIMESTAMP`       | `TIMESTAMP` | Timestamp representing when the result snapshot was written.                                        |

   To create the results table with the correct structure, run the following SQL statement in Databricks:

   ```sql
   CREATE TABLE IF NOT EXISTS <CATALOG_NAME>.<SCHEMA_NAME>.<TABLE_NAME> (
      METRICRESULTID      STRING,
      TREATMENT           STRING,
      DIMENSIONNAME       STRING,
      DIMENSIONVALUE      STRING,
      ATTRIBUTEDKEYSCOUNT BIGINT,
      REQUESTTIMESTAMP    TIMESTAMP,
      MIN                 DOUBLE,
      MAX                 DOUBLE,
      COUNT               BIGINT,
      SUM                 DOUBLE,
      MEAN                DOUBLE,
      P50                 DOUBLE,
      P95                 DOUBLE,
      P99                 DOUBLE,
      VARIANCE            DOUBLE,
      EXCLUDEDUSERCOUNT   BIGINT,
      ASOFTIMESTAMP       TIMESTAMP,
      METRICID            STRING,
      METRICNAME          STRING,
      EXPID               STRING,
      EXPNAME             STRING
   );
   ```
9. Save and activate. Once the test passes, click **Save** to create the connection.

Your Databricks data source can now be used to create assignment and metric sources for Warehouse Native Experimentation.

### Example Databricks configuration

| Setting         | Example                                  |
| --------------- | ---------------------------------------- |
| Data Warehouse  | `Databricks`                             |
| Server Hostname | `dbc-1a2b3c4d-5e6f.cloud.databricks.com` |
| Port            | `443`                                    |
| HTTP Path       | `/sql/1.0/warehouses/a1b2c3d4e5f67890`   |
| Authentication  | `OAuth M2M`                              |
| Catalog         | `ml`                                     |
| Schema          | `demo`                                   |
| Results Table   | `whn_metric_results`                     |

{% @harness-feedback/feedback %}

[^1]: Beta features are feature-complete but still being validated. Available on request.