> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/artifact-registry/3.0/setup/hugging-face.md).

# Hugging Face

Hugging Face - host ML models in Harness Artifacts

Use a **Hugging Face** registry to host private models and datasets for use with Hugging Face Hub client libraries, such as `huggingface_hub`, `transformers`, and `datasets`.

## What you will learn from this topic

By the end of this guide, you will know how to:

* Create a registry in Harness Artifacts.
* Publish a package to the registry.
* Install the package from the registry and confirm that it works.

## Before you begin

* Install the Hugging Face CLI. Running `pip install -U huggingface_hub` installs the `hf` command and the `huggingface_hub` Python library used throughout this quickstart.

{% hint style="info" %}
**Additional installation steps**

Depending on your environment, you might need additional setup steps. To configure your environment correctly, go to [Hugging Face installation](https://huggingface.co/docs/huggingface_hub/en/installation) for details.
{% endhint %}

* Make sure you have access to a Harness account with permission to create registries and generate identity tokens.

## Create a Hugging Face Registry

To create a Huggingface Registry:

1. In Artifact Registry, select **Registries**.
2. Select **+ Create Registry**.
3. Specify the **Registry name**.

{% hint style="info" %}
**Registry name criteria**

The registry name must start with a letter, can only contain lowercase alphanumerics, `_`, `.`, and `-`, and must be unique to your Harness account.
{% endhint %}

4. Optionally, add a **Description**.
5. Choose visibility between **Public** and **Private**. The default is *Private*.
6. Select **Huggingface** as the **Registry type**.
7. Select **Save**.
8. Select **Create** to finalize.

{% hint style="warning" %}
**No upstream proxy support**

Huggingface registries do not support an upstream proxy. Harness Artifacts does not automatically resolve a model or dataset from `huggingface.co` when it is not already present in your registry. Upload the models and datasets your team needs directly to the registry, or point the client at a public endpoint when you need public Hub content.
{% endhint %}

{% hint style="info" %}
**Private registry**

A Huggingface registry serves as your private registry within Harness Artifacts. It does not publish content to the public Hugging Face Hub.
{% endhint %}

## Huggingface client setup

In your Hugging Face Artifact, select the registry, then select **Setup Client**. The dialog gives you an identity token, then setup instructions for publishing and using models and datasets with the `huggingface_hub` library.

### Generate an identity token

An identity token serves as the password for authenticating with the registry.

1. In your Hugging Face Artifact, select **Setup Client**.
2. Select **Generate Token**.
3. Select an **Expiration** for the token:
   * **30 Days**
   * **90 Days**
   * **180 Days**
   * **Custom**
4. Copy the token. You use it in the following steps in place of `<IDENTITY_TOKEN>`.

{% hint style="info" %}
**Token expiration**

The identity token stops working after the expiration you select. Generate a new token and update your client configuration before the current token expires.
{% endhint %}

### Configure the client

Select **Models** or **Datasets** for what you want to publish or use. In the code samples, replace the values in angle brackets with your own values.

{% tabs %}
{% tab title="Models" %}
**1. Configure environment variables**

Set the following environment variables:

```bash
# Recommended timeouts for model resolution.
export HF_HUB_ETAG_TIMEOUT=86400
export HF_HUB_DOWNLOAD_TIMEOUT=86400

# Point the client at your Harness registry endpoint.
export HF_ENDPOINT="https://pkg.harness.io/pkg/<ACCOUNT_ID>/<REGISTRY_NAME>/huggingface"
export HF_TOKEN="<IDENTITY_TOKEN>"
```

{% hint style="info" %}
**Client version requirement**

For Hugging Face client version 0.19.0 and later, these timeout settings enable model resolution through the `transformers` and `tokenizers` libraries.
{% endhint %}

**2. Publish the model**

Upload a model to the registry using the `huggingface_hub` library:

```python
from huggingface_hub import HfApi

api = HfApi()
api.upload_folder(
    folder_path="<FOLDER_PATH>",
    repo_id="<ARTIFACT_NAME>",
    revision="<VERSION>",
    repo_type="model"
)
```

**3. Use the model**

Download the model from the registry:

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="<ARTIFACT_NAME>", revision="<VERSION>", etag_timeout=86400
)
```

{% hint style="info" %}
**Model resolution through client libraries**

With Hugging Face client version 0.19.0 and later, and `HF_HUB_ETAG_TIMEOUT` set, you can resolve models directly through the `transformers` and `diffusers` libraries.
{% endhint %}
{% endtab %}

{% tab title="Datasets" %}
**1. Configure environment variables**

Set the following environment variables:

```bash
export HF_HUB_ETAG_TIMEOUT=86400
export HF_HUB_DOWNLOAD_TIMEOUT=86400
export HF_ENDPOINT="https://pkg.harness.io/pkg/<ACCOUNT_ID>/<REGISTRY_NAME>/huggingface"
export HF_TOKEN="<IDENTITY_TOKEN>"
```

**2. Publish the dataset**

Upload a dataset to the registry using the `huggingface_hub` library:

```python
from huggingface_hub import HfApi

api = HfApi()
api.upload_folder(
    folder_path="<FOLDER_PATH>",
    repo_id="<ARTIFACT_NAME>",
    revision="<VERSION>",
    repo_type="dataset"
)
```

**3. Use the dataset**

Load the dataset directly with the `datasets` library, or download the full dataset repository:

```python
from datasets import load_dataset

dataset = load_dataset("<ARTIFACT_NAME>")
```

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="<ARTIFACT_NAME>", revision="<VERSION>", repo_type="dataset", etag_timeout=86400
)
```

{% hint style="warning" %}
**Dataset caching limitation**

Harness Artifacts fully caches only datasets hosted directly on Hugging Face. A dataset that references external sources does not cache completely, and resolution can fail for the missing files.
{% endhint %}
{% endtab %}
{% endtabs %}

## Setup verification

Confirm that your registry serves the model or dataset you published. To verify your setup:

1. In a new environment with no cached Hugging Face files (clear `HF_HOME`, or point it to an empty directory), use the `huggingface_hub` or `datasets` library to resolve the model or dataset you published.
2. Confirm that the client downloaded the model or dataset from your Harness registry endpoint, which is shown in **Setup Client**.
3. In Harness, open your registry and confirm that the model or dataset, and the revision you published, appear in the package list.

If a step fails, go to [Troubleshooting](#troubleshooting) to resolve common authentication and resolution issues before you retry.

## Pipeline automation (optional)

Publishing from a pipeline gives you a repeatable release process. Run the publish step when you create a version tag, so each release maps to one revision in the registry.

The following Run step uploads a model folder to your registry using the `huggingface_hub` library, with the revision sourced from the pipeline's codebase tag. Store the identity token as a Harness secret, and do not write it into the pipeline YAML.

```yaml
- step:
    type: Run
    name: Publish model
    identifier: publish_model
    spec:
      shell: Sh
      envVariables:
        HF_ENDPOINT: https://pkg.harness.io/pkg/<ACCOUNT_ID>/<REGISTRY_NAME>/huggingface
        HF_TOKEN: <+secrets.getValue("hf_identity_token")>
        PACKAGE_VERSION: <+codebase.tag>
      command: |
        python -c "
        from huggingface_hub import HfApi
        api = HfApi()
        api.upload_folder(folder_path='<FOLDER_PATH>', repo_id='<ARTIFACT_NAME>', revision='$PACKAGE_VERSION', repo_type='model')
        "
```

## Troubleshooting

<details>

<summary>A model or dataset upload fails with `401 Client Error`</summary>

The identity token in `HF_TOKEN` is missing, expired, or revoked. Generate a new token from **Setup Client** and export it again before you retry the upload.

</details>

<details>

<summary>A model or dataset download fails with `RepositoryNotFoundError`</summary>

Hugging Face registries do not support an upstream proxy, so a model or dataset that was never uploaded to your Harness registry cannot resolve from `huggingface.co` automatically. Confirm that the `repo_id` matches an artifact already published to the registry, and that `HF_ENDPOINT` points at the correct registry.

</details>

<details>

<summary>A model or dataset resolution times out</summary>

Increase `HF_HUB_ETAG_TIMEOUT` and `HF_HUB_DOWNLOAD_TIMEOUT`, and verify network egress from your client to the Harness registry endpoint.

</details>

<details>

<summary>A large file upload fails without Git LFS</summary>

Confirm that Git LFS is installed and that the LFS filter is configured for your repository before you retry `api.upload_folder`. The Hugging Face CLI uses Git LFS automatically for large files once the filter is in place.

</details>

<details>

<summary>`load_dataset` fails to resolve an externally hosted dataset</summary>

Harness Artifacts fully caches only datasets hosted directly on Hugging Face. If the dataset script references external sources, download those files from their original location, or use `snapshot_download` to pull the files already cached in your registry.

</details>

## Next steps

* To add a pipeline that publishes a new version for every release tag, go to [Harness CI](https://developer.harness.io/artifact-registry/3.0/platform-integrations/harness-ci.md).
* To control who can publish and consume packages in your registry, go to [RBAC and permissions](https://developer.harness.io/artifact-registry/3.0/reference/rbac-and-permissions.md).
* To set retention and lifecycle rules for old versions, go to [Lifecycle Management overview](https://developer.harness.io/artifact-registry/3.0/lifecycle-management/overview.md).

{% @harness-feedback/feedback %}
