Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.
ArtifactsRegistry TypesGetting Started

Hugging Face

Create a Hugging Face registry, generate an identity token, then host and retrieve private models and datasets with huggingface_hub client libraries.

Use a Hugging Face registry to host private models and datasets for use with Hugging Face Hub client libraries, such as huggingface_hub, transformers, and datasets.

What you will learn from this topic

By the end of this quickstart guide, you will be able to:

  • Create a registry in Harness Artifacts.

  • Publish a package to the registry.

  • Install the package from the registry, and confirm the working.

Before you begin

  • Install the Hugging Face CLI. Running pip install -U huggingface_hub installs the hf command and the huggingface_hub Python library used throughout this quickstart.

Additional installation steps

Depending on your environment, you might need additional setup steps. To configure your environment correctly, go to Hugging Face installation for details.

  • Make sure you have access to a Harness account with permission to create registries and generate identity tokens.

Create a Hugging Face Registry

To create a Huggingface Registry, perform the following steps:

  1. In Artifact Registry, select Registries.

  2. Select + Create Registry.

  3. Specify the Registry name.

Registry name criteria

The registry name must start with a letter, can only contain lowercase alphanumerics, _, ., and -, and must be unique to your Harness account.

  1. Optionally, add a Description.

  2. Choose visibility between Public and Private. The default is Private.

  3. Select Huggingface as the Registry type.

  4. Select Save.

  5. Select Create to finalize.

Private registry

A Huggingface registry serves as your private registry within Harness Artifacts. It does not publish content to the public Hugging Face Hub.

Huggingface client setup

In your Hugging Face Artifact, select the registry, then select Set Up Client. The dialog gives you an identity token, then setup instructions for publishing and using models and datasets with the huggingface_hub library.

Generate an identity token

An identity token serves as the password for authenticating with the registry.

  1. In your Hugging Face Artifact, select Set Up Client.

  2. Select Generate Token.

  3. Select an Expiration for the token:

    • 30 Days

    • 90 Days

    • 180 Days

    • Custom

  4. Copy the token. You use it in the following steps in place of <IDENTITY_TOKEN>.

Token expiration

The identity token stops working after the expiration you select. Generate a new token and update your client configuration before the current token expires.

Configure the client

Select Models or Datasets for what you want to publish or use. In the code samples, replace the values in angle brackets with your own values.

1. Configure environment variables

Set the following environment variables:

Client version requirement

For Hugging Face client version 0.19.0 and later, these timeout settings enable model resolution through the transformers and tokenizers libraries.

2. Publish the model

Upload a model to the registry using the huggingface_hub library:

3. Use the model

Download the model from the registry:

Model resolution through client libraries

With Hugging Face client version 0.19.0 and later, and HF_HUB_ETAG_TIMEOUT set, you can resolve models directly through the transformers and diffusers libraries.

1. Configure environment variables

Set the following environment variables:

2. Publish the dataset

Upload a dataset to the registry using the huggingface_hub library:

3. Use the dataset

Load the dataset directly with the datasets library, or download the full dataset repository:

Setup verification

Confirm that your registry serves the model or dataset you published. To verify your setup, perform the following steps:

  1. In a new environment with no cached Hugging Face files (clear HF_HOME, or point it to an empty directory), use the huggingface_hub or datasets library to resolve the model or dataset you published.

  2. Confirm that the client downloaded the model or dataset from your Harness registry endpoint, which is shown in Set Up Client.

  3. In Harness, open your registry and confirm that the model or dataset, and the revision you published, appear in the package list.

If a step fails, go to Troubleshooting to resolve common authentication and resolution issues before you retry.

Pipeline automation (optional)

Publishing from a pipeline gives you a repeatable release process. Run the publish step when you create a version tag, so each release maps to one revision in the registry.

The following Run step uploads a model folder to your registry using the huggingface_hub library, with the revision sourced from the pipeline's codebase tag. Store the identity token as a Harness secret, and do not write it into the pipeline YAML.

Troubleshooting

A model or dataset upload fails with `401 Client Error`

The identity token in HF_TOKEN is missing, expired, or revoked. Generate a new token from Set Up Client and export it again before you retry the upload.

A model or dataset download fails with `RepositoryNotFoundError`

Hugging Face registries do not support an upstream proxy, so a model or dataset that was never uploaded to your Harness registry cannot resolve from huggingface.co automatically. Confirm that the repo_id matches an artifact already published to the registry, and that HF_ENDPOINT points at the correct registry.

A model or dataset resolution times out

Increase HF_HUB_ETAG_TIMEOUT and HF_HUB_DOWNLOAD_TIMEOUT, and verify network egress from your client to the Harness registry endpoint.

A large file upload fails without Git LFS

Confirm that Git LFS is installed and that the LFS filter is configured for your repository before you retry api.upload_folder. The Hugging Face CLI uses Git LFS automatically for large files once the filter is in place.

`load_dataset` fails to resolve an externally hosted dataset

Harness Artifacts fully caches only datasets hosted directly on Hugging Face. If the dataset script references external sources, download those files from their original location, or use snapshot_download to pull the files already cached in your registry.

Next steps

Last updated

Was this helpful?