Skip to main content

Integrate ML platforms with Harness CI

Last updated on

Harness CI runs your machine learning workload on the platform that already hosts it. A Plugin step submits the training job, deployment, or tracking call to Azure ML, AWS SageMaker, Databricks, Google Vertex AI, or MLflow, so the model moves through the same governed pipeline as the rest of your software.

This page covers the Harness side of each integration. The data science work stays on your ML platform, and each section links to the vendor documentation for the platform-specific detail.

What you will learn from this topic​

  • Plugin configuration: The plugin image and settings for each supported ML platform.
  • Pipeline placement: Where the training step sits relative to evaluation, packaging, and deployment.
  • Credential handling: How to pass cloud credentials to a plugin without writing them into YAML.
  • Evaluation and promotion: How to turn model metrics into a pipeline gate.

Before you begin​

  • Harness CI project access: Permission to create and run pipelines. Go to RBAC in Harness to configure roles.
  • CI pipeline familiarity: Go to CI pipeline components to review stages and steps.
  • Docker connector: Each plugin runs as a container image. Go to Docker connector settings to create one.
  • ML platform account: An active account on your chosen platform, with a prepared workspace or project.
  • Training dataset: Prepare and store training datasets according to your model type and ML framework, and record an immutable path or version identifier for the dataset you train against.
  • Secrets: Store every credential as a text secret before you configure a plugin.
tip

Plugin settings accept Harness expressions. Reference a stage variable with <+stage.variables.trackingUri>, and reference a text secret with <+secrets.getValue("secret_id")>. Use expressions for every credential and for any value that differs between environments.

info

The plugin images in this guide are published to the harnesscommunity Docker Hub organization and are community-maintained. Each image publishes only a latest tag. Before you depend on one in a production pipeline, confirm the image still meets your needs, and consider mirroring it to your own registry so a rebuild upstream cannot change your pipeline behavior without warning.

Prepare your ML platform​

Complete the platform setup before you add a step to Harness. Each platform needs a workspace or project, compute, and credentials that the plugin can use.

  1. Go to Create Azure ML resources to set up your workspace and compute instance.
  2. Note your subscription ID, resource group, and workspace name. The plugin requires all three.
  3. Go to Data concepts in Azure ML to register your dataset as a data asset.

Train a model from your pipeline​

Add a Plugin step to a CI stage. The plugin submits the job to your ML platform and waits for the result, so the pipeline fails when training fails.

              - step:
                  type: Plugin
                  name: Azure ML training job
                  identifier: azure_ml_plugin
                  spec:
                    connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
                    image: harnesscommunity/azure-ml
                    settings:
                      username: <+secrets.getValue("azure_ml_user")>
                      password: <+secrets.getValue("azure_ml_pass")>
                      tenant_id: <+secrets.getValue("azure_ml_tenant")>
                      SUBSCRIPTION_ID: <+secrets.getValue("azure_ml_subscription")>
                      AZURE_ML_WORKSPACE_NAME: my-azure-workspace
                      RESOURCE_GROUP: my-azure-resource-group
                      PROJECT_PATH: https://github.com/Azure/azureml-examples
                      TRAINING_JOB_FILE: azureml-examples/cli/jobs/single-step/scikit-learn/iris/job.yml
                      MODEL_NAME: iris-model-test
                      ENDPOINT_NAME: iris-endpoint-test
                      ENDPOINT_YAML: azureml-examples/cli/endpoints/online/managed/sample/endpoint.yml
                      DEPLOYMENT_NAME: deploy-iris
                      DEPLOYMENT_YAML: azureml-examples/cli/endpoints/online/managed/sample/blue-deployment.yml
                    imagePullPolicy: Always

Settings:

  • connectorRef: A Docker connector.
  • image: harnesscommunity/azure-ml.
  • username, password, tenant_id, SUBSCRIPTION_ID: Azure credentials and subscription. Pass all four as secrets.
  • AZURE_ML_WORKSPACE_NAME, RESOURCE_GROUP: The workspace and resource group you created.
  • PROJECT_PATH: Repository URL for your Azure ML project.
  • TRAINING_JOB_FILE: Path in the project repository to the training job definition.
  • MODEL_NAME: Name to register the trained model under.
  • ENDPOINT_NAME, ENDPOINT_YAML: Endpoint name and its definition file.
  • DEPLOYMENT_NAME, DEPLOYMENT_YAML: Deployment name and its definition file.
Sample training script

This script trains a credit card approval classifier with scikit-learn and logs metrics through MLflow, which Azure ML uses for run tracking in SDK v2. It assumes the dataset is preprocessed, contains features relevant to approval decisions such as income, credit score, and debt level, and has a binary ApprovalStatus target. It also assumes scikit-learn, pandas, and mlflow are installed in the job environment or listed in requirements.txt.

import argparse
import joblib
import mlflow
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import train_test_split

parser = argparse.ArgumentParser()
parser.add_argument("--data", type=str, help="Path to the registered data asset")
args = parser.parse_args()

data = pd.read_csv(args.data)

X = data.drop(columns=["ApprovalStatus"])
y = data["ApprovalStatus"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LogisticRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)

mlflow.log_metric("accuracy", accuracy_score(y_test, y_pred))
mlflow.log_text(classification_report(y_test, y_pred), "classification_report.txt")

joblib.dump(model, "model.joblib")
mlflow.log_artifact("model.joblib")

Adapt the feature engineering, model parameters, and metrics to your use case, and pass the data asset path through the --data argument in your job.yml. Go to Train models with Azure ML to define the job.

Evaluate the trained model​

Evaluation turns a trained model into a promotion decision. The workflow is the same on every platform:

  1. Prepare evaluation data: Store a held-out dataset in a format your model accepts. Confirm the model has not seen it during training, and preprocess it exactly as you preprocessed the training data.
  2. Select metrics: Choose metrics that match the prediction task. Binary classification tasks such as credit card approval commonly use accuracy, precision, recall, F1 score, and area under the ROC curve. Consider business metrics that weigh false positives against false negatives.
  3. Run the evaluation and fail on a miss: Compute the metrics, compare them against your threshold, and exit non-zero when the model misses it. This is what converts a metric into a pipeline gate.
  4. Iterate: Adjust hyperparameters, algorithms, or preprocessing, then retrain and re-evaluate.
info

Model evaluation is iterative and might require multiple rounds of training, evaluation, and tuning to reach the performance you need. Confirm the metrics you choose align with your project objectives before you set a threshold, because a threshold on the wrong metric passes models that fail in production.

The following example computes metrics and enforces a threshold. Run it in a CI Run step after training so a failing model stops the pipeline:

import json
import os
import sys

import joblib
import pandas as pd
from sklearn.metrics import accuracy_score, roc_auc_score

MINIMUM_ACCURACY = 0.85

model = joblib.load(os.path.join(os.environ["MODEL_DIR"], "model.joblib"))

data = pd.read_csv(os.environ["TEST_DATA_PATH"])
X_test, y_test = data.drop(columns=["ApprovalStatus"]), data["ApprovalStatus"]

predictions = model.predict(X_test)
metrics = {
    "accuracy": accuracy_score(y_test, predictions),
    "roc_auc": roc_auc_score(y_test, predictions),
}

with open("evaluation.json", "w") as f:
    json.dump(metrics, f)

print(f"Evaluation metrics: {metrics}")

if metrics["accuracy"] < MINIMUM_ACCURACY:
    sys.exit(f"Accuracy {metrics['accuracy']:.3f} is below the {MINIMUM_ACCURACY} threshold")

Platform-native evaluation tooling is also available:

Go to View metrics for jobs and runs to review logged metrics in Azure ML studio.

Deploy the model and get predictions​

Once a model passes evaluation, deploy it and call its endpoint for predictions. Harness CD manages the deployment and rollback for container and serverless targets, and each ML platform also offers a managed serving path.

Go to Deploy models with online endpoints for managed serving, or to Package and deploy models outside Azure ML to serve from your own infrastructure. The plugin handles endpoint creation when you supply ENDPOINT_YAML and DEPLOYMENT_YAML.

Monitor, improve, and iterate​

Set up monitoring and logging for every deployed model so you detect degradation before it reaches a business metric. Then feed what you learn back into retraining, using triggers or scheduled pipelines to rerun the training stage when drift crosses a threshold.

Troubleshooting​

A Harness CI Plugin step fails immediately with an authentication or permission error against my cloud ML platform

Confirm the credentials are stored as Harness text secrets and referenced with a secrets.getValue expression, and that the underlying cloud identity has permission for both the ML service and the storage bucket the job reads. A job that can start training but cannot write output usually has storage permissions missing rather than ML service permissions.

My ML platform Plugin step cannot pull the harnesscommunity plugin image in a Harness CI pipeline

Confirm the step references a working Docker connector in connectorRef and that the build infrastructure can reach Docker Hub. If your network blocks Docker Hub, mirror the plugin image to your own registry and point connectorRef and image at the mirror.

My model training step passes in the Harness pipeline but the model performs worse in production than in evaluation

Compare the dependency versions between the training image and the serving image, and confirm evaluation data is preprocessed identically to production input. Divergence in either place produces a correct model that returns different predictions once deployed.

MLflow runs from my Harness pipeline do not appear in the MLflow tracking UI

Set MLFLOW_TRACKING_URI to a remote tracking server that the build machine can reach. Without it MLflow writes to the local filesystem of an ephemeral build pod, and the run is discarded when the step ends.

My SageMaker training job in a Harness pipeline fails with an entry point or source directory error

entry_point takes a filename relative to source_dir, not an s3:// URL. Set source_dir to the local directory holding your script and its requirements.txt, and set entry_point to the script filename alone.

Next steps​

Wire the integration into a complete pipeline, then add the governance controls that make model promotion auditable.

Last updated on