> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-integration/3.0/troubleshooting-and-resources/tutorials-and-code-samples/mlops/e2e-mlops-tutorial.md). # Tutorial - End-to-end MLOps CI/CD pipeline with Harness and AWS Build, test, govern, and deploy a machine learning model to AWS Lambda with a Harness CI/CD pipeline. MLOps tackles the complexities of building, testing, deploying, and monitoring machine learning models in real-world environments. Integrating machine learning into the traditional software development lifecycle poses unique challenges due to the intricacies of data, model versioning, scalability, and ongoing monitoring. In this tutorial, you build an end-to-end MLOps CI/CD pipeline that trains a credit card approval model, gates its promotion on accuracy and fairness, and deploys it to AWS Lambda. *** ## What you will learn from this topic * **Model build and scan:** Build the model image, push it to AWS ECR, and scan it with Security Testing Orchestration. * **Quality gates:** Export accuracy and fairness metrics as output variables, then enforce thresholds with Open Policy Agent policies. * **Deployment:** Deploy the model to AWS Lambda with a Harness CD stage. * **Monitoring and triggers:** Alert on a stale model and rerun the pipeline from a Git event. *** ## The story This tutorial uses a fictional bank called *Harness Bank*. Assume that this fictional bank recently launched a website where clients can apply for a credit card. Based on the information provided in the form, the customer's application is approved or denied in seconds. This online credit card application is powered by a machine learning (ML) model trained on data that makes the decision accurate and unbiased. Assume that the current process to update this hypothetical ML model is manual. A data scientist builds a new image locally, runs tests, and manually ensures that the model passes the required threshold for accuracy and fairness. In this tutorial, you automate the model maintenance process and increase the build and delivery velocity. ### Design and architecture Review the MLOps architecture before you start the implementation. ```mermaid graph LR; Data_Preparation --> Train_Model; Train_Model --> Unit_Integration_Test; Unit_Integration_Test --> Deploy_Model; Deploy_Model --> Monitor_Model; ``` For this tutorial, assume you are given a Python data science project, and you are requested to do the following: * Build and push an image for this project. * Run security scans on the container image. * Upload model visualization data to S3. * Publish model visualization data within the pipeline. * Run tests on the model to determine accuracy and fairness scores. * Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model. * Deploy the model. * Monitor the model and ensure the model is not outdated. * Trigger the pipeline based on Git events. * (Optional) Add approval gates for production deployment. For this tutorial, assume that the data is already processed. *** ## Before you begin * **Harness account:** Access to the Continuous Integration, Continuous Delivery, and Security Testing Orchestration modules. Go to the [Harness signup page](https://app.harness.io/auth/#/signup/?\&utm_campaign=cicd-devrel) to create a free account. * **AWS access:** An AWS account and credentials with permission to create ECR, S3, and Lambda resources. Go to [Prepare AWS](#prepare-aws) to complete the setup. * **GitHub access:** A GitHub account and a personal access token. Go to [Prepare GitHub](#prepare-github) to fork the sample repository. * **Harness delegate:** Required for the **Custom** stage that runs policy checks. Go to [Delegate overview](/harness-platform/3.0/harness-platform-resources/delegates/delegate-closed-beta/delegate-overview.md) to review delegate requirements. ### Prepare AWS You need an AWS account with sufficient permissions to create/modify/view resources used in this tutorial. 1. Prepare AWS credentials. This tutorial requires two sets of AWS credentials. One set is for a [Harness AWS connector](/harness-platform/3.0/harness-platform-resources/connectors/cloud-providers/add-aws-connector.md), and the other is for the [AWS ECR scanner for STO](/security-testing-orchestration/3.0/use-sto/sto-scanner-configuration/aws-ecr-scanner-reference.md). You can use an AWS Vault plugin to generate AWS credentials for the AWS connector, and you can use the AWS console to generate the AWS Access Key ID, AWS Secret Access Key, and AWS Session Token, which are valid for a shorter time. **Save these credentials securely and make a note of your AWS account ID and AWS region.**

If you are using a personal, non-production AWS account for this tutorial, you can initially grant admin access for these credentials. Once the demo works, reduce access to adhere to the principle of least privilege.

2. Create ECR repos. From your AWS console, navigate to Elastic Container Registry (ECR) and create two private repositories named `ccapproval` and `ccapproval-deploy`. Under **Image scan settings**, enable **Scan on Push** for both repositories. 3. Create an S3 bucket. Navigate to S3 and create a bucket named something like `mlopswebapp`. Use this bucket to host a static website for the credit card approval application demo, along with a few other artifacts. Make sure all options under **Block public access (bucket settings)** are unchecked, and then apply the following bucket policy: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::YOUR_S3_BUCKET_NAME/*" } ] } ``` After making the bucket public, your bucket page should show a `Publicly accessible` flag. ![S3 bucket is public](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-cea0a24e1ac9b9accfe5ef60b3b33ea816f0c91c%2Fs3bucketpublic.png?alt=media) 4. From your AWS console, go to **AWS Lambda**, select **Functions**, and create a function from a container image using the following configuration: * Name: `creditcardapplicationlambda` * Container image URI: Select **Browse images** and find the `ccapproval-deploy` image. You can choose any image tag. * Architecture: `x86_64` * From Advanced: Select **Enable function URL** to make the function URL public. Anyone with the URL can access your function. For more information, go to the [AWS documentation on Lambda function URLs](https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html). 5. Select **Create function** to create the function. An info banner appears confirming that the function URL is public. ![Lambda Function URL](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-01ee3a4dbb95749583c530804374ea2e1b1d89b9%2Flambda-function-url.png?alt=media) ### Prepare GitHub This tutorial uses a GitHub account for source control management. 1. Fork the [MLops sample app repository](https://github.com/harness-community/mlops-creditcard-approval-model) into your GitHub account. 2. Create a [GitHub personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) with following permissions on your forked repository: * repo/content: read+write * repo/pull requests: read * repo/webhooks: read+write ### Create Harness secrets Store your GitHub and AWS credentials as secrets in Harness. 1. In your Harness account, create or select a [project](/harness-platform/new-to-harness-platform/get-started.md#step-2-create-organization-project-and-invite-collaborators) to use for this tutorial. 2. In your project settings, select **Secrets**, select **New Secret**, and then select **Text**. 3. Create the following [Harness text secrets](/harness-platform/3.0/harness-platform-resources/secrets/add-use-text-secrets.md): * `git_pat` - GitHub personal access token * `aws_access_key_id` - Generated from AWS console * `aws_secret_access_key` - Generated from AWS console * `aws_session_token` - Generated from AWS console * `aws_vault_secret` - Secret access key generated by Vault plugin Make sure the **Name** and **ID** match for each secret, because you reference secrets by their IDs in Harness pipelines. ### Create AWS and GitHub connectors Create Harness connectors to connect to your AWS and GitHub accounts. 1. In your Harness project settings, go to **Connectors**. 2. Select **New Connector**, select the **AWS** connector, and then create an [AWS connector](/harness-platform/3.0/harness-platform-resources/connectors/cloud-providers/add-aws-connector.md) with the following configuration: * Name: `mlopsawsconnector` * Access Key: AWS Vault plugin generated * Secret Key: Use your `aws_vault_secret` secret * Connectivity Mode: Connect through Harness Platform Leave all other settings as is, and make sure the connection test passes. ![Connector connectivity status](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-ced19fe92570ba9a8d0133a086c26d1cf570808f%2Fconnector-connectivity-status.png?alt=media) 3. Create another connector. This time, select the [GitHub connector](/harness-platform/3.0/harness-platform-resources/connectors/code-repositories/ref-source-repo-provider/git-hub-connector-settings-reference.md) and use the following configuration: * Name: `mlopsgithubconnector` * URL Type: `Repository` * Connection Type: `HTTP` * GitHub Repository URL: Enter the URL to your fork of the demo repo, such as `https://github.com/:gitHubUsername/mlops-creditcard-approval-model` * Username: Enter your GitHub username * Personal Access Token: Use your `git_pat` secret * Connectivity Mode: Connect through Harness Platform *** ## Create the Harness pipeline In Harness, you create pipeline to represent workflows. Pipeline can have multiple stages, and each stage can have multiple steps. 1. In your Harness project, [create a pipeline](/continuous-integration/3.0/use-harness-ci/use-harness-ci/prep-ci-pipeline-components.md) named `Credit Card Approval MLops`. 2. Add a **Build** stage named `Train Model`. 3. Make sure **Clone Codebase** is enabled. 4. Select **Third-party Git provider**, and then select your `mlopsgithubconnector` GitHub connector. The repository name should populate automatically. 5. Select **Set Up Stage**. In the following sections of this tutorial, you configure this stage to build and push the data science image, and you add more stages to the pipeline to meet the [tutorial's objectives](#design-and-architecture). {% hint style="success" %} You can find a [sample pipeline for this tutorial in the demo repo](https://github.com/harness-community/mlops-creditcard-approval-model/blob/main/sample-mlops-pipeline.yaml). If you use this pipeline, you must replace the placeholder and sample values accordingly. {% endhint %} ### Build, push, and scan the image Configure your `Train Model` stage to build and push the data science image and then retrieve the ECR scan results. 1. Select the **Infrastructure** tab and configure the build infrastructure for the `Train Model` stage: * Select **Cloud** to use [Harness Cloud build infrastructure](/continuous-integration/3.0/use-harness-ci/use-harness-ci/set-up-build-infrastructure/use-harness-cloud-build-infrastructure.md). * For **Platform**, select **Linux**. * For **Architecture**, select **AMD64**. 2. Select the **Execution** tab to add steps to the stage. 3. Select **Add Step**, select the [Build and Push to ECR step](/continuous-integration/3.0/use-harness-ci/use-harness-ci/build-and-upload-artifacts/build-and-push/build-and-push-to-ecr-step-settings.md), and configure the step as follows: * For **Name**, enter `Harness Training`. * For **AWS Connector**, select your `mlopsawsconnector` AWS connector. * For **Region**, enter your AWS region. * For **Account ID**, enter your AWS account ID. * For **Image Name**, enter `ccapproval`. * For **Tags**, enter `<+pipeline.executionId>`. You can select the **Input type** icon to change the input type to expression (**f(x)**). * For **Dockerfile** (under **Optional Configuration**), enter `Dockerfile_Training_Testing`. 4. Select **Apply Changes** to save the step, and then select **Save** to save the pipeline. Next, add steps to your build stage to retrieve the results of the ECR repo security scan. Because scanning is enabled on your ECR repositories, each image pushed to the repo by the **Build and Push to ECR** step is scanned for vulnerabilities. In order to successfully retrieve the scan results, your pipeline needs to wait for the scan to finish and then request the results. 5. Before adding the step to retrieve the scan result, use a [Run step](/continuous-integration/3.0/use-harness-ci/use-harness-ci/run-step-settings.md) to add a 15-second wait to ensure that the scan is complete before the pipeline requests the scan results. * Select **Add Step** after the `Harness Training` step, and select the **Run** step. * For **Name**, enter `Wait for ECR Image Scan`. * For **Command**, enter the following, and then select **Apply Changes** to save the step. ```bash echo "ECR Image Scan In Progress..." sleep 15 ``` 6. Add an [AWS ECR Scan step](/security-testing-orchestration/3.0/use-sto/sto-scanner-configuration/aws-ecr-scanner-reference.md) to get the scan results. * Select **Add Step** after the `Wait` step, and select the **AWS ECR Scan** step. * For **Name**, enter `Security Scans for ML Model`. * For **Target/Name**, enter `ccapproval-ecr-scan`. * For **Variant**, enter `<+pipeline.executionId>`. You can select the **Input type** icon to change the input type to expression (**f(x)**). * For **Container Image/Name**, enter `ccapproval`. * For **Container Image/Tag**, enter `<+pipeline.executionId>`. * For **Region**, enter your AWS region. * Under **Authentication**, use [Harness expressions](/harness-platform/3.0/harness-platform-resources/variables-and-expressions/harness-variables.md) referencing your AWS credential secrets: * For **Access ID**, enter `<+secrets.getValue("aws_access_key_id")>`. * For **Access Token**, enter `<+secrets.getValue("aws_secret_access_key")>`. * For **Access Region**, enter your AWS region. * For **Log Level**, enter `Info`. * Under **Settings**, add the following key-value pair: `AWS_SESSION_TOKEN: <+secrets.getValue("aws_session_token")>` 7. Select **Apply Changes** to save the step, and then select **Save** to save the pipeline. At this point, you can run the pipeline to test the Build stage. 8. Select **Run Pipeline** to test the Build stage. For **Git Branch**, enter `main`. 9. Wait while the pipeline runs, and then check your `ccapproval` ECR repository to find an image with a SHA matching the pipeline execution ID. Select **Copy URI** to copy the image URI; you need it in the next section. 10. Make sure the image scan also ran. In the Harness [Build details](/continuous-integration/3.0/use-harness-ci/use-harness-ci/viewing-builds.md), you can find the scan results in the **AWS ECR Scan** step logs. For example: ```shell Scan Results: { "jobId": "xlf06YX6a8AupG_5igGA6I", "status": "Succeeded", "issuesCount": 10, "newIssuesCount": 10, "issuesBySeverityCount": { "ExternalPolicyFailures": 0, "NewCritical": 0, "NewHigh": 1, "NewMedium": 5, "NewLow": 4, "NewInfo": 0, "Unassigned": 0, "NewUnassigned": 0, "Critical": 0, "High": 1, "Medium": 5, "Low": 4, "Info": 0, "Ignored": 0 } } ``` You have successfully completed the first part of this tutorial: Configuring a Build stage that builds, pushes, and scans a trained data science image. Continue the tutorial in the next sections and continue building your MLOps pipeline. ### Test and upload artifacts Add another **Build** stage to your pipeline that will run tests, build a Lambda image, and upload artifacts to S3. 1. Edit your MLOps pipeline and add another **Build** stage after the `Train Model` stage. Name the stage `Run test and upload artifacts` and make sure **Clone Codebase** is enabled. 2. On the stage's **Overview** tab, locate **Shared Paths**, and add `/harness/output`. 3. On the **Infrastructure** tab, select **Propagate from existing stage**, and select your `Train Model` stage. 4. On the **Execution** tab, add a **Run** step to run pytest on the demo copebase. Select **Add Step**, select the **Run** step, and configure it as follows: * For **Name**, enter `pytest`. * For **Shell**, select **Sh**. * For **Command**, enter: ``` pytest --nbval-lax credit_card_approval.ipynb --junitxml=report.xml ``` * Under **Optional Configuration**, locate **Container Registry**, and select your `mlopsawsconnector` AWS connector. * Under **Optional Configuration**, locate **Image**, and enter the image URI from your `Train Model` stage execution with the image tag replaced with `<+pipeline.executionId>`. For example: ``` AWS_ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/AWS_ECR_REPO_NAME:<+pipeline.executionId> ``` The data science project includes two Dockerfiles: One for building the source and one for AWS Lambda deployment. Next, add a step to build and push the image using the Dockerfile designed for AWS Lambda deployment. 5. Select **Add Step** and add a **Build and Push to ECR** step configured as follows: * For **Name**, enter `Build and Push Lambda Deployment Image`. * For **AWS Connector**, select your `mlopsawsconnector` AWS connector. * For **Region**, enter your AWS region. * For **Account ID**, enter your AWS account ID. * For **Image Name**, enter `ccapproval-deploy`. * For **Tags**, enter `<+pipeline.executionId>`. * Under **Optional Configuration**, locate **Dockerfile**, and enter `Dockerfile_Inference_Lambda`. The `pytest` command from the **Run** step generates an HTML file with some visualizations for the demo ML model. Next, add steps to upload the visualizations artifact to your AWS S3 bucket and post the artifact URL on the Artifacts tab of the [Build details page](/continuous-integration/3.0/use-harness-ci/use-harness-ci/viewing-builds.md). 6. Select **Add Step**, and add an [Upload Artifacts to S3 step](/continuous-integration/3.0/use-harness-ci/use-harness-ci/build-and-upload-artifacts/upload-artifacts/upload-artifacts-to-s3.md) configured as follows: * For **Name**, enter `Upload artifacts to S3`. * For **AWS Connector**, select your `mlopsawsconnector` AWS connector. * For **Region**, enter your AWS region. * For **Bucket**, enter your S3 bucket name. * For **Source Path**, enter `/harness/output/model_metrics.html`. This is where the model visualization file from the `pytest` step is stored. 7. Use the [Artifact Metadata Publisher plugin](/continuous-integration/3.0/use-harness-ci/use-harness-ci/build-and-upload-artifacts/artifacts-tab.md) to post the visualization artifact URL on the build's Artifacts tab. * Add a **Plugin** step after the **Upload Artifacts to S3** step. * For **Name**, enter `Publish ML model visualization`. * For **Container Registry**, select the built-in **Harness Docker Connector**. * For **Image**, enter `plugins/artifact-metadata-publisher`. * Under **Optional Configuration**, locate **Settings**, and add the following key-value pairs: ``` file_urls: https://S3_BUCKET_NAME.s3.AWS_REGION.amazonaws.com/harness/output/model_metrics.html artifact_file: artifact.txt ``` In addition to the model visualization, the `pytest` command also generates a `shared_env_variables.txt` file to export the model's accuracy and fairness metrics. However, this data is lost when the build ends because Harness stages run in isolated containers. Therefore, you must add a step to export the `ACCURACY` and `EQUAL_OPPORTUNITY_FAIRNESS_PERCENT` values as [output variables](/continuous-integration/3.0/use-harness-ci/use-harness-ci/run-step-settings.md#output-variables). 8. After the **Plugin** step, add a **Run** step configured as follows: * For **Name**, enter `Export accuracy and fairness variables`. * For **Shell**, select **Sh**. * For **Command**, enter: ``` # File path FILE_PATH="/harness/output/shared_env_variables.txt" # Read the file and export variables while IFS='=' read -r key value; do case $key in ACCURACY) export ACCURACY="$value" ;; EQUAL_OPPORTUNITY_FAIRNESS_PERCENT) export EQUAL_OPPORTUNITY_FAIRNESS_PERCENT="$value" ;; *) echo "Ignoring unknown variable: $key" ;; esac done < "$FILE_PATH" echo $ACCURACY echo $EQUAL_OPPORTUNITY_FAIRNESS_PERCENT ``` * Under **Optional Configuration**, locate **Output Variables**, and add the following two output variables: ``` ACCURACY EQUAL_OPPORTUNITY_FAIRNESS_PERCENT ``` 9. Save the pipeline, and then run it. Again, use `main` for the **Git Branch**. 10. Wait while the pipeline runs, and then make sure: * Your `ccapproval` and `ccapproval-deploy` ECR repositories have images with SHAs matches the pipeline execution ID. * Your S3 bucket has `/harness/output/model_metrics.html`. * The URL to the `model_metrics` artifact appears on the Artifacts tab in Harness. ![Artifacts Tab](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-627e6bccc448e95693991c4fc92169f1820afa04%2Fartifacts-tab.png?alt=media) * The output variable values are in the log for the `Export accuracy and fairness variables` step, such as: ```shell 0.92662 20.799999999999997 ``` So far, you have completed half the requirements for this MLOps project: * [x] Build and push an image for this project. * [x] Run security scans on the container image. * [x] Upload model visualization data to S3. * [x] Publish model visualization data within the pipeline. * [x] Run tests on the model to determine accuracy and fairness scores. * [ ] Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model. * [ ] Deploy the model. * [ ] Monitor the model and ensure the model is not outdated. * [ ] Trigger the pipeline based on Git events. * [ ] (Optional) Add approval gates for production deployment. Continue on with policy enforcement in the next section. ### Add ML model policy checks In this section, you author OPA policies in Harness and use a **Custom** stage to add policy enforcement to your pipeline. Harness [Policy As Code](/harness-platform/3.0/harness-platform-resources/governance/policy-as-code/harness-governance-overview.md) uses Open Policy Agent (OPA) as the central service to store and enforce policies for the different entities and processes across the Harness platform. You create individual policies, add them to policy sets, and select the entities (such as pipelines) to evaluate those policies against. For this tutorial, the policy requirements are that the model accuracy is over 90% and the fairness margin for equal opportunity is under 21%. 1. In your Harness project settings, go to **Policies**, select the **Policies** tab, and then select **New Policy**. 2. For **Name**, enter `Check fairness and accuracy scores`. 3. For **How do you want to setup your Policy**, select **Inline**. 4. Enter the following policy definition, and then select **Save**. ```rego package main default allow = false allow { input.accuracy >= 0.9 input.fairnessScoreEqualOpportunity <= 21 } deny[msg] { not allow msg = "Deny: Accuracy less than 90% or fairness score difference greater than 21%" } ``` 5. Select the **Policy Sets** tab, and then select **New Policy Set**. Use the following configuration: * For **Name**, enter `Credit Card Approval Policy Set`. * For **Entity Type that this policy set applies to**, select **Custom**. * For **On what event should the policy set be evaluated**, select **On Step**. * Select **Add Policy**, and select your `Check fairness and accuracy scores` policy. * For **What should happen if a policy fails?**, select **Warn and Continue**. * Select **Finish**, and make sure the **Enforced** switch is enabled. 6. Edit your MLOps pipeline, and add a **Custom** stage after the second **Build** stage. Name the stage `Model Policy Checks`. 7. Select a Harness Delegate to use for the **Custom** stage. The **Build** stages run on Harness Cloud build infrastructure, which does not require a Harness Delegate. However, **Custom** stages cannot use this build infrastructure, so you need a [Harness Delegate](/harness-platform/3.0/harness-platform-resources/delegates/delegate-closed-beta/delegate-overview.md). If you do not already have one, [install a delegate](/harness-platform/3.0/harness-platform-resources/delegates/delegate-closed-beta/install-a-delegate.md). Then, on the **Custom** stage's **Advanced** tab, select your delegate in **Define Delegate Selector**. 8. Add a **Shell Script** step to relay the accuracy and fairness output variables from the previous stage to the current stage. Configure the **Shell Script** step as follows: * For **Name**, enter `Accuracy and Fairness`. * For **Timeout**, enter `10m`. * For **Script Type**, select **Bash**. * For **Select script location**, select **Inline**. * For **Script**, enter the following: ``` accuracy=<+pipeline.stages.Harness_Training.spec.execution.steps.Export_accuracy_and_fairness_variables.output.outputVariables.ACCURACY> fairness_equalopportunity=<+pipeline.stages.Harness_Training.spec.execution.steps.Export_accuracy_and_fairness_variables.output.outputVariables.EQUAL_OPPORTUNITY_FAIRNESS_PERCENT> ``` * Under **Optional Configuration**, locate **Script Output Variables**, and add the following two variables: * `accuracy` - String - `accuracy` * `fairness_equalopportunity` - String - `fairness_equalopportunity`

While you can feed the output variables directly into Policy steps, this Shell Script step is a useful debugging measure that ensures the accuracy and fairness variables are populated correctly.

9. Add a **Policy** step after the **Shell Script** step. * For **Name**, enter `Enforce Fairness and Accuracy Policy`. * For **Timeout**, enter `10m`. * For **Entity Type**, select **Custom**. * For **Policy Set**, select your `Credit Card Approval Policy Set`. * For **Payload**, enter the following: ``` { "accuracy": <+execution.steps.Accuracy_and_Fairness.output.outputVariables.accuracy>, "fairnessScoreEqualOpportunity": <+execution.steps.Accuracy_and_Fairness.output.outputVariables.fairness_equalopportunity> } ``` 10. Save the pipeline. The next time the pipeline runs, the policy is enforced to check if the model accuracy and fairness margins are within the acceptable limits. If not, the pipeline produces a warning and then continues (according to the policy set configuration). You could also configure the policy set so that the pipeline fails if there is a policy violation. If you want to test the response to a policy violation, you can modify the policy definition's `allow` section to be more strict, such as: ```rego allow { input.accuracy >= 0.95 input.fairnessScoreEqualOpportunity <= 19 } ``` Since the model accuracy is around 92% and the fairness margin is around 20%, this policy definition should produce a warning. Make sure to revert the change to the policy definition once you finish experimenting. *** ## Deploy AWS Lambda function In Harness, you can specify the location of a function definition, artifact, and AWS account, and then Harness deploys the Lambda function and automatically routes traffic from the old version of the Lambda function to the new version on each deployment. In this part of the tutorial, you update an existing Lambda function by adding a **Deploy** stage with service, environment, and infrastructure definitions. 1. Edit your MLOPs pipeline, and add a **Deploy** stage named `lambdadeployment`. 2. **Deployment Type**, select **AWS Lambda**, and then select **Continue**. 3. Create a [service definition](https://developer.harness.io/continuous-delivery/3.0/) for the Lambda deployment. * Select **Add Service**. * For **Name**, enter `creditcardapproval-lambda-service`. * For **Set up service**, select **Inline**. * For **Deployment Type**, select **AWS Lambda**. * Under **AWS Lambda Function Definition**, for **Manifest Identifier**, enter `lambdadefinition`, and for **File/Folder Path**, enter `/lambdamanifest`. 4. After creating the manifest under Harness File Store, add the following to the service manifest, and select **Save**: ``` functionName: `creditcardapplicationlambda` role: LAMBDA_FUNCTION_ARN ``` Replace `LAMBDA_FUNCTION_ARN` with your Lambda function's ARN. You can find the **Function ARN** when viewing the function in the AWS console. ![Lambda ARN](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-8a38dc49d8c1eec95b467b0b37cf4a00d60b51da%2Flambda-arn.png?alt=media) 5. Under the **Artifacts** section for the service definition, provide the artifact details to use for the lambda deployment: * Artifact Source Identifier: `ccapprovaldeploy` * Region: YOUR\_AWS\_REGION * Image Path: `ccapproval-deploy` * Value - Tag: `<+input>` ([runtime input](/harness-platform/3.0/harness-platform-resources/variables-and-expressions/runtime-inputs.md)) 6. Create environment and infrastructure definitions for the Lambda deployment. On the **Deploy** stage's **Environment** tab, select **New Environment**, and use the following environment configuration: ``` name: lambda-env type: PreProduction ``` 7. From the `lambda-env`, go to the **Infrastructure Definitions** tab, and add an infrastructure definition with the following configuration: ``` name: `aws-lambda-infra` deploymentType: `AwsLambda` type: AwsLambda spec: connectorRef: `mlopsawsconnector` region: YOUR_AWS_REGION ``` 8. Select **Save** to save the infrastructure definition. 9. On the **Deploy** stage's **Execution** tab, add an **AWS Lambda Deploy** step named `Deploy Aws Lambda` for the name. No other configuration is necessary. 10. Save and run the pipeline. For **Git Branch**, enter `main`, and for **Tag**, enter `<+pipeline.executionId>`, and then select **Run Pipeline**. You need to provide the image tag value because the service definition's **Tag** setting uses runtime input (`<+input>`). While the pipeline runs, you can observe the build logs showing the lambda function being deployed with the latest artifact that was built and pushed from the same pipeline. 11. Test the response from the lambda function. * In your AWS console, go to **AWS Lambda**, select **Functions**, and select your `creditcardapplicationlambda` function. * On the **Test** tab, select **Create new event**, and create an event named `testmodel` with the following JSON: ``` { "Num_Children": 2, "Income": 500000, "Own_Car": 1, "Own_Housing": 1 } ``` * Select **Test** to execute the function with your `testmodel` test event. Once the function finishes execution, you get the result with a **Function URL**. ![Test Lambda Function](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-10a9ef9758dd829d31d1e8ade58b4e0c50966cf7%2Flambda-execution.png?alt=media) 12. Note the **Function URL** resulting from the lambda function test. This is the endpoint that your [ML web application](#use-the-ml-model-in-a-web-application) would call. Depending on the prediction of `0` or `1`, the web application either approves or denies the demo credit card application. *** ## Monitor the model There are many ways to monitor ML models. In this tutorial, you monitor whether the model was recently updated. If it has not been updated recently, Harness sends an email alerting you that the model might be stale. 1. Edit your MLOPs pipeline, and add a **Build** stage after the **Deploy** stage. * For **Name**, enter `Monitor Model stage`. * *Disable* **Clone Codebase**. * On the **Infrastructure** tab, select **Propagate from existing stage** and select the first **Build** stage. 2. Add a **Run** step to find out when the model was last updated. * For **Name**, enter `Monitor Model step`. * For **Shell**, select **Sh**. * For **Command**, enter: ``` # GitHub repository owner OWNER="YOUR_GITHUB_USERNAME" # GitHub repository name REPO="mlops-creditcard-approval-model" # Path to the file you want to check (relative to the repository root) FILE_PATH="credit_card_approval.ipynb" # GitHub Personal Access Token (PAT) TOKEN=<+secrets.getValue("git_pat")> # GitHub API URL API_URL="https://api.github.com/repos/$OWNER/$REPO/commits?path=$FILE_PATH&per_page=1" # Get the current date CURRENT_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ") # Calculate the date 7 days ago SEVEN_DAYS_AGO=$(date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ") # Get the latest commit date for the file LATEST_COMMIT_DATE=$(curl -s -H "Authorization: token $TOKEN" $API_URL | jq -r '.[0].commit.committer.date') # Check if the file has been updated in the last 7 days if [ "$(date -d "$LATEST_COMMIT_DATE" +%s)" -lt "$(date -d "$SEVEN_DAYS_AGO" +%s)" ]; then export model_stale=true else export model_stale=false fi ``` * Under **Optional Configuration**, add `model_stale` to **Output Variables**. 3. After the `Monitor Model` stage, add a **Custom** stage named `Email notification`. This stage will send the email notification if the model is stale. 4. Add an **Email** step to the last **Custom** stage. * For **Name**, enter `Email`. * For **Timeout**, enter `10m`. * For **To**, enter the email address to receive the notification, such as the email address for your Harness account. * For **Subject**, enter `Credit card approval ML model has not been updated in a week.` * For **Body**, enter `It has been 7 days since the credit card approval ML model was updated. Update the model.` * On the step's **Advanced** tab, add a [conditional execution](/harness-platform/3.0/harness-platform-resources/overview/strategy/conditional-execution.md) so the **Email** step only runs if the `model_stale` variable (from the `Monitor Model` step) is `true`: * For **Execute this step**, select **If the stage executes successfully up to this point**. * Select **And execute this step only if the following JEXL Condition evaluates to true**. * Enter the following JEXL condition: ``` <+pipeline.stages.Monitor_Model_Stage.spec.execution.steps.Monitor_Model_Step.output.outputVariables.model_stale> == true ``` 5. Save and run the pipeline. For **Git Branch**, enter `main`, and for **Tag**, enter `<+pipeline.executionId>`. *** ## Trigger pipeline based on Git events So far, this tutorial used manually triggered builds. However, as the number of builds and pipeline executions grow, it is not scalable to manually trigger builds. In this part of the tutorial, you add a [Git event trigger](/harness-platform/3.0/harness-platform-resources/triggers/triggering-pipelines.md). Assume your team has a specific requirement where they want the MLOps pipeline to run *only* if there is an update to the Jupyter notebook in the codebase. 1. In your MLOps pipeline, select **Triggers** at the top of the Pipeline Studio, and then select **New Trigger**. 2. Select the GitHub webhook trigger. 3. On the trigger's **Configuration** tab: * For **Name**, enter `trigger_on_notebook_update`. * For **Connector**, select your `mlopsgithubconnector` GitHub connector. * The **Repository URL** should automatically populate. * For **Event**, select **Push**. 4. Select **Continue** to go to the **Conditions** tab. * For **Branch Name**, select the **Equals** operator, and enter `main` for the **Matches Value**. * For **Changed Files**, select the **Equals** operator, and enter `credit_card_approval.ipynb` for the **Matches Value**. 5. Select **Continue** to go to the **Pipeline Input** tab. * The **Git Branch** should automatically populate. * For **Primary Artifact**, enter `ccapprovaldeploy`. * For **Tag**, enter `<+pipeline.executionId>`. 6. Select **Create Trigger**. The trigger webhook should automatically register in your GitHub repository. If it does not, you need to manually register the webhook: * On the list of triggers in Harness, select the **Link** icon to copy the webhook URL for the trigger. ![Webhook URL](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-6e9d9d8fd22b3902bd2f6f181ee62a7f283b0109%2Ftrigger-webhook.png?alt=media) * In your GitHub repository, navigate to **Settings**, select **Webhook**, and then select **Add Webhook**. * Paste the webhook URL in **Payload URL**. * Set the **Content type** to `application/json`. * Select **Add webhook**. A green checkmark in the GitHub webhooks list indicates that the webhook connected successfully. ![Webhook Success](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-094d526357747042e2a96c1d41c75cfb167c2af1%2Fwebhook-success.png?alt=media) With the trigger in place, whenever you push a change to the `credit_card_approval.ipynb` file on the `main` branch, the MLOps pipeline runs. In the trigger settings, you could adjust or remove the **Conditions** (branch name, changed files, and so on) according to your requirements, if you wanted to use a Git event trigger in a live development or production scenario. *** ## Add an approval gate before prod deployment Your organization might require an approval gate for your CI/CD pipeline before an artifact is deployed to production. Harness offers built-in approval steps for Jira, ServiceNow, or Harness approvals. Assume that you have a different image for production, and a different AWS Lambda function is deployed based on that container image. In your MLOps pipeline, you can create another `AWS Lambda deployment` stage with another `AWS Lambda deploy` step for the production environment and use the approval gate prior to running that production deployment stage. 1. To add the approval gate, add an **Approval** stage immediately prior to the **Deploy** stage that requires approval. * For **Name**, enter `approval-to-prod`. * For **Approval Type**, select **Harness Approval**. 2. Add an **Approval** step to the **Approval** stage. * For **Name**, enter `approval-to-prod`. * For **Timeout**, enter `1d`. * Use the default **Message**. * For **User Groups**, select **Select User Groups**, select **Project**, and select **All Project Users**. 3. Save the pipeline. Next time you run the pipeline, someone from the Harness project must approve the promotion of artifact to the production environment before the final **Deploy** stage runs. *** ## Use the model in a web application In a live MLOps scenario, the ML model would likely power a web application. While this app development is outside the scope of this tutorial, the following animation demonstrates a simple web application developed using plain HTML/CSS/JS. The outcome of the credit card application uses the response from the public AWS Lambda function URL invocation. ![Credit Card Application Demo App](https://1607448479-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FNNjztg4Qwud2Y4c5PCkj%2Fuploads%2Fgit-blob-c766291a063e79ea078f81842bd76e58dffd4e78%2Fcreditcardapprovalapp.gif?alt=media) *** ## Next steps You have accomplished the following in this tutorial: * [x] Build and push an image for this project. * [x] Run security scans on the container image. * [x] Upload model visualization data to S3. * [x] Publish model visualization data within the pipeline. * [x] Run tests on the model to determine accuracy and fairness scores. * [x] Based on those scores, use Open Policy Agent (OPA) policies to either approve or deny the model. * [x] Deploy the model. * [x] Monitor the model and ensure the model is not outdated. * [x] Trigger the pipeline based on Git events. * [x] (Optional) Add approval gates for production deployment. This tutorial trains the model on Harness build infrastructure. If your model already lives on a managed ML platform, the same pipeline shape applies with the training step replaced by a Plugin step. * [Integrate ML platforms with Harness CI](/continuous-integration/3.0/troubleshooting-and-resources/tutorials-and-code-samples/mlops/mlops-integrations.md): Run training and deployment jobs on Azure ML, AWS SageMaker, Databricks, Google Vertex AI, or MLflow. * [MLOps best practices](/continuous-integration/3.0/troubleshooting-and-resources/tutorials-and-code-samples/mlops/mlops-best-practices.md): Review the practices that keep model pipelines reproducible and governed. * [MLOps with Harness](/continuous-integration/3.0/troubleshooting-and-resources/tutorials-and-code-samples/mlops.md): Understand which Harness capability covers each stage of the ML lifecycle. {% @harness-feedback/feedback %}