Skip to main content

Serverless Lambda deployment tutorial

This quickstart shows you how to deploy a Serverless Lambda application to AWS Lambda using Harness. We'll use a publicly-available serverless.yaml and artifact and deploy them to your AWS Lambda service using a Harness Pipeline.

New to Serverless? See Tutorial: Your First Serverless Framework Project from Serverless.

Objectives

  1. Add a serverless.yaml file and ZIP artifact to a Harness Pipeline stage.
  2. Define your AWS Lambda service as the deployment target.
  3. Deploy the Serverless application to Lambda.

Before you begin

Review Harness Key Concepts to establish a general understanding of Harness.* GitHub account: this quickstart uses a publicly available serverless.yaml file, but GitHub requires that you use a GitHub account for fetching files.

  • Kubernetes Delegate cluster with Serverless installed: the Harness Delegate is a worker process that performs all deployment tasks. For this quickstart, we'll install a Kubernetes Delegate running in a Kubernetes cluster.

    • You can use a cluster hosted on a cloud platform or run one in minikube using Docker Desktop locally. The installation steps are the same.
    • The Delegate host must have Serverless installed. We'll add the Serverless installation script to the Delegate YAML file later in this quickstart.
  • AWS User account with required policy: Serverless deployments require an AWS User with specific AWS permissions, as described in AWS Credentials from Serverless. To create the AWS User, do the following:

    • Log into your AWS account and go to the Identity & Access Management (IAM) page.
    • Click Users, and then Add user. Enter a name. Enable Programmatic access by clicking the checkbox. Click Next to go to the Permissions page. Do one of the following:
      • Full Admin Access: click on Attach existing policies directly. Search for and select AdministratorAccess then click Next: Review. Check to make sure everything looks good and click Create user.
      • Limited Access: click on Create policy. Select the JSON tab, and add the JSON using the following code from the Serverless gist: IAMCredentials.json
    {  
    "Statement": [
    {
    "Action": [
    "apigateway:*",
    "cloudformation:CancelUpdateStack",
    "cloudformation:ContinueUpdateRollback",
    "cloudformation:CreateChangeSet",
    "cloudformation:CreateStack",
    "cloudformation:CreateUploadBucket",
    "cloudformation:DeleteStack",
    "cloudformation:Describe*",
    "cloudformation:EstimateTemplateCost",
    "cloudformation:ExecuteChangeSet",
    "cloudformation:Get*",
    "cloudformation:List*",
    "cloudformation:UpdateStack",
    "cloudformation:UpdateTerminationProtection",
    "cloudformation:ValidateTemplate",
    "dynamodb:CreateTable",
    "dynamodb:DeleteTable",
    "dynamodb:DescribeTable",
    "dynamodb:DescribeTimeToLive",
    "dynamodb:UpdateTimeToLive",
    "ec2:AttachInternetGateway",
    "ec2:AuthorizeSecurityGroupIngress",
    "ec2:CreateInternetGateway",
    "ec2:CreateNetworkAcl",
    "ec2:CreateNetworkAclEntry",
    "ec2:CreateRouteTable",
    "ec2:CreateSecurityGroup",
    "ec2:CreateSubnet",
    "ec2:CreateTags",
    "ec2:CreateVpc",
    "ec2:DeleteInternetGateway",
    "ec2:DeleteNetworkAcl",
    "ec2:DeleteNetworkAclEntry",
    "ec2:DeleteRouteTable",
    "ec2:DeleteSecurityGroup",
    "ec2:DeleteSubnet",
    "ec2:DeleteVpc",
    "ec2:Describe*",
    "ec2:DetachInternetGateway",
    "ec2:ModifyVpcAttribute",
    "events:DeleteRule",
    "events:DescribeRule",
    "events:ListRuleNamesByTarget",
    "events:ListRules",
    "events:ListTargetsByRule",
    "events:PutRule",
    "events:PutTargets",
    "events:RemoveTargets",
    "iam:AttachRolePolicy",
    "iam:CreateRole",
    "iam:DeleteRole",
    "iam:DeleteRolePolicy",
    "iam:DetachRolePolicy",
    "iam:GetRole",
    "iam:PassRole",
    "iam:PutRolePolicy",
    "iot:CreateTopicRule",
    "iot:DeleteTopicRule",
    "iot:DisableTopicRule",
    "iot:EnableTopicRule",
    "iot:ReplaceTopicRule",
    "kinesis:CreateStream",
    "kinesis:DeleteStream",
    "kinesis:DescribeStream",
    "lambda:*",
    "logs:CreateLogGroup",
    "logs:DeleteLogGroup",
    "logs:DescribeLogGroups",
    "logs:DescribeLogStreams",
    "logs:FilterLogEvents",
    "logs:GetLogEvents",
    "logs:PutSubscriptionFilter",
    "s3:GetBucketLocation",
    "s3:CreateBucket",
    "s3:DeleteBucket",
    "s3:DeleteBucketPolicy",
    "s3:DeleteObject",
    "s3:DeleteObjectVersion",
    "s3:GetObject",
    "s3:GetObjectVersion",
    "s3:ListAllMyBuckets",
    "s3:ListBucket",
    "s3:PutBucketNotification",
    "s3:PutBucketPolicy",
    "s3:PutBucketTagging",
    "s3:PutBucketWebsite",
    "s3:PutEncryptionConfiguration",
    "s3:PutObject",
    "sns:CreateTopic",
    "sns:DeleteTopic",
    "sns:GetSubscriptionAttributes",
    "sns:GetTopicAttributes",
    "sns:ListSubscriptions",
    "sns:ListSubscriptionsByTopic",
    "sns:ListTopics",
    "sns:SetSubscriptionAttributes",
    "sns:SetTopicAttributes",
    "sns:Subscribe",
    "sns:Unsubscribe",
    "states:CreateStateMachine",
    "states:DeleteStateMachine"
    ],
    "Effect": "Allow",
    "Resource": "*"
    }
    ],
    "Version": "2012-10-17"
    }

    The s3:GetBucketLocation action is required for a custom S3 bucket only.

    • View and copy the API Key and Secret to a temporary place. You'll need them when setting up the Harness AWS Connector later in this quickstart.

Limitations

  • Harness supports all language runtimes that Serverless supports.
  • Harness supports ZIP files and Docker image artifacts only.
    • ZIP files are supported with JFrog Artifactory.
    • Docker images are supported with AWS ECR.

Create the Deploy stage

Pipelines are collections of stages. For this quickstart, we'll create a new Pipeline and add a single stage.

note

Create a Project for your new CD Pipeline: if you don't already have a Harness Project, create a Project for your new CD Pipeline. Make sure that you add the Continuous Delivery module to the Project. See Create Organizations and Projects.

  1. In your Harness Project, click Deployments, and then click Create a Pipeline.

  2. Enter the name Serverless Quickstart and click Start. Your Pipeline appears.

  3. Click Add Stage and select Deploy.

  4. Enter the name Deploy Service, make sure Service is selected, and then click Set Up Stage.

    The new stage settings appear.

  5. In About the Service, click New Service.

  6. Give the Service the name quickstart and click Save.

note

Let's take a moment and review Harness Services and Service Definitions (which are explained below). A Harness Service represents your microservice/app logically.
You can add the same Service to as many stages as you need. Service Definitions represent your artifacts, manifests, and variables physically. They are the actual files and variable values.
By separating Services and Service Definitions, you can propagate the same Service across stages while changing the artifacts, manifests, and variables with each stage.

Once you have created a Service, it's persistent and you can use it throughout the stages of this or any other Pipeline in the Project.

Add the manifest

Next, we can add a serverless.yaml for our deployment. We'll use the publicly-available serverless.yaml file available from Harness.

  1. In Service Definition, in Deployment Type, click Serverless Lambda.

  2. In Manifests, click Add Manifest.

  3. Select Serverless Lambda Manifest, and click Continue.

  4. In Specify Serverless Lambda Manifest Store, click GitHub, and then click New GitHub Connector. The Git Connector settings appear. Enter the following settings.

    • Name: serverless.
    • URL Type: Repository.
    • Connection Type: HTTP.
    • GitHub Repository URL: https://github.com/wings-software/harness-docs.git.
    • Username: Enter your GitHub account username.
    • In Personal Access Token, click Create or Select a Secret.
      • Click New Secret Text.
      • In Secret Name, enter a name for the secret like github-pat.
      • In Secret Value, paste in a GitHub Personal access token.When you're logged into GitHub, these tokens are listed at https://github.com/settings/tokens. For steps on setting up a GitHub PAT, see Creating a personal access token from GitHub.
      • Make sure your PAT has the repo scope selected:

  5. Select Connect through Harness Platform.

  6. Click Finish.

  7. Back in Specify Serverless Lambda Manifest Store, click Continue.

  8. In Manifest Details, enter the following.

    • Manifest Identifier: serverless.
    • Git Fetch Type: Latest from Branch.
    • Branch: main.
    • Folder Path: serverless/artifacts.
    • In Advanced, you can see Serverless Config File Path. Use this setting when your Serverless manifest isn't named serverless.yml|.yaml|.js|.json. This option is the same as the --config option in serverless deploy. See AWS - deploy from Serverless.

You can see the serverless.yaml manifest in Harness.

Here's what the serverless.yaml looks like:

service: <+service.name>  
frameworkVersion: '2 || 3'

provider:
name: aws
runtime: nodejs12.x
functions:
hello:
handler: handler.hello
events:
- httpApi:
path: /tello
method: get
package:
artifact: <+artifact.path>
plugins:
- serverless-deployment-[email protected]

You can see the Harness expression <+artifact.path> in artifact: <+artifact.path>. The expression <+artifact.path> tells Harness to get the artifact from Artifacts section of the Service. We'll add the artifact next.

The expression <+service.name> simply uses the Harness Service name for the deployed service name.

For Docker images, you use the expression <+artifact.image>.

Add the artifact

Currently, Harness supports ZIP file artifacts only. Harness doesn't support Docker images yet.Next, we'll add a publicly-available artifact to your Service. The artifact is a zip file with a JavaScript function hosted in Artifactory.

We'll add a new Artifactory Connector and install a Harness Kubernetes Delegate in a Kubernetes cluster. The Delegate is a worker process that performs the deployment operations. The Delegate will use the URL and credentials you provide in the Connector to connect to Artifactory and fetch the artifact at runtime.

  1. In Artifact, click Add Primary.

  2. In Specify Artifact Repository Type, click Artifactory, and click Continue.

  3. In Artifactory Repository, click New Artifactory Connector.

  4. In Create or Select an Existing Connector, click New Artifactory Connector.

  5. Enter a name for the Connector, such as JFrog Serverless. Click Continue.

  6. In Details, in Artifactory Repository URL, enter https://harness.jfrog.io/artifactory/.

  7. In Authentication, select Anonymous.

  8. In Delegates Setup, click Install new Delegate. The Delegate wizard appears.

  9. Click Kubernetes, and then click Continue.

  10. Enter a name for the Delegate, like serverlesslocal, click the Laptop size.

    With the Laptop size, you can use a local minikube cluster or a small cluster hosted on a cloud platform. If you use minikube, start minikube with the following resources: minikube start --memory 3g --cpus 2.

  11. Click Continue.

  12. Click Download YAML file. The YAML file for the Kubernetes Delegate is downloaded to your computer.

Installing Serverless on the Delegate

Now we need to edit the YAML to install Serverless when the Delegate pods are created.

  1. Open the Delegate YAML in a text editor.

  2. Locate the Environment variable INIT_SCRIPT in the StatefulSet (Legacy Delegate) or Deployment (Harness Delegate) object:

    ...  
    - name: INIT_SCRIPT
    value: ""
    ...
  3. Replace the value with the following Serverless installation script.

    ...  
    - name: INIT_SCRIPT
    value: |-
    #!/bin/bash
    echo "Start"
    export DEBIAN_FRONTEND=noninteractive
    echo "non-inte"
    apt-get update
    echo "updagte"
    apt install -yq npm
    echo "npm"
    npm install -g [email protected]
    echo "Done"
    ...

In cases when the Delegate OS doesn't support apt (Red Hat Linux), you can edit this script to install npm. The rest of the code should remain the same. If you are using Harness Delegate, the base image is Red Hat UBI.Save the YAML file as harness-delegate.yml.

Install and register the Delegate

  1. Open a terminal and navigate to where the Delegate file is located. You'll connect to your cluster using the terminal so you can simply run the YAML file on the cluster.

  2. In the same terminal, log into your Kubernetes cluster. On most platforms, you select the cluster, click Connect, and copy the access command.

  3. Next, install Harness Delegate using the harness-delegate.yml file you just downloaded. In the terminal connected to your cluster, run this command:

    kubectl apply -f harness-delegate.yml

    You can find this command in the Delegate wizard:

  4. In Harness, click Verify. It'll take a few minutes to verify the Delegate. Once it is verified, close the wizard.

  5. Back in Set Up Delegates, in the list of Delegates, you can see your new Delegate and its tags.

  6. Select the Connect using Delegates with the following Tags option.

  7. Enter the tag of the new Delegate and click Save and Continue.

  8. In Connection Test, you can see that the connection is successful. Click Finish.

Add the artifact

  1. Back in Artifactory Repository, click Continue.

  2. Enter the following artifact settings and click Submit. The following image shows how the Artifactory settings correspond to Artifact Details.

    • Repository: lambda.
    • Artifact Directory: serverless.
    • Artifact Details: Value.
    • Artifact Path: handler.zip. When you click one of the settings, the Delegate fetches artifact metadata from Artifactory.
  3. Click Submit.

    The artifact is now in the Service.

  4. Click Continue to view the Infrastructure.

Now that you have configured the Service, we can define the target for our deployment.​

Define the AWS deployment target

  1. In Infrastructure, we'll add an AWS Connector to connect Harness with your Lambda service.

  2. In Infrastructure Details, in Specify your environment, click New Environment. Just like with a Service, you can create a new Environment or select an existing one. We'll create a new one.

  3. In New Environment, enter a name, select Pre-Production, and click Save. The new Environment appears.

  4. In Infrastructure Definition, click AWS.

  5. In Amazon Web Services Details, click in Connector.

  6. In Create or Select an Existing Connector, click New Connector.

  7. Enter the following and click Save and Continue.

    • Name: AWS Serverless.
    • Credentials: AWS Access Key. Enter the AWS access key for the AWS User you created with the required policies in Before You Begin.
    • Enter the secret key as a Harness Text Secret. The Harness Delegate uses these credentials to authenticate Harness with AWS at deployment runtime.
    • Delegates Setup: Only use Delegates with all of the following tags.
    • Select the Delegate you added earlier in this quickstart.
  8. The Connection Test verifies the connection. Click Finish.

  9. Back in Amazon Web Services Details, in Region, enter the region for your AWS Lambda service, such as us-east-1.

  10. In Stage, enter the name of the stage in your service that you want to deploy to, such as dev. This is the same as the --stage option in the serverless deploy command.

    When you run your deployment, you'll see these settings used in the logs. For example: serverless deploy list --stage dev --region us-east-1.

  11. Click Continue. The Execution steps appear.

Add a Serverless AWS Lambda Deploy step

In Execution, you add the steps that define how Harness deploys your Serverless Lambda service.

Harness automatically adds two Serverless Lambda steps to Execution:

  • Serverless Lambda Deploy: this step performs the deployment.
  • Serverless Lambda Rollback: this step performs a rollback in the event of a deployment failure. To see this step, toggle the Execution/Rollback setting.

  1. In Execution, click Serverless Lambda Deploy.

  2. Click the Advanced tab and select the Delegate that you installed in Delegate Selector.

    If you only have one Delegate installed in your Project, then this isn't necessary. But if you have multiple Delegates, you want to make sure the Serverless Lambda Deploy step uses the Delegate where you installed Serverless.

  3. Click Apply Changes.

Now you're ready to deploy.

Serverless Deploy Command Options

In Serverless Deploy Command Options, you can add any serverless deploy command options. See the Serverless AWS - Deploy doc for the list of options.

In the deployment logs, you'll see the options added to the serverless deploy command.

For example, if you add --conceal in Serverless Deploy Command Options you'll see the following:

serverless deploy --stage dev --region us-east-1 --conceal

Deploy and review

  1. Save your Pipeline and then click Run, and then Run Pipeline. The Pipeline executes.

  2. In the Serverless AWS Lambda Deploy step, click Input to see the deployment inputs:

  3. Click Output to see what's deployed:

  4. Click Details or Console View to see the logs.

In the logs you can see the successful deployment.

Deploying..  

Serverless Deployment Starting..
serverless deploy --stage dev --region us-east-1
Serverless: Deprecation warning: Support for Node.js versions below v12 will be dropped with next major release. Please upgrade at https://nodejs.org/en/
More Info: https://www.serverless.com/framework/docs/deprecations/#OUTDATED_NODEJS
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
Switch to it now by setting "provider.lambdaHashingVersion" to "20201221"
More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service artifactFile file to S3 (721 B)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.........
Serverless: Stack update finished...
Service Information
service: myfunction
stage: dev
region: us-east-1
stack: myfunction-dev
resources: 11
api keys:
None
endpoints:
GET - https://85h6zffizc.execute-api.us-east-1.amazonaws.com/tello
functions:
hello: myfunction-dev-hello
layers:
None

Deployment completed successfully.

Congratulations! You have successfully deployed a function using Serverless Lambda and Harness.

Clean up

For steps on deleting the Delgate, go to Delegate a delegate.

Notes

Now that you're done the quickstart, here's some more information to help you extend your Harness Serverless Lambda deployments.

Serverless manifest supports Harness secrets and expressions

The serverless.yaml file you use with Harness can use Harness secret and built-in expressions.

Expression support lets you take advantage of Runtime Inputs and Input Sets in your serverless.yaml files. For example, you could use a Stage variable as a runtime input to change plugins with each stage deployment:

service: <+service.name>  
frameworkVersion: '2 || 3'

provider:
name: aws
runtime: nodejs12.x
functions:
hello:
handler: handler.hello
events:
- httpApi:
path: /tello
method: get
package:
artifact: <+artifact.path>
plugins:
- <+stage.variables.devplugins>

See:

Supported stores for Serverless Lambda YAML files

Harness can fetch your YAML files and packaged code from the following stores:

  • AWS S3 buckets.
    • You can store the serverless.yml and the artifact code in AWS S3, including in the same bucket.
    • You can use the .Zip format to grab the serverless.yaml and the packaged code that has been bundled in .zip and published in S3.
    • Harness will extrapolate the serverless.yaml file and use that for deployment.
    • For S3, you use a Harness AWS Connector. The IAM role permissions required by Harness for S3 are described in AWS Connector Settings Reference.
  • Git providers.

Rollback timestamps

In a Serverless CLI rollback (serverless rollback --timestamp timestamp), you'd have to manually identify and select the timestamp of the last successful deployment. This can be difficult because you need to know which timestamp to use. With multiple developers deploying, there's the possibility of rolling back to the wrong version.

Harness avoids this issue by automatically identifying the last successful deployment using its timestamp. During the event of a rollback, Harness will automatically rollback to that deployment.

You can see the timestamps in the deployment logs:

Serverless: Listing deployments:  
Serverless: -------------
Serverless: Timestamp: 1653065606430
Serverless: Datetime: 2022-05-20T16:53:26.430Z
Serverless: Files:
Serverless: - compiled-cloudformation-template.json
Serverless: - myfunction.zip
Serverless: -------------
Serverless: Timestamp: 1653344285842
Serverless: Datetime: 2022-05-23T22:18:05.842Z
Serverless: Files:
Serverless: - artifactFile
Serverless: - compiled-cloudformation-template.json
Serverless: -------------
Serverless: Timestamp: 1653415240343
Serverless: Datetime: 2022-05-24T18:00:40.343Z
Serverless: Files:
Serverless: - artifactFile
Serverless: - compiled-cloudformation-template.json

If there is no change in code, Serverless doesn't deploy anything new. In the logs you'll see Serverless: Service files not changed. Skipping deployment....

While this is somewhat similar to how rollback is performed in Serverless CLI, Harness performs rollback automatically and always uses the timestamp of the last successful deployment.

During a Harness rollback, you can see the timestamp used to rollback to the last successful deployment (rollback --timestamp 1653415240343 --region us-east-1 --stage dev):

Rollback..  

Serverless Rollback Starting..
serverless rollback --timestamp 1653415240343 --region us-east-1 --stage dev
Serverless: Deprecation warning: Support for Node.js versions below v12 will be dropped with next major release. Please upgrade at https://nodejs.org/en/
More Info: https://www.serverless.com/framework/docs/deprecations/#OUTDATED_NODEJS
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
Switch to it now by setting "provider.lambdaHashingVersion" to "20201221"
More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Updating Stack...

Rollback completed successfully.

Versioning

Serverless Lambda deployments are versioned using the timestamp of their deployment. This versioning has no relation to the versioning in AWS Lambda.

Sidecar artifacts

You reference sidecar artifacts with the format <+artifacts.sidecars.[artifact_Id]>.

The artifact Id comes from the Artifact Details:

You can see it in the artifact list:

Docker sidecars

Here's an example of a serverless.yaml referencing primary and sidecar artifacts:

...  
functions:
hello:
image: <+artifact.image>
hello1:
image: <+artifacts.sidecars.mysidecar>
...

Non-container sidecars

Here's an example of a serverless.yaml referencing primary and sidecar artifacts:

service: my-project-134fadsaez  
frameworkVersion: '2.35.0'

provider:
name: aws
runtime: java8

package:
artifact: <+artifact.path>

functions:
currentTime:
handler: com.serverless.Handler
events:
- httpApi:
path: /time
method: get
layers:
# Ref name is generated by TitleCasing the layer name & appending LambdaLayer
- { Ref: CommonLibsLambdaLayer }

layers:
commonLibs:
package:
artifact: <+artifacts.sidecars.mysidecar>

Plugin support

Harness supports Serverless plugins in your serverless.yaml file.

You simply add the plugin using the standard Serverless plugins format and Harness adds the plugin at runtime.

service: <+service.name>  
frameworkVersion: '2 || 3'

provider:
name: aws
runtime: nodejs12.x
functions:
hello:
handler: handler.hello
events:
- httpApi:
path: /tello
method: get
package:
artifact: <+artifact.path>
plugins:
- serverless-deployment-[email protected]

Serverless YAML for files and images

The serverless.yaml format for files (for example, ZIP, JAR, WAR) is different from the format for Docker images.

Serverless YAML for files

service: example-service  
frameworkVersion: '2 || 3'

provider:
name: aws
runtime: nodejs12.x
functions:
hello:
handler: handler.hello
events:
- httpApi:
path: /tello
method: get
package:
artifact: <+artifact.path>
plugins:
- serverless-deployment-[email protected]

Serverless YAML for Docker images

service: example-service  
frameworkVersion: '2 || 3'

provider:
name: aws

functions:
hello:
image: <+artifact.image>