> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-delivery/use-gitops/security/sops.md). # Manage Kubernetes secrets with Mozilla SOPS Decrypt Git-stored secrets with Mozilla SOPS for Helm-based Harness GitOps applications GitOps uses Git as the source of truth for infrastructure and application configuration, so it needs access to secrets such as authentication tokens and private keys. Storing secrets in Git creates a security risk and must not be allowed, even when the Git repository is private and uses access controls. Mozilla SOPS helps you store encrypted secrets in Git. After you store encrypted secrets in Git, SOPS decrypts them by using keys that you store as Kubernetes secrets. *** ## What you will learn from this topic * How [supported key management systems](#supported-key-management-systems) and [known limitations](#known-limitations) apply to Harness GitOps * How to [prepare a Bring Your Own Argo CD (BYOA) agent](#prepare-the-byoa-agent-only-if-you-are-using-byoa) for SOPS * How to encrypt secrets with [age or PGP](#use-age-or-pgp), [AWS KMS with IRSA](#use-aws-kms-with-irsa), or [GCP KMS](#use-gcp-kms) * How to reference decrypted values files from a Helm GitOps application *** ## Before you begin * **Helm-based GitOps application**: Harness supports SOPS only for Helm-based applications. Go to [Manage GitOps applications](/continuous-delivery/use-gitops/application/manage-gitops-applications.md) to review application setup. * **GitOps Agent access**: You need cluster access to patch the Argo CD repo server in the agent namespace when you mount keys or enable AWS or GCP identity. * **Key management access**: You need permission to create keys in age, PGP, AWS KMS, or GCP KMS, depending on the method you choose. * **BYOA only**: If you migrate an existing Argo CD setup with Bring Your Own Argo CD (BYOA), complete [Prepare the BYOA agent](#prepare-the-byoa-agent-only-if-you-are-using-byoa) before you encrypt secrets. *** ## Supported key management systems Mozilla SOPS supports the following key management systems: * AWS Key Management Service (AWS KMS) * Google Cloud Key Management Service (Cloud KMS) * Azure Key Vault * age file encryption * Pretty Good Privacy (PGP) *** ## Known limitations Harness supports SOPS only for Helm-based applications. {% hint style="warning" %} **Asymmetric keys are not supported** SOPS does not support asymmetric keys for encrypting and decrypting secrets when you use external key management systems such as AWS KMS and Google Cloud KMS. This is a SOPS limitation. {% endhint %} *** ## Encrypt secrets Use the following procedure to implement SOPS: 1. If you use the Bring Your Own Argo CD (BYOA) method to migrate your existing Argo CD setup to Harness GitOps, [prepare the BYOA agent](#prepare-the-byoa-agent-only-if-you-are-using-byoa). 2. Encrypt secrets by using one of the following tools: * age * PGP * AWS KMS (with IAM Roles for Service Accounts, IRSA) * GCP KMS ### Prepare the BYOA agent (only if you are using BYOA) Perform the following steps to prepare the BYOA agent: 1. Patch the Argo CD repo server: ```bash kubectl patch deployment argocd-repo-server -n --patch-file=argocd-repo-server-sops-patch.yaml ``` Use the following YAML configuration in `argocd-repo-server-sops-patch.yaml`: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: argocd-repo-server spec: template: spec: containers: - name: argocd-repo-server env: - name: HELM_PLUGINS value: /helm-sops-tools/helm-plugins/ - name: HELM_SECRETS_CURL_PATH value: /helm-sops-tools/curl - name: HELM_SECRETS_SOPS_PATH value: /helm-sops-tools/sops - name: HELM_SECRETS_KUBECTL_PATH value: /helm-sops-tools/kubectl - name: HELM_SECRETS_BACKEND value: sops - name: HELM_SECRETS_VALUES_ALLOW_SYMLINKS value: "false" - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH value: "true" - name: HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL value: "true" - name: HELM_SECRETS_WRAPPER_ENABLED value: "true" - name: HELM_SECRETS_HELM_PATH value: /usr/local/bin/helm volumeMounts: - mountPath: /helm-sops-tools name: helm-sops-tools - mountPath: /usr/local/sbin/helm subPath: helm name: helm-sops-tools initContainers: - name: sops-helm-secrets-tool image: docker.io/harness/gitops-agent-installer-helper:v0.0.1 imagePullPolicy: IfNotPresent command: [ sh, -ec ] args: - | cp -r /custom-tools/. /helm-sops-tools cp /helm-sops-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /helm-sops-tools/helm chmod +x /helm-sops-tools/* volumeMounts: - mountPath: /helm-sops-tools name: helm-sops-tools serviceAccountName: argocd-repo-server volumes: - name: helm-sops-tools emptyDir: {} ``` 2. Patch the `argocd-cm` ConfigMap: ```bash kubectl patch configmap argocd-cm -n --patch "$(cat argocd-cm-sops-patch.yaml)" ``` Use the following YAML configuration in `argocd-cm-sops-patch.yaml`: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm data: helm.valuesFileSchemes: >- secrets+gpg-import, secrets+gpg-import-kubernetes, secrets+age-import, secrets+age-import-kubernetes, secrets,secrets+literal, https ``` ### Use age or PGP Use the following procedure to encrypt your secrets with age or PGP. Only the first step differs. Later steps are common to age and PGP. Perform the following steps to encrypt a secret with age or PGP: 1. (If using age) Create a key pair and store the keys in a file named `key.txt`: ```bash age-keygen -o key.txt ``` 2. (If using PGP) Create a key pair and store the keys in a file named `key.asc`: 1. Generate a key pair: ```bash gpg --full-generate-key ``` 2. List the generated keys: ```bash gpg --list-secret-keys --keyid-format=long ``` 3. From the output of the previous command, copy the ID (it appears after the forward slash (`/`) in the `sec` field), then use it in place of `ID` in the following command: ```bash gpg --armor --export ID > key.asc ``` 3. Mount the key as a volume on the Argo CD repo server so the agent can access the private key and decrypt the secret. 1. Create a Kubernetes secret from file `key.txt`. For example: ```bash kubectl create secret generic dev-private-key --from-file=key.txt ``` 2. Mount the secret to the Argo CD repo server: ```bash kubectl patch deployment argocd-repo-server -n --type json -p '[ { "op": "add", "path": "/spec/template/spec/volumes/-", "value": { "name": "dev-private-key", "secret": { "secretName": "dev-private-key" } } }, { "op": "add", "path": "/spec/template/spec/containers/0/volumeMounts/-", "value": { "mountPath": "/dev-private-key/", "name": "dev-private-key" } } ]' ``` 4. Verify that the Argo CD repo server has the following YAML configuration: ```yaml repoServer: volumes: - name: dev-private-key secret: secretName: dev-private-key volumeMounts: - mountPath: /dev-private-key/ name: dev-private-key ``` 5. Encrypt the data by using one of the following methods: * Method 1: 1. Create a `.sops.yaml` file in the root directory and set creation rules for the path: ```yaml creation_rules: - path_regex: 'environment/secrets/dev/(.*).yaml' age: 'my-public-key' ``` 2. Run the following command: ```bash sops -e --in-place environment/secrets/dev/values-enc.yaml ``` * Method 2: Run the following command: ```bash sops -e --age [age-public-key] [input-file] > [output-file] ``` 6. Add the following secret reference in the secrets field of the Helm application: ``` secrets+age-import:///dev-private-key/key.txt?secrets.yaml ``` ### Use AWS KMS (with IRSA) Perform the following steps to use AWS KMS to encrypt secrets: 1. Create a symmetric key in AWS KMS. Go to [Create a KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) in the AWS documentation. Use a symmetric key; SOPS does not support asymmetric KMS keys for this flow. 2. Create an IAM role with the following policies and trust relationship. Trust policy: ```json { "Effect": "Allow", "Principal": { "Federated": "" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { ":sub": "system:serviceaccount::argocd-repo-server" } } } ``` Permission policy: ```json { "Sid": "Argo CDKMS", "Effect": "Allow", "Action": [ "kms:Decrypt*", "kms:Encrypt*", "kms:GenerateDataKey", "kms:ReEncrypt*", "kms:DescribeKey" ], "Resource": [ "", "" ] } ``` ```json { "Sid": "AssumeRole", "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "*" } ``` 3. Annotate the `argocd-repo-server` service account in the agent namespace with the role you created: ``` eks.amazonaws.com/role-arn: arn:aws:iam:::role/ ``` 4. Use the following patch command to set `automountServiceAccountToken` in the repo server deployment to `true`: ```bash kubectl patch deployment argocd-repo-server -n sops-setup-test --patch '{"spec":{"template":{"spec":{"automountServiceAccountToken": true}}}}' ``` 5. Restart the `argocd-repo-server` deployment. 6. Encrypt the secret by doing one of the following: * Run the following command (replace placeholders with your KMS key ARN and file paths): ```bash sops -e --kms > ``` * Set up creation rules as follows: ```yaml - path_regex: 'environment/secrets/qa/(.*).yaml' kms: ``` 7. Add the following secret reference in the application: ``` secrets://environment/secrets/qa/values-enc.yaml ``` ### Use GCP KMS Perform the following steps to use GCP KMS to encrypt secrets: 1. Link the `argocd-repo-server` service account to a Google service account. The Google service account must have the `cloudkms.cryptoKeyVersions.useToDecrypt` permission. One way to assign that permission is to grant the role `roles/cloudkms.cryptoKeyDecrypter`. 2. Annotate the `argocd-repo-server` service account in the agent namespace with the Google service account: ``` iam.gke.io/gcp-service-account: ``` 3. Use the following patch command to set `automountServiceAccountToken` in the repo server deployment to `true`: ```bash kubectl patch deployment argocd-repo-server -n sops-setup-test --patch '{"spec":{"template":{"spec":{"automountServiceAccountToken": true}}}}' ``` 4. Restart the Argo CD repo server deployment. 5. Encrypt the secret by doing one of the following: * Run the following command (replace placeholders with your GCP KMS key resource name and file paths): ```bash sops -e --gcp-kms > ``` * Set up creation rules as follows: ```yaml - path_regex: 'environment/secrets/qa/(.*).yaml' gcp_kms: ``` 6. Add the following secret reference in the application: ``` secrets://environment/secrets/qa/values-enc.yaml ``` *** ## Troubleshooting
Helm application sync fails after adding a secrets:// values reference Confirm that `helm.valuesFileSchemes` in `argocd-cm` includes the `secrets` schemes from the BYOA patch (or the equivalent Harness-managed agent configuration), and that the private key or cloud identity is mounted or annotated on `argocd-repo-server`.
SOPS cannot decrypt with AWS KMS or GCP KMS Confirm that you created a symmetric key, that the repo server service account has decrypt permission, and that `automountServiceAccountToken` is set to `true`. Asymmetric KMS keys are not supported by SOPS for this flow.
age or PGP decrypt fails on the repo server Confirm that the Kubernetes secret is mounted at the path referenced in the Helm secrets URI (for example `/dev-private-key/key.txt`) and that the URI scheme matches your key type (`secrets+age-import` or the equivalent GPG import scheme).
*** ## Next steps * [External Secrets Operator integration](/continuous-delivery/use-gitops/security/external-secrets-integration.md): Sync secrets from an external secret store into GitOps. * [Secret injection with the Harness plugin](/continuous-delivery/use-gitops/security/secret-injection-harness-plugin.md): Inject Harness secrets into GitOps manifests. * [GitOps audit trail](/continuous-delivery/use-gitops/security/audit-trail.md): Review GitOps security and audit events. {% @harness-feedback/feedback %}