Secrets Troubleshooting

Diagnose and resolve common issues with secrets in Harness 3.0; secret not found, external manager connection failures, stale cached values, and file decode errors.

Diagnose and resolve common issues with secrets in Harness 3.0. This guide covers the most frequently encountered problems, their root causes, and step-by-step resolution procedures.

Secret not found

Pipeline execution fails with an error indicating the referenced secret cannot be found.

# Common error message <a href="#common-error-message" id="common-error-message"></a>
Error: Secret with identifier [my_secret] not found in scope [project:myproject]

Incorrect scope reference

The secret exists but is being referenced with the wrong scope prefix. For example, referencing an organization-level secret without the org. prefix.

Solution: Verify the scope of the secret in the Harness UI. Use the correct prefix:

# Project scope (no prefix) <a href="#project-scope-no-prefix" id="project-scope-no-prefix"></a>
<+secrets.getValue("my_secret")>

# Organization scope <a href="#organization-scope" id="organization-scope"></a>
<+secrets.getValue("org.my_secret")>

# Account scope <a href="#account-scope" id="account-scope"></a>
<+secrets.getValue("account.my_secret")>

Secret has been deleted

The secret was deleted but is still referenced in a pipeline, connector, or service definition.

Solution: Recreate the secret with the same identifier, or update the referencing entities to use a different secret. Check the audit log to confirm when and by whom the secret was deleted.

Insufficient permissions

The user or service account executing the pipeline does not have the Access permission on the secret.

Solution: Verify that the pipeline execution user or service account has the Access permission for the secret at the appropriate scope. Check role bindings in Access Control settings.

Wrong project or organization context

The pipeline is running in a different project or organization than where the secret was created, and the secret is not accessible from the current scope.

Solution: Either move the secret to a higher scope (organization or account) so it is accessible from the pipeline's project, or create a copy of the secret in the correct project.

---

Connection failed to external secret manager

Pipeline execution fails because Harness cannot connect to the external secret manager to retrieve the secret value.

External secret manager is offline or unreachable

The secret manager endpoint is down, or network connectivity between the Harness Delegate and the secret manager is interrupted.

Solution: Verify that the external secret manager is running and accessible. Check network connectivity from the Delegate host. Verify DNS resolution and firewall rules. Test connectivity using curl or telnet from the Delegate.

Authentication credentials expired

The token or credentials used to authenticate with the external secret manager have expired or been revoked.

Solution: Renew or regenerate the authentication credentials for the secret manager connector. Update the connector configuration in Harness with the new credentials and test the connection.

Secret path has changed

The path or key name in the external secret manager has been modified, but the Harness secret reference still points to the old path.

Solution: Verify the current path of the secret in your external secret manager. Update the secret configuration in Harness to reference the correct path.

Delegate connectivity issues

The Harness Delegate responsible for retrieving the secret is offline, unhealthy, or cannot reach the secret manager endpoint.

Solution: Check the Delegate status in the Harness UI. Verify the Delegate is running and has a heartbeat. Ensure the Delegate has network access to the secret manager. If using Delegate selectors, verify the correct Delegate is selected.

---

Secret value not updating

You have updated a secret value but pipeline executions continue to use the old value.

Secret caching (60-second TTL)

Harness caches resolved secret values for up to 60 seconds to reduce load on secret managers. If you updated the secret and immediately ran a pipeline, the cached value may have been used.

Solution: Wait at least 60 seconds after updating a secret before running a pipeline that references it. The cache will expire and the new value will be retrieved on the next execution.

CACHE DURATION

There is no manual cache invalidation mechanism; the cache expires automatically after 60 seconds.

External manager not synced

If using an external secret manager with replication (e.g., GCP Secret Manager with multi-region replication), the new value may not have propagated to the region where the Delegate is retrieving the secret.

Solution: Verify the replication status in your external secret manager. Wait for replication to complete before running the pipeline. Check that the Delegate is connecting to the correct region endpoint.

Wrong secret updated

If you have secrets with similar names at different scopes, you may have updated the wrong one. For example, updating a project-level secret when the pipeline references an organization-level secret with a similar name.

Solution: Verify the exact identifier and scope of the secret referenced in the pipeline YAML. Ensure you updated the secret at the correct scope level.

---

File secret decode issues

File secrets are stored as base64-encoded content. Common issues arise when the decoded output is not handled correctly in pipeline steps.

File not decoded before use

The file secret value is base64-encoded. If you write the raw value to a file without decoding, the file will contain base64 text instead of the original file content.

Solution: Always pipe the secret value through base64 -d (or base64 --decode) before writing to a file.

File type mismatch

The decoded file content does not match the expected format. For example, a JSON key file was uploaded but the consuming tool expects a different format.

Solution: Verify the original file content matches the expected format. Decode the secret and inspect the output to ensure it matches the expected structure.

#!/bin/bash

# Decode a file secret and write to a file <a href="#decode-a-file-secret-and-write-to-a-file" id="decode-a-file-secret-and-write-to-a-file"></a>
echo '<+secrets.getValue("gcp_sa_key")>' | base64 -d > /tmp/service-account.json

# Verify the decoded file is valid JSON <a href="#verify-the-decoded-file-is-valid-json" id="verify-the-decoded-file-is-valid-json"></a>
cat /tmp/service-account.json | python3 -m json.tool > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "File decoded successfully - valid JSON"
else
  echo "ERROR: Decoded file is not valid JSON"
  exit 1
fi

# macOS alternative (if base64 -d does not work) <a href="#macos-alternative-if-base64-d-does-not-work" id="macos-alternative-if-base64-d-does-not-work"></a>
echo '<+secrets.getValue("gcp_sa_key")>' | base64 -D > /tmp/service-account.json

# Decode and set correct file permissions <a href="#decode-and-set-correct-file-permissions" id="decode-and-set-correct-file-permissions"></a>
echo '<+secrets.getValue("ssh_private_key")>' | base64 -d > /tmp/id_rsa
chmod 600 /tmp/id_rsa

VALIDATE AFTER DECODING

Always validate decoded file secrets before use. For JSON files, use python3 -m json.tool or jq .. For YAML files, use python3 -c "import yaml; yaml.safe_load(open('file.yaml'))". For PEM files, use openssl x509 -in file.pem -noout -text.

---

Diagnostic steps

Follow this five-step process to systematically identify and resolve secret-related issues.

Step 1: Verify the secret exists

Navigate to the Secrets page at the expected scope (Project, Organization, or Account). Search for the secret by name or identifier. Confirm the secret exists and note its exact identifier and scope.

curl -s -X GET \
  "https://app.harness.io/gateway/ng/api/v2/secrets/my_secret?accountIdentifier=abc123&projectIdentifier=myproject" \
  -H "x-api-key: YOUR_API_KEY" | python3 -m json.tool

Step 2: Test secret access with a minimal pipeline

Create a minimal pipeline that references the secret in a shell script step. This isolates the secret access from other pipeline logic.

pipeline:
  name: Secret Test
  stages:
    - name: Test Secret Access
      type: custom
      spec:
        steps:
          - name: Verify Secret
            type: script
            spec:
              shell: bash
              command: |
                # This will print ******** if the secret is resolved
                echo "Secret value: <+secrets.getValue("my_secret")>"
                echo "Secret resolved successfully"

Step 3: Check secret references

Open the secret details and navigate to the References tab. Verify that the pipeline in question is listed as a reference. If it is not listed, the expression syntax in the pipeline may be incorrect.

Step 4: Review the activity log

Open the secret details and navigate to the Activity tab. Review recent events to check for unexpected modifications, deletions, or access failures. Look for error entries that indicate connection issues to the secret manager.

Step 5: Validate permissions

Navigate to Access Control in your project or organization settings. Verify that the pipeline execution user or service account has the Access permission for the secret resource at the correct scope level.

curl -s -X POST \
  "https://app.harness.io/gateway/authz/api/acl" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "principal": {
      "principalIdentifier": "user_or_sa_id",
      "principalType": "USER"
    },
    "resourceScope": {
      "accountIdentifier": "abc123",
      "orgIdentifier": "default",
      "projectIdentifier": "myproject"
    },
    "resourceType": "SECRET",
    "resourceIdentifier": "my_secret",
    "permission": "core_secret_access"
  }'

STILL HAVING ISSUES?

If the diagnostic steps above do not resolve your issue, collect the following information and contact Harness support: the secret identifier and scope, the pipeline YAML with the secret reference, the full error message from the pipeline execution log, the Delegate logs if using an external secret manager, and the audit log entries for the secret.