> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-delivery/use-gitops/resources/troubleshooting.md).
# GitOps troubleshooting
Learn how to resolve Harness GitOps issues.
This topic lists issues encountered when installing and using Harness GitOps and describes how to resolve them.
## What you will learn from this topic
* Understand [installation errors](#installation-errors) and how to resolve them.
* Understand [operational errors](#operational-errors) and how to resolve them.
* Understand [YAML best practices for probe definitions](#yaml-best-practices-for-probe-definitions) to avoid sync failures.
## Installation errors
Error: "Agent took too long to respond"
You might encounter the error `the Agent took too long to respond` when installing the Harness GitOps Agent with an existing Argo CD instance.
The error indicates that the Harness GitOps Agent is unable to connect to Redis or to the Argo CD Repo Server and needs additional `NetworkPolicy` settings.
Add the following `podSelector` settings to the `NetworkPolicy` objects defined in your existing Argo CD *argocd-redis* and *argocd-repo-server* services.
The following table lists the `NetworkPolicy` objects for HA and non-HA Agents, and includes the YAML before and after the new `podSelector` is added.
| **NetworkPolicy** | **HA Agent** |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `argocd-redis-ha-proxy-network-policy` |  |
| `argocd-repo-server-network-policy` |  |
| | **Non-HA Agent** |
| `argocd-redis-network-policy` |  |
| `argocd-repo-server-network-policy` |  |
Agent shows DEGRADED with "Redis Cache Installed" health check failure
The agent shows **DEGRADED** status even though it functions normally (heartbeats are normal, applications reconciling, no Redis errors in logs).
**Cause**
The `highAvailability` flag is set incorrectly for your Redis deployment. For more information, go to [The highAvailability flag](/continuous-delivery/use-gitops/gitops-entities/agents/install-a-harness-git-ops-agent.md#the-highavailability-flag).
**Solution**
Go to [Configure custom Argo CD component names](/continuous-delivery/use-gitops/gitops-entities/agents/install-a-harness-git-ops-agent.md#configure-custom-argo-cd-component-names-byoa) for the fix. The section includes:
* How to check your Redis deployment
* Commands to update the flag
* Examples for standard Redis and High Availability (HA) Redis
Error: "Forbidden: seccomp may not be set provider"
If you see the error `Forbidden: seccomp may not be set provider`, remove the following block from all Argo CD configuration files that have a `kind: deployment` key-value pair.
```yaml
seccompProfile:
type: RuntimeDefault
```
OpenShift Security Context Constraints (SCC) for GitOps agent
On OpenShift clusters, the default `restricted` SCC blocks GitOps agent components from starting. You might see errors like `CreateContainerConfigError`, `Forbidden: seccomp may not be set`, or pods stuck in `CrashLoopBackOff`.
The following table lists the minimum SCC requirements for each GitOps agent component:
| Component | Required SCC | Reason |
| ------------------------------- | ------------ | -------------------------------------------------------- |
| `gitops-agent` | `nonroot-v2` | Runs as a non-root user but requires specific UID/GID |
| `argocd-repo-server` | `nonroot-v2` | Requires writable `/tmp` and non-root execution |
| `argocd-redis` | `anyuid` | Runs as a specific UID (999) by default |
| `argocd-application-controller` | `nonroot-v2` | Requires non-root execution with writable home directory |
To grant the required SCCs to the service accounts in your GitOps namespace:
```bash
# Replace with your GitOps agent namespace (for example, harness-gitops)
oc adm policy add-scc-to-user nonroot-v2 -z argocd-repo-server -n
oc adm policy add-scc-to-user anyuid -z argocd-redis -n
oc adm policy add-scc-to-user nonroot-v2 -z argocd-application-controller -n
oc adm policy add-scc-to-user nonroot-v2 -z gitops-agent -n
```
{% hint style="info" %}
**Verify SCC assignments**
After applying SCCs, verify the assignments with `oc describe scc nonroot-v2 | grep -A 10 'Users:'` and `oc adm policy who-can use scc/nonroot-v2`. Then delete the failing pods to allow them to restart with the correct security context.
{% endhint %}
{% hint style="info" %}
**OpenShift Operator alternative**
If the `CDS_GITOPS_OPERATOR` feature flag is enabled for your account, you can use the Harness GitOps Agent Operator for OpenShift, which handles SCC configuration automatically. For more information, go to [GitOps Agent with OpenShift Operator](/continuous-delivery/use-gitops/gitops-entities/agents/gitops-agent-with-openshift-operator.md).
{% endhint %}
Operator-based manifest support
Before installing the Harness GitOps Agent, be aware of the following limitations regarding operator-based manifests:
* **OpenShift clusters**: Have Operator Lifecycle Manager (OLM) already built-in, so operator-based manifests are supported out of the box.
* **Vanilla Kubernetes clusters**: Require manual setup of OLM, the kubectl-operator plugin, and other CRDs before using operator-based manifests.
**Recommendation**: For vanilla Kubernetes clusters, use the **Helm Chart** or **Kubernetes YAML** manifest options instead of operator-based manifests.
## Operational errors
GitOps application shows Unknown status and cannot be synced or deleted in Harness
When Harness cannot reach the application on Argo CD, open the Delete Application dialog and select **Force delete from Harness**. This checkbox appears only for unreachable applications.
Go to [Manage GitOps Applications](/continuous-delivery/use-gitops/application/manage-gitops-applications.md#force-delete-from-harness) to review step-by-step instructions, minimum component versions, and Terraform delete options.
Error: "Finalizer detected"
The message `failed to delete app in argo: failed to execute delete app task: rpc error: code = Unknown desc = finalizer detected,` indicates that the application you are trying to delete has a finalizer. If a finalizer is used, Argo CD does not delete the application until its resources are deleted. Therefore, the Harness GitOps Agent reconciles the existing application.
To delete the application, remove the finalizer or delete its resources. Removing the finalizer causes the app to be deleted automatically. For more information about the Argo CD app deletion finalizer, go to the [Argo CD documentation](https://argo-cd.readthedocs.io/), switch to the [supported Argo CD version](/continuous-delivery/new-to-continuous-delivery/cd-integrations.md), and then search for the app deletion finalizer.
Error: Unable to delete or create app due to "error: create not allowed while custom resource definition is terminating"
During creation or deletion of a GitOps app, if the process fails with the message `failed to create app in argo: failed to execute create app task: rpc error: code = Unknown desc = error creating application: create not allowed while custom resource definition is terminating`, or a similar message about a CRD stuck in termination state, the cause is most likely a CRD resource pending deletion because it has a finalizer.
For this CRD to complete termination, remove the finalizer from the pending resource.
The CRDs most likely causing this are one of these three: `applications.argoproj.io`, `applicationsets.argoproj.io`, or `appprojects.argoproj.io`.
Run the following command to get pending resources for the CRD that are stuck in termination. You can check the status of any CRD using `kubectl get crd` and then check any of these using `kubectl describe crd `.
```bash
kubectl get -n
```
```bash
kubectl patch -n --type json --patch="[{ \"op\": \"remove\", \"path\": \"/metadata/finalizers\" }]"
```
For example, if the CRD `applications.argoproj.io` is stuck in the `TERMINATING` state in the `harness` namespace, this is how to verify and patch the resource causing it to be stuck.
```bash
kubectl get applications.argoproj.io -n harness
```
```
NAME STATUS SYNC
test-app Unknown Unknown
```
```bash
kubectl patch applications.argoproj.io test-app -n harness --type json --patch="[{ \"op\": \"remove\", \"path\": \"/metadata/finalizers\" }]"
```
```
applications.argoproj.io/test-app patched
```
This lets the CRD `applications.argoproj.io` terminate gracefully.
{% hint style="info" %}
CRDs are cluster-scoped, and the resources themselves can be cluster-scoped or namespace-scoped. Pay attention to the usage of the `-n` (namespace) flag when running these commands.
{% endhint %}
If multiple resources are causing this error, go to [this GitHub issue comment](https://github.com/argoproj/argo-cd/issues/1329#issuecomment-1247176754) for a workaround.
Issue: Agent degraded when installing a Bring Your Own Argo CD (BYOA) agent with a Helm chart
Run the following script with the name of the agent as the argument. The agent name should match the name shown in the Harness GitOps UI.
```bash
#!/bin/sh
# Extract values from the existing ConfigMap argocd-cmd-params-cm
REDIS_SERVER=$(kubectl get configmap -n argocd argocd-cmd-params-cm -o json | jq -r '.data["redis.server"]')
ARGOCD_SERVER_REPO_SERVER=$(kubectl get configmap -n argocd argocd-cmd-params-cm -o json | jq -r '.data["repo.server"]')
configmap_name=$1
agent_name=$1
echo $REDIS_SERVER
echo $ARGOCD_SERVER_REPO_SERVER
kubectl patch configmap -n argocd "$configmap_name" --type merge -p '{"data": {"ARGOCD_SERVER_REPO_SERVER": "'$ARGOCD_SERVER_REPO_SERVER'", "REDIS_SERVER": "'$REDIS_SERVER'", "GITOPS_ARGOCD_REDIS_HA": "redis-ha", "GITOPS_ARGOCD_REDIS_HA_PROXY": "redis-ha-haproxy"}}'
# Comment out the following command if HA mode is not used
kubectl patch deployment $configmap_name -n argocd --type=json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["/app/agent", "--redis", "'"${REDIS_SERVER}"'"]}]'
# Restart agent
kubectl rollout restart deployment -n argocd $agent_name
```
After you run the script, verify that it made the following changes to the ConfigMap. Angle brackets (`<` and `>`) indicate where your release name appears.
```
ARGOCD_SERVER_REPO_SERVER: "-argocd-repo-server:8081"
REDIS_SERVER: "-argocd-redis-ha-haproxy:6379"
GITOPS_ARGOCD_REDIS_HA: "redis-ha"
GITOPS_ARGOCD_REDIS_HA_PROXY: "redis-ha-haproxy"
```
Error: "rpc error: code = InvalidArgument desc = existing cluster spec is different;"
This error indicates that the GitOps entity you are trying to create already exists. It might exist in one of the following locations:
* A different scope (account, organization, or project) in Harness.
* A different Argo CD project that is not mapped to a Harness project.
Issue: "rpc error: code = Unknown desc = error testing repository connectivity: authorization failed"
This error occurs when an agent is unable to connect to a repo:
* Review GitHub account rate limitations before attempting to connect to a repo.
* For more information about managing rate limiting in GitHub, go to [Enabling rate limits for Git](https://docs.github.com/en/enterprise-server@3.10/admin/configuration/configuring-user-applications-for-your-enterprise/configuring-rate-limits#enabling-rate-limits-for-git).
Error: "NOAUTH Authentication required" when the app controller is up before the redis-secret is created
There is a known issue in Argo when the app controller or other Argo CD service pod is up before the redis-secret is created. For more information, go to [Redis NOAUTH failures](https://github.com/argoproj/argo-helm/issues/2836#issuecomment-2636975946). This error happens in particular when using a manifest deployment rather than a Helm deployment, because Helm deployments have hooks that ensure everything comes up in the right order.
The workaround is to restart the app controller, agent, and any other Argo CD service pod or agent that is failing with this error.
Error: GitOps agent pod stuck in CrashLoopBackoff: unauthorized agent
The agent log shows a message similar to: "This is an unauthorized agent. Sleeping for 15 minutes and then shutting down. Check your settings in Harness UI."
This issue occurs because the token in the agent's YAML does not match the public key in the GitOps service database. The GitOps agent goes into a CrashLoopBackoff state due to an authentication failure with the GitOps service.
This problem typically surfaces after re-enabling authentication if the agent previously operated in unauthenticated mode.
* Ensure the agent's YAML file is updated with the correct authentication token, matching the public key in the database. After updating the YAML file, redeploy the agent to authenticate it properly.
Application creation fails due to missing "project" field (Argo CD 2.12 change)
This is a backward-incompatible change in **Argo CD 2.12**. The project field, previously optional, is mandatory for repository access.
With the upgrade to Argo CD 2.12, the project field is mandatory for checking access to repositories at the project-level scope. As a result, Harness GitOps requests that did not explicitly include the project field began to fail, leading to issues in the GitOps application creation flow.
Additionally, Argo CD 2.12 introduced stricter controls on the use of the repository Kubernetes secret. Previously, an Application or ApplicationSet could use any cluster secret that matched the URL specified in the `repoUrl` field. Starting with Argo CD 2.12, the project field of an application must match the project field of the cluster secret for access to be granted.
**Impact**
If a repository is scoped to `project-a`, an application associated with `project-b` can no longer access that repository. To maintain access to the cluster secret across multiple projects, the project field on the repository Kubernetes secret must be `unset` or explicitly scoped for multiple projects.
**Action items**
1. Update application configurations to include the `project` field. When creating new GitOps applications, ensure that the project field is explicitly included in the configuration.
2. Update cluster secrets to support multiple projects. If you have cluster secrets that applications across multiple projects need to access, `unset` the project field in the cluster secret configuration. This ensures that the cluster secret is accessible by applications in different Argo CD projects.
Argo CD AppSet shows "Degraded" status with project-scoped repos
**Argo CD v2.x limitation**
This is a known, recurring issue. For more information, see GitHub issue [#21016](https://github.com/argoproj/argo-cd/issues/21016). There is no fix in Argo CD v2.x; upgrade to v3.x for the resolution.
**Issue**
Your ApplicationSet stays in **Degraded** status and you see errors like:
`rpc error: code = Internal desc = unable to checkout git repo ...` `fatal: could not read Username for 'https://github.com': terminal prompts disabled`
**Details**
* **Expected**: Your GitOps Application should sync normally.
* **Observed**: Sync fails with `ApplicationGenerationFromParamsError` and authentication errors.
* **Impact**: You cannot deploy using GitOps.
* **Reproducibility**: Not a regression. This happens any time you use a project-scoped repo secret with an AppSet.
* **Root cause**: The Argo CD v2.x AppSet controller cannot authenticate when the secret has a `data.project` field (project-scoped credentials).
**Workaround**
1. In the `harness-gitops` namespace, edit your Git repository secret.
2. Remove the `data.project` key so the repo becomes cluster-scoped.
3. Save and allow Argo CD to reconcile. Your ApplicationSet recovers.
## YAML best practices for probe definitions
If you use custom health checks or probes in your GitOps configuration (for example, Argo CD resource health checks with inline scripts), incorrect YAML formatting of multi-line values is a common source of sync failures.
Common error: "cannot unmarshal" in probe definitions
An error like `cannot unmarshal !!str ... into ...` during sync typically points to a YAML syntax issue in a multi-line string value, not a problem with the command itself. This happens when a multi-line command is written inline without proper YAML scalar notation.
**Use block scalars for multi-line commands**
YAML provides two block scalar styles for multi-line strings:
* **Literal block scalar (`|`)**: Preserves line breaks exactly as written. Use this for shell scripts and multi-line commands.
* **Folded block scalar (`>`)**: Folds line breaks into spaces. Use this for long text that should become a single line.
**Before (broken)**: Inline multi-line string causes unmarshaling errors.
```yaml
exec:
command:
- /bin/sh
- -c
- curl -s http://localhost:8080/health
&& curl -s http://localhost:8080/ready
| grep -q '"status":"ok"'
```
**After (correct)**: Using the `|` block scalar keeps each line intact.
```yaml
exec:
command:
- /bin/sh
- -c
- |
curl -s http://localhost:8080/health \
&& curl -s http://localhost:8080/ready \
| grep -q '"status":"ok"'
```
{% hint style="info" %}
**Check for YAML multi-line issues first**
When you see "cannot unmarshal" errors in GitOps sync operations, check your YAML for multi-line values first. The `|` block scalar is almost always what you want for inline shell commands in probe definitions.
{% endhint %}
Applications missing in GitOps for the kube-system namespace
**Issue**
Apps deployed in the reserved `kube-system` namespace do not appear in GitOps reconciliation or in the UI.
**Impact**
Non-system apps in `kube-system` are not visible or reconciled, causing deployment issues.
**Workaround**
* Avoid deploying non-system apps to `kube-system`.
* Use dedicated namespaces, such as `argocd`, instead.
Helm 4 upgrade breaks SOPS-encrypted values after upgrading to Argo CD 3.5.1
Starting with Harness GitOps Agent **0.126.1**, the Agent ships with **Argo CD v3.5.1**, which bundles **Helm v4**. Helm 4 introduces a new, backward-incompatible plugin system. If your GitOps Applications use SOPS-encrypted Helm values through the `helm-secrets` plugin, the Agent must be on a release that includes Helm 4 plugin support, or it fails to render Helm charts that use `helm secrets`.
You are affected only if your GitOps Applications use SOPS-encrypted values through the `helm-secrets` plugin. If you do not use SOPS or `helm-secrets`, no action is needed.
**Issue**
After the Agent picks up Argo CD v3.5.1, affected Applications fail to sync with a manifest-generation error such as:
```
helm template ... failed: Error: unknown command "secrets" for "helm"
```
The error is often shown as `(cached)`. Because Argo CD caches manifest-generation results, a manual hard refresh does not clear it, since the fix requires the upgraded Agent image.
**Cause**
Helm 4 replaced the legacy plugin format with a new plugin API. The previous `helm-secrets` packaging registers under Helm 4 as a getter only, so the `helm secrets` command is no longer available.
**Resolution**
GitOps Agent release **0.126.1** updates the bundled tooling to Helm 4 compatible `helm-secrets` components so the plugin is registered correctly again. No changes to your Applications or repositories are needed.
1. Upgrade the GitOps Agent to **0.126.1** or later, using your usual Agent upgrade path (Helm chart or installer manifest in the Harness UI).
2. Wait for the Agent pods to roll out and become healthy.
3. Re-sync the affected Applications. The previously cached failure clears automatically once the Agent generates valid manifests.
{% hint style="info" %}
**Verify the upgrade**
After the rollout, confirm the plugin is healthy from the repo-server or Agent container:
* `helm version` reports v4.x.
* `helm plugin list` shows `secrets` as an installed plugin, not listed as a legacy getter.
* `helm secrets --help` returns successfully.
* The affected Application syncs without the `unknown command "secrets"` error.
{% endhint %}
This change is internal to the Agent image. Your Application specs, repositories, and SOPS key configuration are unaffected.
For general guidance on moving from Argo CD 3.4 to 3.5, go to [Upgrading Argo CD 3.4 to 3.5](https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/3.4-3.5/).
If issues persist after upgrading, capture the Application's manifest-generation error and the output of `helm plugin list` from the repo-server container when contacting Harness Support.
## Next steps
* Go to [Install a GitOps Agent](/continuous-delivery/use-gitops/gitops-entities/agents/install-a-harness-git-ops-agent.md) to review agent installation options.
* Go to [Manage GitOps Applications](/continuous-delivery/use-gitops/application/manage-gitops-applications.md) to manage and delete applications.
{% @harness-feedback/feedback %}