> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-integration/troubleshooting-and-resources/troubleshoot-harness-ci/debug-mode.md). # Debug with SSH Use debug mode to troubleshoot remote builds {% hint style="info" %} This is an early access feature and is behind the feature flag `CI_REMOTE_DEBUG`. Contact [Harness Support](mailto:support@harness.io) to enable the feature. {% endhint %} You can use the Harness CI **Re-run in Debug Mode** feature to quickly and securely troubleshoot remote builds by SSH-ing into a debug session on the build's remote host. With debug mode, you: * Get temporary, secure access to the build environment without the need for permanent permission changes. * Troubleshoot directly in the build environment, not a recreation on your local machine. * Troubleshoot in real time, rather than waiting for logs to output after each build. You can also use **Re-run in Debug Mode** to troubleshoot local builds. {% embed url="" %} ### Debug mode requirements {% hint style="info" %} Debug mode is available if the following conditions are met: * You have the feature flag `CI_REMOTE_DEBUG` enabled. Contact [Harness Support](mailto:support@harness.io) to enable this feature. * The build fails at a [Run step](/continuous-integration/use-harness-ci/use-harness-ci/run-step-settings.md) with a Bash, Shell, Python, or PowerShell Core (`pwsh`) script in a **Build** (`CI`) stage. * The build runs on Linux or macOS [build infrastructures](/continuous-integration/use-harness-ci/use-harness-ci/set-up-build-infrastructure/which-build-infrastructure-is-right-for-me.md). Windows is not currently supported. Debug mode is never available for a pipeline's first build. You must run the pipeline at least once before you can run the build in debug mode. {% endhint %} If debug mode isn't available for your pipeline, you might consider using [Harness AI](/continuous-integration/troubleshooting-and-resources/troubleshoot-harness-ci/ai.md). ### Enable debug mode {% tabs %} {% tab title="Remote builds" %} If your build meets the [debug mode requirements](#debug-mode-requirements), use these steps to use debug mode for builds on Harness Cloud, self-managed VM, or Kubernetes cluster build infrastructures. 1. Create a [Harness API key and token](/harness-platform/use-harness-platform/automation/api/add-and-manage-api-keys.md) with `pipeline execution` permissions. You must have pipeline execution permissions at the [account scope](/harness-platform/use-harness-platform/platform-access-control.md#permissions-hierarchy-scopes) in order for the token to have those permissions. 2. Navigate to the [**Builds**](/continuous-integration/use-harness-ci/use-harness-ci/viewing-builds.md), **Executions**, or **Execution History** page. 3. Locate the build you want to troubleshoot, select **More Options** (⋮), and select **Re-run in Debug Mode**.
4. Wait while the build runs. If the Run step fails, the build stops and generates log output with an SSH command you can use to SSH into the session on the remote host. The SSH command is formatted as follows. Replace `{harness pat}` with your [token](/harness-platform/use-harness-platform/automation/api/add-and-manage-api-keys.md) that has `pipeline execution` permissions. Copy the exact command from the log — the host shown may be a specific node such as `ssh1.harness.io`. ``` ssh {harness pat}::@ssh.harness.io ```
5. While in the debug session, use the CLI to reproduce and troubleshoot the issue on the remote host. 6. To terminate the debug session, abort the build, and then run a new build to determine if the issue is resolved. Sessions automatically terminate after one hour or at the step timeout limit, whichever occurs first. {% endtab %} {% tab title="Local builds" %} If your build meets the [debug mode requirements](#debug-mode-requirements), use these steps to use debug mode for builds running on [local runner build infrastructure](/continuous-integration/use-harness-ci/use-harness-ci/set-up-build-infrastructure/define-a-docker-build-infrastructure.md), on Linux or macOS. The Run step executes directly on the host machine (or its Docker runtime), so `tmate` must be installed on that host. {% hint style="warning" %} The following are not supported for local runner debug mode: * Windows build infrastructures * Self-Managed Enterprise Edition (SMEE) {% endhint %} {% hint style="info" %} **Network prerequisites** Debug mode uses egress-only connectivity. The runner host must be able to reach `ssh.harness.io` on TCP port 22 (outbound). No inbound connectivity to the runner host is required. {% endhint %} 1. Download the tmate binary matching your runner host's OS and architecture from Harness tmate releases. | Operating System | Architecture | | ---------------- | ------------------------------------------------------------------------------------------------------------------- | | Linux | [x86\_64 (`amd64`)](https://github.com/harness/tmate/releases/download/1.0/tmate-1.0-static-linux-amd64.tar.xz) | | Linux | [ARM64 (`arm64v8`)](https://github.com/harness/tmate/releases/download/1.0/tmate-1.0-static-linux-arm64v8.tar.xz) | | macOS | [Apple Silicon (`arm64`)](https://github.com/harness/tmate/releases/download/1.0/tmate-1.0-static-mac-arm64.tar.xz) |

Apple Silicon (M1/M2/M3) is currently the only validated macOS architecture for local runner hosts.

2. Extract the archive on the runner's host machine, place the `tmate` binary in a dedicated directory (for example `/Users//bin/tmate` on macOS or `/opt/harness/bin/tmate` on Linux), and make it executable: ```bash chmod +x ``` 3. In your CI stage, under Infrastructure, set Platform to the `os` and `arch` that match your runner host (for example, `os: Linux` / `arch: Amd64`, or `os: MacOS` / `arch: Arm64`). 4. Under Infrastructure > Advanced > Shared Paths, add the host directory containing the `tmate` binary (for example, `/Users//bin` on macOS or `/opt/harness/bin` on Linux). This mounts the directory on the runner host so it is accessible inside the step's execution container. 5. In your Run step, expand Environment Variables and add the `TMATE_PATH` variable with the full absolute path to the binary (for example, `/Users//bin/tmate` or `/opt/harness/bin/tmate`). This path must sit inside the Shared Paths directory from the previous step.

TMATE_PATH is required

If TMATE_PATH is omitted, or points to a path outside the directory listed in Shared Paths, debug initialization fails with: TMATE_PATH is empty. When running remote debug in local, you must specify tmate path and set it as shared path.

6. In your CI stage, under Advanced, add a **Delegate Selector** tag matching your local delegate. This pins the build to the host where tmate is installed. Go to [define a Docker build infrastructure](/continuous-integration/use-harness-ci/use-harness-ci/set-up-build-infrastructure/define-a-docker-build-infrastructure.md) to configure delegate selectors. 7. Create a [Harness API key and token](/harness-platform/use-harness-platform/automation/api/add-and-manage-api-keys.md) with `pipeline execution` permissions. You must have pipeline execution permissions at the [account scope](/harness-platform/use-harness-platform/platform-access-control.md#permissions-hierarchy-scopes) in order for the token to have those permissions. 8. Navigate to the [**Builds**](/continuous-integration/use-harness-ci/use-harness-ci/viewing-builds.md), Executions, or Execution History page. 9. Locate the build you want to troubleshoot, select **More Options** (⋮), and select **Re-run in Debug Mode**. ![Builds page with the More Options menu open and Re-run in Debug Mode highlighted](https://4226796345-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FqKtVmwAGTfGQS1MVC97G%2Fuploads%2Fgit-blob-400caa99877143766b880b3590d0a58852476f15%2Fci-rerun-build-in-debug-mode.png?alt=media) 10. Wait while the build runs. If the Run step fails, the build stops and generates log output with an SSH command you can use to SSH into the debug session on the runner host. The SSH command is formatted as follows. Replace `{harness pat}` with your [token](/harness-platform/use-harness-platform/automation/api/add-and-manage-api-keys.md) that has `pipeline execution` permissions. Copy the exact command from the log — the host shown may be a specific node such as `ssh1.harness.io`. ```bash ssh {harness pat}::@ssh.harness.io ``` ![Run step logs showing the SSH command for the debug session](https://4226796345-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FqKtVmwAGTfGQS1MVC97G%2Fuploads%2Fgit-blob-74be90f625874d350e0158b560436a3a86b92966%2Fdebug-remote-build-links.png?alt=media) 11. While in the debug session, use the CLI to reproduce and troubleshoot the issue. 12. To terminate the debug session, abort the build, and then run a new build to determine if the issue is resolved. {% endtab %} {% endtabs %} {% hint style="info" %} Session timeout duration: * **Linux:** The debug session automatically terminates after one hour or at the step timeout limit, whichever occurs first. * **macOS:** The debug session does not terminate automatically; abort the build to end the session. You can force a build to fail if you need to troubleshoot pipelines that appear to build successfully but still need remote troubleshooting. To do this, add a [Run step](/continuous-integration/use-harness-ci/use-harness-ci/run-step-settings.md) with the command `exit 1`. This forces the build to fail so you can re-run it in debug mode. {% endhint %} {% @harness-feedback/feedback %}