Debug with SSH

note

This is an early access feature and is behind the feature flag CI_REMOTE_DEBUG. Contact Harness Support to enable the feature.

You can use the Harness CI Re-run in Debug Mode feature to quickly and securely troubleshoot remote builds by SSH-ing into a debug session on the build's remote host. With debug mode, you:

• Get temporary, secure access to the build environment without the need for permanent permission changes.
• Troubleshoot directly in the build environment, not a recreation on your local machine.
• Troubleshoot in real time, rather than waiting for logs to output after each build.

You can also use Re-run in Debug Mode to troubleshoot local builds.

Debug mode requirements

info

Debug mode is available if the following conditions are met:

• You have the feature flag CI_REMOTE_DEBUG enabled. Contact Harness Support to enable this feature.
• The build fails at a Run step with a Bash, Shell, Python, or PowerShell Core (pwsh) script in a Build (CI) stage.
• The build runs on a Linux-based OS on any build infrastructure (Harness Cloud, a self-managed VM, a Kubernetes cluster, or a local runner).

Debug mode is never available for a pipeline's first build. You must run the pipeline at least once before you can run the build in debug mode.

If debug mode isn't available for your pipeline, you might consider using AIDA.

Enable debug mode

Remote builds

If your build meets the debug mode requirements, use these steps to use debug mode for builds on Harness Cloud, self-managed VM, or Kubernetes cluster build infrastructures.

1. Create a Harness API key and token with pipeline execution permissions. You must have pipeline execution permissions at the account scope in order for the token to have those permissions.

2. Navigate to the Builds, Executions, or Execution History page.

3. Locate the build you want to troubleshoot, select More Options (⋮), and select Re-run in Debug Mode.

   

4. Wait while the build runs. If the Run step fails, the build stops and generates log output with an SSH command you can use to SSH into the session on the remote host.

   The SSH command is formatted as follows. Replace {harness pat} with your token that has pipeline execution permissions.

   ssh {harness pat}:<your-harness-account-ID>:<random-session-token>@tmate.harness.io
   

5. While in the debug session, use the CLI to reproduce and troubleshoot the issue on the remote host.

6. To terminate the debug session, abort the build, and then run a new build to determine if the issue is resolved. Sessions automatically terminate after one hour or at the step timeout limit, whichever occurs first.

Local builds

If your build meets the debug mode requirements, use these steps to use debug mode for builds on local runner build infrastructures.

1. Download and install tmate on the runner's host machine.

2. If necessary, mount the install path. By default, Harness mounts the /addon path. If you installed tmate elsewhere, you need to mount that path.

3. In your CI pipeline, add the environment variable TMATE_PATH to your Run step. Set the value to the tmate mount path, such as /addon/tmate.

4. Create a Harness API key and token with pipeline execution permissions. You must have pipeline execution permissions at the account scope in order for the token to have those permissions.

5. Navigate to the Builds, Executions, or Execution History page.

6. Locate the build you want to troubleshoot, select More Options (⋮), and select Re-run in Debug Mode.

   

7. Wait while the build runs. If the Run step fails, the build stops and generates log output with an SSH command you can use to SSH into the debug session.

   The SSH command is formatted as follows. Replace {harness pat} with your token that has pipeline execution permissions.

   ssh {harness pat}:<your-harness-account-ID>:<random-session-token>@tmate.harness.io
   

8. While in the debug session, use the CLI to reproduce and troubleshoot the issue.

9. To terminate the debug session, abort the build, and then run a new build to determine if the issue is resolved. Sessions automatically terminate after one hour or at the step timeout limit, whichever occurs first.

tip

You can force a build to fail if you need to troubleshoot pipelines that appear to build successfully but still need remote troubleshooting. To do this, add a Run step with the command exit 1. This forces the build to fail so you can re-run it in debug mode.