For the complete documentation index, see llms.txt. This page is also available as Markdown.

Run chaos experiments as GitLab pipelines

Guide to run a chaos experiment as a GitLab pipeline

This tutorial explains how you can create chaos experiments using Harness CE and run them as GitLab pipelines. Chaos experiments in Harness are created the same way in the chaos engineering module, irrespective of where they are invoked from.

  1. Create a chaos experiment in the Harness Chaos Engineering module. Execute this experiment to verify the configuration and ensure that the resilience probes are working as expected. The experiment ID and resilience score determined from this experiment run will be used to integrate the experiment with GitLab.

    chaos experiment with ID and resilience score
  • You will need the account scope details such as ACCOUNT_ID, PROJECT_ID, and CHAOS_EXPERIMENT_ID. You can get these values from your session URL. For example,

  • The strings marked in asterisk are the account ID, project ID, and chaos experiment IDs respectively.

  • To set variables in GitLab CI, go to set variables for a project.

  • For example, make sure the API_KEY is set as protected, masked, and as a secret variable.

  • From your account profile page, generate an API key and copy it to a safe location (you will need it further).

  1. Create a launch script. HCE APIs are used to invoke or launch a chaos experiment from the pipeline.

    To simplify creating an API call with the required secure parameters and data, a CLI tool is provided. Use this tool to create an appropriate API command to include in the pipeline script.

    Below is a sample launch script.

    DEMO

    Go to GitLab demo for a sample configuration of the chaos launch script. You can include this script in the GitLab YAML file. This is a sample to include one single chaos experiment, but the same can be repeated so as to be included in multiple chaos experiments.

  2. Insert chaos experiments into .gitlab-ci.yaml. You can include the above-mentioned launch script in the GitLab pipeline as a stage or a step. In the script section, add the scripts for launching, monitoring and retrieving results. For example:

    The resilience score is the result of the experiment, and it helps decide if a rollback job needs to be invoked.

  3. Retrieve the resilience score using the Harness Chaos API and take appropriate action in the pipeline. An example of how to use the Harness Chaos API is shown below.

Troubleshooting

The chaos job fails and the rollback job runs

The chaos job exits with an error when the obtained resilience score is lower than EXPECTED_RESILIENCE_SCORE, and the rollback job runs because it is set to when: on_failure. Check the resilience score and probe results of the experiment run in Harness. Fix the weakness in the application, or set EXPECTED_RESILIENCE_SCORE to the score you validated when you first ran the experiment.

The launch script returns no notify ID

The launch script pipes the CLI output to jq to read the notify ID, and the sample job installs jq with apt-get in before_script. Make sure the job image provides apt-get, or use an image that already includes jq.

The launch fails for an experiment in an organization other than default

The sample launch script does not pass --org-id, and the CLI uses the organization default when the flag is not set. Add --org-id with your organization ID to the CLI commands in the launch and verify scripts. Go to HCE CLI to review the available flags.

Last updated

Was this helpful?