Skip to main content

PagerDuty Notifications Integration with Spinnaker

Introduction​

PagerDuty is an incident response application that IT departments commonly use to alert, triage, and manage incidents based on different use cases.  PagerDuty empowers organizations with insights to proactively manage events that may impact customers across their IT environment.  It supports multiple integrations from Slack to Atlassian products such as Jira.

In this document, we will look into integrating PagerDuty with Spinnaker Notifications.  Spinnaker Notifications allow simple notification messages to be passed upon pipeline execution completion.  These messages will be passed on to PagerDuty to a named target.  The functionality of the PagerDuty Notifications in Spinnaker operates slightly differently from how the Slack or Email notification systems work and requires additional configuration. For more flexibility with PagerDuty's API systems and the type of data sent to PagerDuty, please explore Spinnaker's Webhook stages.

Prerequisites​

  • Permission to the Spinnaker application* PagerDuty token generated by an administrator

Instructions​

Below is a breakdown of how PagerDuty Notifications can be utilized.  

Table of Contents​

Set-Up​

Administrators will need to set up Spinnaker's UI to allow the feature to be configured.  The following needs to be added to the SpinnakerService.yml file:

spec:
  spinnakerConfig:
    profiles:
      deck:
        settings-local.js:
          window.spinnakerSettings.feature.pagerDuty = true;
 Administrators will also need to make the following changes to the Gate and Echo services in the ```SpinnakerService.yml```
spec:
  spinnakerConfig:
    profiles:
      echo:
        pager-duty: 
          enabled: true 
          token: ***
      gate: 
        pagerDuty:
          base-url: https://api.pagerduty.com 
          enabled: true 
          token: ***

Using the Integration​

  • After applying the config changes, users can log in to Spinnaker and configure their applications.  Users can select the PagerDuty Service when editing the application.* If no services are shown within the drop-down, Gate has an issue loading them, and the token/configurations should be checked.
  • Once a service is selected, Spinnaker will pull the FIRST integration key from the services integration list in PagerDuty.  It then saves the integration key as a new field on the application in Spinnaker pdApiKey.  Orca has a stage that reads this key as part of a task and uses this to call the callback notification service.https://github.com/spinnaker/orca/blob/b34b257f925bde8bc1d25b7c38696329438725cf/orca-echo/src/main/groovy/com/netflix/spinnaker/orca/echo/tasks/PageApplicationOwnerTask.groovy#L36Users can add a stage of the pageApplicationOwner type and edit the JSON, and set it like the below example:
{
    "applications": [
        "demo"
    ],
    "details": {
        "from": "Overnight Monitoring Team"
    },
    "message": "[DEMO] Application ABC has completed",
    "name": "Trigger pager duty",
    "type": "pageApplicationOwner"
}

Note that if the from field is NOT set, it defaults to the triggering via the user's email.  Other PagerDuty settings can be configured in Deck as shown here https://github.com/spinnaker/deck/blob/c768e88fbc893d0bd5dc86959320a7b7d67443e5/packages/core/src/config/settings.ts#L132-L137.

  • Once the Pipeline has executed the stage, the notification will be sent to PagerDuty via the Events API.  If other PagerDuty APIs need to be used, please explore the Webhook stage as an option.