Skip to main content

Ingest Results from Custom or Unsupported Scanners

You can ingest custom Issues from any scanning tool. This topic describes how to ingest data from scan tools that currently have no integration in STO.

By ingesting your custom Issues, you can benefit from STO's refinement, deduplication, and correlation features. Harness handles your issue data the same way as data from supported scanners.

Requirements

  • You have a valid STO license and access to the Harness platform.
  • You have a JSON file of the issues you want to ingest. The data file must match the JSON format specified below.

Required Steps

  1. In your Harness pipeline, go to the Overview tab of the security stage and enter a shared path such as /shared/customer_artifacts.
  2. Generate your issues data in the required JSON format described below and then save it in the shared folder.
    You might want to set up a Run step to generate your scans automatically whenever the pipeline runs. Go to Set up ingestionOnly in an STO pipeline for an example.
  3. Add a Security step and configure the scanner to ingest the results of the scan. These settings are required:
    1. policy_type = ingestionOnly
    2. scan_type — The type the scanned object: containerrepositoryinstance, or configuration
    3. product_name = external
    4. product_config_name = default
    5. manual_upload_filename — The filename of your issues data file.
    6. customer_artifacts_path — The shared folder for the issues data file.

JSON data format reference

The following example illustrates the required format for your data:

{  
"meta":{
"key":[
"issueName",
"fileName"
],
"author":"Jane Doe"
},
"issues":[
{
"scanTool":"MySastTool",
"issueName":"Cross Site Scripting",
"issueDescription":"Lorem ipsum...",
"fileName":"homepage-jobs.php",
"remediationSteps":"Fix me fast.",
"risk":"high",
"severity":8,
"status":"open",
"referenceIdentifiers":[
{
"type":"cwe",
"id":"79"
}
]
}
]
}

The basic schema includes a “meta” section, which requires a “key” — this is the name of the attribute used to deduplicate multiple occurrences of an issue. In the example data file above, "key" = "issueName". Thus if the data includes multiple occurrences of an issue with the same "issueName", the pipeline combines these occurrences into one issue. The resulting issue includes a list of all occurrences and the data for each individual occurrence.

The key used for deduplication must be a Harness field. Do not try to deduplicate based on non-Harness fields.The full JSON takes the form:

"meta":   
{ ... },
"issues": [
{ "issue-1" : "data" },
{ "issue-2" : "data" },
{ "issue-3" : "data" }
}

Required fields

NameFormatDescription
issueNameStringName of vulnerability, license issue, compliance issue, etc.
issueDescriptionString (long)Description of vulnerability, license issue, compliance issue, etc.
scanToolStringThe name of the scanning tool (e.g. “SonarQube”)
severityFloatCVSS 3.0 score (a number from 1.0-10.0)
NameFormatDescription
confidenceFloatDerived from the tool output.
cvssString (long)Derived from the tool output.
fileNameStringRecommended to assist in triaging errors (if present).
hostStringRecommended to assist in triaging errors (if present).
ipStringRecommended to assist in triaging errors (if present).
issueTypeStringType of issue (e.g. vulnerability, license issue, compliance issue, etc.)
lineNumberStringRecommended to assist in triaging errors (if present).
linkStringRecommended to assist in triaging errors (if present).
portIntegerRecommended to assist in triaging errors (if present).
productStringLogical metadata field that can be used for tracking of product(s).
projectStringLogical metadata field that can be used for tracking of project(s)
remediationStepsString (long)Remediation instructions, often provided by the scan tool.
scanSeverityStringThe severity as reported by the scan tool.
scanStatusStringRecommended for measuring scan duration and status.
tagsStringLogical metadata tags, which can be leveraged to describe asset owners, teams, business units, etc.
urlStringRecommended to assist in triaging errors (if present).

Optional fields

NameFormatDescription
authorStringLogical metadata field designed to track the owner of the scan result.
effortString (long)Logical metadata field designed to gauge the required effort to remediate a vulnerability.
exploitabilityScoreFloatDerived from the tool output.
imageLayerIdStringMetadata field to track image layer ID from containers.
imageNamespaceStringLogical metadata field.
impactScoreStringDerived from the tool output.
libraryNameStringDerived from the tool output.
licenseStringDerived from the tool output.
linesOfCodeImpactedStringRecommended to assist in triaging errors (if present).
referenceIdentifiersArrayAn array of Vulnerability identifiers, such as cve, cwe, etc. For example:
“referenceIdentifiers”: [     {“type” : “cve”,“id” : “79”},     {"type" : "cwe", "id" : "83"}]

The type value must be lowercase. |

Custom fields

You can add custom fields to an issue. The only restriction is that you cannot use any of the reserved keywords listed above. To include raw, unrefined data, add the prefix "_raw" to the field name. For example, you can add the following "_raw" fields to an issue:

{  
"testName":"hardcode_tmp_directory",
......
"referenceIdentifiers":[
{
"type":"cwe",
"id":"79"
}
],
"_rawIssueCwe": {
"id" : 377,
"link" : "https://cwe.mitre.org/data.definitions/377.html"
},
"_rawMoreInfo" : "https://bandit.readthedocs.io/en/1.7.4/plugins/b108_hardcoded_tmp_directory.html",
"_rawColOffset":31
}

The custom fields will get grouped together at the end of the issue details like this:

Reserved Keywords

The following keywords are reserved and cannot be used in your JSON file:

  • alertRulesets
  • customerId
  • discoveryIssueId
  • discoveryRunTime
  • discoveryTimespan
  • ignore
  • ignoreRulesets
  • jobId
  • policyId
  • policyName
  • refinementVersion
  • remediationRunTime
  • remediationTimespan
  • runTime
  • scenarioId
  • severityCode
  • target
  • targetId