How does the log verification classifiy errors

Learn how Harness Continuous Verification decides whether a log message is a known error or a new one.

Harness CV log verification answers one question for every log line produced during a deployment: "Have I seen this before in the baseline, or is this a new event I should surface?"

The decision follows a deterministic, three-step process. This page explains exactly what the algorithm keeps, what it discards, how it measures similarity, and how you can tune it with the Sensitivity setting.

---

What you will learn from this topic

• Tokenization: How every log message is broken into meaningful words and what parts are kept versus discarded.

• Baseline learning: How Harness builds a similarity model from your own baseline logs, not a generic dictionary.

• Similarity scoring: How two messages are compared and what makes them the same known error or a new unknown event.

• Sensitivity setting: How to tune the similarity threshold and frequency thresholds for your environment.

• Log feedback: How marking a cluster as Expected is remembered and applied on future runs.

---

Log line classification

The log classification process analyzes each log line, learns patterns from your baseline logs, and assigns a similarity score to determine whether an event matches a known error or represents a new event.

Extract the meaningful parts

Every log line is broken into words. Some parts are automatically discarded because they add noise without adding meaning.

Category	Examples	Treatment
Timestamps	2024-01-15 10:23:45, T10:23:45Z	Discarded
Line numbers, thread numbers, process IDs, ports, HTTP status codes	line 42, pid=1234, :8080, 404	Discarded
Punctuation and short symbols	:, ;, []	Discarded
Class names, method names, package names	com.example.UserService, processRequest	Compared
Message text	not found, connection refused, timeout	Compared
Identifiers such as UUIDs, request IDs, and hashes	550e8400-e29b-41d4-a716-446655440000	Numeric digit runs normalized; letter portions compared

Identifiers such as UUIDs are not discarded, but they are not compared verbatim either. The algorithm replaces digit runs inside each token with positional markers (for example, 550e8400 becomes 0xxe1xx), while letter portions are kept. The effect is that two log lines differing only by a UUID still produce very similar tokens and cluster together.

Learn from your baseline logs

Harness does not use a generic dictionary to judge similarity. For every verification run, Harness learns from your baseline logs which words tend to appear near each other. The comparison is tailored to your service's own vocabulary and naming conventions, not a one-size-fits-all model.

Nothing is carried over across different services or accounts. Every verification learns its own patterns from its own baseline.

Score similarity

Each log message is reduced to a similarity fingerprint. Two messages are compared and given a similarity score. If that score reaches the required threshold, the new message is considered the same error as one already seen in the baseline. If it does not, it is treated as a new, unknown event.

The required threshold depends on the Sensitivity setting. Go to Sensitivity and log classification to review how each level affects the threshold.

---

Common questions

Two errors have the same first line but different stack traces. Are they the same cluster?

The comparison looks at the entire log event, not just the first line. Stack traces contribute a large amount of meaningful content through class names, method names, and package names.

• If only a small part of the trace differs (for example, one or two frames out of a long trace), the messages are still similar enough and are treated as the same known error.

• If the two traces go through genuinely different classes, methods, or code paths, the overall similarity score drops and they are treated as different errors.

Two exceptions with the same top-line message but different call paths are treated as different errors, which is almost always the intended behavior because it usually signals a different root cause.

"ID ABC not found" versus "ID ABC123 not found": does the identifier make them different clusters?

In most cases, no. The algorithm normalizes the digit runs inside identifier tokens (replacing them with positional markers), so two UUIDs that differ only in their numeric portions produce near-identical tokens. When every other word in the message is identical, the normalized identifier does not drop the similarity score below the threshold.

The reverse also applies: if a message is very short, a single differing identifier can represent a large fraction of the message content and may drop the similarity below the threshold, creating a new cluster. For longer messages with stack traces and structured text, this is far less likely.

What is compared, what is discarded, and what is learned?

Category	How it is treated
Timestamps, thread/process/port numbers, HTTP status codes	Discarded; does not affect the comparison
Punctuation and filler symbols	Discarded
Class names, method names, message text, log level	Compared; this drives the match
Identifiers such as UUIDs, request IDs, and hashes	Digit runs normalized; letter portions compared and not discarded
Word relationships (which words normally appear together)	Learned fresh per verification from your own baseline logs

Nothing is carried over across different services or accounts. Every verification learns its own patterns from its own baseline, so results reflect your actual system's behavior.

How do I know it only flags new errors, not old ones?

A message is called "known" only if it is highly similar to something already seen in the baseline. If it is not similar enough to anything in the baseline, it is surfaced as a new, unknown event.

The similarity threshold is intentionally strict by default to avoid the situation where real regressions are silently accepted as known. A message is mis-flagged as new only when it genuinely differs in a meaningful way from anything in the baseline, which usually means the underlying log content has actually changed.

Once you mark a cluster as Expected through log feedback, that decision is remembered and applied automatically on future runs. You do not need to re-classify the same recurring error every time.

To lower the rate of false "new error" flags without changing your feedback, lower the Sensitivity setting. Go to Sensitivity and log classification to review the available levels.

---

Sensitivity and log classification

The Sensitivity setting on the Verify step is the primary control over how strict or lenient log classification is. It affects two things: how similar two messages must be to be considered the same known error, and how much a log's frequency must change before it is flagged as unexpected.

Sensitivity	Similarity required to match a known error	Frequency change required to flag as unexpected	Overall behavior
High	~98% similar	Smaller shifts flagged (for example, ~15–20% increase)	Strictest; catches subtle changes but produces more findings to review
Medium (default)	~96% similar	Moderate shifts flagged (for example, ~25% increase)	Balanced; recommended for most production services
Low	~93% similar	Only large shifts flagged (for example, ~40–50% increase)	Most lenient; only surfaces major anomalies, best for noisy or unstable environments

Use High sensitivity if your engineers are concerned about missing real new errors. Use Low sensitivity if your environment is noisy and you are getting too many findings that turn out to be benign variations such as embedded IDs or minor wording differences.

Sensitivity is set per Verify step, so different services or environments can use different sensitivity levels. Changing sensitivity only affects future verifications. It does not retroactively reclassify past results.

Tune sensitivity before changing other settings

If your team is seeing too many false "new error" flags, try lowering Sensitivity from High to Medium, or from Medium to Low, before assuming the underlying detection logic is unreliable.

---

Log feedback and classification

When you mark a cluster as Expected through log feedback, that decision is remembered and applied automatically on future runs. You do not need to re-classify the same recurring error every time.

Go to Event preference or log feedback to set event preferences for log clusters.

---

Next steps

You now understand how Harness CV log verification tokenizes log messages, learns word relationships from your baseline, and scores similarity to classify errors as known or new. You can tune Sensitivity and use log feedback to improve classification accuracy over time.

• Interpret log verification results: Read the radar chart and event list after a verification run.

• Event preference or log feedback: Mark clusters as expected or risky to improve future verification accuracy.

• Machine learning usage in CV: Learn how the underlying ML model works for both metrics and logs.