Skip to main content

Sentiment & qualitative analysis

With SCM integrations, you can generate an in-depth report that reveals how much faster your team is with AI, assesses code quality, and collects feedback on their overall experience with the AI assistant.

This report provides both qualitative and sentiment analysis of your team's interactions with AI tools, helping you understand productivity and satisfaction improvements.

When to use

This report is ideal for teams that want to:

  • Quantify AI’s impact: Utilize SCM data to understand tangible improvements in metrics like delivery speed, code quality, and collaboration efficiency.
  • Compare AI adoption: Analyze the performance differences between teams using AI tools and those relying on traditional workflows.
  • Understand developer experience: Gather qualitative feedback and sentiment data to evaluate how AI tools enhance or hinder satisfaction and innovation.
  • Drive data-driven decisions: Use measurable SCM-based factors like pull request metrics and commit history to pinpoint areas of success or improvement when adopting AI tools.

Generate the report

  • Go to the AI Productivity Insights section in the left menu of your Harness project.

  • Select the type of report as Qualitative & Sentiment Analysis and click on Continue

Step 1: Create or select SCM integrations

Select the SCM integration from the available list and click on Next.

Step 2: Choose your AI Tool

Choose the AI coding tool that your team is using.

  • Google Gemini Code Assist
  • Harness AI Code Assistant
  • GitHub Copilot

Step 3: Configure the report settings

  • Add a name for the report
  • Define the objective as in what you want to measure out of this report
    • Teams Using AI Tools vs. Teams Not Using AI: Compare performance between teams that use AI tools and those that do not. This analysis helps identify the tangible benefits AI tools bring to the development process.
    • Note that the support for assessing how the performance of the same team changes over a specified time period, such as before and after implementing an AI tool is currently not available but will be added in the future.

Step 4: Select and configure insights

You can configure the following insights to track team velocity and quality:

Velocity measures the team's delivery speed by tracking key performance indicators such as PR Lead Time, Number of Commits, Number of PRs Created, and Number of PRs Merged.

  • PR lead time: The time taken from code being committed to running successfully in production.
  • Number of commits: The average number of code commits made per developer during a specific period.
  • Number of PRs created: The average number of pull requests created by developers within a given time frame.
  • Number of PRs merged: The average number of pull requests successfully merged across all developers within the given time frame.

Velocity metrics & ranges

MetricWorst ValueBest Value
PR Lead Time9 days4 days
Average Commits2.5 commits3.75 commits
Average PRs Created1 PR2 PRs
Average PRs Merged1 PR2 PRs
Average New Lines20 lines400 lines

Step 5: Add Participants and build cohorts

Group participants into the following categories:

  • Using AI Tools: Developers actively using AI tools like GitHub Copilot or Harness AI Code Assistant.
  • Not Using AI Tools: Developers not using any AI assistance tools.

Upload cohort details

Use a CSV to upload participant information such as name and email. Use the provided template if you are unsure about formatting. Note that email is optional and if provided the email will be used to send out the survey in the respective emails.

Step 6: Review and confirm the configuration

Before proceeding, check the following:

  • Ensure the selected categories and questions are correct.
  • Participants are distributed across groups without strict requirements.

Click Create Report once ready.

Step 7: Send the survey

Distribute the survey using the following methods:

  • Send Now: Email the survey to participants immediately.
  • Send Later: Save the survey link for manual distribution at another time.