Sharing experiment results
Overview
The share results functionality allows you to download a copy of your metrics impact results in various formats or share via URL. This enables you to:
Share the outcome of your experiments with colleagues without them needing to visit Split
Keep a record of experiment outcomes in your preferred documentation formats or applications
Conduct further manual analysis or visualizations as desired
Import the metric results into your own internal systems
View results in a tabular format
Sharing results
When viewing the Metrics impact tab, under Results, you can find the Share results dropdown on the right side of the page, above the metric cards.
To generate the results, simply click on Share results to view the dropdown of format options. Click on your chosen format and your browser will begin downloading the file. A green pop up will appear on the page when the file has finished downloading.
Clicking on Copy URL will copy a URL link to your current view in the Split app to your clipboard and will not download any file.
Available formats
You can generate a report containing your metrics impact results in the below formats:
CSV: This format contains the observed metric data and the results of the statistical analyses. The report contains one row per metric and is intended to be human readable as well as appropriate for further manual analyses or reporting.
You can find reference documentation for the fields contained in the CSV reports here.
JSON: This format contains the observed metric data, the results of the statistical analyses, as well as the statistical settings and metric definitions at the time of analysis. This format is intended for ingesting data into other tools and databases.
You can find reference documentation for the fields contained in the JSON reports here.
PDF: This format presents a visual representation of your metrics impact results, similar to the Split UI. This format is ideal for sharing results with stakeholders and for quickly understanding the high level outcomes of your tests.
An example PDF report is shown below.
Copy URL: This copies a URL link to your current view in the Split app to your clipboard.
Reference for fields in the CSV results
There is one row per metric, containing details of the observed data, the results of the statistical analyses and the metadata describing the test.
Metric
string
The metric name
Result
string
The summary of the metric result at the time of report generation
Impact
float
The measured relative impact of the comparison treatment compared to the baseline treatment, as a fraction of the baseline treatment
Impact lies above
float
The lower bound of the confidence interval on the impact
Impact lies below
float
The upper bound of the confidence interval on the impact
Baseline metric value
float
The measured value of the metric for the baseline treatment. For percentage metrics. i.e. percent of unique users, this value is expressed as a fraction
Comparison metric value
float
The measured value of the metric for the comparison treatment. For percentage metrics. i.e. percent of unique users, this value is expressed as a fraction
Comparison metric lies above
float
The lower bound of the confidence interval on the comparison metric value
Comparison metric lies below
float
The upper bound of the confidence interval on the comparison metric value
Baseline metric lies above
float
The lower bound of the confidence interval on the baseline metric value
Baseline metric lies below
float
The upper bound of the confidence interval on the baseline metric value
pValue
float
The p-Value for this metric. If multiple comparison corrections have been applied this will already have been adjusted for to control for the false discovery rate
Baseline sample size
int
Number of unique samples for the baseline treatment in the dataset used to generate this report
Comparison sample size
int
The number of unique samples for the comparison treatment in the dataset used to generate this report
Sample Ratio
string
The result of the sample ratio mismatch calculation. If this is “Mismatch” there is a sample ratio mismatch and results should not be trusted. See here for more details.
Baseline treatment
string
The baseline treatment for the statistical analysis in this report, if there is one
Comparison treatment
string
Comparison treatment for the statistical analysis in this report
Key metric
boolean
Boolean representing whether this metric was a Key metric for this test
Split
string
The feature flag name
Environment
string
The name of the Split environment in this report
Version
int
The feature flag version number in this report
Rule
string
Name of the rule which this report contains data for
Comparison target rollout percentage
float
The target percentage of comparison treatment samples compared to the sum of comparison and baseline samples, expressed in a decimal form. For example, for a 50% : 50% rollout, the targeted rollout percentage would be 0.5
Version start time
datetime
Date and time the version started (local time zone)
Version end time
datetime
Date and time the version ended (local time zone), if it has. This will be null for versions which are still active at the time of report generation.
Experiment duration
datetime
For versions which have already ended, the length of time for which the version was active, in days.
Last calculation time
datetime
Date and time the results were last updated (local time zone). Hence the results in the report are based on data received by Split between Version start time and this Last calculation time. If a calculation has not yet run this field will be equal to the Version start time.
Duration of data collection
float
Period of time over which the data used to generate this report was collected, in days. Note that this may be shorter than the Experiment duration if the results had not been recently calculated when the report was generated.
Review period state
string
The review period state. COMPLETE_BEFORE : The test was ended before a full review period COMPLETE : The test was ended after a full review period INACTIVE_BEFORE : The test is still active, but the report does not contain data for a full review period ACTIVE : The test is active and the report contains data for full review period(s) INACTIVE_BETWEEN : The test is still active and has run for at least one full review period, but the report was generated between review periods If the review period is not “ACTIVE” or “COMPLETE”, interpret your results with caution as they may not be representative of your entire customer base.
Metric tag filters applied
array
Any metric tags filters used to filter down the metrics in this report
Metric owner filters applied
array
Any metric owner filters used to filter down the metrics in this report
Experiment significance threshold
float
The significance threshold used for the statistical analyses in this report, as specified by the experiment settings at the time of report generation
Minimum sample size
int
Minimum sample size required in each treatment before statistical comparisons are run for a metric. Specified by the experiment settings at the time of report generation
Multiple comparison correction
boolean
A boolean reflecting whether multiple comparison corrections were applied to the statistical analyses in this report. Specified by the experiment settings at the time of report generation
Experimental review period
int
The experimental review period, as specified by the experiment settings at the time of report generation
Metric spread
string
ACROSS : Metric is measured across the whole sample and statistical comparisons are not possible PER : Metric is measured per unique key e.g. per user
Metric aggregation
string
TOTAL : Metric is a sum of values COUNT : Metric is a count of values RATE : Metric measures the percent of the sample who have done the event AVERAGE : Metric is an average of values NONE : Metric definition unknown
Metric ID
string
ACROSS : Metric is measured across the whole sample and statistical comparisons are not possible PER : Metric is measured per unique key e.g. per user
Traffic type
string
Traffic type
Workspace
string
The name of the Split project that contains the feature flag
Report generated on
datetime
The date and time the report was generated (local timezone)
URL
string
A link to view these results in the Split UI
Reference for fields in the JSON results
The top level JSON fields contain metadata for the report, The metrics section contains an array for each metric in the report, each containing the data describing the measurements and results of the statistical analyses for this metric.
Inside each metric is a metricDefinition array containing fields which describe the definition of the metric at the time the report was downloaded.
For metadata:
organizationName
string
The name of the Split account that contains the feature flag
split
string
The feature flag name
splitDescription
string
The feature flag description
SplitVersionNumber
int
The feature flag version in this report
versionStartTime
int
The time when the version started (unix milliseconds)
versionEndTime
int
The time when the version ended (unix milliseconds), if it has ended. This will be null for versions which are still active at the time of report generation.
lastUpdateTime
int
The time the results were last updated (unix milliseconds). Hence the results in the report are based on data received by Split between versionStartTime and lastUpdateTime.
versionDurationAtLastUpdateInMs
int
Duration of data collection, in milliseconds.
versionDurationAtLastUpdate
float
Duration of data collection, in days.
experimentDuration
int
For versions which have already ended, the length of time for which the version was active, in milliseconds.
environmentName
string
The name of the Split environment
workspaceName
string
The name of the Split project
trafficTypeName
string
The traffic type
baselineTreatment
string
The baseline treatment for the statistical analysis in this report, if there is one
comparisonTreatment
string
The comparison treatment for the statistical analysis in this report
ruleName
string
Name of the feature flag targeting rule
sampleSizeOfBaseline
int
The number of unique samples for the baseline treatment in the dataset used to generate this report
sampleSizeOfComparison
int
The number of unique samples for the comparison treatment in the dataset used to generate this report
minimumSampleSize
int
The minimum sample size required in each feature flag treatment before statistical comparisons are run for a metric. This value is specified by the experiment settings at the time of report generation
filterTags
array of strings
Any metric tags filters used to filter down the metrics in this report
filterOwners
array of strings
Any metric owner filters used to filter down the metrics in this report
srmPValue
float
The p-value output of the sample ratio mismatch calculation
significanceThreshold
float
The significance threshold used for the statistical analyses in this report, as specified by the experiment settings at the time of report generation
mcc
boolean
A boolean reflecting whether multiple comparison corrections were applied to the statistical analyses in this report. This value is specified by the experiment settings at the time of report generation
experimentalReviewPeriodState
string
The review period state. COMPLETE_BEFORE : The test was ended before a full review period COMPLETE : The test was ended after a full review period INACTIVE_BEFORE : The test is still active, but the report does not contain data for a full review period ACTIVE : The test is active and the report contains data for full review period(s) INACTIVE_BETWEEN : The test is still active and has run for at least one full review period, but the report was generated between review periods If the review period is not “ACTIVE” or “COMPLETE”, interpret your results with caution as they may not be representative of your entire customer base.
targetedRatio
float
The target fraction, of the sum of unique samples in the two selected treatments, who receive the comparison treatment. For example, for a 50% : 50% rollout, the targetedRatio would be 0.5
currentRatio
float
The actual fraction of all unique samples who received the comparison treatment
splitLink
string
A link to view these results in the Split UI
sampleRatio
string
The result of the sample ratio mismatch calculation. If this is “Mismatch” there is a sample ratio mismatch and results should not be trusted. See here for more details.
experimentalReviewPeriod
int
The experimental review period configured for this test, as specified by the experiment settings at the time of report generation
reportGenerationTime
int
The time when the report was generated (unix milliseconds)
metricsTotal
int
The total number of metrics in the report
metricsWithDesiredImpact
int
The number of metrics in the report with a statistically significant desired impact
metricsWithUndesiredImpact
int
The number of metrics in the report with a statistically significant undesired impact
metricsWithInconclusiveImpact
int
The number of metrics in the report with a statistically inconclusive impact
metricsThatNeedMoreData
int
The number of metrics in the report which need more data before a statistical comparison can be run
metricsWithComparisonNotPossible
int
The number of metrics in the report for which a statistical comparison was not possible
metricsUnavailable
int
The number of metrics in the report for which data is unavailable
metrics
Array
An array containing the measured data and the results of the statistical comparisons for each metric. See below.
For metrics:
id
int
Unique identifier for the metric
name
string
The metric name
stateSummary
string
Summary of the metric result at the time of report generation
stateDetailed
string
More granular categorization of the metric result at the time of report generation
impact
float
The measured relative impact of the comparison treatment compared to the baseline treatment, as a fraction of the baseline treatment
impactLower
float
The lower bound of the confidence interval on the impact
impactUpper
float
The upper bound of the confidence interval on the impact
baselineMetricUpper
float
The upper bound of the confidence interval on the baseline metric value
baselineMetricLower
float
The lower bound of the confidence interval on the baseline metric value
comparisonMetricUpper
float
The upper bound of the confidence interval on the comparison metric value
comparisonMetricLower
float
The lower bound of the confidence interval on the comparison metric value
sampleSizeBaseLine
int
The number of unique samples contributing to the metric result for the baseline treatment
sampleSizeComparison
int
The number of unique samples contributing to the metric result for the comparison treatment
baselineMetricValue
float
The measured value of the metric for the baseline treatment. For percentage metrics. i.e. percent of unique users, this value is expressed as a fraction
comparisonMetricValue
float
The measured value of the metric for the comparison treatment. For percentage metrics. i.e. percent of unique users, this value is expressed as a fraction
pValue
float
The p-Value for this metric. If multiple comparison corrections have been applied this will already have been adjusted for to control for the false discovery rate
mlde
float
The minimum likely detectable effect of this metric
endOfReviewPeriodMlde
float
The expected minimum likely detectable effect of this metric at the end of the next review period, assuming metric characteristics and rate of increase in the sample size remains consistent
format
string
Formatting guide for the metric result
comparisonRemainingNormality
int
The remaining number of unique samples receiving the comparison treatment needed for the comparison treatment to reach the minimum sample size
baselineRemainingNormality
int
The remaining number of unique samples receiving the baseline treatment needed for the baseline treatment to reach the minimum sample size
baselineZero
boolean
A boolean representing whether the metric value for the baseline was zero. If True, the impact will be an absolute impact rather than a relative impact
metricDefinition
array
This section contains information on the metric definition at the time of report generation. See below
key
boolean
A boolean representing whether this metric was a Key metric for this test
For metric definitions:
id
int
Unique identifier for the metric
name
string
The metric name
stateSummary
string
Summary of the metric result at the time of report generation
stateDetailed
string
More granular categorization of the metric result at the time of report generation
impact
float
The measured relative impact of the comparison treatment compared to the baseline treatment, as a fraction of the baseline treatment
impactLower
float
The lower bound of the confidence interval on the impact
impactUpper
float
The upper bound of the confidence interval on the impact
baselineMetricUpper
float
The upper bound of the confidence interval on the baseline metric value
baselineMetricLower
float
The lower bound of the confidence interval on the baseline metric value
comparisonMetricUpper
float
The upper bound of the confidence interval on the comparison metric value
comparisonMetricLower
float
The lower bound of the confidence interval on the comparison metric value
sampleSizeBaseLine
int
The number of unique samples contributing to the metric result for the baseline treatment
sampleSizeComparison
int
The number of unique samples contributing to the metric result for the comparison treatment
baselineMetricValue
float
The measured value of the metric for the baseline treatment. For percentage metrics. i.e. percent of unique users, this value is expressed as a fraction
comparisonMetricValue
float
The measured value of the metric for the comparison treatment. For percentage metrics. i.e. percent of unique users, this value is expressed as a fraction
pValue
float
The p-Value for this metric. If multiple comparison corrections have been applied this will already have been adjusted for to control for the false discovery rate
mlde
float
The minimum likely detectable effect of this metric
endOfReviewPeriodMlde
float
The expected minimum likely detectable effect of this metric at the end of the next review period, assuming metric characteristics and rate of increase in the sample size remains consistent
format
string
Formatting guide for the metric result
comparisonRemainingNormality
int
The remaining number of unique samples receiving the comparison treatment needed for the comparison treatment to reach the minimum sample size
baselineRemainingNormality
int
The remaining number of unique samples receiving the baseline treatment needed for the baseline treatment to reach the minimum sample size
baselineZero
boolean
A boolean representing whether the metric value for the baseline was zero. If True, the impact will be an absolute impact rather than a relative impact
metricDefinition
array
This section contains information on the metric definition at the time of report generation. See below
key
boolean
A boolean representing whether this metric was a Key metric for this test
For example, this metric definition as shown in the UI would have the below metricDefinition in a JSON report output:
Last updated
Was this helpful?