Experiment velocity planner

Know when a test is safe.

Compare your current test with a process that rules out meaningful harm sooner.

One-sided Non-inferiority P80 forecast

01 / Business baseline

Enter your business baseline.

Monthly sessions are required. Every missing funnel value uses an illustrative benchmark.

Monthly sessions

sessions

All rates use sessions as their denominator.

Average order value

$ optional

AOV changes revenue impact, not test duration.

Session funnel

Enter a count or rate. Reset any value to restore estimation.

Illustrative defaults

Metric	Monthly count	Session rate	Source	Actions

02 / Decision lens

Match the metric to the change.

Common events answer faster. Downstream events stay closer to revenue.

Test surface

Eligible traffic 80%

Effective sample 90%

Homepage default: PDP views are closer to the change and occur more often than purchases.

03 / Operating guardrails

Fix two. Calculate the third.

The forecast assumes the variation performs like the control.

Decision duration

Calculate

days

Days until 80% of simulated tests reach a safe decision.

Evidence threshold

Calculate

90%

Required confidence that the variation stays within the acceptable loss.

Acceptable loss

Calculate

5% relative

0.15 percentage points at this baseline.

Planning statement

04 / Process model

Compare your testing policies.

Both policies receive the same simulated stream of test-ready changes.

Current process

Winner-only

Editable

Changes tested 70%

Maximum test duration 60 days

Untested changes

Measured velocity

Rule out meaningful harm

Modeled

Decision metric: PDP view rate
Safe decision: 90% evidence
Loss limit: 5% relative
Expected duration

Program assumptions

Test-ready changes per month 4

Concurrent test capacity 1

Winners out of ten 2

Losers out of ten 3

Typical winner lift 5%

Typical loser loss 7%

Proxy-to-purchase link 70%

Five of ten changes have no purchase effect.

05 / Annual result

Compare your annual outcomes.

Neutral releases count toward throughput and add no revenue.

Business impact delta $0 Median realized revenue over 12 months

Completed decisions 0 Measured velocity

Harmful changes caught 0 Median across modeled years

Likely range $0 - $0 P10 to P90 impact delta

Cumulative business impact

Current process compared with measured velocity.

Current process Measured velocity

P10-to-P90 bands show model uncertainty. The lines show medians.

Operating target

Before launch

Check your measurement before launch.

Confirm each measurement check before you plan. The planner cannot inspect production data.

Assignment unit is defined Returning units keep one variation Exposure logging covers both arms Metric definitions match across arms Traffic allocation is documented Bot and internal traffic rules exist

Ready to planSix of six checks confirmed

Safe decisions

Know what a safe result proves.

A two-sided superiority test asks whether the variation differs in either direction. Use it when proof of improvement controls the release.

A one-sided non-inferiority test asks whether the variation avoids a loss you chose in advance. Use it when the feature has another reason to ship.

Better decision rules cannot repair invalid measurement. Valid measurement does not require every test to prove a purchase lift.