A/B testing video funnels: a primer

A/B testing video isn't quite the same as A/B testing a landing page. Watch time matters, drop-off is non-linear, and CTAs can be time-coded. Here's how Clipora handles it.

Bucketing

Every visitor gets a sessionId on first load. We hash the sessionId modulo two and bucket them into Variant A or B. The same visitor sees the same variant on every visit. No flicker, no flip-flop, no weird mid-funnel switches.

Counters

Per variant we track impressions and conversions (where conversion = the funnel goal). The impression counter increments on first slide view; the conversion counter on the goal-fulfilment event (subscribe, apply, purchase, etc.).

Significance

We compute a two-proportion z-test on (impressions_A, conversions_A) vs (impressions_B, conversions_B), surface the p-value, and recommend a winner once we're above 95% confidence and at least 200 conversions per arm. We're frequentist for v1; Bayesian variant on the roadmap.

// Pseudocode of the significance helper
function zTestTwoProportions(
  impressions: { a: number; b: number },
  conversions: { a: number; b: number },
) {
  const p1 = conversions.a / impressions.a;
  const p2 = conversions.b / impressions.b;
  const pPool =
    (conversions.a + conversions.b) /
    (impressions.a + impressions.b);
  const se = Math.sqrt(
    pPool * (1 - pPool) * (1 / impressions.a + 1 / impressions.b),
  );
  const z = (p1 - p2) / se;
  return { z, pValue: 2 * (1 - normalCdf(Math.abs(z))) };
}

Promoting the winner

When you click 'Promote variant', Clipora copies B's slide content over A's, archives B, and stops the experiment. The next visitor lands on the new control. We keep the experiment record forever for audit.