Analytics & Experimentation — North Star, Activation, Retention & A/B Testing

Measurement is a design responsibility, not a post-launch afterthought. The person who designs a flow decides what "working" means and whether it can ever be proven. Instrument as you draw. This doc gives the frameworks and the rules to apply them.

The North Star Metric Framework

A North Star Metric (NSM) is the single output metric that best captures the value your product delivers to customers. Everyone can name it; teams argue less about priorities because they share one number. It is a leading indicator of durable revenue, not revenue itself.
The NSM is an outcome you cannot touch directly. You move it only by moving its inputs — the 3–5 factors your teams can influence day to day. Amplitude's heuristic for choosing inputs: cover breadth (how many users), depth (how much each does), frequency (how often), and efficiency (how fast/cheaply). A good input set is complementary and collectively "moves the star."
Guardrail metrics sit alongside: numbers you must not damage while chasing the star (churn, refund/return rate, support-ticket volume, P95 latency, error rate, unsubscribe rate, margin, NPS/CSAT). Guardrails exist because any single metric can be gamed — a team can inflate the star by hurting something else. Name the guardrails before you run anything.
Structure: one North Star → a handful of inputs (what you build against) → guardrails (what you protect). This is the whole framework.
Examples of good North Stars (value-aligned, not vanity):
Spotify — time spent listening (proxy for value received, predicts retention/subscription).
Airbnb — nights booked (both sides of the marketplace realize value).
Facebook (classic) — daily active users; Amplitude's own often-cited example is weekly learning consumers / weekly active queries style engagement measures.
Slack — messages sent within active teams; Duolingo — daily active users sustained by streaks.
Anti-patterns (bad North Stars): raw signups, pageviews, total registered users, cumulative downloads, revenue-only. These reward acquisition and volume while saying nothing about whether anyone got value — you can grow them while the product quietly dies.
Test a candidate NSM with three questions: (1) Does it express customer value, not company desire? (2) Does moving it reliably lead to revenue later? (3) Can teams actually influence its inputs? If any answer is no, keep looking.

Activation & Retention: The Metrics That Actually Matter

Acquisition is rented; retention is owned. Two metrics predict a product's fate more than any other: activation (do new users reach value?) and retention (do they keep coming back?).

Activation = The Aha Moment

Activation is the moment a new user first experiences the product's core value — the "aha moment," or the setup milestone that reliably precedes it. Not signup. Not a tour completed. The point where the promise becomes real for that user.
Define activation as a specific, observable event or short sequence, e.g. Slack's classic "team sends 2,000 messages," Dropbox "file placed in a synced folder on a second device," Facebook "7 friends in 10 days," a scheduling app "first meeting actually booked through a shared link."
The validity test for an activation definition: users who hit it should retain at least ~2x better than users who do not. If your chosen milestone doesn't split retention that sharply, it isn't the real aha moment — find the behavior that does.
Benchmarks (SaaS, 2026): median activation rate hovers around 30%, mean around 36% — but ranges are wide by category: developer tools ~18–28%, analytics ~12–22%, collaboration ~30–45%, vertical SaaS ~35–50%. Treat these as sanity checks, not targets; your own retention split matters more than any benchmark.
Activation rate = (users who reached the activation milestone) ÷ (users who signed up), measured within a sensible window (often 1–7 days, sometimes 30 for complex B2B).
Time-to-value (TTV) is activation's twin: how long from signup to the aha moment. Shorter TTV lifts activation and retention together. Track median TTV as a first-class metric and treat every onboarding step as a tax on it — cut anything that doesn't move the user toward first value.

Engagement & Stickiness

DAU/MAU ratio (the "stickiness ratio") measures how many monthly users show up on a given day. ~0.2 (20%) is respectable for many products; best-in-class daily-habit products reach 0.5+. Judge it against your product's natural cadence — a weekly tool shouldn't chase a daily-app ratio.
Pair stickiness with feature adoption (share of active users using the core value feature) to distinguish "logged in" from "got value."

Retention & the Retention Curve

Plot retention as a curve: the % of a cohort still active by day/week N after signup. The shape tells you everything.
A curve that flattens into a horizontal plateau above zero is the single clearest signal of product-market fit. Bad-fit users churn early (the initial drop), then a stable core keeps returning — that plateau height is your terminal retention rate. A curve that decays toward zero means no PMF, no matter how good acquisition looks.
The "smile" (upward) curve — retention that dips then rises as dormant users return — is the rarest and strongest pattern, typical of network-effect and habit products. Track it on a power-user curve (retention of already-engaged users) to see it clearly.
Amplitude's ~7% D7 heuristic: when at least ~7% of a new cohort is still active on day 7, the product sits in the top quartile for activation; a large share of products with strong D7 retention also show strong 3-month retention. Use D7 as a fast early read while long curves accumulate.
Pick the right frequency. Daily-use products (social, messaging) measure DAU/daily retention; weekly tools measure weekly; infrequent-by-nature products (tax, travel) should not be judged on daily curves — match the cadence to the natural usage rhythm or you'll declare false failure.

Choosing a Retention Definition

Three common definitions answer different questions — pick deliberately and state which you use:

N-day retention — active on exactly day N. Strictest; right for products meant to be used every day (fitness, streaks). Punishes normal skipped days.
Unbounded ("full") retention — active on day N or later. Most forgiving; good for churn/resurrection analysis.
Bracket / range retention — active at least once within a window (e.g. week 2). Best default for most weekly or occasional-use products; tolerates natural gaps without overcounting.
Rule: report the definition alongside the number. "40% retention" is meaningless without saying which of the three, over what window, on which event.

The Metric Hierarchy (Pirate Metrics + Retention)

Organize every metric into the funnel so no team optimizes a stage in isolation. The canonical stages (AARRR, plus engagement broken out):

Acquisition — how users find and arrive (traffic, signups, CAC, channel mix).
Activation — first value / aha moment (activation rate, time-to-value).
Engagement — depth and habit (sessions/user, feature adoption, DAU/MAU stickiness ratio).
Retention — repeat usage over time (Nth-day/week curves, churn rate).
Revenue — monetization (conversion to paid, ARPU, LTV, expansion).
Referral — users bringing users (invites sent/accepted, viral coefficient, NPS).

Rule: fixing a downstream stage before an upstream one leaks is wasted work. Retention above a leaking activation just fills a bucket with a hole. Diagnose top-down, fix bottom-up.
Map your North Star to this hierarchy so everyone sees how their stage feeds the star.

Leading vs. Lagging, Actionable vs. Vanity

Lagging indicators (revenue, churn, LTV) confirm results but arrive too late to steer by. Leading indicators (activation rate, week-1 engagement, time-to-value) move first and predict the laggards — steer with these. The NSM should be a leading indicator of revenue; that's its whole point.
A metric earns its place only if a change in it changes a decision. If no plausible number would make you act differently, it's a vanity metric — cut it from dashboards. Prefer rates and ratios (activation rate, retention %, conversion %) over raw cumulative counts (total signups, total downloads), which only ever go up and hide decay.
Segment before you conclude. An aggregate that looks flat often hides one cohort surging and another collapsing. Cut by acquisition channel, plan, platform, and signup cohort before declaring a trend.

Cohort Analysis

Group users by the period they joined (or the action they took) and follow each group over time. Aggregate metrics mix new and old users and mask reality; cohorts reveal whether the product is getting better or worse for comparable users.
Read cohorts as a triangle/heatmap: each row a signup cohort, each column an age (week 1, 2, 3…). Improving retention shows as later cohorts holding higher at the same age. This is how you prove an onboarding change actually worked.

A/B Testing Done Right (and Its Pitfalls)

Reference standard: Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (Cambridge, 2020) — distilled from Microsoft, Google, and LinkedIn running tens of thousands of experiments a year. The hard part isn't running a test; it's trusting the result.

The Logic (why controlled experiments work)

A/B testing is the only method that establishes causation, not correlation. Randomly split users into control (A) and treatment (B); the randomization makes the groups statistically identical on everything except the change, so any reliable difference in the metric is caused by the change. Everything below exists to protect that inference.
Statistical significance (p < 0.05) answers "how likely is a difference this big if there were really no effect?" It does not measure business importance or the size of the effect. A statistically significant 0.1% lift may be worthless; report the effect size and confidence interval, not just the p-value.
Power (usually 80%) is the chance of detecting a real effect of your MDE size. Under-powered tests fail to find real wins and produce false "no difference" verdicts — which is why sample size is non-negotiable.

Before You Start

Compute sample size and MDE up front. The Minimum Detectable Effect is the smallest lift worth acting on. Smaller MDE → exponentially larger sample and longer runtime. Decide the MDE from business impact before looking at data, then use a sample-size calculator (Evan Miller's is standard) with your baseline rate, chosen power (usually 80%), and significance (α = 0.05).
Fix the decision rule in advance: one primary metric, the guardrails, the sample size, and the stop date. Write it down. This prevents rationalizing after the fact.

The Pitfalls (memorize these)

Peeking / early stopping. Repeatedly checking significance and stopping the moment p < 0.05 massively inflates false positives — with continuous peeking you can eventually hit "significance" on a pure A/A test. Fix: commit to the pre-computed sample size and end date, or use a method built for continuous monitoring (sequential testing / always-valid p-values, Bayesian). Never stop early just because it looks good.
Novelty effect. A change wins at first purely because it's new — returning users click it out of curiosity, then the lift decays. Fix: run long enough for novelty to wear off; segment new vs. returning users.
Primacy effect. The mirror image — a change loses at first because habituated users are disrupted, then recovers as they adapt. Fix: same — run long enough, watch the trend over time, not just the pooled average.
Twyman's Law. "Any figure that looks interesting or different is usually wrong." A result that's too good (a "50% lift") almost always signals instrumentation error, a logging bug, or a broken assignment — not a breakthrough. Fix: treat spectacular results as suspect until independently verified; run A/A tests to validate your platform detects no difference when there is none.
Simpson's Paradox. A treatment can win in every subgroup yet lose overall (or vice versa) when segments have different sizes or ramp-up traffic is unevenly split. Fix: watch for it whenever traffic allocation changed mid-test; analyze consistent cohorts; don't pool across periods with different splits.
Sample Ratio Mismatch (SRM). If a 50/50 split arrives as, say, 52/48 beyond chance, assignment is broken and the whole test is untrustworthy. Fix: run an SRM check automatically; a failed SRM invalidates the result — investigate, don't interpret.
Multiple comparisons. Testing 20 metrics at α = 0.05 yields ~1 false "winner" by chance. Fix: pre-declare one primary metric; correct (e.g. Bonferroni) for the rest, treat them as directional.

Guardrails & Duration

Always ship guardrail metrics inside the experiment (latency, errors, crash rate, unsubscribe, revenue-per-user). A conversion "win" that quietly raises latency or churn is a loss.
Minimum duration ≥ one to two full business cycles. Almost always this means at least 1–2 full weeks so both weekday and weekend behavior are captured; round partial weeks up to whole weeks. For products with slower return cycles, extend to 2–4 weeks so returning users are exposed multiple times. Cap around 4–8 weeks (cookie churn and external drift degrade longer tests).

Rolling Out & Reading the Result

Ramp traffic gradually (e.g. 1% → 5% → 50%) to catch catastrophic bugs cheaply — but remember: the ramp-up period is not your measurement window. Analyze only the stable full-allocation period; mixing ramp phases invites Simpson's Paradox.
Report a result trustworthily: state the effect size and its confidence interval, the primary metric, every guardrail's movement, the SRM check result, the sample size, and the exact dates. A "win" without guardrails and an SRM check is not a decision-ready result.
A flat (non-significant) result is information, not failure — it means the change didn't move the needle by at least your MDE. Ship it for other reasons or move on; don't torture the data until it confesses.

When NOT to A/B Test

A/B testing needs volume. A rough floor: ~1,000+ visitors and ~50+ conversions per variant per week for the target page; reliable small-lift detection wants far more (tens of thousands of visitors, thousands of conversions per arm).
Below that, don't A/B test small changes — you'll never reach significance, and a "result" is noise. Instead:
Prioritize qualitative research — usability tests (5 users surface most issues), session replays, interviews, surveys. Judgment beats an underpowered test.
Test only big, bold swings (full redesigns, radically different value props) where the effect is large enough to detect with modest traffic.
Ship well-reasoned changes and monitor trend metrics rather than gating on inconclusive experiments.
Work on growing traffic and conversions first so experimentation becomes viable later.

Rules for Designers: Instrument the Flows You Build

You are the last person who knows exactly what each state and interaction means. Bake measurement into the design, not on top of it.

Define success before you design. For every flow, write the one event that means "this worked" (the activation milestone, the completed job). If you can't name it, the flow has no goal.
Design the aha moment explicitly. Onboarding's only job is to get users to activation fast. Cut every step that doesn't move them toward first value; measure time-to-value as a first-class metric.
Spec events, not just screens. In the design handoff, list the events to fire, the properties each carries (source, variant, plan, step index), and the exact trigger. Name them consistently (object_action, e.g. meeting_booked, invite_sent) — a rename later orphans historical data.
Instrument the whole funnel, including failure. Track drop-off, validation errors, empty states, and back-navigation — not only the happy path. The states you'd rather not think about are where retention leaks.
Track guardrails for your own work. A slicker screen that adds a step or a load spinner can lower completion. Watch the counter-metrics on any redesign.
Tie every screen to a hierarchy stage. Know whether a given design serves acquisition, activation, engagement, retention, revenue, or referral — and which North Star input it feeds.
Give experiments a fair shot. Ensure variants are cleanly assignable, mutually exclusive, and logged identically; avoid designs where treatment and control differ in ways you can't isolate. Confounded designs produce untrustworthy tests.
Respect the analytics contract. Don't silently change an event's meaning, rename properties, or reuse an event name for a new purpose — downstream dashboards and cohort definitions depend on stability.
Don't ship instrumentation debt. A flow launched without its events is a flow you can never evaluate or improve. Missing tracking is a bug, not a nice-to-have — block launch on it the same way you'd block on a broken submit button.
Design for the experiment, not around it. If a redesign is worth doing, it's worth measuring. Ensure the old and new experiences can run side by side cleanly so the change can be A/B tested rather than shipped on faith.

A Starter Event Taxonomy (adapt per product)

signup_completed — props: source, plan, referrer
onboarding_step_viewed / onboarding_step_completed — props: step_name, step_index
activation_reached — the single aha event (define it deliberately) — props: time_since_signup
core_action_performed — the repeatable value event that feeds retention — props: feature
invite_sent / invite_accepted — the referral loop — props: channel
subscription_started / subscription_canceled — props: plan, mrr
Fire one event per meaningful outcome, name it object_action, keep property names identical across events, and never overload one event with two meanings.

Sources

Amplitude — About the North Star Framework; Good vs. Bad North Star Metric
Amplitude — What is Activation Rate
Lenny Rachitsky — What is a good activation rate
Sequoia — Retention
Kohavi, Tang & Xu — Trustworthy Online Controlled Experiments; companion site experimentguide.com
Nielsen Norman Group — A/B Testing 101