Skip to content
testcritic

A/B Testing & Feature Flags

Ship behind a flag, measure the difference, roll back in seconds.

13 specialist tools0 free or open source4 self-hostable5 usable without code

What this category is really about

Feature flags and experimentation are the same primitive used for two goals. A flag decouples deploy from release so you can ship dark and roll back instantly. An experiment takes the same split and attaches a statistical test to it.

The distinction that costs money is client-side versus server-side. Client-side tools inject a variant in the browser — trivial to install, visible flicker, and increasingly blocked. Server-side flags are evaluated in your own code, work anywhere, and require engineering to instrument.

Statistics deserve real scrutiny. Fixed-horizon tests stopped early produce false positives at rates far above the advertised 5%. If a vendor lets you watch a live p-value and stop when it dips, ask which sequential-testing correction they apply.

What to evaluate

01

Evaluation model

Server-side SDK, edge, or client script. This decides flicker, bot exposure and what you can experiment on.

02

Statistics engine

Sequential testing or CUPED beat a raw p-value. Ask how peeking is handled.

03

Flag hygiene

Stale-flag detection and code references. A codebase with 400 dead flags is a real maintenance tax.

04

Pricing metric

MTU, events, or seats. The same traffic can differ 10x in cost between vendors.

Common mistakes

  • Calling a test after two days because the numbers look good.
  • Running six overlapping experiments on the same funnel and not modelling the interaction.
  • Measuring a proxy metric (clicks) that moved while the real metric (revenue) did not.
  • Never removing flags, so every code path has four historical branches.

13 tools built for this

Sorted by how widely adopted they are, not by preference. Tick the compare boxes to put two or more side by side.

Filter these

Tools that also cover this

These are built primarily for something else but handle this category as part of a broader platform — often a way to consolidate two subscriptions into one.