The small-sample illusion — why a 5-star average from 4 reviewers is not a 5-star product

By Ryan Siegal · Founder and Principal

Published 2026-04-27

The Gates Foundation small-schools cautionary tale

Around 2000 the Bill & Melinda Gates Foundation funded a major initiative to break large American high schools into smaller ones, on the strength of evidence that small high schools dominated rankings of best-performing schools. The statistician Howard Wainer pointed out the obvious-in-retrospect flaw: small high schools also dominated rankings of worst-performing schools. Both ends of the distribution were enriched in small schools, because small schools had small-sample variance. The Gates Foundation eventually wound down the small-schools push when more careful analyses showed the high-performance result was largely a sampling artifact.

Among the smallest 50 schools in the United States, six were in the top 50 in eighth-grade math. But among the smallest 50, six were also in the bottom 50. Variance, not virtue.
— Howard Wainer, "The most dangerous equation," American Scientist 2007

The same effect in product reviews

A product with 4 reviewers averaging 4.9 stars looks superb. The 4.9 is a real number, but it's an estimate of the product's population-mean rating with substantial uncertainty. Quadrupling the sample size halves the standard error of that mean estimate; from 4 reviewers, the standard error is half the population standard deviation. From 80 reviewers, it's less than 12% of the population standard deviation. The thick-sample 4.7 is a much more precise estimate of the underlying quality than the thin-sample 4.9.

Kahneman's formalisation

Daniel Kahneman and Amos Tversky's 1971 paper Belief in the law of small numbers documented the cognitive bias underneath this: even professional statisticians, when shown small-sample data, over-estimated how reliably it represented the population. The bias was strongest for samples that looked like they ought to be telling a clear story — exactly the cognitive trap a 4-of-4 5-star review array is designed to trigger.

People view a sample randomly drawn from a population as highly representative of that population in all essential characteristics — even when the sample is absurdly small.
— Kahneman & Tversky, Psychological Bulletin 1971

How Rankquant encodes the fix

The global and in-cohort percentiles rank the measurement as made: the empirical-CDF rank of the product's mean reviewer z-score, with every qualifying reviewer counting equally. The sample-size protection is published beside them as a separate figure — the AI-adjusted percentile:

adj  =  Ẑ  ·  n / ( n + K )        with K = 53

  Ẑ    mean reviewer z-score for the product
  n    number of qualifying (calibrated) reviewers
  K    the shrinkage constant, 53 — the median sample size across
       everything we rank, so the factor is exactly 1/2 at the median item
       (constants pinned in /methodology)

  adj is then ranked by empirical CDF on the same scale as the global percentile

Because per-reviewer normalization puts the corpus average at exactly 0 on the z-scale, multiplying by n/(n+K) is pulling the product toward the corpus average — hard when the sample is thin, barely at all when it is thick. A product with 3 calibrated reviewers keeps about 5% of its measured distance from average; one with 3,000 keeps 98%. It is one line of arithmetic: no model, no training, nothing learned from anything.

A worked head-to-head

Product A:  n =  4 reviewers,  Ẑ = +2.10
   shrink factor  =   4 / ( 4 + 53)  =  0.070
   adjusted       =  +2.10 · 0.070   =  +0.147

Product B:  n = 20 reviewers,  Ẑ = +1.80
   shrink factor  =  20 / (20 + 53)  =  0.274
   adjusted       =  +1.80 · 0.274   =  +0.493

Product C:  n = 80 reviewers,  Ẑ = +1.60
   shrink factor  =  80 / (80 + 53)  =  0.602
   adjusted       =  +1.60 · 0.602   =  +0.962

Rank by raw mean Ẑ:  A > B > C  (2.10  > 1.80  > 1.60)
Rank by adjusted:    C > B > A  (0.962 > 0.493 > 0.147)

The reordering is the point — and Rankquant publishes both orderings rather than quietly picking one. Product A might genuinely be exceptional, but with only 4 reviewers we can't tell apart "exceptional quality" from "lucky small sample", so the adjusted figure holds it near the corpus average until the sample grows. Products B and C have earned more of their measured distance from average. Product A's global percentile still reports the +2.10 exactly as measured: the adjustment is a second number standing beside the first, not a silent edit to it.

Why 53, and why one constant everywhere

K sets how much sample it takes to keep half your measured distance from average: at n = K the factor is exactly 1/2. We read K off the data rather than choosing it — 53 is the median sample size across everything we rank, so by construction the median item is shrunk by half. Per-vertical medians differ widely, and one constant is used everywhere rather than six, precisely so the adjustment cannot be retuned category by category once the results are in. The constant is published and version-stable; any change requires a public version bump.

What the n/(n+53) factor does at each sample size.
n = 4 reviewers	Keeps 4/57 ≈ 7% of its measured distance from the corpus average. A +2.10 mean enters the adjusted ranking at +0.15.
n = 16 reviewers	Keeps 16/69 ≈ 23%. Four times the sample, roughly three times the retained distance.
n = 53 reviewers	Keeps exactly 1/2 — this is the median sample size across everything we rank, which is why K is 53.
n = 256 reviewers	Keeps 256/309 ≈ 83%. The discount is now small enough that ordering by the adjusted figure rarely differs from ordering by the mean.
n = 1000 reviewers	Keeps 1000/1053 ≈ 95%. Once you have a thousand qualifying reviewers, the adjusted figure is essentially the mean.

What the n/(n+53) factor does at each sample size.

The 90% floor is still published — as a diagnostic

The other classical answer to the same problem is a confidence-interval floor: report a defensibly pessimistic lower bound instead of the point estimate. Rankquant computes it for every item and ships it as a per-item field — ciFloorZ in the catalog exports:

floor  =  Ẑ  −  1.645  ·  ( 1 / √n )

  1.645    one-tailed 90% z-critical value
  1 / √n   the standard error of Ẑ — the scale factor is 1 because
           z-scores have unit variance by construction

It is there to be read, not sorted on: no published percentile is a function of it. Both it and the shrinkage factor are answers to the same question, and they agree on the direction — thin samples get discounted — but only one of them sets a rank, and that one is the AI-adjusted percentile.

Where this leaves new products

New products with genuinely good early reviews are structurally penalised until they accumulate sample. We own that. The alternative — letting four-reviewer products fake-rank ahead of established ones — is exactly the hole that fake-review farms exploit on Amazon. We'd rather under-rank a real new winner for a few months than over-rank a manipulated one indefinitely.

The item page surfaces this structurally:

The calibrated-reviewer count n is printed next to the adjusted percentile on every item page, so the size of the sample behind the number is never hidden.
Items with too few calibrated reviewers for the adjustment to mean anything publish an em dash, never a 0 — a missing measurement must not read as a bad one.
The unadjusted percentile stays on the page beside the adjusted one, so a new entrant with a genuinely strong mean can be seen being held back rather than quietly demoted.

Frequently asked questions

Doesn't this give established products an unfair head start?+

It gives them a sample-size head start in one of the three published figures. That's a real advantage and we don't pretend otherwise. The structural alternative — treating a 4-reviewer mean and a 4000-reviewer mean as equally trustworthy — would be statistically wrong and operationally exploitable. The n/(n+53) factor is the same discount applied to every item, and new entrants close the gap automatically as their reviewer counts grow. The global and in-cohort percentiles apply no such discount at all: there, a new product's mean is ranked exactly as measured.

What about products with high variance among reviewers?+

A product can be controversial — half its reviewers love it, half hate it — and still land on a centred mean z-score. Be clear about what we do and do not do with that: no published percentile charges a product for reviewer disagreement. The global and in-cohort percentiles rank the mean as measured, and the AI-adjusted percentile corrects for how thin the sample is, not how split it is. What every item page publishes instead is the material for spotting the case yourself — the calibrated-reviewer count the mean was taken over, and the raw-average percentile printed beside the z-normalized one, so you can see how many reviewers stand behind the number and how far per-reviewer normalization moved it.

Why is the published floor one-tailed instead of two-tailed?+

Because the diagnostic answers a one-tailed question: "what is a defensibly low estimate of this product's quality?" The upper bound adds nothing — no product deserves credit for a lucky ceiling — and a two-tailed interval would only report the same uncertainty in a wider band. Note that the floor is a per-item field to be read, not a rank: no published percentile is computed from it.

Is the AI-adjusted percentile Bayesian shrinkage?+

It is the same arithmetic in closed form. adj = Ẑ · n/(n+K) is precisely the shrinkage a normal-normal posterior applies when the prior sits on the corpus average with strength K — and per-reviewer z-normalization puts that average at exactly 0, which is why the formula is this short. The frequentist confidence floor is its close dual: it pulls toward the bottom of an interval rather than toward a prior, and under similar parameter choices the two orderings are nearly identical. K = 53 is not a taste we ask you to take on trust; it is the median sample size across everything we rank, published and version-stable. And despite the chip's name: nothing here is machine learning. No parameter is fitted, no data is trained on — one multiplication by a published constant, then a rank.

Series: ← Simpson's paradox · Hub · Next: Pre-committed constants →