Founding metrics — the five statistics primitives behind every Rankquant percentile

By Ryan Siegal · Founder and Principal

Published 2026-04-27

Short answer

Rankquant rests on five textbook statistics primitives, each chosen because it is the right tool for a specific job. The z-scorestrips out per-reviewer scale bias by re-expressing every rating in standard-deviation units relative to that reviewer's own distribution. The standard error of the mean (SE = σ / √n) tells us how much the sample mean can wobble around the true mean and shrinks at the rate of √n. The 90% one-tailed confidence interval gives a defensibly pessimistic estimate of a product's true normalized score, and we publish it per item as the uncertainty figure rather than ranking on it. The empirical cumulative distribution function turns the mean z-score into a calibrated 0–100 percentile without assuming any parametric distribution — and, run a second time on Ẑ · n/(n + 53), produces the AI-adjusted percentile, the figure that actually discounts an item for the confidence its sample supports. The Kish design effect prices what unequal weighting would cost — which is why every qualifying reviewer counts equally here and the effective sample size is simply the reviewer count. Each is taught in any first-year statistics course; their composition is what is novel-as-applied.

Why this page exists

The methodology page tells you what Rankquant does. The theory hub tells you why each step of the pipeline is rigorous. This page sits one level deeper — it tells you why the five underlying primitives are the right ingredients in the first place. Read it if you want to understand the methodology from first principles, debate the choice of estimator with us, or reimplement the pipeline from the published constants and swap a primitive for one you think is better. Every primitive below is paired with the canonical statistical reference it traces back to.

1. The z-score — scale-free comparison across reviewers

The z-score is a re-expression of a value as the number of standard deviations it sits above or below the mean of its reference distribution:

z = ( x − μ ) / σ

For Rankquant: z_{u,i}  =  ( r_{u,i}  −  μ_u )  /  σ_u

  r_{u,i}  reviewer u's rating of product i
  μ_u      reviewer u's personal mean across all their ratings
  σ_u      reviewer u's personal Bessel-corrected SD (df = n_u − 1)

Why this is the right primitive for combining reviewers

Two reviewers who agree on relative quality but disagree on absolute scale produce numerically different raw ratings of the same product but identical z-scores. A reviewer whose rating range is 90–96 and a reviewer whose rating range is 75–88 will both produce z ≈ +1.3 for their personal favorites. The z-score captures the only thing reviewers actually share: ordering and relative spread within their own rating universe.

The standard reference is psychometrics: per-rater normalization is the foundational move in essay scoring (where graders use different personal scales) and in inter-laboratory measurement comparison (where instruments have different baselines). The same logic shows up in standardized testing (SAT/GRE/GMAT score scaling), grade curves at universities, and Olympic-style judging where each judge's scoring tendencies are corrected before medals are awarded.

Why we pool z-scores cross-category

A reviewer who rates 50 wines and 50 books has one personal scale, not two. People don't reset their generosity bias when they switch domains. We pool a reviewer's entire rating history to estimate μ_u and σ_u rather than computing a separate z-scale per category. The technical justification — and the n_u ≥ 2 admission rule — is detailed at /theory/degrees-of-freedom.

2. The standard error of the mean — why √n is everything

The standard error of a sample mean tells you how far the sample mean is likely to be from the population mean it is trying to estimate:

SE(x̄)  =  σ / √n

For z-scores (which have unit variance by construction): SE(Ẑ)  =  1 / √n

This single equation is why a 4-reviewer mean and an 80-reviewer mean are not the same kind of claim. Quadrupling the sample size halves the SE. Going from 4 reviewers to 80 reviewers shrinks the SE by a factor of √(80/4) ≈ 4.47×. The mean of 80 reviewers is 4.47 times more precise than the mean of 4 — and any honest publication has to say so somewhere. Here it is said twice: in the 90% floor printed beside every item, and in the AI-adjusted percentile, which is the figure that lets the difference change an order.

A worked example

Product A:  4 reviewers, mean z = +2.1
   SE(Ẑ_A)  =  1/√4   =  0.500

Product B: 80 reviewers, mean z = +1.6
   SE(Ẑ_B)  =  1/√80  =  0.112

Product A has a HIGHER mean but FOUR TIMES the wobble.
The wobble is what makes its rank uncertain.

The standard error is the foundation of every defensible aggregation method we know of, from clinical trials to election polling to FiveThirtyEight's sports models. It is also, fittingly, where the central limit theorem earns its rent: as n grows, the distribution of the sample mean converges on a normal distribution with standard deviation SE — which is what lets us compute confidence intervals without needing the population distribution itself to be normal.

3. The 90% one-tailed confidence interval — what we publish

A confidence interval for an estimate Ẑ is a range of values that, with stated confidence, contains the true population value. The one-tailed floor of a 90% CI is:

floor  =  Ẑ  −  z_{α=0.10}  ·  SE(Ẑ)
       =  Ẑ  −  1.645 · SE(Ẑ)

The CI-floor is the answer to the question: given the data we've observed, what is a defensibly pessimistic estimate of this product's true normalized score? Rankquant computes it for every item and publishes it with that item — products with high means but few reviewers have wide intervals and low floors; products with slightly lower means but many reviewers have tight intervals and higher floors. It is a figure to read, not a sort key: the headline percentile ranks Ẑ itself, and the correction for sample size is applied by the AI-adjusted percentile at step 4. The full derivation, including the choice of 90% over 95%, is at /theory/confidence-intervals.

The CI-floor and the shrinkage behind the AI-adjusted percentile sit in the same well-known family of small-sample-safe techniques.
Wilson score interval (binary outcomes)	Reddit's "best comments" sort and Yelp's internal ranking, where the lower bound is the sort key. Same idea applied to up/down vote binomials. Edwin Wilson, 1927.
Bayesian shrinkage (Beta-Binomial / Normal-Normal)	IMDb's Top 250 formula and credibility theory in actuarial science. Pulls thin-sample estimates toward a prior. This is the family Rankquant's AI-adjusted percentile belongs to — Ẑ · n/(n + 53), with the prior fixed at the corpus average of zero — and it is mathematically dual to a CI-floor under standard assumptions.
Hodges–Lehmann estimator	Rank-based estimate of a location parameter. Robust to non-normality; influences our empirical-CDF step.
DerSimonian–Laird random-effects estimator	Standard meta-analysis aggregation that combines within-study and between-study variance. A future Rankquant methodology upgrade.

The CI-floor and the shrinkage behind the AI-adjusted percentile sit in the same well-known family of small-sample-safe techniques.

4. The empirical CDF — distribution-free percentiles

The empirical cumulative distribution function is the data-driven estimate of the true CDF. Given n observations, it is:

F̂(x)  =  ( 1 / n )  ·  Σ  I( X_i ≤ x )

For Rankquant:  percentile(P)      =  F̂( Ẑ_P )              ·  100
                AI-adjusted(P)     =  F̂( Ẑ_P · n/(n+53) )   ·  100

Each product's headline percentile is the fraction of all products whose mean z-score is at or below its own, scaled to 0–100. There is no parametric assumption about the distribution of those means. This matters because the population of mean z-scores is not normal: in practice it's right-skewed (a few exceptional products) and bounded above (no product can score better than "everyone agrees this is exceptional"). The same nonparametric step is run a second time over the shrunk values to produce the AI-adjusted percentile, so both figures land on the same 0–100 scale and the size of the correction is readable as the gap between them.

The empirical CDF is, by the Glivenko–Cantelli theorem, a uniformly consistent estimator of the true CDF — meaning as our database grows, the percentiles we publish converge on what they would be against the full population of products in each category. It is the foundation of nonparametric statistics and the reason rank-based methods are robust where parametric ones break.

Why not the normal CDF (Φ)?

We could compute percentiles via p = Φ(Ẑ)— feeding the mean z-score through the standard-normal CDF. We don't, because that assumes the population of product means is normal. Our data shows it isn't, and assuming normality where it doesn't hold introduces bias we can't justify. The empirical CDF makes no such assumption.

5. The Kish design effect — what weighting would cost

If an aggregate applies unequal weights wto its raters, the simple sample size N overstates the precision of the resulting mean. The right count is the effective sample size, given by Kish's design effect:

N_eff  =  ( Σ w )²  /  Σ w²

  Equality (N_eff = N_raw) only when every w is equal.
  Otherwise N_eff < N_raw — sometimes much less.

A weighted mean of 100 reviewers where 90 carry weight 1 and 10 carry weight 10 has N_eff = (190)² / (90·1 + 10·100) = 36,100 / 1,090 ≈ 33. It is statisticallya 33-reviewer mean, even though it's nominally a 100-reviewer mean — two-thirds of the sample spent on the weighting, and the SE that feeds the published floor at step 3 has to say so.

Rankquant sets every weight to 1. No reviewer and no source counts for more than any other, so N_eff = N_raw exactly and the standard error at step 2 needs no correction. That is what this primitive is doing here: it is the reason equal weighting is the cheap option rather than the modest one. The Kish design effect originates in survey statistics — Kish (1965), Survey Sampling— and is the standard correction in any weighted-aggregation context (epidemiology, polling, meta-analysis). It is also the check to run on anyone else's weighted aggregate: without it, a credibility-weighted score publishes optimistically narrow intervals and over-ranks items whose reviewer mix leans on a few heavily weighted raters.

How the five primitives compose into the pipeline

The five primitives, their pipeline role, and their canonical references.
1. Z-score	Per-reviewer normalization. Removes scale bias before aggregation. Reference: psychometrics, e.g. Cronbach (1951); SAT/GRE score scaling.
2. Standard error	Quantifies sampling uncertainty in each aggregate. SE(Ẑ) = 1/√N for z-scores. Reference: central limit theorem; any first-year statistics text.
3. 90% CI-floor	Defensibly pessimistic bound on the same mean, Ẑ − 1.645 · SE, published per item as the uncertainty diagnostic. Same family as Wilson score; nothing is ranked on it. The sample-size correction that does act on the rankings is the AI-adjusted percentile's n/(n + 53) shrinkage.
4. Empirical CDF	Distribution-free percentile mapping, applied to Ẑ for the global and in-cohort figures and to Ẑ · n/(n + 53) for the AI-adjusted one. F̂(x) = (1/n) Σ I(X_i ≤ x). Reference: Glivenko–Cantelli (1933); nonparametric statistics.
5. Kish design effect	Effective sample size under unequal weights. N_eff = (Σw)² / Σw². Rankquant's weights are all equal, so N_eff = N_raw; the primitive is here as the check on that, and on anyone else's weighted score. Reference: Kish (1965).

The five primitives, their pipeline role, and their canonical references.

Where the primitives run but the data runs out

All five compose in the verticals with a usable reviewer graph — Films & TV, Hotels, Wines, Cruises and Books — with primitive 5 confirming N_eff = N rather than correcting anything, because the weights are equal everywhere. Products (score schema reviewer-z-r3-mean-v2) runs primitives 1–4 the same way: per-reviewer standardization, the standard error, the published confidence floor, and the empirical CDF of the mean z. What is genuinely constrained here is coverage, not the estimator. Amazon captures read review page 1 only and are sharded ASIN-disjoint, so 1,594,196 of 4,721,328 reviewers rated two or more products and can supply a personal μ_u and σ_u; that leaves 1,215,077 of 1,352,809 published products with a computable percentile. The remainder are published unscored rather than scored by a substitute statistic. The full specification is at /methodology#scope.

What is novel here

None of these primitives is novel. Combining them as Rankquant does — per-reviewer z-score → equal-weight mean → empirical-CDF percentile, with the 90% floor published beside that percentile and the sample size charged in a third, shrunk figure — is a textbook composition. What is novel is publishing it as the basis of a consumer review site. Every other review surface we are aware of publishes either a raw average or a weakly-shrunken average and stops. Rankquant publishes the math.

If you disagree with any of the five primitives, every constant you would need to swap one out is published at /methodology. Replace the empirical CDF with the normal CDF; move the sample-size correction out of the AI-adjusted percentile and into the headline figure by ranking the 90% floor instead of the mean, as we did until 29 July 2026; take the floor to 95% while you're there; replace the per-reviewer z-score with a per-source z-score. Run the pipeline on the same data. Compare the rankings. The methodology is meant to be argued with — that is the whole point of publishing it.

Frequently asked questions

Why frequentist confidence intervals instead of Bayesian credible intervals?+

Both are here, doing different jobs. The frequentist floor is the figure we publish per item: it needs no prior strength k, is scale-free across z-scored data, and a reader can reproduce it by hand from the mean, 1.645 and the reviewer count. The Bayesian side is what acts on the rankings — the AI-adjusted percentile multiplies the same mean z by n / (n + 53), exactly a shrinkage weight pulling thin samples toward the corpus average of zero, which is a credible-interval move in everything but name. Under standard noninformative priors the two agree closely about which items a thin sample cannot carry.

Why one-tailed instead of two-tailed?+

The floor answers a one-tailed question: "what is a defensibly low estimate of this product's quality?" The upper bound is not interesting — a lucky ceiling is not evidence of anything, and no figure on this site reads one. Quoting a two-tailed bound as the floor would overstate how little we know about products whose uncertainty only matters in one direction.

Why pool reviewers cross-category for the z-score?+

A reviewer's personal scale is a property of the reviewer, not of the category. Someone who rates everything 90+ on a 100-point wine scale also rates everything 4.5+ on Goodreads. Pooling cross-category increases the sample size for estimating μ_u and σ_u, which in turn makes the z-score itself less noisy.

What about ordinal-vs-interval scale concerns?+

Strictly speaking, consumer ratings are ordinal — the gap between 4 and 5 stars is not necessarily the same as between 3 and 4. Treating them as interval is a common simplifying assumption that's defensible given the regularity of rating distributions across millions of users. The empirical-CDF step at the end implicitly recovers a rank-based interpretation; the percentile is invariant to monotone transformations of the underlying score.

What's the smallest N where this all works?+

Per-reviewer z-score requires n_u ≥ 2 with σ_u > 0. The CI-floor numerically works at any N, though SE = 1/√N goes to 1.0 at N = 1 and the floor sits about 1.6 below the mean. Empirical-CDF percentiles need a populated database — at least ~30 products in the same comparison group for the percentile to be stable. We declare a category "ready" once the thinnest cohort hits ~30 products.

Where do these primitives break down?+

Three places. (1) When the z-score assumption that reviewers have monotone preferences is violated — a reviewer who likes both red and white wines but for different reasons may have multimodal preferences not captured by μ_u, σ_u. (2) When a category is too narrow for the empirical CDF to converge — sub-50-product categories produce noisy percentiles. (3) When μ_u and σ_u are themselves estimated from very few ratings: at n_u = 2 a reviewer's personal scale is a single data point, so their z-scores are noisy, and that noise is only booked later, at the product level — in the width of the published 90% interval, and in the n/(n + 53) shrinkage the AI-adjusted percentile applies.

Continue: Degrees of freedom → · Confidence intervals → · Inter-rater reliability →

Or: Statistics can lie — here's how, and what we do about it →