Degrees of freedom in the Rankquant pipeline

Q: Why does sample variance divide by n−1 instead of n?

Because the sample mean x̄ is estimated from the same data. When you use the sample mean as your center, the deviations (x_i − x̄) are constrained to sum to zero, so only n−1 of them vary independently. Dividing by n−1 (Bessel's correction) gives an unbiased estimator; dividing by n underestimates true variance.

Q: Is n_u = 2 really enough to normalize a reviewer?

Just barely. With n_u = 2 a reviewer's σ_u has df = 1 and is extremely noisy. We admit them because (a) excluding them would throw away a lot of crowd data, and (b) the noise they introduce is booked at the product level rather than hidden: it widens SE(Ẑ) = 1/√N_eff and so the 90% floor we publish, and where the item's whole sample is thin the AI-adjusted percentile shrinks its mean by n/(n + 53). Reviewers with larger n_u contribute more stable z-scores, which is exactly what we want.

Q: What happens to reviewers with only one review?

They're on file but excluded from all three published figures. A single-review reviewer has no personal distribution — μ_u and σ_u are undefined — so there's no z-score to contribute. Their rating is still visible on individual product pages as raw data.

Q: What's the difference between N_raw and N_eff?

N_raw is the count of qualifying reviewers for a product. N_eff accounts for non-uniform weighting via the Kish formula N_eff = (Σw)² / Σw². Rankquant weights every qualifying reviewer identically, so N_eff = N_raw everywhere on this site. The distinction is worth knowing for reading aggregates elsewhere: unequal weights can collapse an impressive-looking N to a fraction of itself, and the CI is only honest if it uses the collapsed number.

Q: Does the AI-adjusted figure bias the ranking?

Yes, deliberately and in a known direction: it pulls every item toward the corpus average in proportion to how thin its sample is, so a four-reviewer item cannot hold a percentile its df does not support. The correction shrinks as n grows — at n = 53 an item keeps half its measured distance from the average, at n = 500 about 90%. It is one published constant of arithmetic, not a model, and the unadjusted global percentile stays visible next to it.

Q: Does pooling wine and bourbon reviewers into one pool violate df logic?

No. Pooling is a modeling assumption, not a df cheat. We're assuming the same reviewer uses roughly the same personal scale across categories — an assumption that loses a little per-category resolution in exchange for much better σ_u estimates (n_u is larger). We validate the assumption empirically by checking that within-reviewer rating dispersion is similar across categories for cross-category reviewers.

By Ryan Siegal · Founder and Principal

Published 2026-04-24

Short answer

Degrees of freedom (df) counts the number of independent values in a dataset after constraints. Rankquant normalizes per-reviewer, so the critical df is df = n_u − 1on each reviewer's personal standard deviation σ_u (Bessel's correction). We require n_u ≥ 2 for admission — which means df ≥ 1, the minimum possible. Reviewers with thin df are admitted on the same footing as everyone else, and thinness is handled downstream rather than at the door: it widens the product-level 90% CI-floor we publish, via SE(Ẑ) = 1/√N_eff, and it is charged in the AI-adjusted percentile, which shrinks the same mean by n/(n + 53). Because every qualifying reviewer counts equally — there is no source or reviewer multiplier anywhere in the pipeline — the effective sample size N_effis simply the reviewer count; Kish's design-effect formula is what shows that.

1. The classical definition

Degrees of freedom counts the dimensionality of a parameter space minus the number of constraints applied to the sample. For a sample x₁, x₂, …, x_n with computed sample mean x̄, the deviations (x_i − x̄) sum to zero:

Σ (x_i − x̄) = 0                  ← one linear constraint
therefore exactly (n − 1) of the (x_i − x̄) are free to vary;
the nth is determined by the constraint.

That constraint is why sample variance divides by n − 1 rather than n. Dividing by n produces an estimator biased low; dividing by n − 1 (Bessel's correction) yields an unbiased estimator of the population variance:

σ̂²  =  (1 / (n − 1)) · Σ (x_i − x̄)²      ← Bessel's correction, E[σ̂²] = σ²

The estimated variance of a sample underestimates the true variance of the population it was drawn from. The correction factor is n/(n−1).
— Bessel, Astronomical notes, 1830s (paraphrased)

2. Where df enters Rankquant's pipeline

Each estimator's df and the consequence when data is thin.
σ_u (reviewer personal SD)	df = n_u − 1. Bessel-corrected. Reviewer normalization requires σ_u > 0, which is automatic once a reviewer has any rating variance across their history.
Admission rule n_u ≥ 2	Minimum df = 1 for σ_u to be defined at all. A reviewer with n_u = 1 has no μ_u / σ_u — they go on file and are excluded from every published figure. Reviewers with n_u ≥ 2 but σ_u = 0 have no defined z-score either (division by zero), so they are excluded too.
Aggregation df	df = N − 1 for the per-product mean z-score, where N is the count of qualifying reviewers of that product. That mean is what the global and in-cohort percentiles rank, unaltered. The df shows up in SE(Ẑ) = 1/√N (scale factor 1 because z-scores have unit variance by construction), which sets the width of the 90% CI-floor published alongside the item.
Effective df (Kish)	N_eff = (Σ w)² / Σ w². Every qualifying reviewer carries the same weight, so N_eff = N exactly. The formula is here to show what unequal weighting would cost, not to describe a step we run.
AI-adjusted df	The third published figure multiplies the same mean z by n / (n + 53), so a thin-df item keeps less of its measured distance from the corpus average. Shrinkage arithmetic on the same equally-weighted pool — no second reviewer pool, no weighting.

Each estimator's df and the consequence when data is thin.

3. Why reviewer-level normalization (not source-level)

The choice to normalize at the reviewer grain rather than the source grain is a df choice. A wine publication might have 40 staff critics over 20 years; pooling them into one "source distribution" would throw away the fact that each critic uses a different personal scale. Per-reviewer normalization gives us one μ_u and one σ_u per critic — finer-grained, more honest.

The cost: reviewers with few reviews have noisy μ and σ estimates. A critic with n_u = 2 has df = 1 on σ_u, which means their σ_uis essentially a single data point. We admit them anyway — but the z-scores they produce are noisy, and that noise propagates to the product-level aggregate, where it widens the SE and pushes the published 90% floor further below the mean. If it is the item's whole sample that is thin, the AI-adjusted percentile charges for that directly. Thinness is reported and charged at the product level, not handled through exclusion.

4. Effective sample size when every reviewer counts equally

Any weighted aggregation raises a classical survey-statistics question: what's the effective sample size of a non-uniformly-weighted sample? Kish's (1965) design-effect formula is the standard answer:

N_eff  =  ( Σ_u w_u )²  /  Σ_u w_u²

  where w_u is the weight reviewer u carries in the aggregate.

Rankquant sets every w_u to 1. No reviewer is worth more than another and no source is worth more than another, so the formula collapses: N_eff = (N · 1)² / (N · 1) = N. The effective sample size is the plain count of qualifying reviewers, and SE(Ẑ) = 1/√N with nothing hidden inside it.

That is a df result, not a house style. The moment weights stop being equal, the effective sample size drops below the raw one — which is the cost a credibility multiplier would charge:

Equal weights, 12 reviewers:
  Σ w = 12,  Σ w² = 12   →   N_eff = 144 / 12 = 12      (= N_raw)

Hypothetical unequal weights, 100 reviewers
(90 at weight 1, 10 at weight 10):
  Σ w = 190, Σ w² = 1090 →   N_eff = 36,100 / 1,090 ≈ 33

  A nominally 100-reviewer mean that is statistically a 33-reviewer mean.

Under equal weights the published floor uses SE ≈ 1/√12 ≈ 0.289 for that 12-reviewer product. Under the unequal scheme, a 100-reviewer product would have to be booked at SE ≈ 1/√33 ≈ 0.174 rather than 1/√100 = 0.100 — two-thirds of the sample spent on the weighting, and an honest interval has to say so. Equal weighting is the only weighting that spends none of it, and it is what every figure on this site uses.

5. Why the AI-adjusted figure is a df trade

The third published figure, the AI-adjusted percentile, takes the same equally-weighted mean z and corrects it for how much confidence its df actually supports: adj = Ẑ · n / (n + 53). An item keeps that fraction of its measured distance from the corpus average (z = 0) and surrenders the rest, then the adjusted values are re-percentiled on the same scale as the global figure. K = 53 is the median sample size across everything we rank, so an item with a median-sized sample keeps half its distance; at n = 500 it keeps about 90%.

The trade is explicit: thin df buys a smaller claim. A product with four reviewers averaging z = +2.0 has almost no df behind that mean, and the adjustment moves it to +2.0 · 4/57 ≈ +0.14 — near the middle, where four reviewers can support a position. As n grows the correction vanishes and the adjusted figure converges on the unadjusted one. This is shrinkage arithmetic with one published constant — no model, no training, no inference — and the unadjusted global percentile stays on the page beside it so the size of the correction is visible.

6. The t-vs-z question under per-reviewer normalization

A standard statistics instinct says: "for small N, use Student's t instead of zfor the critical value." Under per-reviewer normalization, that instinct mostly doesn't apply. The z-scores z_u,iare already approximately unit-variance by construction. The sampling distribution of the mean z-score is asymptotically normal under the Central Limit Theorem, with small-N departures driven by (a) reviewer-σ estimation noise and (b) skew in the underlying raw-rating distribution.

We confirmed via simulation that for N ≥ 6 (the minimum sample size we admit), the 90% CI coverage of Ẑ − 1.645·(1/√N) is within 1.5 percentage points of nominal on realistic review distributions. For N < 6 we flag the product as "limited coverage" and show the CI-floor with a visible warning.

SE(Ẑ)  =  1 / √N_eff

  N_eff = 4     →  SE ≈ 0.500    (limited-coverage flag shown)
  N_eff = 6     →  SE ≈ 0.408    (interval coverage near nominal from here up)
  N_eff = 30    →  SE ≈ 0.183    (acceptable; coverage near nominal)
  N_eff = 100   →  SE ≈ 0.100    (comfortable)
  N_eff = 1000  →  SE ≈ 0.032    (floor essentially equals mean)

Frequently asked questions

Why does sample variance divide by n−1 instead of n?+

Because the sample mean x̄ is estimated from the same data. When you use the sample mean as your center, the deviations (x_i − x̄) are constrained to sum to zero, so only n−1 of them vary independently. Dividing by n−1 (Bessel's correction) gives an unbiased estimator; dividing by n underestimates true variance.

Is n_u = 2 really enough to normalize a reviewer?+

Just barely. With n_u = 2 a reviewer's σ_u has df = 1 and is extremely noisy. We admit them because (a) excluding them would throw away a lot of crowd data, and (b) the noise they introduce is booked at the product level rather than hidden: it widens SE(Ẑ) = 1/√N_eff and so the 90% floor we publish, and where the item's whole sample is thin the AI-adjusted percentile shrinks its mean by n/(n + 53). Reviewers with larger n_u contribute more stable z-scores, which is exactly what we want.

What happens to reviewers with only one review?+

They're on file but excluded from all three published figures. A single-review reviewer has no personal distribution — μ_u and σ_u are undefined — so there's no z-score to contribute. Their rating is still visible on individual product pages as raw data.

What's the difference between N_raw and N_eff?+

N_raw is the count of qualifying reviewers for a product. N_eff accounts for non-uniform weighting via the Kish formula N_eff = (Σw)² / Σw². Rankquant weights every qualifying reviewer identically, so N_eff = N_raw everywhere on this site. The distinction is worth knowing for reading aggregates elsewhere: unequal weights can collapse an impressive-looking N to a fraction of itself, and the CI is only honest if it uses the collapsed number.

Does the AI-adjusted figure bias the ranking?+

Yes, deliberately and in a known direction: it pulls every item toward the corpus average in proportion to how thin its sample is, so a four-reviewer item cannot hold a percentile its df does not support. The correction shrinks as n grows — at n = 53 an item keeps half its measured distance from the average, at n = 500 about 90%. It is one published constant of arithmetic, not a model, and the unadjusted global percentile stays visible next to it.

Does pooling wine and bourbon reviewers into one pool violate df logic?+

No. Pooling is a modeling assumption, not a df cheat. We're assuming the same reviewer uses roughly the same personal scale across categories — an assumption that loses a little per-category resolution in exchange for much better σ_u estimates (n_u is larger). We validate the assumption empirically by checking that within-reviewer rating dispersion is similar across categories for cross-category reviewers.

Next: The 90% CI-floor →