Statistics & methodology glossary

Every statistical concept behind Rankquant's normalized percentiles, in plain language. Each entry is linkable as #term and exposes schema.org/DefinedTerm structured data for AI retrieval.

Normalized percentile: A 0–100 number indicating where a product's mean z-score ranks relative to every other product's mean z-score in the Rankquant database. 90 means top 10%; 50 means median; 10 means bottom 10%. Derived from the empirical cumulative distribution function (empirical CDF) of all products' mean z-scores.
Global percentile: The first of the three figures Rankquant publishes per item: a product's normalized percentile against everything we rank. Unweighted — it is built from the mean z-score alone, with every qualifying reviewer counting equally. Always stable because the reference set is very large. Useful for big-picture "is this product any good" questions.
In-cohort percentile: The second published figure: the same mean z-score, purely re-ranked against only products in the same category and within ±20% of its list price. Mathematically it is a re-ranking of the quantity behind the global percentile — a smaller reference set, not a separate computation. Useful for "is this good for what it costs?" questions.
Rating inflation: The phenomenon where online review scores cluster in a compressed high range (Amazon 4.4/5, Yelp 4.2/5, Goodreads 4.1+). Caused by self-selection bias, social signaling, and platform incentives. Makes raw averages nearly useless for consumer buying decisions — the problem Rankquant's per-reviewer normalization is designed to solve.
Per-reviewer z-score normalization: Rankquant's first operation: for each reviewer u, subtract their personal mean μ_u and divide by their personal SD σ_u. Converts a reviewer's rating to a z-score z_{u,i} = (r_{u,i} − μ_u) / σ_u in their personal units. Removes the reviewer main effect so different reviewers' scores become commensurable.
Z-score: The number of standard deviations a value sits above or below its mean. In Rankquant, z-scores are computed per reviewer so each reviewer's ratings become unit-variance in their personal distribution. A reviewer's z of +1.6 for a product means that product sits 1.6 personal-SDs above their personal mean.
Reviewer personal mean (μ_u): The arithmetic mean of all ratings a reviewer u has produced in our dataset, pooled across every category they've rated. Computed once and updated as new reviews arrive.
Reviewer personal standard deviation (σ_u): The Bessel-corrected sample standard deviation of all ratings a reviewer u has produced, pooled cross-category. Must be > 0 for the reviewer to qualify: if σ_u = 0 (the reviewer gives every product the same rating) there is no personal scale to divide by, so the reviewer is excluded.
Cross-category reviewer pooling: Rankquant's decision to compute a reviewer's μ_u and σ_u across every category they've rated rather than per-category. Assumes a person's rating calibration is roughly stable across product types — an assumption we continue to validate empirically. Keeps n_u large and σ_u stable.
Qualifying reviewer: A reviewer with n_u ≥ 2 and σ_u > 0. Only qualifying reviewers enter the aggregate, and every one of them counts exactly the same — no reviewer and no source is weighted above another. Reviewers with σ_u = 0 (never varies) or n_u = 1 (μ and σ undefined) are excluded.
Reviewer main effect: The portion of raw-rating variance explained by "which reviewer is rating" rather than "which product is being rated." Per-reviewer z-score normalization mathematically zeros the reviewer main effect, leaving only the product-quality signal plus residual noise.
Mean z-score (the aggregate): Rankquant's one aggregate, and the quantity every published figure is built from. The unweighted mean z-score across every qualifying reviewer of a product. Every reviewer counts equally; there is no credibility weighting and no per-source multiplier anywhere in the pipeline. Answers: "What does the crowd think, relative to each reviewer's own scale?"
AI-adjusted percentile: The third published figure: the same reviewer-normalized mean, discounted for how much confidence the sample actually supports. An item keeps the share n / (n + 53) of its measured distance from the corpus average, so thin samples are pulled hard toward that average while well-sampled items keep most of their distance. The rule is identical in all six verticals, and it is the only place sample size is charged for — no published percentile is ranked on a confidence-interval bound, though each item also publishes its own 90% CI lower bound as a diagnostic that quantifies the same uncertainty. It is shrinkage arithmetic — no model, no training, no machine learning of any kind.
Source weight (w_s): A per-source multiplier some aggregators apply to make reviewers from favoured publications count for more than others. Rankquant does not use one. Every qualifying reviewer is normalized onto their own z-scale and then counted equally, so no source is weighted above any other and there is no weight table to publish.
CI-floor (90% confidence-interval lower bound): A pessimistic lower bound on an aggregate: floor = Ẑ − 1.645 · SE(Ẑ). It sits further below the mean the thinner the sample, because a 4-reviewer +2.1 mean has a much wider SE than an 80-reviewer +1.6 mean. Rankquant computes it for every item and publishes it as a per-item diagnostic — ciFloorZ in the catalog exports, zCiLower and zMargin on products — but nothing is ranked by it. The global and in-cohort percentiles rank the mean z-score itself, and the AI-adjusted percentile charges for sample size by shrinkage instead. Same family as Wilson score (Reddit) and IMDb Top 250 shrinkage.
Standard error (SE): The standard deviation of a sampling distribution. For a mean z-score, SE(Ẑ) = 1/√N_eff. Shrinks as √N, so N=100 cuts the SE to one-third of N=10. The CI-floor Rankquant publishes for each item sits 1.645 · SE below the mean, so that diagnostic scales linearly with SE.
Effective sample size (N_eff): In Rankquant, N_eff is simply the count of contributing qualifying reviewers, because the aggregate is unweighted — every reviewer carries the same weight, so nothing collapses the sample. Weighted aggregates (which Rankquant does not compute) always have N_eff ≤ N_raw; the ratio tells you how much the weighting collapses the effective sample.
Kish design-effect formula: The standard survey-statistics formula for the effective sample size of a weighted mean: N_eff = (Σ w)² / Σ w². When weights are equal, N_eff = N. When one observation dominates the weights, N_eff collapses toward 1. It has no work to do at Rankquant — with equal weights throughout, N_eff = N by construction.
One-tailed critical value: The z or t value defining a one-sided rejection region. Rankquant uses z = 1.645 (90% one-tailed) for the CI floor it publishes beside each item's percentiles — a defensibly pessimistic lower bound on the mean z-score. We use one-tailed rather than two-tailed because we care about the downside of the estimate, not both tails.
Wilson score interval: The continuity-corrected confidence interval for a binomial proportion. Used by Reddit's "best" sort and Yelp's internal ranking. Rankquant's CI-floor is the continuous-scale analog — the same family of small-sample-safe primitives, published here as a per-item diagnostic rather than used as the ranking basis.
Empirical cumulative distribution function (empirical CDF): A step function that counts the fraction of sample points below any given value. Rankquant computes the empirical CDF of all mean z-scores in the database and uses it to convert each product's mean z-score to a 0–100 percentile. Robust to non-normality of the underlying distribution.
Rank-based ranking: A ranking scheme that uses a value's rank among peers rather than its absolute magnitude. Robust to monotone transformations, outliers, and non-normality. Rankquant's percentile is a rank-based transformation of the mean z-score.
Cohort (±20% price, same category): The narrow peer set used for Rankquant's in-cohort percentile: products in the same category and within a symmetric ±20% price band. A $100 product's cohort is $80–$120. When the natural band produces fewer than 20 cohort members, we expand the band until ≥20 are in scope and log the expansion on the product page.
Cohort re-ranking: Converting a product's percentile from global (against everything we rank) to in-cohort (against cohort members only) by pure re-ranking of the same mean z-score. No new computation. Guarantees cohort rank is a deterministic function of mean z-scores and cohort membership — auditable by hand.
Degrees of freedom (df): The number of independent values in a dataset after constraints. In Rankquant, σ_u uses df = n_u − 1 (Bessel's correction). For the product-level aggregate, the minimum admission rule is df ≥ 1 per reviewer (n_u ≥ 2). Deep dive at /theory/degrees-of-freedom/.
Bessel's correction: The n−1 denominator in the sample-variance formula (rather than n). Corrects the downward bias in σ² that would otherwise arise when the sample mean is estimated from the same data. Rankquant uses Bessel-corrected σ_u for every reviewer.
Standard deviation (σ): The square root of variance: a measure of spread around the mean. In Rankquant, each reviewer has a personal σ_u, and z-score normalization uses σ_u as that reviewer's scale factor.
Bias–variance tradeoff: The fundamental statistical tension: reducing bias in an estimator typically increases variance and vice versa. Rankquant's AI-adjusted percentile takes that trade explicitly: shrinking toward the corpus average accepts a little bias to buy a large variance reduction on thin-sample products. The same intuition drives Bayesian shrinkage generally.
Reviewer fixed effects: A regression-modeling framework in which each reviewer is assigned their own intercept, effectively removing their main effect from the product-quality estimate. Rankquant's per-reviewer normalization is a normalized analog: subtracting μ_u and dividing by σ_u is equivalent to absorbing a reviewer-specific intercept and scale.
Central Limit Theorem: The theorem that the distribution of a sample mean becomes approximately normal as sample size grows, regardless of the underlying distribution's shape. Justifies treating the sampling distribution of Ẑ (the product-level mean z-score) as approximately normal for the CI-floor computation once N ≥ ~6.
Intraclass correlation coefficient (ICC): A measure of reliability for continuous ratings. ICC = σ²_between / (σ²_between + σ²_within). In Rankquant, we compute reviewer-level ICC(1,1) on the z-scored data to quantify how much reviewers agree about product quality after personal-scale differences are removed. Deep dive at /theory/inter-rater-reliability/.
Cohen's kappa (κ): Chance-corrected agreement statistic for two raters on a categorical scale. κ = 0 means chance-level agreement; κ = 1 means perfect. Used for binary reviewers (Rotten Tomatoes fresh/rotten, Michelin star/no-star) that can't be z-score normalized in the standard way.
Fleiss' kappa: A generalization of Cohen's κ to more than two raters rating the same items categorically. Rankquant uses it for panels of binary reviewers where pairwise Cohen's would produce combinatorially many numbers.
Variance decomposition: Splitting total variance into additive components — in review data typically between-product, between-reviewer, and residual. Per-reviewer normalization mathematically zeros out the between-reviewer component, leaving between-product + residual for ICC computation.
Self-selection bias: A bias in voluntary review systems: consumers with strongly positive (or sometimes strongly negative) experiences are much more likely to leave reviews than consumers with middling experiences. Drives rating inflation. Per-reviewer normalization partially corrects for it by rescaling each reviewer's distribution to unit variance.
Right-skew: A distributional shape where most observations cluster near the top of the scale. Online review distributions are typically right-skewed (in the sense of ceiling-clustered). Rank-based percentile mapping is robust to right-skew; mean-based aggregation is not.
Outlier filtering: Removing reviews that diverge extremely from the bulk distribution — typically obvious manipulation (review bombs, 1-star content-farm reviews) before normalization. Rankquant applies a simple IQR-based filter on the raw-rating distribution before computing reviewer statistics.
Review bomb: A coordinated effort to lower a product's review score via large numbers of 1-star reviews from newly-created or low-activity accounts. Detected via reviewer-behavior heuristics (n_u = 1 with σ_u undefined excludes these reviewers from the aggregate automatically).
Reproducibility: The property that running Rankquant's normalization on the same inputs (reviewer ratings + published constants) produces identical outputs. All inputs are published, and every product page shows its own intermediates (mean z-score, reviewer count, the shrinkage applied), so any percentile can be recomputed by hand from published numbers. The reference implementation itself is not distributed as a public repository.
Maximum likelihood estimation (MLE): A method for estimating parameters by maximizing the likelihood of observing the data under a model. Rankquant uses Bessel-corrected variance (the unbiased estimator, df = n−1) rather than the MLE variance (biased, uses n) because unbiasedness matters more than asymptotic efficiency at small sample sizes.
Empirical Bayes: A framework where prior parameters are estimated from the data rather than specified a priori. Rankquant's shrinkage constant K = 53 is empirical-Bayes in spirit: it is read off the data — the median sample size across everything we rank — rather than chosen in advance.
Shrinkage estimator: An estimator that moves raw observations toward a central value (shrinkage target) to trade bias for reduced variance. IMDb's Top 250 formula uses Bayesian shrinkage toward a global mean. Rankquant publishes both flavours but ranks on only one. The CI-floor is a close cousin of shrinkage — instead of pulling the estimate toward a prior it subtracts a standard-error term — and it ships as a per-item diagnostic figure. The AI-adjusted percentile shown as the third figure on every detail page is literal shrinkage, and it is the one that gets ranked: adj = mean_z · n/(n+53), which pulls each item toward 0 — the corpus average on the per-reviewer z-scale — in proportion to how thin its sample is, with K = 53 set to the median sample size across all 139,430 ranked items so the median item is pulled exactly halfway. A 4-reviewer +2.1 mean lands at +0.15; an 80-reviewer +1.6 mean keeps +0.96. Both flavours account for small samples in a principled way; neither involves a model or any machine learning.
James–Stein estimator: Stein's 1956 finding that when simultaneously estimating three or more means, shrinking them all toward a common grand mean dominates the unshrunken sample means in total squared error. Rankquant's empirical-CDF percentile step is a rank-based analog of this shrinkage intuition.
Fixed-effects vs random-effects aggregation: In meta-analysis, fixed-effects models assume all studies share one true effect; random-effects models allow per-study heterogeneity. Rankquant's reviewer-level aggregation is fixed-effects (we treat each reviewer as producing noisy observations of the same product-quality signal). A random-effects upgrade is a future methodology version.
Bootstrap resampling: A nonparametric method: draw B resamples (with replacement) from the original data, compute the statistic on each, use the resulting distribution as the sampling distribution. Rankquant uses bootstrap to validate analytical CI-floor coverage at small N and to estimate uncertainty in the percentile rank itself.
Kolmogorov–Smirnov test: A nonparametric test for whether a sample comes from a specified distribution (one-sample) or whether two samples share a distribution (two-sample). Rankquant uses KS in its pipeline diagnostics to flag reviewer distributions with unusual shapes (strong bimodality, hard truncation) that merit editorial inspection.
Skewness: A measure of distributional asymmetry. Positive skew = long right tail; negative skew = long left tail; 0 = symmetric. Online review distributions have strong negative skew (ceiling clustering). Per-reviewer normalization does not remove skew — it removes scale and location — which is why we use rank-based percentile mapping instead of a z-to-percentile normal CDF in Step 4.
Kurtosis: A measure of tail heaviness. Normal distribution has kurtosis = 3 (excess kurtosis = 0). Review-bombed products have low kurtosis (bimodal, thin tails around the middle); well-calibrated reviewer distributions have kurtosis close to normal.
Interquartile range (IQR): The range between the 25th and 75th percentile (Q3 − Q1). A robust measure of spread, unaffected by extreme outliers. Used in Rankquant's pre-normalization outlier filter: ratings more than 1.5·IQR outside Q1/Q3 are flagged for editorial review.
Median absolute deviation (MAD): A robust alternative to standard deviation: MAD = median(|x_i − median(x)|). Resistant to outliers. Rankquant uses MAD as a cross-check on σ_u when a reviewer's distribution is suspected of contamination (e.g. bot-pattern ratings).
Cohen's d: Standardized effect size: d = (μ_1 − μ_2) / σ_pooled. A Rankquant global-90th-percentile product has Cohen's d of roughly +1.3 vs the database mean — a large effect, meaning the product is genuinely distinguishable from average.
Statistical power (1 − β): The probability of correctly rejecting a false null hypothesis. In Rankquant's context, power is the probability that two meaningfully-different-quality products receive distinguishable CI-floors. At N = 30 per product, power to distinguish effect sizes of d = 0.5 at our 90% confidence level is roughly 0.75.