The theory, the evidence, and the methodology behind Rankquant's normalized review percentile. Pillar pieces designed to be the authoritative reference on rating inflation, Bayesian averaging, and review-source credibility.
Statistical methods are tools; tools can be misused. The hub for an ongoing series cataloguing the most common ways aggregated review scores cheat — Simpson's paradox, the small-sample illusion, p-hacked weights, survivorship bias, ecological fallacies — and the specific pre-commitments in Rankquant's methodology that defend against each.
A wine that's #1 in Burgundy can be #100 globally. A movie that's the best in its decade can sit mid-table when pooled with the entire IMDb catalog. Simpson's paradox — sub-group rankings flipping when sub-groups are aggregated — is the most common way naive review averages mislead. Here's the math, the worked example, and how Rankquant's cohort percentiles avoid it.
Kahneman and Tversky called it the "law of small numbers": the human tendency to over-trust patterns in tiny samples. Small-school test scores have higher variance at both extremes — and so do small-sample product ratings. Here's what the small-sample illusion is, the worked statistical case for ignoring 4-reviewer 5-star averages, and how Rankquant's AI-adjusted percentile encodes the principle in code: an item keeps only the share n / (n + 53) of its measured distance from the corpus average.
Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. The same logic, applied to review aggregation, says that any methodology operator who reserves the right to retune the numbers has implicitly given themselves the ability to pick winners after the fact. Rankquant weights no source and no reviewer — every qualifying reviewer counts equally, so there is no weight table to retune — and pre-commits in writing to every constant the pipeline depends on: the K = 53 shrinkage denominator behind the AI-adjusted percentile, the z = 1.645 multiplier behind each item's published confidence floor, the n_u ≥ 2 and σ_u > 0 reviewer-admission rules, and the ±20% cohort price band. Each is version-bumped publicly when it changes, with every historical version queryable. Here's why this self-imposed constraint is the difference between an editorial product and a marketing one.
Why almost nothing on Amazon, Yelp, or Goodreads is rated below 4-and-change — and what the right statistical response is. A plain-English introduction to the rating-inflation problem and why raw averaging is the wrong tool for consumer buying decisions.
How Bayesian prior adjustment protects review aggregation from thin-sample manipulation (three 5-star reviews no longer fake-ranks a product #1). The math, the prior-strength choice, and why IMDb Top 250 has been doing this since 2008.
Goodreads literary fiction averages 4.1+. Genre fiction averages 4.3+. Fewer than 3% of books on the platform rate below 3.5. A data-driven look at how friend-network rating behavior and a 1-5 scale destroyed the signal in book reviews.
The two most influential wine-rating publications disagree surprisingly often. A category-level analysis of the scoring distributions, where they diverge, and what the z-score tells you when they fight.
A field guide to every major wine-scoring scale in use today — Parker 100-pt, Jancis 20-pt, Vivino 1-5, retailer stars — and how they map against each other in practice. Includes the rescaling constants Rankquant uses to normalize across scales.
The fresh/rotten binary throws away 80% of the information in a movie review. A statistical critique of RT's methodology and why Metacritic's weighted aggregate, despite its own problems, carries more signal per percentage point.
A category-by-category breakdown of the review sources that actually matter, ranked by editorial rigor, manipulation resistance and inflation pressure. Covers wines, movies, books, Amazon consumer products, hotels, and restaurants — plus the short list of platforms Rankquant actually reads today.
Cross-platform analysis of rating inflation across the six largest consumer-review surfaces. Distribution compression, 20-year drift, information-loss in bits per review. Monthly updates.
Why a 2019 Bordeaux and a 2013 Bordeaux should never be compared against the same peer-set mean, and how Rankquant's adaptive-cohort normalization handles vintage variation.
The statistical case for normalization in the value tier ($10–$30). At that price the raw 100-point scale collapses to 3 usable points — normalization is the only way to tell a great $20 wine from an ordinary one.
Natural wines present unique challenges for aggregated scoring: thin professional coverage, stylistic polarization, rapid category growth. Here's how we normalize them without either dismissing or over-promoting the category.