The 7 review sources that dominate every category (and how much to trust each)
By Ryan Siegal · Founder and Principal
The pattern across every category
Every consumer category falls into a similar review-source ecology:
- Professional critics — editorial publications with paid reviewers and editorial standards. Small sample size but high per-review information density.
- Verified-purchase aggregators — platforms where reviews are tied to actual transactions. Large samples, medium rigor, high inflation pressure.
- Enthusiast crowd platforms — community-driven sites where passionate hobbyists rate. Large samples, medium rigor, slightly less inflation than general consumer platforms.
- General consumer platforms — anyone-can-review sites (Google Reviews, Yelp). Very large samples, low rigor, highest inflation pressure.
The right meta-methodology is to weight all four tiers, Bayesian-adjust thin-sample products, and z-score within category. Rankquant's source weights reflect this tiered credibility.
Wines — source weights
| Robert Parker / Wine Advocate / Vinous | Weight 10 — oldest professional 100-pt scale; historically highest credibility. |
|---|---|
| Wine Spectator | Weight 10 — institutional panel reviews; staff tastings. |
| Jancis Robinson MW | Weight 9 — 20-pt scale with less inflation pressure; rigorous editorial. |
| Decanter | Weight 8 — international panel coverage. |
| James Suckling | Weight 7 — prolific; known to be slightly generous. |
| Jeb Dunnuck | Weight 7 — growing credibility; narrower coverage. |
| CellarTracker (crowd, enthusiast) | Weight 3 — deep for specific wines. |
| Vivino (crowd, general) | Weight 2 — largest sample; lowest rigor per review. |
| Retailer reviews (Wine.com, Total Wine, etc.) | Weight 1 — retail bias; low weight. |
Movies & TV — source weights
| Metacritic | Weight 8 — pre-weighted professional aggregate. |
|---|---|
| Letterboxd (crowd, cinephile) | Weight 6 — enthusiast audience; cleaner than IMDb. |
| Rotten Tomatoes (critics) | Weight 6 — binary fresh/rotten loses information. |
| IMDb weighted average | Weight 5 — huge sample; genre ballot-stuffing pressure. |
Books — source weights
| NYT Book Review | Weight 9 — rigorous editorial; highest-profile professional reviews. |
|---|---|
| Kirkus Reviews | Weight 8 — pre-publication professional reviews; industry standard. |
| Publishers Weekly | Weight 8 — trade-press professional editorial. |
| Booklist / LitHub / NPR | Weight 6-7 — professional critics; somewhat narrower coverage. |
| Goodreads (crowd) | Weight 3 — massive sample; severe inflation. |
| Amazon book reviews | Weight 2 — known manipulation; low weight. |
Amazon consumer products — source weights
| RTINGS / Wirecutter / Consumer Reports | Weight 9 — lab-tested; professional editorial. |
|---|---|
| Amazon verified purchase | Weight 4 — transaction-verified but self-selection biased. |
| Reddit topical subreddits (aggregated) | Weight 3 — noisy but directionally informative. |
Hotels — source weights
| Michelin Keys / Forbes Travel | Weight 9 — professional; rigorous standards. |
|---|---|
| Booking.com (verified stays) | Weight 5 — verified but 8.4/10 average inflation. |
| TripAdvisor | Weight 4 — older, more gaming pressure. |
| Google Hotels | Weight 4 — lower verification strictness. |
Restaurants — source weights
| NYT / Michelin / professional critics | Weight 9 — rigorous editorial. |
|---|---|
| Resy / OpenTable (verified diners) | Weight 5 — verified but inflated. |
| Google Reviews | Weight 4 — high volume; lower verification. |
| Yelp | Weight 3 — most-gamed platform in the category. |
Why we publish the weights at all
Every other review aggregator treats its weighting as proprietary secret sauce. Metacritic doesn't tell you why a New York Times review is weighted more than a Variety review. Rotten Tomatoes doesn't explain which reviewer gets which weight.
We publish everything because the methodology is the product. If a reader disagrees with our weight for James Suckling (7) versus Wine Spectator (10), they can fork the open-source code at github.com/rankquant/normalize and re-run the normalization with their own preferred weights. The output is deterministic: same inputs, same weights, same score.
That transparency is also how we defend against accusations of bias. If a brand claims Rankquant is under-ranking their product, we can point to the source weights, the raw scores, the Bayesian adjustment, and the peer set — and demonstrate that the normalized output is mechanically correct given the inputs.
Frequently asked questions
How do you decide source weights?+
Three factors: (1) editorial rigor (professional staff > individual crowd reviewers), (2) manipulation resistance (verified purchase > anonymous), and (3) inflation pressure (lower-inflation scales get a modest boost). The specific numbers come from evaluating each source's historical correlation with outcomes that matter downstream — resale value for wines, audience retention for movies, durability for consumer products.
Can source weights change?+
Yes, but only via a public version bump. If a source gets acquired, loses its editorial staff, changes scoring methodology, or demonstrably starts being gamed, we revisit its weight and publish a changelog. Historical scores remain computable at the weights that were active when they were calculated.
What if a source you've weighted highly publishes an obviously biased review?+
The Bayesian prior pulls heavily-disagreeing single reviews toward the peer-set mean, so an outlier from a single highly-weighted source doesn't dominate the final score. Combined with multiple sources for most products, this limits single-source-bias damage.
What if a product only has coverage from one source?+
Then the Bayesian adjustment pulls its normalized score hard toward the category prior — a single-source wine with no category context gets a score very close to the peer-set median, flagged with "Limited coverage" on the product page.
Related: The full methodology · Rating inflation explained · Bayesian averaging