The 7 review sources that dominate every category (and how much to trust each)
By Ryan Siegal · Founder and Principal
The pattern across every category
Every consumer category falls into a similar review-source ecology:
- Professional critics — editorial publications with paid reviewers and editorial standards. Small sample size but high per-review information density.
- Verified-purchase aggregators — platforms where reviews are tied to actual transactions. Large samples, medium rigor, high inflation pressure.
- Enthusiast crowd platforms — community-driven sites where passionate hobbyists rate. Large samples, medium rigor, slightly less inflation than general consumer platforms.
- General consumer platforms — anyone-can-review sites (Google Reviews, Yelp). Very large samples, low rigor, highest inflation pressure.
Our answer to that ecology is not to rank the tiers against each other. It is to read whichever of them we can reach, normalize every reviewer against their own history — subtract their personal mean, divide by their personal standard deviation — and rank inside a peer cohort, with every qualifying reviewer counting equally. The method is written out at /methodology. The tiers in the tables below are a separate thing: an argument about which sources are worth your attention, not a coefficient anything gets multiplied by.
What Rankquant reads today
None of the professional publications in the tables that follow is ingested anywhere, in any category — so those tables are landscape analysis, not a description of a shipped computation. The live inputs are crowd platforms:
| Wines | Vivino reviewer pool. |
|---|---|
| Movies & TV | IMDb, blended with Rotten Tomatoes on the subset of titles that carries it. |
| Books | Goodreads and Amazon. |
| Hotels | TripAdvisor, Booking.com, Agoda, Expedia, Trip.com, Ostrovok. |
| Cruises | Cruise Critic. |
| Products | Amazon customer reviews. |
| Reviewer weighting | None. Every qualifying reviewer counts exactly the same, in every category; the differentiation comes from per-reviewer normalization, not from ranking the platforms or the people inside them. |
So the rest of this article is an assessment of the review landscape: which sources repay your attention when you go read them yourself, and which do not. None of it feeds a Rankquant number.
Wines — who to trust
| Robert Parker / Wine Advocate / Vinous | Oldest professional 100-pt scale; historically the highest credibility in the category. |
|---|---|
| Wine Spectator | Institutional panel reviews; staff tastings. |
| Jancis Robinson MW | 20-pt scale with less inflation pressure; rigorous editorial. |
| Decanter | International panel coverage. |
| James Suckling | Prolific; known to be slightly generous. |
| Jeb Dunnuck | Growing credibility; narrower coverage. |
| CellarTracker (crowd, enthusiast) | Deep for specific wines. |
| Vivino (crowd, general) | Largest sample; lowest rigor per review. |
| Retailer reviews (Wine.com, Total Wine, etc.) | Retail bias; the least informative of the set. |
Movies & TV — who to trust
| Metacritic | Professional aggregate, pre-weighted before you see it — that weighting is Metacritic's, and it is undisclosed. |
|---|---|
| Letterboxd (crowd, cinephile) | Enthusiast audience; cleaner than IMDb. |
| Rotten Tomatoes (critics) | Binary fresh/rotten loses information. |
| IMDb weighted average | Huge sample; genre ballot-stuffing pressure. |
Books — who to trust
| NYT Book Review | Rigorous editorial; highest-profile professional reviews. |
|---|---|
| Kirkus Reviews | Pre-publication professional reviews; industry standard. |
| Publishers Weekly | Trade-press professional editorial. |
| Booklist / LitHub / NPR | Professional critics; somewhat narrower coverage. |
| Goodreads (crowd) | Massive sample; severe inflation. |
| Amazon book reviews | Known manipulation; read the raw average with suspicion. |
Amazon consumer products — who to trust
| RTINGS / Wirecutter / Consumer Reports | Lab-tested; professional editorial. |
|---|---|
| Amazon verified purchase | Transaction-verified but self-selection biased. |
| Reddit topical subreddits (aggregated) | Noisy but directionally informative. |
Hotels — who to trust
| Michelin Keys / Forbes Travel | Professional; rigorous standards. |
|---|---|
| Booking.com (verified stays) | Verified but 8.4/10 average inflation. |
| TripAdvisor | Older, more gaming pressure. |
| Google Hotels | Lower verification strictness. |
Restaurants — who to trust
| NYT / Michelin / professional critics | Rigorous editorial. |
|---|---|
| Resy / OpenTable (verified diners) | Verified but inflated. |
| Google Reviews | High volume; lower verification. |
| Yelp | Most-gamed platform in the category. |
Why publish an opinion no score depends on
Every other review aggregator treats its weighting as proprietary secret sauce. Metacritic doesn't tell you why a New York Times review is weighted more than a Variety review. Rotten Tomatoes doesn't explain which reviewer gets which weight.
The opposite move is to have no secret sauce, because there is no weighting to keep secret. Every qualifying reviewer we read counts once, and the whole computation — admission rules, per-reviewer z-scores, the confidence correction and the percentile mapping — is written out at /methodology, so a reader can rerun it line by line. The tier lists above are an opinion you can disagree with at no cost to the arithmetic, precisely because no score reads them. The output is deterministic: same inputs, same arithmetic, same score.
The same transparency is how we answer accusations of bias on the scores that are live. If a brand claims Rankquant is under-ranking their product, we can point to the reviewer set, the raw ratings, each reviewer's own baseline, the published CI-floor diagnostic and the peer set — and demonstrate that the normalized output is mechanically correct given the inputs.
Frequently asked questions
How are these trust tiers decided?+
Which of these sources actually feeds a Rankquant score?+
Can the tiers change?+
What stops one biased review from moving a score?+
What if an item has almost no usable reviewers?+
Related: The full methodology · Rating inflation explained · Bayesian averaging