Inter-rater reliability under per-reviewer normalization

By Ryan Siegal · Founder and Principal

Published 2026-04-24

1. What per-reviewer normalization does to inter-rater disagreement

A classical decomposition of a review dataset splits observed variance into three sources: between-product variance (σ²_P), between-reviewer variance (σ²_R, also called the "reviewer main effect"), and residual variance (σ²_e).

Observed rating:   r_{u,i}  =  μ  +  α_u  +  β_i  +  ε_{u,i}

  μ        = grand mean
  α_u      = reviewer main effect    (reviewer u's personal offset)
  β_i      = product true quality    (the signal we want)
  ε_{u,i}  = residual (noise + reviewer-by-product interaction)

Variance decomposition:
  Var(r)  =  σ²_R  +  σ²_P  +  σ²_e

Per-reviewer z-score normalization subtracts each reviewer's personal mean and divides by their personal SD. That operation:

• Zeros out α_u: subtracting μ_u removes the reviewer's offset.
• Standardizes σ_u: dividing by σ_u forces every reviewer to unit variance.
• Preserves β_i: the product-quality signal — the thing we actually care about — is the only remaining systematic effect.

What's left in the z-scored dataset is just between-product variance plus residual noise. ICC computed on this residual captures agreement about product quality, not agreement about scale usage.

2. Reviewer-level ICC(1,1) in the z-scale

For z-scored data z_u,i, the one-way random-effects single-measurement intraclass correlation is:

ICC(1,1)  =  σ²_P  /  (σ²_P  +  σ²_e)

where σ²_P and σ²_e are estimated from the ANOVA decomposition of z_{u,i}.

Interpretation (Koo & Li, 2016):
  ICC < 0.50    →  poor reliability        (category is genuinely polarizing)
  0.50–0.75     →  moderate reliability    (typical for luxury wine, arthouse film)
  0.75–0.90     →  good reliability        (typical for consumer electronics, hotels)
  ICC ≥ 0.90    →  excellent reliability   (rare; only in very objective categories)

3. What ICC tells us about a category

These are reviewer-level ICCs, computed on cross-reviewer z-scored data, which is why they differ from published source-pair ICCs (the older approach that compared Wine Spectator and Wine Advocate source means directly). Reviewer-level ICCs are lower than source-level ICCs because per-reviewer normalization exposes individual-critic disagreement that source-level averaging hides.

4. What low ICC does to a single headline number

When reviewer-level ICC is high, the mean z-score is a reliable single number: the reviewers who differed only about scale have already been reconciled, and what remains is a consensus worth quoting. When ICC is low, that same mean still reports the crowd's consensusbut cannot tell you whether a dissenting minority is the better-informed one. No aggregation rule repairs that. Promoting some reviewers above others would only substitute the publisher's opinion about whose taste counts for the reader's, which is why every qualifying reviewer here counts exactly once.

What that leaves as the practical low-ICC signal is the spread inside the reviewer pool. Where an item's z-normalized score and its raw average diverge sharply, its reviewers disagree systematically once their personal baselines are removed: the item is polarizing, and the decision depends on whose taste aligns with yours. Both figures sit on every item page for exactly that reason.

5. Cohen's κ for binary rater decisions

Some reviewers produce only binary decisions (Rotten Tomatoes fresh/rotten, Michelin star/no-star). Z-score normalization doesn't directly apply to a single binary observation. The right agreement statistic for those raters is pairwise Cohen's κ on shared products:

κ  =  (p_o − p_e) / (1 − p_e)

  p_o = observed agreement rate on binary decisions
  p_e = agreement rate expected by chance

Interpretation (Landis & Koch, 1977):
  κ < 0.00     →  poor
  0.00 – 0.20  →  slight
  0.21 – 0.40  →  fair
  0.41 – 0.60  →  moderate
  0.61 – 0.80  →  substantial
  0.81 – 1.00  →  almost perfect

Binary reviewers are admitted only when paired with at least two other binary decisions (so they have a personal "mean" of 0 or 1 and a non-zero σ). Their z-scores saturate at ±1, so they move an aggregate less than a continuous-scale reviewer can — a property of the arithmetic, not a weight we assign them.

6. What reviewer-level ICC is used for — and what it is not

ICC is a diagnostic we report, not a multiplier we apply. It is estimated per category and used to say plainly how much agreement sits behind the numbers on that category's pages. It never becomes a coefficient on anyone's rating. No reviewer and no source is weighted above another anywhere in the pipeline: the global percentile, the in-cohort re-ranking and the AI-adjusted figure are all computed from the same pool, one vote per qualifying reviewer.

The reason is that the alternative is unauditable from the reader's side. A credibility multiplier is an editorial opinion wearing a number's clothes — it moves the ranking by an amount only the publisher can check. An equal-weight mean of z-scores is reproducible from the published constants: take the reviewers, take their personal means and SDs, average. What we do let vary is confidence, not credibility — a thin sample is corrected for how much it can actually support, which is arithmetic anyone can redo. The constants are at /methodology, and any change to them produces a public methodology version bump in the changelog with historical percentiles preserved as computed.

← Confidence intervals · Back to Theory overview