Why we never change our published constants after the fact (and publish them on day one)

By Ryan Siegal · Founder and Principal

Published 2026-04-27·Last updated 2026-08-06

Correction, 6 August 2026. An earlier version of this article described Rankquant as assigning per-source credibility weights, and printed a table of them. That was wrong. No published Rankquant score has ever applied a source weight or any other per-source multiplier: the pipeline computes one unweighted mean of per-reviewer z-scores in which every qualifying reviewer counts equally. The weight tables have been withdrawn site-wide and the withdrawal is logged at /changelog. The article has been rewritten around the constants that do exist.

Goodhart's Law and Campbell's Law

The British economist Charles Goodhart formulated the rule that bears his name in 1975, in the context of UK monetary policy: any observed statistical regularity will tend to collapse once pressure is placed on it for control purposes. Donald Campbell, working independently in social-policy evaluation, formulated a stronger version: the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Achievement tests may well be valuable indicators of general school achievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways.

What this looks like in review aggregation

In an aggregator that weights its sources, the weight table is the highest-leverage knob on the board: raise one publication's weight and everything that publication likes moves up. Rankquant does not have that knob. Reviewers are normalized onto their own z-scale and then averaged with no weights at all — no source, platform or publication is worth more than any other, and neither is any individual reviewer. Admission to the pool is the only distinction the pipeline makes, and once admitted, everyone counts once.

That removes one class of manipulation and leaves another. The pipeline still depends on constants, and a constant is a number somebody chose:

• K = 53. The shrinkage denominator in the AI-adjusted percentile, which ranks mean_z · n/(n + K) instead of the mean alone. It is the median calibrated-reviewer count across the ranked catalog, so the median item is pulled exactly halfway toward the corpus average.
• z = 1.645. The 90% one-tailed multiplier behind the confidence floor published on each item beside its score, as a diagnostic rather than as the headline ranking key.
• n_u ≥ 2 and σ_u > 0. The reviewer admission rules. A reviewer needs two ratings before they have a personal distribution at all; a reviewer who rated everything identically has no dispersion to divide by and is admitted, where they are admitted, on an imputed one.
• ±20% of list price. The band that defines a product cohort, and so which peers the in-cohort percentile is a re-ranking against.

Each is a place where a number could be moved to move rankings. K is the clearest case. Because per-reviewer normalization puts the corpus average at exactly zero on the z-scale, multiplying a mean by n/(n + 53) pulls thin-sample items toward the middle: an item with three calibrated reviewers keeps about 5% of its measured distance from average. Set K to 5 instead and the same item keeps 38%. An operator who wanted to elevate boutique items carried by a handful of enthusiastic reviewers could do it that way without changing a single line of the aggregation code. There would be no statistical evidence of it on any individual item's page — the arithmetic is the same; only the constant moved.

Which is why the inventory matters. Wine scores come from the Vivino reviewer pool, films from IMDb blended with Rotten Tomatoes where it exists, books from Goodreads and Amazon, hotels from six booking and travel platforms, cruises from Cruise Critic, products from Amazon customer reviews. No professional critic publication is ingested. There is no credibility ladder anywhere in that list and no plan to add one: a source's contribution is however many qualifying reviewers it brings, each counted once. The constants are the entire tunable surface — which is what the rest of this article is about.

The defence is pre-commitment

Pre-registration — the practice of publishing your analysis plan beforeseeing the data — is the standard defence against this kind of post-hoc manipulation in clinical trials and increasingly in academic social science. The replication-crisis literature documented how flexibility in analytical choices (which covariates to include, which subgroups to analyse, which outliers to drop) lets researchers find "significant" effects in noise. The fix is to commit in writing, in advance, to what you will do.

Rankquant's constants are pre-registered in the same sense. Each one is written out at /methodology inside the step that uses it — the admission rules in the normalization step, K = 53 under the AI-adjusted percentile, z = 1.645 under the published confidence floor, the ±20% band under cohorts — and the specification is complete, because there is no weight table left off it. The changelog at /changelog records every methodology change with a date, a numerical diff, and a written rationale, including the August 2026 corrections that withdrew the source-weight tables. Every detail page records the methodology version it was scored under, so that if we change a constant in the future, anyone can read a score against the specification that produced it.

The rules we have committed to

The hard cases — when a constant legitimately needs to change

Pre-commitment doesn't mean constants can never change. It means changes are public events with stated reasons. There are three cases where we expect to bump one over time:

• The corpus moves under the constant. K = 53 was set at the median calibrated-reviewer count across the ranked catalog, which is what makes the median item shrink exactly halfway. As the catalog grows that median moves, and at some point K stops being the number it was chosen to be. Re-deriving it means publishing the new median and the new K together, in the same entry.
• The published confidence level changes. z = 1.645 is 90% one-tailed. Moving to another confidence level changes a published number, so it gets a version bump, the new multiplier, and the argument for why the old one had stopped being informative — not a quiet edit to a formula.
• The data tells us a rule is wrong. If the ±20% price band is producing cohorts too thin to rank against in a category, or the n_u≥ 2 admission rule turns out to be admitting reviewers whose personal standard deviation is mostly noise, we'd publish the analysis and the new rule together. The analysis is the rationale.

The bar for changing a constant is: can we publish a paragraph in the changelog explaining why?If yes, we can change it. If no — if the only honest answer is "we wanted a different ranking" — we can't.

Why this matters more than it might seem

Most review aggregators don't publish their weighting at all. Yelp's ranking algorithm is a black box. Amazon's "helpful" review promotion is opaque. Google Reviews don't expose any weighting. The few aggregators that do publish weights — Metacritic publishes critic weights; Rotten Tomatoes weights publication clout — reserve the right to retune those weights without public notice.

The result is that for any of those services, you can't verify that a product's score wasn't engineered. You have to trust the editorial team. Trust is fine when it's well-founded; it scales poorly when it's the only defence. We have no weighting to publish because we apply none; what we publish instead is every constant that does touch a score. The structural alternative — published constants, version-bumped publicly, historical versions queryable — gives you a verification surface that doesn't depend on trusting us.

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

The published-constants layer

The constants themselves are in the open, on /methodology, inside the prose of the step that uses each one. They are not numbers we hold privately and apply behind the scenes: every value is on a public page that anyone can read, archive, or diff against the Wayback Machine. If we ever change a constant without a corresponding changelog entry, the archived copies of that page will show it — and would be cited against us by anyone watching.

That's the strongest version of the defence. Editorial pre-commitment plus publicly published constants plus a versioned changelog forms a three-way verification surface that even a determined operator would have a hard time spoofing without leaving evidence. It is a smaller surface than this article used to claim, in the useful direction: a weight table would be one more set of numbers you had to take on trust, and there isn't one.

Series: ← The small-sample illusion · Hub · Coming next: Survivorship bias — why we publish the full database, not just the winners