How we check our numbers,
and what we found wrong.

Before publishing any claim, we audited twelve of our own effect sizes against their primary sources. None of the twelve was publishable as stored. This page explains what happened, what the rules are now, and what is still unresolved.

Last updated 2026-07-30.

The bug that mattered

Our evidence base stores an effect size and a metric name. A function turned that pair into a human-readable label. It tested whether the metric name began with relative_risk before testing whether it described a reduction.

Every metric named relative_risk_reduction… also begins with relative_risk. So a stored 15% relative risk reduction was rendered as “RR 0.15” — a risk ratio of 0.15, which reads as an 85% reduction. Olive oil’s stored 12% reduction rendered as an 88% mortality reduction.

A validator stack had reviewed these claims thousands of times and did not catch it, because there was nothing wrong with the data. This is why we do not cite our own review counts as evidence of quality.

Four kinds of error, four different fixes

The audit found that “wrong number” is not one problem. A numeric sanity check catches only the first of these. The fourth, which we found last, is caught by nothing but reading the source.

1. A percentage stored where a ratio was expected

A 15% reduction and a risk ratio of 0.15 are both plausible-looking numbers. Only the metric name distinguishes them. Caught by: flagging any ratio outside 0.5–2.0.

2. Direction inverted

Grip strength was stored as a hazard ratio of 1.16 attached to a node named “top quartile”. The paper reports 1.16 per 5 kg lower grip. As stored, it read as strong grip increases mortality. Invisible to any range check, because 1.16 is a perfectly ordinary hazard ratio. Caught by: comparing the direction of the ratio against whether the exposure is protective or harmful.

3. The wrong comparison

The number is right and what it is measured against is wrong. Gait speed was stored as a threshold (“>1.2 m/s”) when the paper reports a continuous per-0.1-m/s increment. VO2max was stored as “top quartile” when the paper uses percentile bands. This survives every numeric check and can only be caught by reading the source. Caught by: refusing to publish any effect size with no recorded contrast.

4. The number is not in the paper it cites

The worst category, and the one we found last. When we went looking for missing sample sizes, six of nine claims turned out to cite a paper that does not contain the stored figure. One cited a narrative review that reports no estimate at all. One cited a 1953 study that published crude rates and no relative risk. One was a Bayesian shrunken posterior — a legitimate internal quantity — presented as though the paper had printed it. And one matched nothing in the literature anywhere: the closest real finding measured a different exposure entirely. That one was retired, because there was no number to correct it to.

No automated check finds this. The only method is opening the paper and searching it for the number.

The rules now

1. Every figure is read at its primary source. Not from a review, not from a press release, not from another site’s summary. Where a paywall blocks the full text, we use the abstract and say so.
2. No claim publishes without a confidence interval and a sample size. Two claims in our corpus currently lack one or both. They are excluded rather than published with the gap hidden.
3. If the source reports no interval, we publish none. Doll’s 1951–2001 doctors study gives life-expectancy gains with no intervals. Our corpus had stored “8–11 years”, which appears nowhere in the paper. It was deleted, not softened.
4. The population is named, including when it is narrow. Our sauna figure comes from 201 middle-aged Finnish men. Our religious attendance figure is women only. Our VO2max figure is a clinical referral population, not the public. Each says so.
5. The comparison is stated. “Sauna 4–7×/week” is meaningless alone; the comparator is 1×/week, because that cohort has no non-sauna group.
6. Descriptive statistics are not effect sizes and are labelled so they can never be rendered as one.
7. Corrections are printed, not quietly applied. Each claim shows what we had wrong.

What is still unresolved

Publishing this list is part of the method. A methodology page with no open problems is not describing real work.

• Nine claims were sent for sample-size verification, and six came back citing a paper that does not contain the stored figure. One — a walkability and cognitive-decline figure — matched nothing in the literature at all, and was retired rather than corrected because there was no number to correct it to. Two remain without a sample size and stay excluded: both are ecological designs where no individual-participant sample exists to record.
• Three exposure–outcome pairs hold multiple estimates measuring genuinely different contrasts — per-standard-deviation versus quintile for sleep regularity, three separate cohorts for social connection. Choosing one per claim is an editorial judgement we have not made yet.
• One metric is ambiguous at source. A peer-support claim is recorded as “hazard ratio or odds ratio”. It cannot be both, and a reader cannot interpret it without knowing which, so it is flagged for re-extraction rather than guessed.
• Not every effect size in the corpus has been individually audited. Twenty-one were — twelve proposed for publication, then nine more sent for sample-size verification. Six of that second nine turned out to cite a paper that does not contain the stored number. We have no basis for assuming the unaudited remainder is cleaner, which is why this page lists what has been checked rather than claiming the corpus is clean.

Why this page exists

Two reasons, and the second is the honest one about incentives.

First, an effect size without its interval, sample size and population cannot be evaluated. Most health content strips exactly those, which is how a 19% risk reduction becomes “olive oil slashes mortality”. If we publish numbers, they have to arrive with the means to check them.

Second, we found these errors in our own work and it is cheaper to say so than to be caught. The corpus was not sloppy — the data was right and a display function was wrong. But the number a reader sees is the only number that exists, so a display bug is a data error with extra steps.

← See the published claim set

We apply this same standard when we assess a real community. See whether yours is a match — for operators of communities that already exist.