Publication Bias in Anxiety Research: What We Don't See Shapes What We Believe

Imagine reading only the race results of runners who finished in the top half. You would conclude that the average pace was fast and that most people who enter a race run well. You would be wrong, because you never saw the slower finishers. This is, roughly, what happens in anxiety research when negative trials go unpublished.

What publication bias is

Publication bias is the selective publication of research based on the nature of the results. Studies with positive, statistically significant findings are more likely to be submitted by authors, accepted by journals and cited by other researchers. Studies that find no effect, or a negative effect, are more likely to sit in file drawers, be rejected by reviewers, or be published in obscure journals years later.

The result is a published literature that systematically overstates treatment effects.

The scale of the problem in medication trials

The most striking demonstration came from Turner and colleagues, who compared what pharmaceutical companies submitted to the FDA for antidepressant approval with what was eventually published [1]. Of 74 trials submitted to the FDA, the agency considered 38 positive and 36 negative or questionable. Of the positive trials, 37 were published, and all but one reported positive results. Of the negative trials, 22 were not published at all. Of the 14 that were published, 11 were written in ways that presented the results as positive.

The consequence: the published literature showed 94% positive trials, while the actual rate was 51%. The published effect size was 0.37 (a moderate effect); the true effect size across all trials was 0.15 (a small effect). The drugs worked, but the published literature made them look roughly twice as effective as they actually were.

A similar analysis specific to anxiety pharmacotherapy found evidence of publication bias in meta-analyses of several drug classes, with funnel plot asymmetry and inflated effect sizes in smaller studies [6].

The problem in psychotherapy trials

Publication bias in psychotherapy research is harder to quantify because there is no equivalent of the FDA that receives all conducted trials. But indirect evidence suggests it exists at a similar scale.

An analysis of NIH-funded psychotherapy trials for depression found that, as with medication trials, published results were more favorable than the full set of completed trials [2]. A meta-analysis of CBT for depression that assessed publication bias found that correcting for it reduced the pooled effect size substantially [3].

For psychotherapy specifically, additional forms of bias operate:

• Researcher allegiance. Researchers who developed a therapy tend to find larger effects for it. This may reflect genuine expertise or subtle biases in study design and execution.
• Control group selection. Choosing a weak comparator (like a waitlist) inflates apparent effects.
• Outcome selection. When a study measures multiple outcomes, the primary outcome reported may shift post hoc to whichever showed the best result.

Trial registration: a partial solution

Since 2005, the International Committee of Medical Journal Editors has required that clinical trials be registered before enrolment begins for publication in major journals. ClinicalTrials.gov serves as the primary registry. Registration records the study design, primary outcome and planned analyses before results are known, making it harder to suppress entire studies or switch outcomes.

But compliance is incomplete. A study comparing registered and published trials found that about a third of published trials had discrepancies between their registered and published primary outcomes [4]. Some changed their primary outcome. Some added outcomes that were not pre-specified. Some dropped pre-specified outcomes that showed no effect.

Registration also does not solve the problem for the many smaller psychotherapy trials that are not registered, or for the gray area of pilot studies, case series and unregistered studies that feed into narrative reviews and clinical guidelines.

How to detect publication bias in a meta-analysis

Several tools exist, none perfect:

Funnel plots. A graph of effect size against study size. In the absence of bias, small studies scatter symmetrically around the pooled estimate. Asymmetry — with small positive studies present but small negative studies missing — suggests bias. However, funnel plot asymmetry can have other explanations, including genuine heterogeneity [5].

Egger’s test. A statistical formalization of funnel plot asymmetry. A significant result suggests bias, but the test has low power when there are few studies.

Trim-and-fill. Estimates the number of “missing” studies needed to make the funnel plot symmetric, imputes them, and recalculates the pooled effect. This provides an adjusted estimate but assumes a specific model of the missing data.

p-curve and p-uniform. Newer methods that examine the distribution of p-values to assess whether the published results contain genuine signal or are artifacts of selective reporting.

What to do with this information

Publication bias does not mean treatments don’t work. It means the published effect sizes are likely overestimates. Practical implications:

1. When a meta-analysis reports an effect size, check whether publication bias was assessed. If not, mentally discount the estimate.
2. A “small” published effect may be a “very small” true effect.
3. Treatments supported only by small studies from enthusiastic research groups should be viewed with particular caution.
4. Large, pre-registered trials provide more reliable estimates than the pooled results of many small trials.

For how to interpret the effect sizes that remain after bias correction, see understanding effect sizes. For how the placebo response further complicates interpretation, see the placebo response in anxiety trials.

When to look for more support. Publication bias is a problem for researchers and guideline writers, not for patients. Treatments recommended in guidelines like NICE or APA have been evaluated with these biases in mind. If anxiety is affecting your life, talk to a GP. If you are in crisis, call or text 988 (US) or Samaritans on 116 123 (UK and Ireland).