Questionnaire Habits May Be Driving Apparent Metacognitive Bias in Mental Health
Large online studies have been treating links between psychiatric traits and decision confidence as if they reflected altered metacognition. This PNAS analysis argues for a far less glamorous explanation: some of those correlations may be built on how people answer questionnaires, not on how they think about their own thoughts.
The central concern is straightforward. If a person has a habitual tendency to choose extreme positive or negative responses on self-report scales, that style can spill over into confidence ratings. If another respondent is inattentive and answers carelessly, the same pattern can inflate rare symptom endorsements and create a false impression of high symptom burden paired with high confidence. In both cases, the resulting statistics can mimic a psychiatric signal even when the underlying signal is mainly behavioral noise.
Rating Style Can Travel Across Very Different Question Types
The analysis drew on existing and newly collected large-scale datasets from healthy adults and examined whether previously described associations between transdiagnostic psychiatric dimensions and metacognitive measures would survive closer scrutiny. The answer was mixed in a way that matters clinically and methodologically. Two dimensions that had been framed as meaningful mental health constructs — “anxiety-depression” and “compulsivity and intrusive thought” — lined up neatly with two surface-level questionnaire behaviors: systematic rating bias and inattentive responding.
This is not a trivial statistical nuisance. In self-report research, confidence ratings are often treated as a window into how people evaluate their own decisions or knowledge. Yet those ratings are still answers on a scale, and scale behavior can be shaped by personality-like response habits, fatigue, disengagement, or a preference for extreme options. Once those habits are shared across separate measures, correlations become easy to manufacture.
A respondent who prefers higher numbers may look more confident regardless of actual task performance. Another who clicks through with little attention may endorse unusual symptoms at an elevated rate, creating a spurious association between rare symptom reporting and confidence. In a dataset large enough, those artifacts can look clean, stable, and theoretically exciting.
The illusion is powerful because it fits existing narratives about psychiatric cognition.
Confidence Ratings Were Correlated With Independent Response Bias Measures
The strongest counterweight in the paper comes from the preregistered experiment. There, decision confidence and self-reported obsessive-compulsive tendencies were both correlated with independent measures of inattentive and biased responding. The design matters because it moved beyond retrospective pattern matching and tested whether the apparent associations could be traced to separate behavioral tendencies that operate outside any true metacognitive effect.
The result was not a claim that psychiatric symptoms never relate to confidence or self-evaluation. It was narrower and more unsettling: once questionnaire-filling behavior is measured directly, it can account for a substantial share of the observed relationship. In practical terms, what looked like altered metacognition may instead reflect a combination of response style and inattention. For mental health research that relies heavily on self-report, that distinction changes how findings should be interpreted.
Why This Matters for Transdiagnostic Psychiatry Research
Transdiagnostic models aim to identify dimensions that cut across traditional diagnostic categories, and they have become especially attractive in studies of anxiety, depression, compulsivity, and intrusive thought. Confidence ratings have been folded into that framework because they seem to probe a deeper cognitive layer than symptom checklists alone. If confidence and symptom reports are both influenced by the same noisy response habits, the transdiagnostic structure may be less about shared mental processes and more about shared measurement artifacts.
That possibility is clinically important because metacognitive bias has been used to support theories of how patients monitor errors, evaluate certainty, and regulate behavior. If the observed effect is partly or largely driven by biased responding, then downstream theories about self-monitoring in mental health need to be rebuilt on cleaner ground. The paper does not argue that metacognition is irrelevant; it argues that the measurement strategy must be tougher before any mechanistic claim is trusted.
The challenge is especially acute in large online samples, where scale completion happens outside the controlled setting of a lab or clinic. When attention is variable and incentives for careful responding are limited, the same participant may display both a generalized positivity or negativity bias and a tendency to answer inconsistently or hastily. Those patterns can be statistically elegant and biologically empty at the same time.
Concrete Safeguards for Future Studies
The paper calls for precautionary measures to control for these biases, and the logic behind that call is compelling. Researchers studying confidence, self-monitoring, or transdiagnostic traits should treat inattentive responding and rating bias as core variables rather than nuisance covariates. Embedded attention checks, response-consistency metrics, and explicit modeling of scale-use tendencies can help separate true cognitive effects from questionnaire style.
A cleaner approach would also distinguish between symptom endorsement and response disposition more aggressively in study design. If rare symptom items are especially vulnerable to careless clicking, then any association between those items and confidence should be viewed with caution unless independent response-quality measures are included. The same holds for studies that interpret extreme confidence ratings as evidence of altered insight or impaired error monitoring.
This paper does not close the case on metacognition in mental health. It does something more useful: it shows how easily the case can be contaminated. Once biased and inattentive responding are measured directly, the apparent association between psychiatric traits and confidence may shrink into something far less dramatic than a cognitive abnormality — and far more consistent with how people complete questionnaires when attention wanders or response habits dominate.