scientific-critical-thinking

Assess study design appropriateness, internal validity (randomization, confounding control, attrition), external validity (sample representativeness), and measurement quality. The skill provides structured checklists covering each validity dimension with specific questions to ask.

GRADE rates evidence as high, moderate, low, or very low quality. RCTs start high and observational studies start low, then ratings are downgraded for risk of bias, inconsistency, indirectness, imprecision, or publication bias, and upgraded for large effects or dose-response relationships.

Check for preregistration, compare published outcomes to registered protocols, look for suspicious p-value clustering just below .05, and verify whether hypotheses were stated a priori. Undisclosed multiple analyses and post hoc subgroup claims are key warning signs.

Cochrane Risk of Bias assesses individual study quality across domains like randomization and blinding. GRADE evaluates the overall body of evidence for a question, incorporating risk of bias plus consistency, precision, and directness across studies.

Observational studies can support causation when effects are large, show dose-response relationships, replicate consistently across populations and methods, have biological plausibility, and lack plausible confounders. Frameworks like Bradford Hill criteria guide this judgment.