EBM 2.0 Interpretation Principles

Be Savagely Skeptical:

• Assume all positive study findings are false until rigorously proven otherwise through repeated, high quality independent replications.

Ask the right question:

• What is the overall probability that our hypothesis is true given everything we already know including what is likely to be hidden from us?
• Note this is not 1 minus the p value (a common and serious misconception).

Answer the right question:

– Bias

• Use the best of EBM 1.0 principles to always closely review studies for any sources of visible bias and assume invisible bias is present. 
• Realise that bias changes the very meaning of the p value calculation
  • Therefore discount findings for the presence of bias and reject them entirely if there is substantial bias.

– Chance:

• Abandon EBM 1.0 statistical measures
  • End the use of the dichotomous concept of “statistical significance” with arbitrary meaningless thresholds for p-values and confidence intervals
• Use Bayesian Analysis 
  • Use p values as likelihood ratios (aka Bayes Factors) to convert pre-test probabilities (aka prior probability) of a finding being real into post-test probabilities (aka posterior probability), where the study is the “test” and the p value is viewed as the “accuracy” of the test.
    • See this post for how to calculate
    • Be deliberately conservative when choosing baseline pre-test probabilities. See this post for guidance on choosing a baseline pre-test probability that a clinical hypothesis is true.
• Explicitly “price-in” bias
  • As post test probabilities assume zero bias, discount probabilities for visible bias and assumed invisible bias.

Accept & Manage Uncertainty:

• Accept uncertainty over the seductive lure of “positive” or”negative”.
• View evidence as merely incrementally changing levels of uncertainty.
• Use real world cost-benefits of therapy to determine acceptable level of certainty required before practice change.
  • e.g. therapies with low net benefit and high costs need the highest levels of certainty to consider utilising – this can only be derived from supportive evidence that is repeatedly reproducible in high quality independent trials.