EBM 2.0 Interpretation Principles
Be Savagely Skeptical:
- Assume all positive study findings are false until rigorously proven otherwise through repeated, high quality independent replications.
Ask the right question:
- What is the overall probability that our hypothesis is true given everything we already know including what is likely to be hidden from us?
- Note this is not 1 minus the p value (a common and serious misconception).
Answer the right question:
- Use the best of EBM 1.0 principles to always closely review studies for any sources of visible bias and assume invisible bias is present.
- Realise that bias changes the very meaning of the p value calculation
- Therefore discount findings for the presence of bias and reject them entirely if there is substantial bias.
- Abandon “Statistical Significance”
- End the use of the dichotomous concept of “statistical significance” with arbitrary meaningless thresholds for p-values and confidence intervals
- Use the Bayes Factor approach
- Use p values as likelihood ratios (aka Bayes Factors) to convert pre-test probabilities (aka prior probability) of a finding being real into post-test probabilities (aka posterior probability), where the study is the “test” and the p value is viewed as the test result and accuracy.
- Explicitly “price-in” bias
- As post test probabilities assume zero bias, discount probabilities for visible bias and assumed invisible bias.
Accept & Manage Uncertainty:
- Accept uncertainty over the seductive lure of “positive” or”negative”.
- View evidence as merely incrementally changing levels of uncertainty.
- Use real world cost-benefits of therapy to determine acceptable level of certainty required before practice change.
- e.g. therapies with low net benefit and high costs need the highest levels of certainty to consider utilising – this can only be derived from supportive evidence that is repeatedly reproducible in high quality independent trials.