How to convert p values into something useful
This is a supplementary post to the talk Evidence Based Fraud & The End of Statistical Significance – you may wish to start there, if you haven’t already.
There it was revealed that the common belief that p values (specifically 1 minus the p value) tell us the likelihood that a study finding is true was in fact a serious and dangerous misunderstanding. Instead it was revealed that p values are best conceptualised as “likelihood ratios” or “Bayes Factors” that convert pre-test probabilities (Pre-TP; aka prior probability) that a studied hypothesis is true into post-test probabilities (Post-TP; aka posterior probability) where the study in question is effectively the “test” and the p value attempts to represent the accuracy of this “test”. So the key to determining the meaning of a p value for a given finding, is first attempting to estimate what you believe the pre-test probability was before the study, based on biological plausibility and prior research.
In the landmark article by Nuzzo (Nature 2014), this was beautifully represented in the following infographic.
Benjamin & Berger (American Statistician 2019) have provided recent guidance in converting p values into Bayes Factors (essentially likelihood ratios) that allow clinicians to calculate Post-TP from Pre-TP, as demonstrated in the above infographic.
Of note, under this methodology, they are referred to as Bayes Factor Bounds which means they are “maximum” possible Bayes Factors for a given p value, so should be considered the very “best case scenario” for Post-TP. In addition, if there is any bias in the study, this affects the validity of the p value calculation so any Post-TP calculated using these flawed p values are likely to be substantial over-estimates. So use this process as an optimistic guide only and heavily discount your figures for visible and presumed invisible bias, or reject the p values entirely as inapplicable if the bias is substantial.
Finally, never rely on the p values from a single study. An iron clad rule of EBM (and scientific enquiry more generally) is the need for repeated independent reproducibility of the data before you believe it, even if the Post-TP calculated from a single study’s p-value is appealing.
Bayesian Methodology Using P Values:
1. Estimate a pre-test probability that the hypothesis is true
- Base this on biological/logical plausibility and prior research.
2. Convert the relevant study finding p value to a Bayes Factor (likelihood ratio). One prominent methodology uses the calculation, Bayes Factor = 1/(-ep ln(p)). This provides the following Bayes Factors in this table:
3. Use Bayes Factors to convert the hypothesis Pre-TP into Post-TP
- Bayes Factors can be multiplied by pre-test odds to provide post-test odds. However probabilities are somewhat more intuitive and familiar to clinicians than odds, and converting Pre-TP into odds, applying Bayes Factor to reveal post test odds and then converting back to Post-TP adds unnecessary intellectual hassle when online calculators already exist that more simply calculate Post-TP from Pre-TP using likelihood ratios (like this one). Using these Bayes Factors and entering them into the calculators as likelihood ratios, reveals the following estimates of the absolute maximum Post-TP given a Pre-TP and p-value.
Questions, comments and collaboration are welcome via comments below or our Contact page.
[This post was first published on EDguidelines.com 13/5/20 and transferred to this new EBM 2.0 project site on 8/10/20]