In a past interview for research funding (a Fellowship award), I boldly stated

“Statistics makes the research-world go round.”

Whilst my eagerness to impress the interviewers is now embarrassingly obvious to me, I don’t think the statement is all that far from the truth. Here’s one case in point: it is well established that confidence intervals (CIs) communicate more useful information than P values in hypothesis testing.

What does a confidence interval communicate? It indicates the lack of precision (uncertainty) in your estimate of interest. Significantly, it uses the same units as your estimate of interest too. So for example, if you specify a mean value of a random sample of a population, a 95% CI of the mean indicates how confident we can be that the true population mean lies within the calculated confidence interval. Additonally, by stating a CI, the P value can be roughly determined even if it is not explicitly stated. For example, if your 95% CI does not include zero (or if not zero, the value stated in the null hypothesis), then your P value will be less than 0.05. Similarly if your 99% CI doesn’t include zero, P < 0.01.

P values alone can be more easily misinterpreted.

To say that P describes the probability that an observed effect is not a real one is mistaken. (It’s certainly real—you observed it!) An accurate statement is that a P value indicates the probability that such data would have arisen by chance if the null hypothesis is true. So a P value, interpreted correctly, can tell us about the strength of evidence against the null hypothesis. If the P value is low, the evidence against the null hypothesis is strong. If the P value is higher (above our arbitrary threshold of say, 0.05), the evidence against the null hypothesis is poor, and we fail to reject it.

So let’s say we have P < 0.05, from which we decide to reject the null hypothesis (we think the effect which we observed is likely to be present in the whole population). Do we have any information about the size of the effect which we measured? Is it likely to represent the true size of the effect in the whole population? These data are communicated by a CI, and are important in determining the clinical significance of effects which we observe which we have decided are likely to be real ones.

How do we calculate a confidence interval for a mean? We calculate the confidence interval boundaries: the mean minus a calculation of uncertainty, to the mean plus that calculation of uncertainty:

s.e.m stands for the standard error of the mean, SD is the standard deviation of the sample, and n is the sample size. The standard error of the mean is is the value we would expect if we took several samples, and computed all their means, and then took the standard deviation of those means. The s.e.m is multiplied by the so-called critical value of t. t is found in look-up tables (reach for your closest stats book) and depends on the sample size minus one (so-called degrees of freedom), and on the CI you want to calculate (e.g. 95% CI with P < 0.05). It is necessary to multiply the s.e.m by t because the maximum possible distance (in standard errors) between the sample mean and the true population mean depends on these quantities. Usually, the t look-up tables which are used correspond with two-tailed probability P (when the real value you want to estimate can go in either direction—up or down).

For example:
mean = 122
SD = 9
n = 30
degrees of freedom = 30-1 = 29
critical t (look-up, two-tailed) = 2.045 (for 95% CI, P < 0.05)
s.e.m = 9/√30 = 1.64

95% CI = 122-(2.045×1.64) to 122+(2.045×1.64)
95% CI = 119 to 125.

We can be 95% confident that the true population mean lies between 119 and 125.

Confidence intervals can be calculated for many different statistical quantities (proportions, difference between two means or two proportions, relative risks, odds ratios, et cetera), but the method of calculation varies.

Btw—I got that Fellowship!

Post a Comment | Trackback URI