Cohen D Calculator: Effect Size and CI

This will allow you to interpret statistics propery.

For instance, you may read that some new teaching method increased scores by 15 points. Sounds good! But then you notice it was a test out of 100, and the control group also increased their score by 10 points. The raw difference here is five. Feels like nothing. But when variation across classes are small, those five points might be a huge jump in performance.

How to Use the Calculator Correctly

Cohen’s d standardizes the difference. It allows you to compare a reading test to a math test. Did a change occur? How large of a change was that in comparison to all the other stuff going on?

Then you input your summary stats into the calculator and it does the work for you. It removes all the pesky calculation mistakes from doing it by hand.

First thing, pick what type of design you have. Two different groups? Then you have two sets of means and standard deviations (e.g., one group on a medication vs. One group is a control group on a placebo). Then it will pool those variances together as a reference baseline. It’s assuming that the variation between groups are about the same. In many cases, this won’t be true and so you’ll want to re-think your design. In general though, pooling the standard deviation produce a strong estimate of spread for practical purposes.

The math changes a bit when you’re doing one-sample comparisons. In this case, you’re testing whether a particular group differs from some known benchmark (e.g., a history target, or the national average). Your denominator is now just the sample’s standard deviation. That’s a subtle but critical difference. Using an incorrect variance estimator will either magnify or shrink size of your effect, which results in overconfidence.

The calculator automatically corrects for degrees of freedom, so that the confidence intervals takes into account the true number of samples and not some idealized population.

The paired design has its oddities too. If you’re measuring the same people before and after an intervention, what you want to know is how much they changed. So you need the standard deviation of the differences. Do not use standard deviations from the separate pre and post measures. Many people make this mistake. As a consequence, you’ll typically underestimate the effect size if you compute it using the wrong SD. It neglects to take into account that first and second measurements is correlated.

In this mode the tool ask you for the SD of the differences. This keeps you on the right track.

But what do we do with that d value? How do we interpret it? There’s no hard-and-fast rule here. Instead, consider these as guidelines: Small = 0.2, Medium = 0.5, Large = 0.8. That means that a d value of 0.5 in one study would count as “medium” but a d value of 0.05 would be a tiny effect.

Context is everything. Saving the lives of thousands of people could be a tiny effect in a massive public health initiative. But finding a cure for a rare disease in a lab setting could be completely meaningless. The reference table on the page lays this out. And then use your own discretion… Determine whether a medium effect justifies a change in opinion or not.

It even computes something called Hedges’ g. Hedges’ g is a correction factor for reducing bias in small samples. When sample sizes gets really tiny, Cohen’s d tends to exaggerate. Hedges’ g pulls it back down to earth. You’ll notice the difference shrinks a bit. That’s a good thing. Better to be careful, rather then to say there’s a miracle when there is only a small improvement.

The confidence interval and standard error provide a range of plausible values. If that range contain zero, then your result isn’t statistically significant. This remains true, regardless of how large the point estimate appears.

But always examine the assumptions. In most cases that means checking normality and homogeneity of variance.

For well behaved data though, Cohen’s d is still the best measure when you’re trying to communicate impact. It makes you consider size, not just importance. A p-value tells you there is an effect. A d value tells you whether or not it matters.

Leave the arithmetic to the software; let them worry about telling the story with the numbers. Only half the battle is knowing the difference. Half the battle is knowing if that difference is big enough to care about, and you should of checked earlier.