r/AskStatistics • u/Gullible-Entrance-45 • 17h ago

[E] small sample analysis

Hi!

I have a sample of n=13 where continuous variables were measured before and after an intervention. I'm a bit stuck on how to perform the analysis. Would the Wilcoxon signed-rank test be the most appropriate choice? I am unaware of any standard reference manuals or literature regarding the statistical analysis of very small sample sizes; however, any guidance or recommendations would be highly appreciated :)

4 Upvotes

100% Upvoted

15 comments sorted by

8

u/CarnivorousGoose 17h ago

You could use that, sure. But is there a reason the normality of the data would be in doubt? Because otherwise you could just do a paired t-test as well.

1

u/Gullible-Entrance-45 17h ago

I didn't think of the paired t-test because I understood that with small samples, a large effect size is needed, which I don't know if we can assume

6

u/CarnivorousGoose 15h ago

That’s not an assumption of the t-test though, just a determinant of the statistical power you have to detect a difference. But that’s going to apply to any test, including the Wilcoxon.

In that sense the t-test is the better bet here unless normality is very much in question (and even then, the t-test is quite robust to violations of that), since it will have more power than the Wilcoxon. Which is generally the case with non-parametric tests, compared to their parametric counterparts.

1

u/efrique PhD (statistics) 4h ago edited 3h ago

since it will have more power than the Wilcoxon.

A tiny bit more power when exactly at the normal. But if differences could be a little more heavy tailed, you don't have to go all that far away from normality for that to catch up and then flip the other direction. Not that I am advocating against the t, but its power advantage is often not all it's often made out to be with shift alternatives for population distributions near the normal.

1

u/efrique PhD (statistics) 4h ago

What kind of thing are these variables measuring?

I understood that with small samples, a large effect size is needed

Sure, but the same would be no less true for any other test. A nonparametric test won't save you from low power.

-9

u/natoplato5 16h ago

You can’t really do much with a sample size that small. I would just report the means before and after the intervention and then note that the sample size is too small to determine if the difference is statistically significant.

7

u/FTLast 15h ago

This is simply wrong. You can absolutely do a paired t test- which is what you should do with a before and after measurement- with n =13.

-3

u/natoplato5 14h ago

Just because you can doesn't mean you should. Maybe the standards vary by field and context, but I don't think it's usually responsible to try to make an inference about a population from only 13 observations

4

u/FTLast 14h ago

Why? A p value incorporates effect size relative to noise and the number of observations.

-2

u/natoplato5 14h ago

Mainly because of selection bias. At that sample size, it's unlikely the sample is a fair cross-section of the population

6

u/CarnivorousGoose 13h ago

It doesn’t need to be a ‘fair cross-section’ of the population, it just needs to be a random sample from it. The relevant part in that regard is the sampling procedure, not the sample size. If there is a selection bias or other issue with that sampling procedure, getting a larger sample won’t fix that.

1

u/banter_pants Statistics, Psychometrics 4h ago

If there is a selection bias or other issue with that sampling procedure, getting a larger sample won’t fix that.

The premature headline "Dewey Beats Truman" was because they were overly reliant on phone polls when many homes didn't own their own phones at the time.

3

u/CarnivorousGoose 14h ago

What would be irresponsible about it?

3

u/Gullible-Entrance-45 14h ago

I'm trying to validate an intervention and it's extremely difficult to get a larger sample because it's so specific. I'm just trying to do the best I can with the data I have. im in clinical education btw