What can happen:
Decision | ||
---|---|---|
Truth | Do not reject | Reject null |
Null true | Correct | Type I error |
Null false | Type II error | Correct |
Tension between truth and decision about truth (imperfect).
[1] 14.487469 5.014611 6.924277 5.201860 8.852952 10.835874 3.686684
[8] 11.165242 8.016188 12.383518 1.378099 3.172503 13.074996 11.353573
[15] 5.015575
x
from population with mean 10 or not (over):
One Sample t-test
data: x
t = -1.8767, df = 14, p-value = 0.08157
alternative hypothesis: true mean is not equal to 10
95 percent confidence interval:
5.794735 10.280387
sample estimates:
mean of x
8.037561
tibble(sim = 1:1000) %>%
rowwise() %>%
mutate(my_sample = list(rnorm(15, 8, 4))) %>%
mutate(t_test = list(t.test(my_sample, mu = 10))) %>%
mutate(p_val = t_test$p.value) %>%
count(p_val <= 0.05)
We correctly rejected 422 times out of 1000, so the estimated power is 0.422.
tibble(sim = 1:1000) %>%
rowwise() %>%
mutate(my_sample = list(rnorm(40, 8, 4))) %>%
mutate(t_test = list(t.test(my_sample, mu = 10))) %>%
mutate(p_val = t_test$p.value) %>%
count(p_val <= 0.05)
Power is (much) larger with a bigger sample.
prop.test
.
1-sample proportions test with continuity correction
data: 422 out of 1000, null probability 0.5
X-squared = 24.025, df = 1, p-value = 9.509e-07
alternative hypothesis: true p is not equal to 0.5
95 percent confidence interval:
0.3912521 0.4533546
sample estimates:
p
0.422
95% CI for power: 0.391 to 0.453
Change 1000 to eg 10,000:
1-sample proportions test with continuity correction
data: 4353 out of 10000, null probability 0.5
X-squared = 167.18, df = 1, p-value < 2.2e-16
alternative hypothesis: true p is not equal to 0.5
95 percent confidence interval:
0.4255594 0.4450905
sample estimates:
p
0.4353
0.426 to 0.445, about factor \(\sqrt{10}\) shorter because number of simulations 10 times bigger.
power.t.test
. Input delta
is difference between null and true mean:Method | Power |
---|---|
Simulation (10000) | 0.4353 |
power.t.test |
0.4378 |
n=
, replaced by a power=
:
One-sample t test power calculation
n = 33.3672
delta = 2
sd = 4
sig.level = 0.05
power = 0.8
alternative = two.sided
n
as categorical so that colour
works properly.delta
is true difference in meanspower.t.test
, so take as 14.
Comments
mean=10
, that is, the true mean equals the null mean, \(H_0\) is actually true, and the probability of rejecting it then is \(\alpha = 0.05\).