FRM Exam Part I · Hypothesis Testing

p-Values, Type I and Type II Errors, and Power

Updated 11 October 2026 · Fact-checked

The p-value is the probability of seeing a test statistic at least as extreme as yours if the null hypothesis is true. You reject the null when p ≤ α. Type I error is rejecting a true null (probability α). Type II error is failing to reject a false null (probability β). Power = 1 − β.

Understand p-Values, Type I and Type II Errors, and Power

A hypothesis test starts with a null hypothesis (H0), such as "the mean daily return is zero". You collect a sample, compute a test statistic, and ask whether the data are too unusual to fit H0.

The p-value measures that. It is the probability, assuming H0 is true, of getting a result at least as extreme as the one you observed. A small p-value means the data would be rare under H0, so you doubt H0. It is not the probability that H0 is true. It is also not the probability that your result happened by chance in some general sense.

The significance level α is the cutoff you set before the test, often 1%, 5% or 10%. If p ≤ α, reject H0. Equivalently, reject when the test statistic falls in the rejection region. α is also the probability of a Type I error: rejecting H0 when it is actually true. The confidence level of the matching confidence interval is 1 − α. A two-sided test at 5% gives the same decision as checking whether the hypothesised value lies outside the 95% confidence interval.

A Type II error is failing to reject H0 when it is false. Its probability is β. The power of a test is 1 − β: the probability of correctly rejecting a false H0. Power depends on the true value, so it rises as the true parameter moves further from the null value.

There is a tradeoff. For a fixed sample size, lowering α (a stricter test) shrinks the rejection region, so β rises and power falls. To cut both errors at once, you need more data. Larger samples reduce the standard error and raise power. Power also rises with a larger true effect and lower data variability.

Key formulas to remember

Decision rule using p-value: Reject H0 if p-value ≤ α; Set α before looking at the data. A smaller p-value means stronger evidence against H0.
Type I error probability: P(reject H0 | H0 true) = α; Equals the significance level.
Type II error probability: β = P(fail to reject H0 | H0 false); Depends on the true parameter value, sample size and α.
Power of a test: Power = 1 − β; Probability of rejecting H0 when it is false.
Confidence level and significance: Confidence level = 1 − α; A two-sided test at α rejects H0 exactly when the hypothesised value lies outside the (1 − α) confidence interval.
Two-sided p-value for a z statistic: p = 2 × P(Z > |z|); For a one-sided test, use just one tail: p = P(Z > z) or P(Z < z).
Test statistic for a mean: t = (x̄ − μ0) ÷ (s ÷ √n); Use the z critical values when the population standard deviation is known or n is large.

How to solve p-Values, Type I and Type II Errors, and Power questions

Use this order for any question on p-values, errors or power.

1Write H0 and the alternative HA. Note whether the test is one-sided or two-sided.
2Identify α (or the confidence level, where α = 1 − confidence level).
3Compute the test statistic, or read the p-value given in the question.
4Compare: reject H0 if p ≤ α, or if the statistic lies beyond the critical value. For a two-sided test, compare p with α, not α ÷ 2, once p is already two-tailed.
5State the conclusion in words: reject or fail to reject H0. Never say "accept H0".
6If asked about errors, match the case: rejecting a true H0 is Type I (α); failing to reject a false H0 is Type II (β).
7If asked about power, use 1 − β. Then check the direction of any change: larger n, larger effect or larger α raises power.

Quickest way: Compare p with α, then name the error

When to use it: Use for conceptual multiple-choice questions where numbers are given or implied.

Find p and α. If p ≤ α, reject; otherwise do not reject.
Ask: is H0 actually true in the scenario? True and rejected = Type I. False and not rejected = Type II.
For power, remember: power = 1 − β, and it moves opposite to β.
For tradeoffs: lower α means higher β and lower power, with n fixed. Only a larger n improves both.
Eliminate any option that says "the probability H0 is true" or "accept H0".

Common mistakes in p-Values, Type I and Type II Errors, and Power

Saying the p-value is the probability that H0 is true.
The wording sounds like the probability of the hypothesis.
Fix: The p-value is calculated assuming H0 is true. It is the probability of data this extreme, not the probability of H0.
Swapping Type I and Type II errors.
The names are easy to confuse.
Fix: Type I = reject a true null (false positive, α). Type II = fail to reject a false null (false negative, β).
Writing "accept the null hypothesis".
It feels like the natural opposite of rejecting.
Fix: Say "fail to reject". A non-rejection means the evidence is insufficient, not that H0 is proven.
Thinking power equals α or 1 − α.
Several symbols look alike.
Fix: Power = 1 − β. The confidence level is 1 − α. They are different quantities.
Believing that lowering α improves the test with no cost.
Fewer false rejections sounds purely good.
Fix: With n fixed, lower α raises β and cuts power. Only more data reduces both errors.
Thinking a tiny p-value means a large or important effect.
Small p-values feel like strong results.
Fix: With a large sample, a trivial effect can give a tiny p-value. The p-value shows statistical evidence, not economic size.

Worked examples

Example 1

A risk analyst tests H0: mean daily P&L = 0 against HA: mean ≠ 0. The two-sided p-value is 0.034. Decide at α = 5% and at α = 1%. If H0 is in fact true, which error occurs where you reject?

Show the solution

Compare p = 0.034 with α = 0.05: 0.034 ≤ 0.05, so reject H0.
Compare p = 0.034 with α = 0.01: 0.034 > 0.01, so fail to reject H0.
If H0 is true and you reject it (the 5% case), you reject a true null. That is a Type I error.

Answer: Reject at 5%, fail to reject at 1%. Rejecting a true H0 is a Type I error.

Example 2

A test has a Type II error probability of β = 0.28 when the true mean is 0.5. The analyst keeps the sample size fixed and changes α from 5% to 1%. What is the power at α = 5%, and what happens to power after the change?

Show the solution

Power = 1 − β = 1 − 0.28 = 0.72, or 72%.
Lowering α from 5% to 1% shrinks the rejection region.
With n fixed, a smaller rejection region makes it harder to reject a false H0, so β rises.
Since power = 1 − β, power falls below 72%.

Answer: Power is 72% at α = 5%; it decreases when α is lowered to 1%.

Exam tips

Expect wording tests: check each option for "probability H0 is true" or "accept H0" and discard it.
Memorise the link: lower α means higher β and lower power for fixed n; only a larger sample helps both.
When a question gives a confidence level, convert it: α = 1 − confidence level (95% gives 5%).
For two-sided tests, confirm whether the p-value given is already two-tailed before comparing with α.
Power questions often ask what raises it: bigger sample, bigger true effect, higher α, or lower variance.

Practice questions from Hypothesis Testing

p-Values, Type I and Type II Errors, and Power: frequently asked questions

What is the difference between Type I and Type II error?

A Type I error rejects a null hypothesis that is true, and its probability is α. A Type II error fails to reject a null hypothesis that is false, and its probability is β. Power is 1 − β.

How do I interpret a p-value in hypothesis testing?

It is the probability of getting a test statistic at least as extreme as yours if H0 is true. If it is at or below your significance level, you reject H0. It does not give the probability that H0 is true.

What is the relationship between significance level and confidence level?

Confidence level = 1 − α. A 5% significance level pairs with a 95% confidence interval. For a two-sided test, you reject H0 at 5% exactly when the hypothesised value falls outside the 95% interval.

How can I increase the power of a test?

Increase the sample size, accept a higher α, or test where the true effect is larger or the data less noisy. Increasing the sample size is the only way to lower both Type I and Type II error risk together.