Actuarial Statistics · Hypothesis testing and goodness of fit

Non-Parametric and Permutation Tests Explained

Updated 11 October 2026 · Fact-checked

Non-parametric tests use signs or ranks instead of raw values, so they need no normality assumption. Choose the sign or Wilcoxon signed-rank test for one sample or pairs, Mann-Whitney for two independent samples, and Spearman or Kendall for association. Rank the data, find the statistic, then compare it with tables or the null distribution.

Understand Non-Parametric and Permutation Tests

A parametric test assumes the data come from a named family, usually the normal. The t-test is an example. A non-parametric test (also called distribution-free) makes much weaker assumptions. It usually works on signs or ranks, not on the actual values.

You use a non-parametric test when the sample is small and clearly not normal, when the data are only ordinal (ranks or grades), or when outliers would distort a mean. The price is some loss of power when the data really are normal. If normality holds, the t-test is more powerful.

The sign test looks only at whether each observation is above or below a hypothesised median. The Wilcoxon signed-rank test also uses the size of each difference through its rank, so it is more powerful. It assumes the differences are roughly symmetric about the median. The Mann-Whitney test (the same test as the Wilcoxon rank-sum test) compares two independent samples by ranking all observations together.

Spearman's rank correlation and Kendall's tau measure how strongly two variables move together in order. They detect monotonic relationships, not only straight lines, and are not distorted by outliers in the way Pearson's coefficient is.

A permutation test builds the null distribution from the data itself. You list all the ways the group labels could be reassigned, compute the statistic each time, and find the proportion at least as extreme as the observed value. That proportion is the p-value. The rank tests above are permutation tests applied to ranks.

Key rules to remember

Sign test statistic: S = number of positive differences; under H0, S ~ Binomial(n, ½); Drop zero differences first and reduce n. For a test of a median m0, take differences x − m0. Use the binomial table or tail probabilities.
Wilcoxon signed-rank statistic: T+ = sum of ranks of positive differences; T− = sum of ranks of negative differences; T+ + T− = n(n + 1) ÷ 2; Rank the absolute differences, giving tied values the average rank. Drop zeros. Tables usually use the smaller of T+ and T−, and you reject if it is at or below the critical value.
Normal approximation for signed-rank: E(T) = n(n + 1) ÷ 4; Var(T) = n(n + 1)(2n + 1) ÷ 24; Z = (T − E(T)) ÷ √Var(T); Use for larger n, with a continuity correction of 0.5. Ties need an adjusted variance.
Mann-Whitney U: U_X = R_X − m(m + 1) ÷ 2; U_X + U_Y = mn; X has size m, Y has size n, and R_X is the sum of X's ranks in the combined ranking. Check with U_X + U_Y = mn. Some books use W = R_X (rank-sum form) instead of U; the tests are equivalent.
Mann-Whitney normal approximation: E(U) = mn ÷ 2; Var(U) = mn(m + n + 1) ÷ 12; For the rank sum W = R_X: E(W) = m(m + n + 1) ÷ 2 and the variance is the same as for U.
Spearman rank correlation: r_s = 1 − 6Σd² ÷ (n(n² − 1)); d is the difference between the two ranks of each item. This shortcut holds only when there are no ties. With ties, compute Pearson's correlation on the ranks.
Kendall's tau: τ = (C − D) ÷ (n(n − 1) ÷ 2); C is the number of concordant pairs and D the number of discordant pairs, with no ties. Values lie between −1 and +1. Under H0 of independence, Var(τ) = 2(2n + 5) ÷ (9n(n − 1)) for the normal approximation.
Permutation test p-value: p = (number of permutations with statistic at least as extreme as observed) ÷ (total number of permutations); For a two-sided test, count extreme values in both tails. If you sample random permutations, the p-value is only an estimate.

How to solve Non-Parametric and Permutation Tests questions

Use this order for any non-parametric question. It keeps the working clear and protects method marks.

1Identify the design: one sample or paired data, two independent samples, or a measure of association. This tells you which test applies.
2State H0 and H1 in words and symbols (for example, median difference = 0 against ≠ 0). Say whether the test is one-sided or two-sided.
3Compute the differences where needed. Remove zeros and reduce n accordingly.
4Rank carefully. Rank absolute differences for signed-rank, and all observations combined for Mann-Whitney. Use average ranks for ties. Check your ranks add up to n(n + 1) ÷ 2.
5Calculate the statistic (S, T, U or W, r_s, τ). Use the check relationships, such as T+ + T− = n(n + 1) ÷ 2 or U_X + U_Y = mn.
6Compare with the exact table or binomial probability for small samples. For large samples, standardise and use the normal approximation with a continuity correction.
7Conclude in context. State whether you reject H0 at the stated level, and state any assumptions, such as symmetry or independence.

Quickest way: Rank, check, compare

When to use it: Use this when time is short, especially in multiple-choice questions and small-sample written questions.

Pick the test from the data layout: pairs go to sign or signed-rank, two groups go to Mann-Whitney, two variables go to Spearman or Kendall.
Write the data in a small table with a rank column, and sort first so ranking is easy.
Use the check sums (n(n + 1) ÷ 2 for signed-rank, mn for U) to catch ranking errors before going further.
Take the smaller of the two rank sums or U values when the table asks for it. Do not mix a one-sided and a two-sided critical value.
For Spearman with few items, d = rank difference, so Σd² is quick to find. Sanity-check the sign of r_s against the data pattern.

Common mistakes in Non-Parametric and Permutation Tests

Keeping zero differences in the sign or signed-rank test.
Students treat a zero as a normal observation and rank it.
Fix: Discard zero differences and use the reduced n in the table, the binomial or the formulas.
Ranking the signed differences instead of the absolute differences in the signed-rank test.
It feels natural to rank the numbers as they appear.
Fix: Rank |d| from smallest to largest, then attach the sign of d to each rank to form T+ and T−.
Mixing up U and W, or using R_X as if it were U.
Mann-Whitney and Wilcoxon rank-sum are two forms of the same test, and books use different notation.
Fix: Say which form you use. Convert with U_X = R_X − m(m + 1) ÷ 2 and check that U_X + U_Y = mn.
Using the Spearman shortcut formula when there are tied ranks.
The formula 1 − 6Σd² ÷ (n(n² − 1)) looks universal.
Fix: Give tied items the average rank. If ties exist, compute Pearson's correlation on the ranks.
Saying non-parametric tests make no assumptions.
The phrase distribution-free is read too literally.
Fix: State the real assumptions: independent observations, and symmetry of differences for the signed-rank test. Mann-Whitney compares medians only if the two distributions have the same shape.
Reading a significant rank correlation as proof of cause.
A high r_s or τ looks like a strong link.
Fix: Say it shows monotonic association only. Comment on possible confounding and sample size.

Worked examples

Example 1

Eight policyholders' annual premium changes after a re-rating (in ₹) are: +30, −10, +40, +20, −50, +60, +70, +80. Use the Wilcoxon signed-rank test to test H0: median change = 0 against H1: median change ≠ 0 at the 5% level. The two-sided 5% critical value for n = 8 in the Wilcoxon table is 3 (reject if the smaller of T+ and T− is at or below 3).

Show the solution

There are no zero differences, so n = 8.
Absolute values in order: 10, 20, 30, 40, 50, 60, 70, 80. These take ranks 1, 2, 3, 4, 5, 6, 7, 8 (no ties).
Negative differences are −10 (rank 1) and −50 (rank 5). So T− = 1 + 5 = 6.
Check: n(n + 1) ÷ 2 = 8 × 9 ÷ 2 = 36, so T+ = 36 − 6 = 30.
The test statistic is the smaller value, min(T+, T−) = 6.
Compare with the critical value: 6 is greater than 3, so we do not reject H0.

Answer: T− = 6 and T+ = 30. Since 6 > 3, we do not reject H0 at the 5% level. There is no significant evidence that the median premium change differs from zero.

Example 2

Six risks are ranked by two underwriters. Underwriter A ranks them 1, 2, 3, 4, 5, 6. Underwriter B gives the same risks the ranks 2, 1, 4, 3, 6, 5. Calculate Spearman's rank correlation and Kendall's tau.

Show the solution

Find d = rank A − rank B: −1, 1, −1, 1, −1, 1.
Square: each d² = 1, so Σd² = 6.
Spearman: r_s = 1 − 6 × 6 ÷ (6 × (36 − 1)) = 1 − 36 ÷ 210 = 1 − 0.1714 = 0.8286.
For Kendall, list B's ranks in A's order: 2, 1, 4, 3, 6, 5. Count discordant pairs (a later value smaller than an earlier one): (2,1), (4,3), (6,5). So D = 3.
Total pairs = 6 × 5 ÷ 2 = 15, so C = 15 − 3 = 12.
τ = (12 − 3) ÷ 15 = 9 ÷ 15 = 0.6.
Compare r_s with the table value for n = 6 at the chosen level before deciding on significance.

Answer: r_s ≈ 0.829 and τ = 0.6. Both show strong positive agreement between the two underwriters' rankings. Kendall's tau is smaller than Spearman's here, which is typical.

Exam tips

Show the ranking table in full. Examiners award method marks for correct ranks and the check sum even if the final decision is wrong.
Always state H0, H1 and whether the test is one-sided or two-sided, and use the matching critical value.
In multiple-choice questions, pick the test from the data layout first: pairs, two independent groups or two variables. Many options can be removed this way.
In a written question, comment on why a non-parametric test is suitable, such as a small sample, skewed claims data or ordinal ranks, and mention the power trade-off.
For computer-based work, set out the function call, the statistic and the p-value, then interpret the result in words. Say how ties were handled.

Practice questions from Hypothesis testing and goodness of fit

Non-Parametric and Permutation Tests: frequently asked questions

What is the difference between parametric and non-parametric tests?

Parametric tests assume a specific distribution, usually normal, and use the actual values to estimate parameters such as the mean. Non-parametric tests use signs or ranks and need weaker assumptions. They are safer for small, skewed or ordinal data but are somewhat less powerful when normality really holds.

Is the Mann-Whitney U test the same as the Wilcoxon rank-sum test?

Yes, they are the same test written in two forms. The rank-sum form uses W = R_X, the sum of one sample's ranks. The U form uses U_X = R_X − m(m + 1) ÷ 2. For example, with X = 12, 15, 9 and Y = 14, 18, 20, 11, we get R_X = 1 + 3 + 5 = 9 and U_X = 9 − 6 = 3. Then U_Y = 12 − 3 = 9 and the total is mn = 12.

When should I use the sign test instead of the Wilcoxon signed-rank test?

Use the sign test when you can only say whether each value is above or below the median, or when differences are not symmetric. Use the signed-rank test when the sizes of the differences are meaningful and roughly symmetric, because it uses more information and is more powerful.

How do I handle ties in rank tests?

Give tied observations the average of the ranks they would have taken. For example, two values tied for ranks 3 and 4 each get 3.5. Ties change the exact null variance, so the standard normal approximation needs an adjustment, and the simple Spearman shortcut no longer applies.

Spearman or Kendall: which should I use?

Both measure monotonic association from ranks. Spearman is the Pearson correlation of the ranks and is quick to compute. Kendall's tau counts concordant and discordant pairs and has a simple interpretation as a difference in probabilities. Use whichever the question asks for, and expect the two values to differ.