Skip to content

Risk Modelling and Survival Analysis · Stochastic processes

Markov Chains and Transition Probabilities: n-Step Matrix Method

Updated 11 October 2026 · Fact-checked

A Markov chain is a process where the next state depends only on the current state. You store one-step probabilities in a matrix P. The n-step probabilities are the entries of Pⁿ, which follow from the Chapman-Kolmogorov equations. To solve a question, build P, then multiply matrices or sum over paths.

Understand Markov Chains and Transition Probabilities

A stochastic process is a set of random variables indexed by time. A Markov chain is one with discrete states and discrete time. Its key feature is the Markov property: given the present state, the past tells you nothing more about the future.

In symbols, P(Xₙ₊₁ = j | Xₙ = i, Xₙ₋₁ = iₙ₋₁, ..., X₀ = i₀) = P(Xₙ₊₁ = j | Xₙ = i). Think of a no claims discount system. Your next discount level depends on your current level and whether you claim this year. It does not depend on how you reached that level.

The one-step transition probability is pᵢⱼ = P(Xₙ₊₁ = j | Xₙ = i). If these do not depend on n, the chain is time-homogeneous. You then collect them in the transition matrix P. Each row is a probability distribution, so every entry is between 0 and 1 and every row sums to 1. Columns need not sum to 1.

The n-step transition probability pᵢⱼ⁽ⁿ⁾ = P(Xₘ₊ₙ = j | Xₘ = i) is the chance of moving from i to j in exactly n steps. To go from i to j in m + n steps, you pass through some state k after m steps. Summing over all k gives the Chapman-Kolmogorov equations. In matrix form, P⁽ⁿ⁾ = Pⁿ. If the starting state is random, the distribution at time n is the row vector π₀Pⁿ.

Key rules to remember

Markov property
P(Xₙ₊₁ = j | Xₙ = i, Xₙ₋₁, ..., X₀) = P(Xₙ₊₁ = j | Xₙ = i)
Future depends only on the present state.
One-step transition probability
pᵢⱼ = P(Xₙ₊₁ = j | Xₙ = i)
For a time-homogeneous chain this does not depend on n.
Row sum condition
Σⱼ pᵢⱼ = 1 for every state i
Rows sum to 1. Columns do not have to.
Chapman-Kolmogorov (elementwise)
pᵢⱼ⁽ᵐ⁺ⁿ⁾ = Σₖ pᵢₖ⁽ᵐ⁾ pₖⱼ⁽ⁿ⁾
Sum over all intermediate states k at the middle time.
Chapman-Kolmogorov (matrix)
P⁽ᵐ⁺ⁿ⁾ = P⁽ᵐ⁾ P⁽ⁿ⁾, so P⁽ⁿ⁾ = Pⁿ
Valid for time-homogeneous chains. For inhomogeneous chains, multiply the matrices for each step in order.
Distribution at time n
πₙ = π₀ Pⁿ
π₀ is a row vector of initial probabilities.

How to solve Markov Chains and Transition Probabilities questions

Use this method for any question on transition probabilities over one or several steps.

  1. 1List the states and read the Markov assumption from the question. Check whether the chain is time-homogeneous.
  2. 2Write the one-step matrix P. Check that each row sums to 1. If a row does not, you have misread the data.
  3. 3Identify what is asked: a specific n-step probability pᵢⱼ⁽ⁿ⁾, a distribution at time n, or a probability of a particular path.
  4. 4For a path probability, multiply the one-step probabilities along the path. For a probability over n steps with free intermediate states, use Pⁿ or sum over paths.
  5. 5For small n, split n as m + (n − m) and use Chapman-Kolmogorov, computing only the row or entries you need. Do not compute the whole matrix.
  6. 6Combine with the initial distribution if the starting state is random: πₙ = π₀Pⁿ.
  7. 7Check the answer lies between 0 and 1, and that any full row you computed still sums to 1. State the assumptions used.

Quickest way: Row-vector multiplication

When to use it: Use when you need one start state and a small number of steps, such as 2 to 4. It avoids full matrix powers.

  1. Write the start state as a row vector, for example (1, 0, 0).
  2. Multiply that vector by P to get the distribution after one step.
  3. Multiply the result by P again for each further step.
  4. Read off the required entry after n steps.
  5. For a two-step answer, use p⁽²⁾ᵢⱼ = Σₖ pᵢₖpₖⱼ directly, listing only the non-zero terms.

Common mistakes in Markov Chains and Transition Probabilities

  • Squaring each entry of P instead of multiplying matrices.

    P² looks like it should mean pᵢⱼ² for each entry.

    Fix: Use matrix multiplication: (P²)ᵢⱼ = Σₖ pᵢₖpₖⱼ, row i of P times column j of P.

  • Using columns that sum to 1, or reading the matrix as P(from j to i).

    Confusion with other conventions or with the transpose.

    Fix: In IAI work, row i is the current state and column j is the next state. Rows sum to 1.

  • Ignoring intermediate states or missing one in the Chapman-Kolmogorov sum.

    Students stop after finding one convenient path.

    Fix: List every state k and include every term with non-zero probability, or compute the matrix product.

  • Assuming the process is Markov because transitions look simple.

    The Markov property is a modelling assumption, not automatic.

    Fix: If the next state depends on more history, such as the last two years' claims, redefine the states to include that history so the new process is Markov.

  • Using Pⁿ when the chain is not time-homogeneous.

    Students apply the standard result without checking.

    Fix: If the matrix changes with time, multiply the matrices for each step in order: P⁽⁰,¹⁾P⁽¹,²⁾ and so on.

  • Wrong order when multiplying distribution and matrix.

    Writing Pπ instead of πP with a row vector.

    Fix: With a row vector π₀, always write π₀P. Matrix products are not commutative.

Worked examples

Example 1

A time-homogeneous Markov chain has states 1, 2, 3 and transition matrix with rows: (0.5, 0.3, 0.2), (0.4, 0.4, 0.2), (0, 0.6, 0.4). Find P(X₂ = 3 | X₀ = 1).

Show the solution
  1. Use Chapman-Kolmogorov: p₁₃⁽²⁾ = p₁₁p₁₃ + p₁₂p₂₃ + p₁₃p₃₃.
  2. Substitute: 0.5 × 0.2 + 0.3 × 0.2 + 0.2 × 0.4.
  3. Compute: 0.10 + 0.06 + 0.08 = 0.24.

Answer: 0.24

Example 2

Using the same chain as above, the chain starts at time 0 with P(X₀ = 1) = 0.6 and P(X₀ = 3) = 0.4. Find P(X₁ = 2).

Show the solution
  1. The initial distribution is π₀ = (0.6, 0, 0.4).
  2. P(X₁ = 2) is the second entry of π₀P, so it equals Σᵢ π₀(i) pᵢ₂.
  3. Compute: 0.6 × p₁₂ + 0 × p₂₂ + 0.4 × p₃₂ = 0.6 × 0.3 + 0.4 × 0.6.
  4. This is 0.18 + 0.24 = 0.42.

Answer: 0.42

Exam tips

  • Show the general formula first, such as the Chapman-Kolmogorov sum, then substitute numbers. Method marks depend on it.
  • Write out every non-zero path term in two-step questions. It makes errors easy for you and the examiner to spot.
  • If a question describes a system with memory, such as a no claims discount depending on the last two years, define new states so the Markov property holds.
  • In Paper B, build P as a matrix in R or Excel and use matrix multiplication for powers. Check that rows sum to 1 before calculating.
  • For multiple-choice questions, compute only the entry you need, not the full matrix power.

Practice questions from Stochastic processes

Markov Chains and Transition Probabilities: frequently asked questions

What is the Markov property in simple terms?

It means that given the current state, the earlier history does not change the probabilities of future states. Only where you are now matters. This is why a single matrix can describe the whole process.

How do I calculate n-step transition probabilities?

Raise the one-step matrix to the power n for a time-homogeneous chain. The entry in row i and column j of Pⁿ is the probability of going from i to j in n steps. For small n, you can multiply a row vector by P repeatedly.

What do the Chapman-Kolmogorov equations say?

They say the probability of going from i to j in m + n steps equals the sum over all intermediate states k of the probability of going i to k in m steps and k to j in n steps. In matrix form, P⁽ᵐ⁺ⁿ⁾ = P⁽ᵐ⁾P⁽ⁿ⁾.

Do the columns of a transition matrix sum to 1?

Not in general. Each row sums to 1 because from any state the chain must go somewhere. Column sums can be any non-negative number.