Risk Modelling and Survival Analysis · Markov processes

Discrete-Time Markov Chains and Transition Probabilities

Updated 11 October 2026 · Fact-checked

A discrete-time Markov chain is a process where the next state depends only on the current state. You store one-step probabilities in a transition matrix P. The n-step probabilities are the entries of Pⁿ, by the Chapman-Kolmogorov equations. To solve questions, build P, then multiply matrices or follow paths.

Understand Discrete-Time Markov Chains and Transition Probabilities

A stochastic process is a set of random variables observed over time. A discrete-time Markov chain has a countable set of states and time points 0, 1, 2, and so on. Its key feature is the Markov property: given the present state, the past does not affect the future.

In symbols, P(Xₙ₊₁ = j | X₀ = i₀, ..., Xₙ = i) = P(Xₙ₊₁ = j | Xₙ = i). This is why a Markov model is easy to handle. You only need to know where you are now, not how you got there.

The one-step transition probabilities p_ij = P(Xₙ₊₁ = j | Xₙ = i) go into a transition matrix P. Row i holds the probabilities of moving from state i. Every entry is between 0 and 1, and every row sums to 1. Columns do not need to sum to 1.

A chain is time-homogeneous if p_ij does not depend on n. Then one matrix P describes every step. In a non-homogeneous chain the probabilities change with time, so you have a different matrix for each step, written P(n, n+1). To get a multi-step probability you multiply the matrices for the correct steps in order.

A classic use is the no-claims discount (NCD) system. States are discount levels, for example 0%, 25% and 50%. A claim-free year moves the policyholder up a level. A claim moves them down. The next level depends only on the current level and whether a claim occurs, so the Markov property holds. The model is then used to find the distribution of discount levels after several years.

Key rules to remember

Markov property: P(Xₙ₊₁ = j | X₀, X₁, ..., Xₙ = i) = P(Xₙ₊₁ = j | Xₙ = i); Future depends only on the present state.
One-step transition probability: p_ij(n) = P(Xₙ₊₁ = j | Xₙ = i); For a time-homogeneous chain, p_ij does not depend on n.
Row sum rule: Σⱼ p_ij = 1 for every state i; Rows sum to 1. Columns need not.
n-step probability: p_ij⁽ⁿ⁾ = P(Xₘ₊ₙ = j | Xₘ = i) = (Pⁿ)_ij; Valid for time-homogeneous chains.
Chapman-Kolmogorov equations: p_ij⁽ᵐ⁺ⁿ⁾ = Σₖ p_ik⁽ᵐ⁾ × p_kj⁽ⁿ⁾; Sum over all intermediate states k. In matrix form, Pᵐ⁺ⁿ = Pᵐ Pⁿ.
Non-homogeneous multi-step matrix: P(m, m+n) = P(m, m+1) × P(m+1, m+2) × ... × P(m+n−1, m+n); Keep the matrices in time order.
Distribution at time n: π⁽ⁿ⁾ = π⁽⁰⁾ Pⁿ; π is a row vector of state probabilities. Multiply on the left.

How to solve Discrete-Time Markov Chains and Transition Probabilities questions

Use this method for any question on transition matrices, n-step probabilities or NCD models.

1Define the states clearly and list them in a fixed order.
2Write down the one-step probabilities from the question. For an NCD model, work out what a claim and a claim-free year each do to the state.
3Build the matrix P with rows as 'from' and columns as 'to'. Check each row sums to 1.
4Decide whether the chain is time-homogeneous. If not, note which matrix applies to which year.
5Identify what is asked: a specific n-step probability, a distribution at time n, or an expected value such as average premium.
6For a small n, list the paths from the start state to the end state and add the products of probabilities. For a larger n, use Pⁿ or Chapman-Kolmogorov with a convenient split.
7Write the answer with the notation p_ij⁽ⁿ⁾ and check it lies between 0 and 1.
8If asked for an expected cost, multiply each state probability by its value and add.

Quickest way: Path listing and vector multiplication

When to use it: Use it when n is 2 or 3, the matrix has 3 or 4 states, and you only need one probability or one distribution.

Do not compute the full Pⁿ. Start with the initial row vector, for example (1, 0, 0) if you begin in state 1.
Multiply the vector by P once to get the distribution after one step.
Multiply the result by P again for each further step. Each product needs only the entries of that vector.
Read off the required state. Check the new vector sums to 1 at every step.
If the start state is fixed, this is faster than squaring the matrix and gives the same answer.

Common mistakes in Discrete-Time Markov Chains and Transition Probabilities

Rows that do not sum to 1, or reading the matrix by columns.
Students mix up 'from' and 'to' when filling the matrix.
Fix: Rows are from-states, columns are to-states. Check every row sums to 1 before doing anything else.
Squaring each entry instead of using matrix multiplication.
Pⁿ looks like a power of numbers.
Fix: Pⁿ means P multiplied by itself n times with row-by-column multiplication. Entry (i, j) is Σₖ p_ik p_kj for n = 2.
Multiplying the vector on the wrong side.
Students write Pπ instead of πP.
Fix: With π as a row vector, use π⁽ⁿ⁾ = π⁽⁰⁾ Pⁿ. Check dimensions before multiplying.
Using the same matrix for every year in a non-homogeneous chain.
Most examples are time-homogeneous, so students assume it.
Fix: Read the question for time-dependent probabilities. Multiply the year-specific matrices in order.
Missing a path in a two- or three-step calculation.
Students stop after finding the obvious route.
Fix: List every intermediate state k, including the start and end states themselves. Staying put is a valid move if p_ii > 0.
Setting up NCD transitions wrongly at the top or bottom level.
The rule says 'move up after a claim-free year' but a policyholder at the top level cannot move up.
Fix: At the top level, a claim-free year keeps you there. At the bottom level, a claim keeps you there. Apply the rules from the question exactly, including any 'move back two levels' rules.

Worked examples

Example 1

A motor insurer uses an NCD system with three levels: 0%, 25% and 50% discount (states 1, 2 and 3). The probability of a claim-free year is 0.8 for every policyholder. After a claim-free year a policyholder moves up one level, or stays at 50%. After a claim the policyholder moves down one level, or stays at 0%. Write the transition matrix and find the probability that a policyholder at 0% is at 50% after two years.

Show the solution

Let q = 0.8 (claim-free) and 1 − q = 0.2 (claim).
From state 1 (0%): claim-free to state 2 with 0.8, claim stays at state 1 with 0.2.
From state 2 (25%): claim-free to state 3 with 0.8, claim to state 1 with 0.2.
From state 3 (50%): claim-free stays at state 3 with 0.8, claim to state 2 with 0.2.
P has rows (0.2, 0.8, 0), (0.2, 0, 0.8) and (0, 0.2, 0.8). Each row sums to 1.
We need p₁₃⁽²⁾ = Σₖ p₁ₖ p_k₃.
k = 1: 0.2 × 0 = 0. k = 2: 0.8 × 0.8 = 0.64. k = 3: 0 × 0.8 = 0.
Sum = 0.64.

Answer: p₁₃⁽²⁾ = 0.64

Example 2

Using the NCD chain above, a new policyholder starts at 25% (state 2). Find the distribution of discount levels after two years, and the expected discount after two years.

Show the solution

Initial vector π⁽⁰⁾ = (0, 1, 0).
After one year: π⁽¹⁾ = π⁽⁰⁾P = row 2 of P = (0.2, 0, 0.8).
After two years: π⁽²⁾ = π⁽¹⁾P.
State 1: 0.2 × 0.2 + 0 × 0.2 + 0.8 × 0 = 0.04.
State 2: 0.2 × 0.8 + 0 × 0 + 0.8 × 0.2 = 0.16 + 0.16 = 0.32.
State 3: 0.2 × 0 + 0 × 0.8 + 0.8 × 0.8 = 0.64.
Check: 0.04 + 0.32 + 0.64 = 1.
Expected discount = 0 × 0.04 + 25% × 0.32 + 50% × 0.64 = 8% + 32% = 40%.

Answer: π⁽²⁾ = (0.04, 0.32, 0.64); expected discount = 40%

Exam tips

Always state the states, the matrix and the Markov assumption. Examiners award marks for setup before any arithmetic.
Check row sums at every stage. It catches most arithmetic and set-up errors quickly.
In written answers, name the Chapman-Kolmogorov equations when you split an n-step probability at an intermediate time.
For NCD questions, read the rules for the top and bottom levels twice. Many marks are lost on boundary states.
Questions often ask you to comment on the model: the Markov assumption ignores claim history beyond the current level, and claim probability may vary by policyholder. Be ready to say this in one or two lines.

Practice questions from Markov processes

Discrete-Time Markov Chains and Transition Probabilities: frequently asked questions

What is the difference between time-homogeneous and non-homogeneous Markov chains?

In a time-homogeneous chain, the transition probabilities do not depend on time, so one matrix P applies at every step. In a non-homogeneous chain they vary with time, so you need a matrix for each step. Multi-step probabilities are then products of the different matrices in time order.

How do I calculate n-step transition probabilities?

For a time-homogeneous chain, compute Pⁿ by matrix multiplication and read off entry (i, j). For small n you can instead add the products of probabilities along all paths from i to j. Both methods give the same answer.

What do the Chapman-Kolmogorov equations say?

They say the probability of going from i to j in m + n steps equals the sum over all intermediate states k of the probability of going i to k in m steps and then k to j in n steps. In matrix form, Pᵐ⁺ⁿ = Pᵐ Pⁿ.

Why is a no-claims discount system a Markov chain?

The discount level next year depends only on the current level and whether a claim is made this year. The earlier history does not matter once the current level is known. This is exactly the Markov property.