Risk Modelling and Survival Analysis · Stochastic processes

Applications: Multi-State Models and Random Walks

Updated 11 October 2026 · Fact-checked

These are applications of the Markov property. List the states, find the transition probabilities (or intensities for continuous time), then use the matrix, stationary distribution or Kolmogorov equations to get probabilities and expected values. A random walk moves up or down by one step at each time. Barriers turn it into a ruin problem.

Understand Applications: Multi-State Models and Random Walks

A Markov model says the future depends only on the present state, not on how you got there. Once you accept this, every question becomes: what are the states, and how do you move between them?

The no claims discount (NCD) system is a discrete-time Markov chain. Each state is a discount level, for example 0%, 25% or 50%. At the end of each policy year you move up a level if you made no claim and down a level if you did. The state next year depends only on this year's level and whether you claim. That is the Markov property. The model is then used to find the long-run share of policyholders at each level and the average premium.

A multi-state model is the continuous-time version. A person can be in states such as Healthy, Sick and Dead, and can move at any moment. This is a Markov jump process. Moves are governed by transition intensities μ_ij, where μ_ij × h is approximately the probability of moving from i to j in a short time h. The time spent in a state before leaving is exponential, with a rate equal to the total intensity out of that state. When the process leaves state i, it goes to state j with probability μ_ij ÷ (total intensity out of i).

A simple random walk starts at some integer and moves +1 with probability p and −1 with probability q = 1 − p at each step. It is a Markov chain with infinitely many states. Put absorbing barriers at 0 and N and it becomes a gambler's ruin model: you want the probability of hitting one barrier before the other. This is the classic ruin-probability question.

For all three, the exam wants you to set up the model clearly first. Most marks go to correct states and transitions. The arithmetic is usually short.

Key rules to remember

Markov property (discrete time): P(X_{n+1} = j | X_n = i, X_{n-1}, ..., X_0) = P(X_{n+1} = j | X_n = i); Rows of the transition matrix must each sum to 1.
n-step probabilities: P(n) = Pⁿ, and p_ij(m+n) = Σ_k p_ik(m) p_kj(n); This is the Chapman-Kolmogorov equation. Multiply matrices in the right order.
Stationary distribution: π = πP, with Σ π_i = 1; Use the normalising condition. Check the chain is irreducible and aperiodic if you claim the chain converges to π.
Expected long-run NCD premium: Σ π_i × (full premium) × (1 − discount_i); Weight each level's premium by its stationary probability.
Transition intensity: P(X_{t+h} = j | X_t = i) = μ_ij × h + o(h), for j ≠ i; Intensities are rates, not probabilities. They can exceed 1.
Holding time in state i: Time in i ~ Exp(λ_i), where λ_i = Σ_{j≠i} μ_ij; mean = 1 ÷ λ_i; For constant intensities. The next state is j with probability μ_ij ÷ λ_i.
Kolmogorov forward equations: d/dt p_ij(t) = Σ_{k≠j} p_ik(t) μ_kj − p_ij(t) Σ_{k≠j} μ_jk; Inflow into j minus outflow from j. The backward equations use the starting state instead.
Probability of staying in a state: p_ii(t) = exp(−λ_i t) when the process cannot leave and return; Not valid if you can come back to i. Then you need the full equations.
Random walk with absorbing barriers, p ≠ q: P(hit 0 before N | start i) = ((q/p)^i − (q/p)^N) ÷ (1 − (q/p)^N); Here the walk goes up with probability p. The probability of hitting N first is 1 minus this.
Symmetric walk (p = q = ½): P(hit 0 before N | start i) = 1 − i/N; expected steps to absorption = i(N − i); Do not use the p ≠ q formula here, as it gives 0 ÷ 0.

How to solve Applications: Multi-State Models and Random Walks questions

Use the same routine for NCD, health-insurance and random walk questions. It keeps your answer structured and protects method marks.

1Define the state space in words. Name every state, such as discount levels or Healthy, Sick, Dead. Decide whether time is discrete or continuous.
2Write the transition rules. For discrete time, build the matrix and check each row sums to 1. For continuous time, draw the state diagram with intensities on the arrows.
3State your assumptions: Markov property, constant intensities, time-homogeneity, claim probability constant and independent across years.
4Decide what is being asked: a few-step probability (use matrix powers or direct paths), long-run behaviour (stationary distribution), a holding time or expected value (use the exponential rates), or a ruin probability (use the barrier formula).
5Do the calculation. For π, write the equations π = πP, express all π_i in terms of one, and then normalise. For Kolmogorov, write inflow minus outflow for the probability needed.
6Check the answer: probabilities between 0 and 1, π sums to 1, intensity-based probabilities sum to 1.
7Give the final answer in context, with units and rupee amounts where relevant, and comment briefly if the question asks you to.

Quickest way: Fast routes for stationary and ruin questions

When to use it: Use this when time is short in the MCQ section or when a written part only asks for a single number.

For NCD chains where you step up one level or down one level, balance flows between neighbouring levels. In the long run, flow up from i equals flow down from i+1, so π_i × (up probability) = π_{i+1} × (down probability).
With a one-step-up, one-step-down structure, the π values form a geometric pattern. Set the lowest as 1 unit, build the others, then divide by the total.
For two paths to the same state in two steps, just multiply along each path and add. You do not need the full matrix squared.
For a ruin problem, first check whether p = q. If so, use 1 − i/N. If not, compute r = q/p and plug into the formula.
For a holding-time question, add up the intensities out of the state and take the reciprocal for the mean. Do not solve the full Kolmogorov equations unless asked.

Common mistakes in Applications: Multi-State Models and Random Walks

Using the wrong direction in the transition matrix, or confusing rows and columns.
Students copy the layout from one example and do not check which index is 'from'.
Fix: Fix the convention: row i is the current state, column j is the next state. Check every row sums to 1 before you continue.
Treating transition intensities as probabilities.
The symbol μ_ij looks like a probability and the diagram arrows carry numbers.
Fix: Remember μ_ij is a rate per unit time. Probability over a short time h is about μ_ij × h. Over longer periods you need the Kolmogorov equations or the exponential holding time.
Getting the NCD rules wrong at the top and bottom levels.
Students forget that a policyholder at the highest level who makes no claim stays there, and one at the lowest who claims stays there.
Fix: Write out every transition from every state, including the self-loops at the ends, before building the matrix.
Forgetting to normalise the stationary distribution.
The equations π = πP have infinitely many solutions, so students stop once they have ratios.
Fix: Always finish with Σ π_i = 1 and divide through. Then check by confirming πP = π.
Using the p ≠ q ruin formula for a symmetric walk, or mixing up which barrier is which.
The formula is memorised without the conditions, and p and q are swapped.
Fix: Check p = q first. Define up probability p, r = q/p, and say which barrier you mean. Check the answer makes sense: if p > q, ruin should be less likely than in the symmetric case.
Writing p_ii(t) = exp(−λ_i t) when the state can be re-entered.
It is correct in a simple model with no return, so students apply it everywhere.
Fix: This only gives the probability of staying continuously in i. If the process can leave and come back (for example Healthy to Sick and back), the probability of being in i at time t is larger. Use the Kolmogorov equations.

Worked examples

Example 1

A motor insurer uses three NCD levels: 0%, 25% and 50%. After a claim-free year a policyholder moves up one level (staying at 50% if already there). After a year with a claim the policyholder moves down one level (staying at 0% if already there). The probability of a claim-free year is 0.8 each year, independently. The full premium is ₹10,000. (a) Write the transition matrix. (b) Find the stationary distribution. (c) Find the long-run expected premium per policy.

Show the solution

States: 0 = 0% discount, 1 = 25% discount, 2 = 50% discount. Assume claim behaviour is the same each year and independent of past years, so the level is a Markov chain.
Transitions: from 0, up to 1 with 0.8, stay at 0 with 0.2. From 1, up to 2 with 0.8, down to 0 with 0.2. From 2, stay at 2 with 0.8, down to 1 with 0.2.
Matrix P (rows are current state, in order 0, 1, 2): row 0 = (0.2, 0.8, 0); row 1 = (0.2, 0, 0.8); row 2 = (0, 0.2, 0.8). Each row sums to 1.
Stationary equations: π0 = 0.2π0 + 0.2π1, so 0.8π0 = 0.2π1 and π1 = 4π0.
π2 = 0.8π1 + 0.8π2, so 0.2π2 = 0.8π1 and π2 = 4π1 = 16π0.
Normalise: π0 + 4π0 + 16π0 = 21π0 = 1, so π0 = 1/21, π1 = 4/21, π2 = 16/21.
Check the middle equation: π1 = 0.8π0 + 0.2π2 = 0.8/21 + 3.2/21 = 4/21. ✓
Expected premium = 10,000 × (1/21 × 1 + 4/21 × 0.75 + 16/21 × 0.5) = 10,000 × (1 + 3 + 8)/21 = 10,000 × 12/21 = 10,000 × 4/7.
This equals ₹5,714.29 (to the nearest paisa).

Answer: Stationary distribution (1/21, 4/21, 16/21). Long-run expected premium = ₹10,000 × 4/7 ≈ ₹5,714.29 per policy.

Example 2

A gambler starts with ₹2 (take one unit as ₹1). Each round they win ₹1 with probability 0.6 or lose ₹1 with probability 0.4, independently. They stop when they reach ₹5 or ₹0. Find the probability that they are ruined (reach ₹0 before ₹5).

Show the solution

Model: a simple random walk on {0, 1, ..., 5} with absorbing barriers at 0 and 5. Start at i = 2, N = 5, up probability p = 0.6, down probability q = 0.4.
Since p ≠ q, use the formula. Let r = q/p = 0.4/0.6 = 2/3.
P(ruin) = (r^i − r^N) ÷ (1 − r^N).
r² = 4/9 = 108/243. r⁵ = 32/243.
Numerator: 108/243 − 32/243 = 76/243. Denominator: 1 − 32/243 = 211/243.
P(ruin) = 76/211 ≈ 0.360.
Check: the probability of reaching ₹5 first is (1 − r²) ÷ (1 − r⁵) = (135/243) ÷ (211/243) = 135/211. And 76/211 + 135/211 = 1. ✓
Sense check: with a symmetric walk, ruin would be 1 − 2/5 = 0.6. Here the walk is favourable (p > q), so a smaller ruin probability of about 0.36 is reasonable.

Answer: P(ruin) = 76/211 ≈ 0.360.

Exam tips

In written answers, show the state diagram or matrix and the assumptions first. Examiners give marks for the model set-up even if the arithmetic goes wrong.
For multi-state questions, check whether the question wants a probability, a mean time or an expected cost. Each uses a different tool: Kolmogorov equations, 1 ÷ λ, or a sum over states.
In the computer-based paper, state the matrix and method in a comment or note, then compute with matrix powers or by solving π = πP in R or Excel. Check rows sum to 1 and π sums to 1.
Always say whether the random walk is symmetric before choosing a ruin formula. State your up probability and which barrier your answer refers to.
For MCQs on NCD, trace the first one or two steps by hand. It is often quicker than building the whole matrix.

Practice questions from Stochastic processes

Applications: Multi-State Models and Random Walks: frequently asked questions

Is the NCD system really a Markov chain?

Yes, if the next level depends only on the current level and whether a claim is made. This holds when the rules move you up or down by fixed steps. It breaks if the rule also depends on, for example, how many years you spent at a level, unless you expand the states to include that history.

What is the difference between a Markov chain and a Markov jump process?

A Markov chain changes state at fixed time steps and is described by a transition matrix. A Markov jump process can change state at any time and is described by transition intensities. Both have the Markov property.

How do I find the probability of being in a state at time t in a multi-state model?

Write the Kolmogorov forward equations for that probability: inflow minus outflow. If the state cannot be re-entered, it simplifies to an exponential, such as exp(−λt) for staying in the same state. Otherwise you solve the differential equations, often using an integrating factor.

When do I use 1 − i/N for ruin?

Use it only for the symmetric random walk where the up and down probabilities are both ½, with absorbing barriers at 0 and N. If p and q differ, use the formula with r = q/p.