Skip to content

Risk Modelling and Survival Analysis · Markov processes

Stochastic Processes and the Markov Property Explained

Updated 11 October 2026 · Fact-checked

A stochastic process is a collection of random variables {Xt} indexed by time, taking values in a state space. It has the Markov property if the future depends only on the present state, not on the past path. To test it, check whether extra history changes the conditional probability.

Understand Stochastic Processes and the Markov Property

A stochastic process is a family of random variables {Xt : t ∈ T}. The index t is usually time. Each Xt describes the system at time t, such as a policyholder's discount level or whether a life is alive or dead.

You classify a process in two ways. The state space S is the set of values Xt can take. It is discrete if the values are countable (for example {Healthy, Sick, Dead}) and continuous if they fill an interval (for example the value of an asset). The time set T is discrete if it is a set like {0, 1, 2, ...} and continuous if it is an interval like [0, ∞). This gives four combinations. Each has standard examples.

  • Discrete time, discrete state: a simple random walk, a no claims discount (NCD) system observed each year.
  • Continuous time, discrete state: a Poisson process, a multi-state health model.
  • Discrete time, continuous state: a time series such as an AR(1) model.
  • Continuous time, continuous state: Brownian motion.

The Markov property says that, given the present, the past is irrelevant for predicting the future. For a process with discrete states in discrete time: P(Xn+1 = j | X0 = i0, ..., Xn = i) = P(Xn+1 = j | Xn = i). Only the current state matters. This is the memoryless idea in terms of path history. It does not say the past has no effect at all. The past matters only through where you are now.

A Markov process with discrete state space and discrete time is a Markov chain. A Markov process with discrete state space and continuous time is a Markov jump process. The chain moves at fixed time steps. The jump process can move at any instant, and it stays in a state for a random holding time. For the jump process to be Markov, the holding times must be exponential, which is the continuous memoryless distribution, and the state it jumps to must depend only on the current state. Together with the Markov property, these two conditions characterise a Markov jump process.

Many processes are not Markov as given, but can be made Markov by enlarging the state. For example, an NCD system that depends on the last two years of claims can be made Markov by defining the state as the pair of last two years' outcomes.

Key rules to remember

Markov property (discrete time)
P(Xn+1 = j | X0 = i0, X1 = i1, ..., Xn = i) = P(Xn+1 = j | Xn = i)
Holds for all n, all states and all histories with non-zero probability. Only the current state matters.
Markov property (continuous time)
P(Xt = j | Xs1 = i1, ..., Xsn = in, Xs = i) = P(Xt = j | Xs = i), for s1 < ... < sn < s < t
Conditioning on the most recent known state is enough.
Transition probability (time-homogeneous chain)
pij = P(Xn+1 = j | Xn = i), independent of n
Time-homogeneous means the probabilities do not depend on when the step happens. This is an extra assumption, not part of the Markov property.
Memoryless property of the exponential
P(T > s + t | T > s) = P(T > t)
In a Markov jump process, holding times must be exponential, and the next state must depend only on the current state. The memoryless property means the time already spent in a state does not change the remaining time.
Two-step probability by conditioning
P(Xn+2 = k | Xn = i) = Σj pij pjk
Uses the Markov property and the law of total probability. This is the Chapman-Kolmogorov idea.

How to solve Stochastic Processes and the Markov Property questions

Use this method for classification questions and for questions that ask you to test or justify the Markov property.

  1. 1Identify the time set: discrete (steps 0, 1, 2, ...) or continuous (any t ≥ 0).
  2. 2Identify the state space: countable list of states, or a continuum of values.
  3. 3Name the process type from these two answers, for example continuous time with discrete states is a jump process.
  4. 4Write down what determines the next move. Ask whether it needs only the current state or also earlier states.
  5. 5If earlier states matter, try to enlarge the state (for example, record the last two outcomes) so the new process is Markov. Define the new states clearly.
  6. 6For a jump process, check that holding times are exponential and that the next state depends only on the current state.
  7. 7State any extra assumption, such as time-homogeneity, separately from the Markov property.
  8. 8Write the conclusion in one sentence and link it to the context of the question.

Quickest way: Two-question check

When to use it: Use this in multiple-choice questions or when you have little time to classify a process.

  1. Ask: is time stepped or continuous? Ask: are the states a list or a range of values?
  2. Ask: to predict the next step, do I need anything beyond the current state? If no, it is Markov. If yes, try enlarging the state.
  3. Remember that a random walk with independent increments is Markov, and a process with a cumulative or lagged dependency usually needs a bigger state.

Common mistakes in Stochastic Processes and the Markov Property

  • Saying the Markov property means the past has no effect on the future.

    The word memoryless is taken too literally.

    Fix: Say that the past affects the future only through the current state. Given the present state, extra history adds no information.

  • Confusing Markov with time-homogeneous.

    Both ideas appear together in most examples.

    Fix: Markov is about dependence on history. Time-homogeneous is about the transition probabilities not changing with time. A process can be Markov but not time-homogeneous.

  • Mixing up the time set and the state space when classifying.

    Students describe the process as continuous because the values look continuous, ignoring the time index.

    Fix: Always state both separately, for example discrete time with discrete state space.

  • Calling a Markov jump process a Markov chain.

    Both are Markov processes with discrete states.

    Fix: Use chain for discrete time and jump process for continuous time. In a jump process, transitions can occur at any time.

  • Declaring a process non-Markov without trying to enlarge the state space.

    The question seems to depend on past years, so the student stops.

    Fix: If dependence is on a finite history, define a new state holding that history. The enlarged process is Markov.

  • Assuming any process with discrete states in continuous time is Markov.

    Poisson and multi-state models are common examples.

    Fix: Check the Markov property directly. If holding times are not exponential, or the next state depends on duration or past path, it is not Markov.

Worked examples

Example 1

For each process, state the time set and the state space, and name the type: (a) the number of claims received by an insurer up to time t, t ≥ 0; (b) the NCD level of a motor policyholder at each annual renewal, with levels 0%, 20% and 40%.

Show the solution
  1. (a) Time t ≥ 0 can take any value, so time is continuous.
  2. (a) The number of claims is 0, 1, 2, ..., which is countable, so the state space is discrete.
  3. (a) Continuous time with discrete state space: a jump process type (for example a Poisson process).
  4. (b) Observation is at annual renewals, so time is discrete.
  5. (b) The states 0%, 20% and 40% are a finite list, so the state space is discrete.
  6. (b) Discrete time with discrete state space: a Markov chain type if the next level depends only on the current level and the claims.

Answer: (a) Continuous time, discrete state space. (b) Discrete time, discrete state space.

Example 2

A NCD system has the rule: the discount in year n + 1 depends on whether a claim was made in each of years n − 1 and n. Let Yn = 1 if there was a claim in year n and 0 otherwise, and let Dn be the discount level in year n. Assume claims in different years are independent. Is the discount level Dn on its own a Markov chain? If not, how can you make a Markov chain?

Show the solution
  1. The discount Dn+1 is a function of the last two claim indicators (Yn−1, Yn). Write Dn+1 = f(Yn−1, Yn). It follows that Dn = f(Yn−2, Yn−1).
  2. So Dn is built from the claims of years n − 2 and n − 1. If different claim pairs give the same level, Dn does not reveal Yn−1, yet Yn−1 is needed to predict Dn+1. Whether this breaks the Markov property depends on the specific f, so test a concrete rule.
  3. Take this rule: the discount is 20% if there was no claim in either of the last two years, and 0% otherwise. Let q = P(Yn = 1), with 0 < q < 1. Then Dn = 20% exactly when Yn−2 = Yn−1 = 0, and Dn+1 = 20% exactly when Yn−1 = Yn = 0.
  4. Condition on Dn = 0 only. Then P(Dn = 0) = 1 − (1 − q)² = q(2 − q). The event {Dn = 0, Yn−1 = 0} means Yn−2 = 1 and Yn−1 = 0, with probability q(1 − q). So P(Yn−1 = 0 | Dn = 0) = (1 − q) ÷ (2 − q). Hence P(Dn+1 = 20% | Dn = 0) = (1 − q)² ÷ (2 − q). For q = 0.5 this is 1/6, which is greater than 0.
  5. Now add history: Dn−1 = 20% and Dn = 0. Dn−1 = 20% means Yn−3 = Yn−2 = 0. Then Dn = 0 forces Yn−1 = 1. This history has probability (1 − q)² × q, which is greater than 0. Since Yn−1 = 1, Dn+1 = 0 for certain. So P(Dn+1 = 20% | Dn = 0, Dn−1 = 20%) = 0.
  6. The two conditional probabilities differ, (1 − q)² ÷ (2 − q) versus 0. So for this rule Dn alone is not Markov: earlier levels change the distribution of Dn+1 even given Dn. The reason is that Dn−1 and Dn together reveal Yn−1, but Dn alone does not.
  7. Note that the claim indicators Yn are independent, so the sequence Yn is trivially a Markov chain. The problem is the discount level Dn, not Yn.
  8. Enlarge the state. Define Zn = (Yn−1, Yn). It has four values: (0,0), (0,1), (1,0), (1,1).
  9. The next state Zn+1 = (Yn, Yn+1) uses Yn, which is in the current state Zn, and the new indicator Yn+1. Since Yn+1 is independent of the past, the distribution of Zn+1 depends only on Zn.
  10. So Zn is a Markov chain. From state (a, b) it moves to (b, c) with probability P(Yn+1 = c), and to no other state. The discount is then Dn+1 = f(Zn).

Answer: For the rule 'discount only after two claim-free years', Dn alone is not Markov: given Dn = 0, the probability that Dn+1 = 20% is (1 − q)² ÷ (2 − q), but it is 0 if we also know Dn−1 = 20%. Whether Dn is Markov in general depends on the specific rule f. Define the state as Zn = (Yn−1, Yn). Then Zn is a Markov chain with four states, provided claims in different years are independent, and the discount level is a function of Zn.

Exam tips

  • Always state both the time set and the state space. Examiners award separate marks for each.
  • When asked to test the Markov property, write the defining equation and compare it to the conditional probability given the full history.
  • Use examples from the question context, such as NCD levels or health states, in your answer.
  • Keep Markov and time-homogeneous separate in your wording.
  • In Paper B style tasks, define states and time steps clearly before building a transition matrix.

Practice questions from Markov processes

Stochastic Processes and the Markov Property: frequently asked questions

What is the Markov property in simple words?

Given the current state, the future does not depend on how you got there. Only the present state is needed to predict what happens next. The past matters only through the present state.

What is the difference between a Markov chain and a Markov jump process?

A Markov chain moves at discrete time steps, such as every year. A Markov jump process runs in continuous time and can change state at any instant. Its holding times in each state are exponential.

Is a random walk a Markov process?

A simple random walk with independent steps is a Markov chain. The next position depends only on the current position plus a new independent step. The earlier path adds no extra information.

Does a Markov process have to be time-homogeneous?

No. Time-homogeneity is a separate assumption that transition probabilities do not depend on time. Many exam questions assume it, but you should state it as an assumption.