Skip to content

Risk Modelling and Survival Analysis · Markov chains

Stochastic Processes and the Markov Property Explained

Updated 11 October 2026 · Fact-checked

A stochastic process is a collection of random variables {X_t} indexed by time t. It is classified by its state space (discrete or continuous) and its time set (discrete or continuous). It has the Markov property if, given the present, the future is independent of the past. Time-homogeneous means transition probabilities depend only on the time gap.

Understand Stochastic Processes and the Markov Property

A stochastic process is a family of random variables {X_t : t ∈ T}. Each X_t is the value of something random at time t. Examples: the claims paid in year t, the number of claims by time t, a policyholder's health status at age x, or a share price at time t.

Two things classify a process. The state space S is the set of values X_t can take. The time set T is the set of times at which you observe the process. Each can be discrete or continuous. This gives four combinations. A no claims discount level each year is discrete state, discrete time. The number of claims up to time t is discrete state, continuous time. A share price observed daily is continuous state, discrete time. Brownian motion is continuous state, continuous time.

A sample path (or realisation) is one possible outcome of the whole process over time. The process is a random choice among many such paths.

The Markov property says the future depends on the past only through the present. If you know the current state, the earlier history adds no information about what happens next. For a discrete-time process with discrete states: P(X_n+1 = j | X_0 = i_0, ..., X_n = i_n) = P(X_n+1 = j | X_n = i_n). A process with this property is a Markov chain (discrete time) or Markov jump process (continuous time, discrete states).

A Markov process is time-homogeneous if the transition probability from state i at time s to state j at time s+t depends only on the gap t, not on s. Then P(X_n+1 = j | X_n = i) is the same for every n, and you can write it as p_ij. Do not confuse this with the Markov property. A process can be Markov but not time-homogeneous, for example when mortality rises with age.

Key rules to remember

Stochastic process
{X_t : t ∈ T}, with X_t taking values in state space S
Always state S and T when asked to describe or classify a process.
Markov property (discrete time)
P(X_n+1 = j | X_0 = i_0, ..., X_n = i_n) = P(X_n+1 = j | X_n = i_n)
Holds for all n and all states and histories with non-zero probability.
Markov property (general)
P(X_t ∈ A | X_s1 = x_1, ..., X_sn = x_n, X_s = x) = P(X_t ∈ A | X_s = x), for s1 < ... < sn < s < t
Use this form for continuous time. A is any subset of the state space.
Time-homogeneity
P(X_s+t = j | X_s = i) = P(X_t = j | X_0 = i) for all s ≥ 0
Transition probabilities depend only on the elapsed time t.
One-step transition probability
p_ij = P(X_n+1 = j | X_n = i)
Used for a time-homogeneous Markov chain. Each row sums to 1: Σ_j p_ij = 1.

How to solve Stochastic Processes and the Markov Property questions

Use this method when you are asked to classify a process or to decide whether it is Markov.

  1. 1Write down the random variable X_t and say what it measures.
  2. 2State the state space S. Say whether it is discrete (countable) or continuous.
  3. 3State the time set T. Say whether it is discrete (for example t = 0, 1, 2, ...) or continuous (t ≥ 0).
  4. 4To test the Markov property, ask: if I know X_n, does the earlier history X_0, ..., X_n-1 change the probability of X_n+1? If yes, the process is not Markov as defined.
  5. 5If not Markov, try to make it Markov by enlarging the state, for example by including the last two values as the state.
  6. 6To test time-homogeneity, check if the transition probabilities depend on n (or on calendar time or age). If they do not, it is time-homogeneous.
  7. 7Write a clear conclusion using the correct terms, with a one-line reason.

Quickest way: Three-question check

When to use it: Use this in MCQs and short written parts where you must classify a process or judge the Markov property quickly.

  1. Ask what values X can take: countable or an interval? This gives the state space type.
  2. Ask when you observe it: at set steps or at any time? This gives the time set type.
  3. Ask: does the next move depend on anything except where I am now? If it depends on earlier values, it is not Markov. If the rule uses only the current state, it is Markov. Then check if the rule changes with time.

Common mistakes in Stochastic Processes and the Markov Property

  • Saying the Markov property means the future is independent of the past.

    The short slogan drops the condition.

    Fix: Say the future is independent of the past given the present. Past and future can still be strongly dependent overall.

  • Treating Markov and time-homogeneous as the same thing.

    Most examples in notes are both.

    Fix: Check them separately. Markov concerns dependence on history. Time-homogeneity concerns whether probabilities change with time.

  • Mixing up the state space and the time set.

    Both can be described as discrete or continuous.

    Fix: State space is the set of values X_t takes. Time set is the set of times t. Name both separately.

  • Calling a process Markov when its next step depends on the last two values.

    Students check only the one-step rule and ignore the history it uses.

    Fix: If the rule uses X_n-1 as well as X_n, it is not Markov in X_n alone. Redefine the state as the pair (X_n-1, X_n) to restore the property.

  • Assuming a continuous-time process must have a continuous state space.

    The word continuous gets applied to everything.

    Fix: A Poisson process counts events in continuous time but has state space {0, 1, 2, ...}, which is discrete.

Worked examples

Example 1

Classify each process by state space and time set: (a) the number of claims received by an insurer in each calendar month; (b) the number of claims received up to time t, where t is measured continuously in days; (c) the closing value of the Sensex each trading day.

Show the solution
  1. (a) X_n is the claim count in month n. It takes values 0, 1, 2, ..., so the state space is discrete. Months are observed one at a time, so time is discrete.
  2. (b) N_t is the cumulative count of claims by time t. It takes values 0, 1, 2, ..., so the state space is discrete. t ≥ 0 can be any real time, so time is continuous.
  3. (c) The index closing value can take any positive real value in practice, so treat the state space as continuous. It is observed once per trading day, so time is discrete.

Answer: (a) discrete state space, discrete time. (b) discrete state space, continuous time. (c) continuous state space, discrete time.

Example 2

A process has state space {0, 1}. Let Y_n = 1 if it rains on day n and 0 otherwise. Suppose P(Y_n+1 = 1 | Y_n = 1) = 0.7 and P(Y_n+1 = 1 | Y_n = 0) = 0.2 for every n, whatever happened on earlier days. (a) Is Y a Markov chain? Is it time-homogeneous? (b) Find the one-step transition matrix. (c) If it rains on day 5, find the probability that it does not rain on day 6.

Show the solution
  1. (a) The stated probabilities depend only on Y_n, not on earlier days. So the Markov property holds. The probabilities are the same for every n, so the chain is time-homogeneous.
  2. (b) p_11 = 0.7, so p_10 = 1 − 0.7 = 0.3. p_01 = 0.2, so p_00 = 1 − 0.2 = 0.8. With rows and columns ordered 0 then 1, row 0 is (0.8, 0.2) and row 1 is (0.3, 0.7). Each row sums to 1.
  3. (c) We need P(Y_6 = 0 | Y_5 = 1) = p_10 = 0.3.

Answer: (a) Yes, it is a time-homogeneous Markov chain. (b) The matrix has rows (0.8, 0.2) and (0.3, 0.7). (c) 0.3.

Exam tips

  • In definition questions, write the Markov property as an equation and also in words, with the phrase given the present.
  • When asked to classify a process, always give both the state space type and the time set type. Marks are usually split between them.
  • If a process is not Markov, show why with a concrete dependence on earlier values. Then say how redefining the state could fix it.
  • Check that every row of a transition matrix sums to 1 before using it. This catches arithmetic slips.
  • In Paper B, set out the state space and time-homogeneity assumption in a comment or a line of text before you code a chain.

Practice questions from Markov chains

Stochastic Processes and the Markov Property: frequently asked questions

What is the state space of a stochastic process?

It is the set of all values the random variable X_t can take. For a no claims discount scheme it might be {0%, 20%, 40%}. It can be discrete or continuous.

What is the difference between a discrete-time and a continuous-time process?

In a discrete-time process you observe the process at set times such as t = 0, 1, 2, ... In a continuous-time process the time set is an interval such as t ≥ 0, so the process exists at every moment. This is independent of whether the states are discrete or continuous.

What does the Markov property mean in simple words?

Once you know the current state, the earlier history tells you nothing more about the future. The next step depends only on where you are now. This is often called being memoryless in the history.

Is every stochastic process Markov?

No. A process whose next value depends on several past values, such as a moving average of recent results, is not Markov in the current value alone. You can sometimes make it Markov by enlarging the state to include the needed history.

What does time-homogeneous mean?

It means the transition probabilities depend only on the length of the time gap, not on when it starts. For a discrete chain, p_ij is the same for every step n. A Markov chain need not be time-homogeneous.