hidden markov model for regime detection

A hidden Markov model treats the market as switching between a few unobserved states, each with its own return and volatility profile, and learns how likely the market is to stay in a state or move to another. That persistence is the reason to use it over a volatility threshold: regimes last weeks or months, and the model carries that assumption instead of flipping on every noisy day. The one I run in production has three states (bull trend, bear trend, sideways), fitted with hmmlearn's GaussianHMM on three scaled features, and it outputs a label plus a probability every trading day.

hidden markov model market regime detection trading strategy

In my system the regime never opens or closes a position by itself. It scales position size on hedges, and only when the model is confident: below a 60% state probability the multiplier is 1.0 and the regime is ignored. Above it, the ETF book uses 0.9 in a bull trend, 1.2 sideways and 1.3 in a bear trend, because hedges are worth more when the market is falling or going nowhere. Keeping it a multiplier rather than a switch is what made it safe to run unattended.

regime detection models

The two I have used seriously are hidden Markov models and clustering on Wasserstein distance between return distributions. The HMM gives you transition probabilities and a state probability for free, which is what a live system needs to decide whether to trust today's label. Wasserstein clustering is better at finding regimes you did not name in advance. What decided it for production was not accuracy but plumbing: the HMM retrains in seconds, scores in milliseconds, and its probability output plugs straight into a confidence gate.

hidden markov model market regime detection finance paper

The starting point is Hamilton (1989), "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle", Econometrica 57(2), which introduced regime-switching models to finance. For the clustering alternative, Horvath, Issa and Muguruza (2021), "Clustering Market Regimes Using the Wasserstein Distance", arXiv:2110.11848, later in the Journal of Computational Finance. Neither paper tells you how to use the output in a live book, which is the part this post covers.

Market regime detection in production: what the model actually changes

A while ago I wrote up two ways to detect market regimes: hidden Markov models and clustering on Wasserstein distance. That post was research on a toy S&P 500 series. This one is what actually runs, every trading day, inside the trading system behind Ansaar.

The short version: the model turned out to be the small part. What made it safe to run without me watching was everything around it.

Three states, three features

The model is a GaussianHMM from hmmlearn with three states. I tried more. The Bayesian information criterion kept picking three, and three is also the number a human can act on.

State	What it looks like
Bull trend	Positive returns, moderate volatility
Bear trend	Negative returns, elevated volatility
Sideways	Near-zero returns, volatility all over the place

It sees three features, and only three:

ret_5d, the 5-day log return
realized_vol_21d, the 21-day rolling standard deviation, annualised
return_vol_ratio, the first divided by the second

I had a longer list at one point. Every feature I added made the fit look better in-sample and the labels worse out of it. Three features is enough to separate "going up calmly" from "going down violently" from "going nowhere", and that is the whole job.

The bug that scaling fixed

The first version fed those three features to the model raw. It trained fine, the states had names, and then I looked at the state statistics and the bear trend had a positive average return.

The cause was scale. return_vol_ratio moves over a much wider range than a 5-day return, and unscaled it had about 19 times the influence of ret_5d. The model was clustering on the ratio and mostly ignoring the return. The tell was March 2020: the COVID crash, the most obvious bear regime in the training window, was classified as sideways.

A StandardScaler in front of the model fixed it, and the verification is the part I keep:

March 2020: 100% of days classified as bear trend
23 March 2020, the worst single day: bear trend, a -13.93% return at 70.3% volatility, with 100% state probability

from hmmlearn.hmm import GaussianHMM
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X = scaler.fit_transform(features[["ret_5d", "realized_vol_21d", "return_vol_ratio"]])

model = GaussianHMM(n_components=3, covariance_type="full", random_state=42)
model.fit(X)

# Keep the scaler with the model. Scoring unscaled data later is the same bug again.

If you take one thing from this post, take that: check the model against a day you already know the answer to.

Train on four indices, not one

A regime is a property of the market, not of one index. So the model trains on Nifty 50, Bank Nifty, the IT index and the Pharma index together, with a seven-year lookback. That gives about 14,900 training rows instead of the roughly 3,700 a single index would.

It matters more than it sounds. With one index the model has seen a handful of bear regimes. With four it has seen the same bear regimes expressed four different ways, which is closer to what "bear" means.

The learned transition matrix comes out strongly diagonal: whatever state the market is in today, it is most likely in the same state tomorrow. That is the reason to use an HMM rather than a volatility threshold. A threshold flips on any noisy day. The model has been told, in its structure, that regimes persist, and it estimates how much.

Crypto gets its own model. Its regimes do not line up with equities and pretending they do just adds noise to both.

Retrain monthly, score daily

The model retrains on the first of every month at 6 AM. Classification runs daily at 2:30 AM, over the previous day's consolidated index data, and writes a label and a probability for that day. Scoring after the data is settled rather than at the close is deliberate: a label computed from a half-built row is worse than no label.

The state statistics from the current fit, which is what I sanity-check after every retrain:

State	Average daily return	Average volatility
Bull trend	+0.53%	17.5%
Bear trend	-0.39%	28.2%
Sideways	+0.01%	14.8%

If a retrain ever produces a bear state with a positive return again, I know before anything downstream does.

The rule that matters: ignore the model below 60%

This is the part that is not in any paper. The regime label is only allowed to change a position size when the model's probability for that state is at least 0.6. Below that the multiplier is 1.0 and the regime does not exist as far as the book is concerned.

Above it, each domain has its own multipliers. For the ETF book, which is mostly hedges:

REGIME_ADJUSTMENTS = {
    "BULL_TREND": 0.9,   # less hedging needed
    "SIDEWAYS":   1.2,   # hedges earn their keep
    "BEAR_TREND": 1.3,   # hedges are the point
}

def regime_multiplier(label, prob, adjustments=REGIME_ADJUSTMENTS):
    if prob < 0.6:
        return 1.0
    return adjustments.get(label, 1.0)

So a confident bear call scales hedge sizing up 30%. A confident bull call scales it down 10%. That is the entire effect the model is permitted to have. It never decides what to hold. It nudges how much, in one direction, on one part of the book.

I want to be honest about why it is this small. A regime model is right often enough to be useful and wrong often enough that giving it a switch would hurt. A multiplier with a confidence gate captures most of the value and caps the damage. The first version was a switch. It did not survive its first false bear call.

What the regime is not allowed to do

Three more guard rails, each of which exists because something went wrong without it:

It reads the last available label, not today's. On a holiday there is no label for the date, and an exact-date lookup returned nothing, which the sizing code read as "no regime", which was silently the 1.0 path. The lookup is now as-of: the most recent label on or before the date. A missing day should look like the last known state, not like calm.

It is not the risk manager. Volatility limits come from India VIX through a separate path. The regime model scales hedges. The risk manager caps exposure. If the two disagree, the cap wins.

A human sees every flip. The daily newsletter prints today's regime next to yesterday's. Most days they match and nobody reads that line. On the day they differ, it is the first line anyone reads.

If you are building one

Fewer features than you think. Three was enough. Ten was worse.
Scale them, and keep the scaler with the model.
Train across instruments, not on one.
Check the fit against a day you already know the answer to. March 2020 is a good one.
Gate on probability. Below your threshold, the model does not exist.
Make the output a multiplier, not a switch.
Retrain on a calendar, not on a feeling.

None of this is clever. It is the difference between a model that looks good in a notebook and one you can leave running.

The theory and the code for both approaches are in the earlier post. The system this runs inside is described on the Ansaar page, and the pipeline underneath it on /systems/.