Home / Projects

POMDP as a Proxy for Humans Making Life Decisions

How partially observable Markov decision processes are a fitting system to understand and model human behavior. Writing, 2026.

1. Humans are complex, so are their environments

Theoretically, this should create tremendous possibilities, but people can moderately be described as a product of their environment. In a way, human biology and society¹ force us to strongly react to the aspects of our environment that relate to our odds of survival, either directly (food, shelter) or indirectly (status).

2. Why humans as state machines make partial sense

The main drivers of human behavior look something like:

• Motivation: biological and social drivers.
• Energy: your battery, though not linear and not fully charged for every task.
• Personality: compass, morals, and innate neural pathways (nature and nurture).²
• Past reinforcement per task: gets stronger the narrower the task is.³

Given such factors, if we know the current state and the potential next states, the next state becomes fairly predictable. If I am energized and want to take on a difficult task I am interested in, I do it. Same task when I am tired, I don't.

An MDP also adds choice, whereas a state machine makes it sound like humans are solely pushed around by their environment.

3. Life pushes back: why a Markov⁶ decision process beats a state machine

• Probability: we don't know exactly what will happen, though we typically have an idea of the main possible outcomes.
• States: the set of conditions described above, and more.
• Action set: the paths we are considering from the current state.
• Rewards: the potential benefit (and punishment) from our actions.

By expanding the analogy from state machine to MDP, we get a better diagram of a human proxy.

4. Life also confuses: why a partially observable MDP is more accurate than an MDP

You can't observe the exact details of the situation you are in. As a human you are always deciding in partial darkness:

• Other people's thoughts and motives.
• Unknown facts about the world, other parties, ongoing developments.
• So you develop guesses and correct them as you go, mostly subconsciously.

We don't act on reality directly, but on what we take to be reality from our perception (loosely Kantian), and reconcile as we go. In a POMDP this is the belief state: you can't know your true state, but you maintain an estimate of it.

One interesting insight from Ilya Sutskever is that innate emotions are a useful heuristic for fast, accurate decisions in a system where we lack both self-knowledge and knowledge of the outside: "the value function of humans is modulated by emotions in some important way that's hardcoded by evolution and maybe that is important for people to be effective in the world." This may help explain how humans succeed in complex environments, while making the task of modeling them markedly harder.⁴

A counterargument: in an MDP, mathematically, the only thing that matters for the next step is the current state; any past step is irrelevant. But the current state doesn't have to be a person dropped into a situation. It can just as well include how the person was shaped by their past, carrying the past within the person. I am of the opinion that we live in a hauntological⁵ reality, where life is like a song in which the current note only makes sense in relation to what you just heard and what you expect to hear next.

5. Why any of this matters

We went from a state machine, which describes the broad state a human is in and what changes it, to MDPs, where we consider actions and their consequences, and arrived at POMDPs, where we introduce uncertainty about the current situation itself.

If we ever try to model human behavior in a given setting, this is a good place to start: a low-fidelity playground for figuring out which variables are worth including.

Notes

1. Values hardcoded in humans, like sharing and group survival, shape how society forms, which in turn shapes how humans develop in ways that are also hardcoded by biology. Shame, for instance, is a mechanism for staying in line with group morals to increase odds of survival. Biology dictates how we build societies and interact with them as individuals, which in turn feeds back into society. Perhaps many cross-cultural commonalities, like the concept of family, can be attributed to this. ↩
2. Extroversion, for example, is detectable in the womb by reaction to light. ↩
3. If you are learning a new card game and just learned that a move is "good", you are very likely to repeat it. The wider the environment variables, the less predictable and recallable this becomes. ↩
4. The patient who lost his emotional cortex could complete almost all tasks as before, but struggled with simple decisions like which socks to wear. ↩
5. Derrida on hauntology; also temporal consciousness. ↩
6. What got me into Markov chains is this sentence, generated by Claude Shannon in 1948: THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED. ↩

Home Readings Poems Projects