World Models for Personal Agents

LLMs are a good interface. They are not a state-transition system.

A personal agent needs to know what is true now, predict what changes if it acts, and stop when the prediction is too uncertain. That layer is what world models are really pointing at.

Yann LeCun’s useful provocation is not “LLMs are useless.” It is sharper: a reliable agent cannot just generate the next message. It has to model the next state.

“I do not understand how you can even think of building an agentic system without … the ability of predicting the consequences of its actions.” — Yann LeCun, Welch Labs interview

That sentence is the whole product requirement.

Memory vs State vs World Model

Memory is what happened. State is what is true now. A world model is what the system expects to happen next.

An agent can remember the right fact and still act on the wrong state. That is why “more memory” does not automatically create agency.

Layer	Question	Example
Memory	What happened before?	“Ray prefers concise PR reviews.”
State	What is true now?	“This branch has two uncommitted files.”
World model	What happens if I do X?	“If I edit this file, the Hugo build should still pass.”
Planner	Which X should I choose?	“Draft first, run Hugo, then ask Claude to proofread.”
Guardrails	Which X is disallowed?	“Do not publish if citations are missing.”
Observability	Did reality match prediction?	“The build failed; update state and recover.”

Agent memory systems handle recall. World models handle the next problem: consequence prediction.

What “World Model” Means

A world model is an internal model of state transitions.

For robots, the state might include positions, forces, objects, collisions, and goals. For a personal AI OS, the state is more boring and more dangerous:

files
branches
calendars
reminders
invoices
inboxes
permissions
drafts
public posts
private commitments
user preferences
irreversible side effects

The model does not need to be mystical. It needs to answer:

state_now + possible_action -> predicted_state_next

Then the agent can ask a second question:

predicted_state_next == acceptable_state?

If not, do not act. Ask, revise, or simulate a different path.

Why JEPA Enters The Conversation

JEPA means Joint Embedding Predictive Architecture.

The short version:

predict missing or future embeddings from context embeddings, not raw pixels, tokens, or frames.

In I-JEPA , the system predicts latent representations of masked image regions instead of reconstructing pixels. V-JEPA moves the same idea into video: predict visual features in latent space, not every future pixel.

LeCun describes the operational version like this:

“You take an observation in the world and then the next observation in the world. You run them through encoders … and then you have a predictor that tries to predict the state at time t+1 from the state at time t, and you might condition this on an action, and now you have a world model.” — Yann LeCun, Welch Labs interview

That is the important shape:

observe_t ──encode──> state_t
observe_t+1 ──encode──> state_t+1
state_t + action? ──predict──> predicted_state_t+1
loss = distance(predicted_state_t+1, state_t+1)

The model is not rewarded for drawing every pixel. It is trained to capture the latent structure that survives the encoder.

Why Latent Prediction Beats Pixel Prediction

Video is brutally multimodal.

If a ball can go left, right, stop, bounce, or disappear behind a cup, a raw pixel predictor has a stupid incentive: average the possible futures. The result is blur. It is not just ugly. It is evidence that the target is wrong.

A latent predictor can ignore irrelevant surface noise and focus on action-relevant structure:

object identity
rough location
motion
contact
affordance
possible next state
constraint violation

That does not make the model automatically intelligent. It gives the planner a better substrate than “generate the next pretty frame.”

For personal agents, the equivalent is not pixels. It is state.

A chat agent that predicts the next sentence but not the next repository state will eventually do dumb shit with confidence.

The Control Stack

For self.md, the useful stack is:

# pseudocode — real implementations vary by agent stack
state = {
    "branch": "draft-world-models",
    "dirty_files": ["content/concepts/jepa-world-models.md"],
    "citations_checked": False,
    "publish_target": "self.md",
    "risk": "medium"
}

action = {"type": "publish", "target": "self.md"}

prediction = predict_next_state(state, action)

if prediction["confidence"] < 0.85:
    escalate_to_human(action, prediction)
elif prediction["risk"] == "high":
    block(action, prediction)
else:
    execute(action)
    observed = observe_state()
    compare(prediction, observed)

The code is fake. The shape is not.

A personal agent needs this loop:

read current state
propose an action
predict the state after the action
block or escalate risky transitions
act
observe reality
compare prediction with reality
update the state model

This is where world models meet agentic loops , agent guardrails , human-on-the-loop , and agent observability .

LeCun’s LLM Critique, Minus The Theater

LeCun is deliberately spicy about LLMs. Some of it is useful. Some of it is a bit sport.

The useful quote:

“They’re really good in domains where the language itself is the substrate of reasoning.” — Yann LeCun, Welch Labs interview

That is a fair boundary. LLMs are excellent when the task lives in language: code, prose, summaries, protocol glue, tool-call planning. They are weaker when the task depends on persistent state, hidden constraints, physical dynamics, or consequences that need to be simulated before action.

So the self.md reading is not:

LLMs bad, world models good.

It is:

LLM = interface + language reasoning
world model = state + transition prediction
planner = search over possible actions
observability = reality check

LeCun puts the final move this way:

“The inference process now becomes a search as opposed to just an autoregressive prediction.” — Yann LeCun, Welch Labs interview

That is the agent jump. Not “what token comes next?” but “what action sequence gets the world into an acceptable state?”

Search requires a cost signal. LeCun’s AMI architecture names this explicitly: a cost module scores proposed states as acceptable or dangerous. The world model predicts the transition; the cost function evaluates it; the planner searches over action sequences that minimize cost within constraints.

For personal agents, that cost is concrete. “Does the PR pass CI?” “Was this email approved before sending?” “Are the citations verified?” Those are cost functions. They are also the difference between a reactive tool and something that can be trusted with irreversible actions.

Why Self-Supervised Learning Matters

LeCun’s cake metaphor explains the training bias:

If intelligence is a cake, self-supervised learning is the cake, supervised learning is the icing, and reinforcement learning is the cherry.

Labels and rewards are sparse. Observation is dense. Human beings learn a huge amount by watching the world and predicting what will happen next.

That is why representation learning matters here. Before JEPA became the headline, the field had to fight representation collapse: the failure mode where a model maps everything to the same useless embedding.

Several methods attacked that problem:

Work	Useful idea
Barlow Twins	reduce redundancy by pushing a cross-correlation matrix toward identity
VICReg	preserve variance, enforce invariance, reduce covariance
DINO	self-supervised ViTs learn useful visual structure without labels
DINOv2	scaled self-supervised vision features become broadly reusable

Do not read this as one clean family tree. It is messier than that. The shared direction is enough: learn useful representations without hand labels and without reconstructing every surface detail.

What Breaks Without A World Model

Most agent failures are not pure language failures. They are state failures.

The agent acts as if:

the file still exists
the branch is clean
the approval was granted
the calendar slot is open
the user meant the same thing as yesterday
the tool did what it claimed
the world did not change between plan and execution

One concrete version: the agent sends a calendar invite, marks the step done, then composes the follow-up email with the confirmed time. The invite was rejected. The agent has no world model — it compared nothing. The email goes out with the wrong date. The failure is not language. It is unobserved state.

That is why agent failure modes show up as hallucinated completion, stale context, bad tool assumptions, and silent integration failure.

A world-model layer makes the failure explicit. It asks:

what state am I acting on?
what state do I expect after the action?
how confident am I?
what would prove me wrong?
when should I ask the human?

The answer does not have to be a giant neural video model. For a personal OS, it can start as boring infrastructure: state schemas, event logs, simulators, typed tool outputs, confidence thresholds, rollback paths, and tests.

The model can be learned later. The state-transition discipline has to exist now.

Builder Pattern

Start without magic:

define the state your agent is allowed to act on
log every action and observed result
write a predicted state before execution
require confidence on risky transitions
escalate low-confidence or irreversible actions
measure prediction mismatch rate
treat mismatches as training data

A minimal state-transition record can be this plain:

before:
  branch: main
  dirty_files: 0
  target: content/concepts/jepa-world-models.md

action:
  type: edit_file
  intent: expand_article

prediction:
  dirty_files: 1
  build_should_pass: true
  broken_links: 0
  risk: low

observed:
  dirty_files: 1
  build_passed: true
  broken_links: 0

mismatch:
  none

This is not glamorous. It is the difference between an agent and a roulette wheel with Markdown skills.

What It Is Not

World models are not an LLM replacement.

LLMs are still useful for language, planning surfaces, code, explanation, and tool calls. A world model does a different job: it predicts action-relevant state transitions.

World models also do not solve hallucinations by magic. They can still be wrong. The difference is that the system has a place to check the prediction against reality.

V-JEPA 2 reports self-supervised video models that combine large-scale video pretraining with a smaller amount of robot trajectory data for understanding, prediction, and planning. V-JEPA 2.1 pushes dense video features further. LeWorldModel explores stable end-to-end JEPA-style world models from pixels.

Good research. Not a permission slip for blind autonomy.

The self.md Takeaway

If you cannot name the state, you do not have a world model.

If you cannot predict the transition, you do not have planning.

If you cannot compare prediction with observation, you do not have reliability.

You have a chat log wearing a lab coat.

Sources

Next: Yann LeCun