Preference Learning: AI That Adapts to You

Table of content

Standard AI alignment optimizes for population averages. RLHF and DPO train models to satisfy most users most of the time. Your specific preferences for tone, reasoning style, verbosity, and formatting get averaged away.

Preference learning fixes this. Instead of one-size-fits-all responses, the AI infers what you want from your interactions and adapts without explicit configuration.

Why Population-Level Alignment Fails

Current alignment methods treat users as interchangeable:

What alignment optimizes	What you need
Average user preferences	Your specific style
Safety for all edge cases	Your risk tolerance
General helpfulness	Your domain expertise
Consistent persona	Your preferred interaction mode

The gap shows up immediately. You want terse code reviews; the model gives verbose explanations. You want direct answers; it hedges with disclaimers. You want technical depth; it explains from first principles.

Per-user fine-tuning could solve this, but it’s computationally expensive and doesn’t scale. What works: learning preferences on the fly from interaction patterns.

The POPI Framework

POPI (Personalizing LLMs via Optimized Natural Language Preference Inference) introduces a preference inference layer between you and the model.

The trick: distill your interaction history into a compact natural language preference summary. This summary conditions the generation model without per-user fine-tuning.

User signals      →  Preference LLM  →  Summary  →  Generation LLM
(history, clicks,    "Prefers terse     ↓            Personalized
 corrections)        responses,         Injected     response
                     technical depth,   into prompt
                     no hedging"

POPI shrinks user context from thousands of tokens to tens while maintaining or improving personalization accuracy. On benchmarks, it achieves +20-30% win rate over raw prompting approaches.

The summaries stay transparent. You can inspect what the system learned about you:

Inferred preferences:
- Tone: Direct, no pleasantries
- Code style: Functional, minimal comments
- Explanations: Skip basics, focus on edge cases
- Format: Bullet points over paragraphs

Memory Systems for Preference Tracking

Preference learning requires memory that persists across sessions. Three approaches:

Explicit Memory Extraction

Mem0 extracts key facts from conversations and stores them in a dedicated memory layer:

from mem0 import MemoryClient
client = MemoryClient(api_key="...")

# Add interaction to memory
client.add([
    {"role": "user", "content": "Always use TypeScript, not JavaScript"},
    {"role": "assistant", "content": "Understood. I'll use TypeScript for all code examples."}
], user_id="user_123")

# Retrieve relevant memories for new context
memories = client.search("code style preferences", user_id="user_123")
# Returns: "User prefers TypeScript over JavaScript"

Mem0 reports 26% accuracy improvements over baseline approaches by selectively storing and retrieving conversational facts.

Graph-Based Relationships

A-MEM (Agentic Memory) organizes preferences as interconnected knowledge graphs:

preferences/
├── coding/
│   ├── language: TypeScript
│   ├── style: functional
│   └── testing: property-based
├── communication/
│   ├── tone: direct
│   └── format: bullets
└── domains/
    ├── frontend: advanced
    └── databases: intermediate

Graph structure captures relationships between preferences. “Prefers TypeScript” links to “uses React” links to “favors hooks over classes.”

Compression-Based Memory

Alex Newman’s claude-mem treats preference storage as a compression problem:

Capture everything Claude observes during a session
Compress observations using AI summarization
Inject relevant compressed context into future sessions

The progressive disclosure pattern saves tokens:

search       →  Compact index (~50-100 tokens/result)
timeline     →  Chronological context
get_details  →  Full observation (~500-1000 tokens/result)

Benchmarks

The LaMP benchmark tests how well models adapt to individual users across seven tasks: sentiment analysis, headline generation, product review writing, and more.

LaMP results:

Approach	Average improvement
RAG-based personalization	14.92%
Per-user fine-tuning (PEFT)	1.07%
RAG + PEFT combined	15.98%

Retrieval-augmented personalization outperforms fine-tuning by a wide margin. The bottleneck isn’t model capacity; it’s getting relevant user context into the prompt.

The Dynamic Evaluation Framework extends this to multi-session scenarios. It models user personas with evolving preferences and tests whether agents can track changes over time.

Implementation

A minimal preference tracker:

import json
from pathlib import Path

class PreferenceTracker:
    def __init__(self, user_id: str):
        self.path = Path(f"~/.prefs/{user_id}.json").expanduser()
        self.path.parent.mkdir(exist_ok=True)
        self.prefs = self._load()

    def _load(self) -> dict:
        if self.path.exists():
            return json.loads(self.path.read_text())
        return {"explicit": {}, "inferred": [], "history": []}

    def save(self):
        self.path.write_text(json.dumps(self.prefs, indent=2))

    def add_explicit(self, key: str, value: str):
        """User-stated preference: 'I prefer TypeScript'"""
        self.prefs["explicit"][key] = value
        self.save()

    def add_interaction(self, prompt: str, response: str, feedback: str = None):
        """Store interaction for later inference"""
        self.prefs["history"].append({
            "prompt": prompt,
            "response": response,
            "feedback": feedback  # thumbs up/down, correction, etc.
        })
        self.save()

    def infer_preferences(self, llm) -> str:
        """Use LLM to extract preference patterns from history"""
        if not self.prefs["history"]:
            return ""

        history_sample = self.prefs["history"][-20:]  # Recent interactions
        prompt = f"""Analyze these interactions and extract user preferences.
        Focus on: tone, verbosity, technical depth, formatting, domain expertise.

        Interactions:
        {json.dumps(history_sample, indent=2)}

        Output a concise bullet list of inferred preferences."""

        inferred = llm.complete(prompt)
        self.prefs["inferred"] = inferred
        self.save()
        return inferred

    def get_system_context(self) -> str:
        """Generate context to inject into prompts"""
        parts = []
        if self.prefs["explicit"]:
            parts.append("Explicit preferences:")
            for k, v in self.prefs["explicit"].items():
                parts.append(f"- {k}: {v}")
        if self.prefs["inferred"]:
            parts.append(f"\nInferred from interactions:\n{self.prefs['inferred']}")
        return "\n".join(parts)

Usage:

tracker = PreferenceTracker("user_123")

# Capture explicit preferences
tracker.add_explicit("code_language", "TypeScript")
tracker.add_explicit("explanation_style", "skip basics")

# Track interactions
tracker.add_interaction(
    prompt="How do I sort an array?",
    response="[detailed explanation]",
    feedback="too verbose"
)

# Periodically infer patterns
tracker.infer_preferences(llm)

# Inject into new conversations
system_prompt = f"""You are a helpful assistant.

{tracker.get_system_context()}

Respond according to these preferences."""

Preference categories

What actually matters:

Category	Examples	Impact
Communication style	Tone, verbosity, formality	Every response
Technical level	Expertise assumptions, depth	Explanations
Output format	Bullets vs prose, code comments	Structure
Domain context	Industry, role, tools used	Relevance
Interaction mode	Socratic vs direct, exploratory vs task-focused	Conversation flow

Start with communication style and output format. These affect every interaction and are easy to infer from corrections.

Privacy

Preference tracking creates a detailed profile of how you think and work. Protect it:

Risk	Mitigation
Profile leakage	Store preferences locally, never in cloud
Over-inference	Let users audit and delete inferred preferences
Manipulation	Don’t let inferred preferences override explicit ones
Lock-in	Export preferences in standard format

Khoj’s approach offers a model: open source, local-first, user-controlled. Your preferences should be portable across AI systems, not locked into one vendor.

Combining with personal search

Preference learning complements personal search. Search retrieves what you wrote; preference learning adapts how the AI responds to it.

Together:

def personalized_query(question: str, user_id: str):
    # Get relevant personal documents
    docs = personal_search.search(question, limit=5)

    # Get user preferences
    prefs = PreferenceTracker(user_id).get_system_context()

    prompt = f"""User preferences:
{prefs}

Relevant documents:
{docs}

Question: {question}

Answer according to user preferences, citing documents."""

    return llm.complete(prompt)

The search provides domain knowledge. The preferences shape delivery.

The payoff

Preference learning moves AI from tool to environment. The system remembers how you work. Each interaction refines its understanding. Less correction, more flow.

This isn’t “remember my name” personalization. It’s “understand my thinking patterns” personalization. The difference matters for building a personal operating system. Your AI shouldn’t require constant instruction. It should learn your defaults and get out of the way.

Next: Personal Search