the creepy line: where ai privacy boundaries should be

Table of content

the original line

eric schmidt, former google CEO, once said: “google policy is to get right up to the creepy line and not cross it.”

the creepy line is the point where a technology stops feeling helpful and starts feeling invasive. where convenience becomes surveillance. where personalization becomes manipulation.

google crossed it years ago. so did facebook, amazon, and apple.

AI tools are crossing it faster.

where the line is

it’s not the same for everyone. what feels creepy depends on: → cultural norms — some cultures expect privacy, others expect community
→ personal history — people who’ve been surveilled are more sensitive
→ trust in institutions — if you trust the company, you tolerate more
→ perceived benefit — the more value you get, the more access you’ll grant

but there are common patterns.

not creepy:
→ AI summarizes a document you explicitly uploaded
→ AI answers questions based on context you provided
→ AI remembers preferences you set

creepy:
→ AI reads your emails without asking
→ AI listens to your conversations and surfaces insights you didn’t request
→ AI makes decisions on your behalf without transparency

the line is somewhere between “you control what it sees” and “it sees everything by default.”

most AI tools use opt-out consent.

the default is: we collect everything, you can disable it in settings (if you can find them).

this is backwards. consent should be opt-in: → by default, the AI has no access
→ you explicitly grant permissions
→ permissions are granular and revocable

right now, installing an AI assistant is like giving someone full access to your house and hoping they only look at the rooms you wanted to show them.

the ambient surveillance trap

ambient AI requires persistent access to be useful. screencast tools, voice assistants, always-on agents — they watch everything.

this is fine if: → data stays local — never leaves your device
→ you control what gets recorded — explicit boundaries, not blanket access
→ you can audit and delete — inspect what was captured, purge it anytime

most ambient AI tools fail at least one of these.

the third-party problem

you give an AI tool access to your data. the AI tool is owned by a company. the company: → trains models on your data — even if “anonymized”
→ shares data with partners — for analytics, ads, integrations
→ gets acquired — now a different company owns your data
→ gets hacked — your data leaks

you can’t trust the creepy line to stay where it was when you signed up. companies move it.

the only real control: don’t give them the data in the first place. use local models, local storage, local execution.

the inference problem

even if the AI doesn’t have direct access to something, it can infer it.

you don’t tell the AI your salary. but it sees: → where you work
→ your job title
→ the neighborhood you live in
→ your spending habits

it can infer your salary with decent accuracy. and your political views. and your health issues. and whether you’re likely to quit your job.

this is shadow data — information derived from information you shared. you never consented to the inference, but it happened anyway.

there’s no good defense against this except limiting what the AI can see in the first place.

the screencast dilemma

tools like Rewind, Granola, and Otter record your screen and audio. they make everything searchable.

incredibly useful. also incredibly invasive.

questions no one’s answering: → what about passwords? — most tools have blocklists, but they’re not foolproof
→ what about video calls? — are you recording the other person without their consent?
→ what about private browsing? — if you open an incognito tab, should the screencast tool respect that?
→ what about medical/legal/financial data? — should these be auto-excluded, or is it on you to disable recording?

the default should be: recording off, explicitly enabled for specific apps/contexts. instead, the default is: recording on, disable if you remember.

the notification boundary

some AI tools send you proactive suggestions or reminders.

“you haven’t followed up on that email.”
“this meeting conflicts with another one.”
“your code has a bug in this function.”

helpful? yes. but also: the AI is monitoring you and judging your behavior.

there’s a creepy line between “helpful reminder” and “surveillance with suggestions.”

the fix: transparency + control. the AI should tell you what it’s monitoring and let you disable it per-category. not “all notifications on/off,” but “email reminders: on, code suggestions: off, calendar conflicts: on.”

the voice assistant problem

voice assistants (alexa, siri, google) are always listening. they only “activate” on wake words, but the microphone is always on.

this means: → they hear everything — even if they “don’t record” it
→ false activations happen — they think they heard the wake word, start recording
→ accidental recordings — you say something that sounds like the wake word, it captures 10 seconds of conversation

the creepy part: you can’t audit what was heard but not recorded. you only see what made it to the cloud.

local-first voice processing (on-device wake word detection, no cloud until you opt in) would fix this. most companies don’t do it because cloud = data = training material = competitive advantage.

the agent-to-agent creep

future scenario: your AI talks to someone else’s AI to coordinate a meeting.

your AI shares: your availability, location preferences, dietary restrictions.
their AI shares: same.

now both AIs know things about you that you didn’t explicitly tell the other person.

this is transitive disclosure — sharing data via intermediaries. happens in human conversation too (“oh, sarah mentioned you’re vegan”), but AI will do it at scale, automatically, without you noticing.

consent models don’t account for this yet.

the memory decay question

should AI memory be permanent or ephemeral?

some argue: everything should be remembered. perfect recall, infinite history, never forget.

others argue: memory should decay. what you did 5 years ago shouldn’t define you today. let old data expire.

right now, most AI tools default to permanent. once recorded, it’s in the database forever (unless you manually delete it, and even then, maybe not really).

human memory fades. we forget things. this is a feature, not a bug. it lets us change, move on, not be haunted by old versions of ourselves.

AI memory doesn’t fade. should it? and if so, how fast?

most privacy laws (GDPR, CCPA) were written for databases and ad networks, not AI agents.

they give you rights: → right to access — see what data is stored
→ right to deletion — delete your data
→ right to portability — export your data

but AI introduces new problems: → right to de-training — if a model trained on your data, can you remove your influence?
→ right to explanation — if an AI made a decision about you, can you know why?
→ right to inference limits — can you prevent the AI from inferring things you didn’t share?

the law hasn’t caught up. until it does, you’re relying on companies to self-regulate. good luck.

the transparency fix

one way to make the creepy line clearer: show what the AI sees.

a dashboard that lists: → every data source the AI has access to
→ every inference it made about you
→ every third party it shared with
→ every action it took on your behalf

most tools hide this. not because it’s technically hard, but because showing it would scare users.

“wait, you’ve been reading my emails for 6 months?”

yes. you clicked “agree” on the EULA. did you read it? no. would it have mattered? probably not.

transparency alone doesn’t fix consent. but it helps.

the local-first answer

the only way to avoid crossing the creepy line: don’t send data to the cloud in the first place.

→ run models locally (Llama, Mistral, Whisper)
→ store data on-device (SQLite, local embeddings)
→ sync peer-to-peer if needed (no central server)

this is technically feasible. it’s just slower, more expensive to build, and harder to monetize.

companies default to cloud because it’s easier and more profitable. users tolerate it because they don’t know there’s an alternative.

local-first AI is the future. just not the one most companies are building toward.

the illich frame

Ivan Illich distinguished between tools and manipulative institutions.

tools enhance autonomy. you control them.
institutions create dependency. they control you.

most AI systems are institutions, not tools. they require: → constant internet connection
→ subscription fees
→ vendor lock-in
→ trust in the company’s good behavior

a tool would: → work offline
→ store data locally
→ let you switch providers without losing history
→ be transparent and auditable

we have the technology to build tools. we mostly build institutions.

questions worth asking

Topics: privacy boundaries consent surveillance ethics