Yesterday Apple announced a big step towards deploying real AI in their Siri ecosystem. The deal describes a partnership with Google to inject that company’s advanced LLM models into Siri. In some ways this is good and inevitable: Siri is one of the world’s most preeminent voice agents, and it would probably be good if it didn’t suck. The idea that Apple would boost its capabilities with models from a frontier LLM lab wasn’t so much a matter of “if”, but a question of “when” and “who“.

The who turns out to be Google: Apple looks like it will use some combination of Google Gemini models, combined with Google’s Confidential Inference and Apple’s own Private Cloud Compute for private hosting. These systems will process both your questions and evaluate private data from your devices. Apple pitches the advantages as follows:

1. First, since your phone already has context about you — i.e., your private information, schedules, email, text messages — Google-Siri can potentially offer much more useful and personalized answers to your practical questions than other LLMs. Want to schedule a reservation for next week’s birthday party? In theory, a future Siri-AI might know who’s coming, and what cuisine they like.
2. Of course, what Apple calls “context” is the raw data of your life. It can’t just be shipped to random adtech companies (or Sam Altman) for processing. This data needs to be protected, and Apple is a privacy company.

Apple has addressed this apparent contradiction with a service it calls Private Cloud Compute, or PCC. PCC was introduced in 2024 as a private model inference system that ran entirely on Apple Silicon, using a set of “trusted” hardware security modules running in Apple’s datacenters. The goal of this system is to ensure that your data never leaves Apple’s hardware: it’s encrypted from your phone to the server, and then it disappears once a response reaches your phone. This ensures (in principle) that even Apple can’t see what you’re doing with it.

Apple has since “expanded” PCC to encompass Google’s hardware as well. I will confess that I find the details of the new “expanded” PCC just a tad vague. It sounds a lot like Apple is really just going to rely on Google’s existing confidential compute (running in Google datacenters) to process this data, but they’re bolting on a new layer of technical security to control which software is actually running. In any case, this is all fine. Security experts can argue about whether this is good enough to keep Cozy Bear away from your data. What I will grant is that it’s certainly good enough to keep Google and Apple from accessing your stuff, which is what most people are worried about in the first place.

So why am I so nervous?

Private inference is nice, but to be useful, agents need to talk to people.

Let me walk you through the future of personal agents. Or rather, not the future. But one possible future that you might experience over the next couple of years. To illustrate how agents might work, it’s helpful to consider an example use case.

Let’s imagine that you’re planning a business dinner for six people. This involves several subtasks:

1. You need to juggle the participants’ schedules, know when they’re in town and available to meet.
2. You need to choose the appropriate restaurant based on menu and location. This might depend on what you know about the participants’ preferences: Mike is wildly allergic to szechuan peppercorn, for example, which rules out many options.
3. With these time/cuisine/location constraints in place, you’ll need to search for a restaurant that actually has a table for six in the right place.
4. Finally, you’ll need to book the reservation, mark your calendar, and alert your attendees.

In the past, this type of scheduling required a significant amount of human effort. The beauty of AI agents is that, in theory, this is exactly the sort of project that can be automated. The agent can scan your recent conversations to answer the questions of steps (1) & (2), then perform the searches in step (3). With a nod from you, it can even author the calendar invites and text messages required to complete step (4).

So what’s the problem here?

A first observation is that being really useful on (1) requires your agent to have context, which means: relatively unrestricted access to your private data. You know about your invitees’ availability because they texted it to you. You know about Mike’s allergy because you’ve talked about it with him or jotted it down somewhere. (This could mean iMessages, email, contacts, or personal notes.) Re-entering all of this data into an agent would be annoying and time consuming and the whole point of an agent is to save you time. The winning personal assistant doesn’t win just because it’s smart: it wins because it “already knows” the things you need it to know, like a personal assistant who sits next to your desk.

For example: the agent might scan your messages database to learn the parameters needed to schedule your dinner. Or, in a more token-efficient system, it might read your messages continuously and store a “memory” that distills useful facts that it might need later. Both are functionally equivalent, but one produces an artifact that may be highly sensitive. And keep in mind that the set of facts that might be useful is very broad. For example, Mike’s allergy. And perhaps that other conversation you had where you discovered that Mike was having an affair. Memory or not, this data will all be within the agent’s view, and you’ll have to hope that it doesn’t share it.

Whatever it contains, this intensely private data is leaving your device. The agent (which is really an LLM running on a server in a data center somewhere) wil read (conduct inference on) some prompt and the data, and ship back a result. At least so far this is mostly safe. The entire purpose of Apple’s PCC and Google’s Confidential Inference is ensure that this data, and any results from inference, is kept private. The inputs and outputs should be wiped as soon as inference is done, and the only remaining copy of any of it should exist on your phone.

And so far I find this to be a compelling story, as long as you never plan to do anything else beyond inference.

The problem with agents that can actually do things

The problem is obvious when you consider our scenario above. Of course we need to do things beyond simple inference. Private inference alone gives us very little capability. Imagine a personal assistant who can see read private files, but is otherwise locked in a windowless room with no Internet access and no phone. Your data is perfectly safe, but your assistant is worthless for all but the simplest tasks: for example, summarizing inbound messages for your consumption, or helping draft text messages. (In short, what Apple Intelligence does today.)

Now picture a personal assistant who can actually get things done. This assistant will need Internet access: at minimum the ability to query search engines, or more realistically, search LLMs like Gemini or ChatGPT. Ideally you’ll expect them to schedule public calendar invites and draft messages to share with your contacts. This assistant is now useful, but the beautiful guarantees of “no private data leave your device” are also gone. The privacy of your data now depends on your assistant’s discretion and judgement.

Moving back to our business dinner: to accomplish step (3) your agent will likely need to visit an open search engine, perhaps asking it as many as a dozen queries, each of which leaks some information about your specific requirements. The nature of the data leakage really depends on how cautious the “private” agent is in authoring its external queries. A very reasonable case would be for the model to simply collect a series of known facts, and upload them to a more powerful “open” LLM like Gemini, ChatGPT or Claude, as follows:

“Hey, LLM search engine, here is a list of thirty detailed facts about my attendees and the purpose of this meeting, find me a restaurant that works for everyone.“

Notice that private inference does nothing to prevent this data from leaving your system, and the system is largely useless if these searches can’t be performed. Put differently: private inference can work perfectly, and still useful (and monetizable) data can still flow outward to a public search engine or LLM, simply because the agent was programmed to do its job well.

In short: the privacy guarantees these systems have to offer are less related to how Nvidia or Apple design their silicon, and everything to do with how the model is programmed (prompted) to protect your data. Who do you think will handle that programming?

Ok, so search engines will probably learn some private data. So what?

You probably don’t care very much if a search engine learns that Mike has an allergy. But there are things you really do care about. In security parlance, they both have to do with different adversaries.

Let’s consider the most obvious “adversary”. Imagine you’re Mark Zuckerberg or Sundar Pichai, or whoever runs Apple’s advertising business. You have billions of users with piles of deeply useful data stored on their phones. This data is incredibly valuable for targeted advertising, something that is about to become wildly more lucrative thanks to generative AI. At the same time, a big chunk of this data is inaccessible, simply because users don’t love the idea of you scanning their private conversations. And so you might have access to email, public social media, or browsing data. But not private text message conversations, the trillions of WhatsApp or iMessage messages, the interactions with other companies’ services (Meta can’t read your Google data, and vice versa).

Now imagine deploying an agent to users’ phones. That agent will see all that data. It’ll have access to everything the user does. To do its job, it will literally need to divine each user’s preferences and then operationalize them into queries that will repeatedly hit your search engine or “search LLM”. Whoever operates this search engine will learn a vast amount of useful information about the users’ desires, some of which will come from the most intimate private conversations. These conversations may even have occurred years in the past, so that the user will have forgotten about them entirely.

My point here is not that our data is necessarily about to be mined, simply that intention matters a lot here. The real threat is not “how well does private inference work”, but instead: how carefully does my private agent model (and prompting) care about leakage? A non-privacy-forward agent will be an advertiser’s dream: it could upload reams of raw memory-facts and let the open search engine figure out what to do with them. The question of which agent you’ll get mainly depends on the interests of its designers. These may or may not intersect with your interests.

If your agent can talk to people, then strangers may talk to it.

Some folks will shrug at the threat of Google learning more about their interests. While I don’t subscribe to this resignation, I find it hard to argue with. From the public side, at least, Google has been a reasonably good steward of our data. To my knowledge, there have been no major data breaches where our most intimate Google searches were dumped all over the Internet (in the style of AOL‘s search breach.) The company deserves a lot of credit for this.

So maybe you don’t care about your search engine or a social media company poking through your data for useful advertising data. Still, this is not the worst thing we need to worry about with private agents. Earlier, I mentioned that there were multiple adversaries we should care about. The second adversary isn’t a corporation, it’s all the people who will talk to your agent.

Simon Willison describes a condition that he calls the lethal trifecta. This occurs when you have a combination of (a) access to private data, (b) untrusted content an LLM must parse, and (c) the ability to send external communications. These together create the perfect storm for data-exfiltration attacks, where a remote attacker simply “tricks” the LLM by sending it instructions to ship out confidential data. Although the technology is getting better, it’s still quite common for even frontier LLMs to fall to prompt injection attacks in which a malicious user includes text (as part of a website or a piece of data) that causes your LLM to reveal things it should not. Even the most advanced models have not solved these issues: OpenAI recently unveiled a “lockdown mode” feature, where ChatGPT is restricted from making web searches due to the risk that it might upload your sensitive documents.

Agents, whether they use confidential inference or not, are really a worst-case for lethal trifecta issues.

These systems will need to ingest a vast amount of data, much of which will come from highly untrustworthy sources: think incoming emails and text messages. They will have access to everything on your system, like your encrypted messages and documents. And, to be useful, they will need to handle all sorts of actions that have visible external effects, like scheduling calendar invites and sending text messages.

The result is that your private data isn’t just vulnerable to the person who controls the agent, it’s potentially vulnerable to anyone who can cause your agent to misbehave. This problem exists regardless of how well-designed the private inference engines are. And OpenAI’s recent example illustrates that it’s far from solved. It’s possible that we’ll be able to solve these problems technically, or through some careful element of human observation — read all your calendar invites carefully — but right now these are the threats that should concern you about agents.

In short: private agents will be a massive target for prompt injection attacks. If you think spam directed at humans is bad, wait until it’s spam directed at agents.

Who does your agent really work for?

So far we’ve discussed two adversaries: the poorly-incentivized designers of private agents (such as search operators), and the possibility of remote prompt injection attacks. But of course, in any discussion of technical privacy systems we need to talk about the last elephant in the room: your government.

We live in a society, and that society has laws. If an agent has access to all of your data, messages and actions, then technically speaking it has the ability to detect criminal activity. That criminal activity might include sharing of CSAM, or terrorism-related activity, or it could include tax fraud or any other form of crime. These agents make a perfect one-stop shop for crime detection, since they can identify patterns of bad behavior and also report them.

Is this farfetched? Well, as I’m fond of repeating on this blog, this is more or less what existing rules published by the UK’s OFCOM require, and there have been proposals in the EU Commission to do similar things. The UK also maintains a vigorous regime of Technical Capability Notices (TCNs) that allow it to demand that providers make changes to their systems, changes that could potentially affect devices worldwide. Apple is in the midst of a battle with the UK over its other encrypted services.

Traditionally in the United States we’ve shied away from this sort of thing, partly because it’s creepy and mostly because it seems like a direct attack on the Fourth Amendment. With that said, the Fourth Amendment applies only to governments: in theory a private company like Apple or Google could configure their agents to report crimes to them, and then pass along the serious ones to the government. This is more or less what Apple proposed to do in 2021, when they designed a system to monitor photos for CSAM.

At the risk of saying more obvious things, the difference between a helpful private agent, a corporate advertising bot, and a government spy comes down mainly to a matter of prompting, and maybe a bit of model fine-tuning. Once you combine private data access and the ability to send messages, there is essentially no technical protection that private inference alone can offer.

So what does this have to do with cryptography?

For decades the point of cryptography has been to remove trust: to replace “I promise not to look” with “I can’t.”

Private inference is the most ambitious version of that promise. Against the adversary it was designed for — the provider who performs the inference itself — I believe that it probably does what it says. All I’m trying to say in this post is that this adversary is a very small piece of any agentic system.

The adversaries we care about are the ones that deal with the model directly, or even the ones who designed the model or specified its technical requirements. There is no cryptographic primitive that protects you from “upload your search facts to Google” or “report anything suspicious to the government because I programmed you that way.” That protection, if it exists at all, lives in law and politics and corporate incentives: the exact messy human institutions that cryptography was invented to let us stop trusting.