Private document AI is a simple idea: point an AI model at your own file, a PDF, a long article, a set of meeting notes, and get a summary or a specific answer, without that file ever leaving your device. Most tools that let you "chat with your PDF" today do the opposite. They upload the document to a server, run a large cloud model over it, and keep a copy. This guide explains how on-device AI plus retrieval (often called RAG) can summarize and query documents offline on an iPhone, why that matters for privacy, and the honest limits of doing it on a phone. It builds on our complete guide to local AI chat and the on-device vs cloud comparison.
Want the short version? Jump to the summary table. Want to be first to try private, on-device AI when it ships? PocketLLM runs the model fully on-device with zero telemetry. Join the launch list.
Private document AI keeps the file, the model, and the computation all on your device, so nothing is uploaded or retained by a third party. On an iPhone, a small model (roughly 1B to 3B parameters) can summarize and answer questions about documents offline, using retrieval to pull only the relevant passages rather than reading the whole file at once. The trade-offs are real: limited context length, limited memory, and lower accuracy on very long or highly technical documents than a frontier cloud model. For short and medium documents, it is genuinely useful and private by construction.
PocketLLM is launching soon. Private, on-device AI, starting on iPhone and iPad with more platforms planned. No account, no tracking, no cloud. Join the launch list and be first in.
Join the launch listWhat "private document ai" actually means
There are only two places the work can happen: on a server, or on your device. Cloud document tools send your file to their infrastructure, where a large model reads it. On-device tools do the opposite: the document is parsed, indexed, and answered locally. Private document AI is simply that second approach, where your file, the model weights, and every calculation stay on hardware you own.
The word "private" here is architectural, not a promise on a policy page. A cloud vendor can write a careful policy, but the file still travels to a server the moment you upload it. On-device, there is no upload to write a policy about, and that distinction is the whole point.
The privacy problem with "chat with your PDF"
When you drag a document into a cloud AI tool, you hand over a full copy of that file. What happens next depends on the vendor's terms: how long they keep it, whether it trains their models, and who can access it. We walk through that handoff in what happens to your data with ChatGPT, and the mechanics are similar across most cloud services.
For a casual document, that may be a reasonable trade. The problem is that the things people most want to summarize, personal notes, downloaded research, drafts, a book chapter, are exactly the things they would least like sitting on someone else's server. The cleanest fix is to not upload at all. If you are auditing tools by this standard, our roundup of AI apps that do not collect your data is a good companion.
How on-device retrieval (RAG) works
A phone-class model cannot hold a long document in memory at once, so private document AI leans on retrieval-augmented generation, or RAG. Every step happens locally:
- Parse the document into plain text on the device (extracting text from the PDF, stripping layout).
- Chunk that text into small overlapping passages, a few paragraphs each.
- Embed each chunk into a numeric vector with a small embedding model, again on the device.
- Retrieve the chunks whose vectors are closest to your question when you ask it.
- Generate an answer by feeding only those chunks plus your question to the local model.
The model never sees the whole file, only the slice relevant to the current question, which keeps the work inside a phone's memory budget while letting you query a document far larger than the model's context window. For a plain summary, the app can walk the document chunk by chunk and combine partial summaries, slower but reliable on constrained hardware.
The real limits on a phone: context length and model size
Two constraints shape everything about on-device document AI, and being honest about them saves disappointment.
Context length. This is how much text the model can consider at once. Cloud models advertise very large context windows; on-device, the practical window is much smaller, often a few thousand tokens once you account for memory. That is why retrieval matters: rather than stretching the context to fit a whole document (which balloons memory use), you feed only the retrieved passages. The KV cache grows with the amount of text you load, so a longer context directly costs more RAM.
Model size. File size is not the same as memory need. As a rough rule at 4-bit quantization, weights run around 0.6 GB per billion parameters, but the model also needs headroom for the KV cache, the runtime, and the operating system, and iOS caps how much memory a single app may use. So a file that looks like "2 GB" wants meaningfully more than 2 GB of RAM free to run comfortably, which keeps the phone-class sweet spot around 1B to 3B parameters. For the full mechanics of shrinking models to fit, see our explainer on local AI chat.
| Model family | Params | Approx. size (Q4) | Comfortable on a modern phone? | Source |
|---|---|---|---|---|
| SmolLM2 | 135M–1.7B | ~0.1–1 GB | Yes, very light | Model card |
| Llama 3.2 | 1B–3B | ~0.7–2 GB | Yes (3B on ~8 GB phones) | Estimate |
| Phi-3.5 Mini | ~3.8B | ~2.2 GB | Borderline on 6 GB, fine on 8 GB | Model card |
| Qwen 2.5 | 7B | ~4.5 GB (wants ~8 GB RAM) | High-RAM / desktop territory | Estimate |
| gpt-oss | 20B–120B | ~16–80 GB memory | No, desktop/workstation only | Model card |
Recent iPhone "Pro" tier devices carry roughly 8 GB of RAM, with base tiers commonly in the 6 to 8 GB range (Apple does not always publish per-model figures, so treat these as approximate and teardown-sourced). Practically, a 1B to 3B model is the sweet spot for document work on a phone, a 7B model suits a well-equipped Mac, and OpenAI's open-weight gpt-oss models are workstation-class.
Cloud vs on-device document AI: the summary
Here is the same question set applied to a cloud document tool, an on-device approach, and PocketLLM. The PocketLLM column reflects its on-device architecture; the document features described here are a direction, not a shipped capability yet.
| Question | Cloud document AI | On-device document AI | PocketLLM |
|---|---|---|---|
| Where your file goes | Uploaded to a server | Stays on the device | Stays on the device |
| Account required | Usually | Depends on the app | No account |
| Works in airplane mode | No | Yes | Yes |
| Document size handled | Very large | Bounded by context + RAM | Bounded by device |
| Who can read the file | Vendor, per policy | Only you | Only you |
| Telemetry on content | Varies by vendor | Depends on the app | Zero on conversations |
The trade is clear. Cloud tools win on raw capability and document size; on-device wins on privacy, offline use, and the fact that there is no server-side copy to leak or repurpose. For a fuller side-by-side, our on-device vs cloud AI breakdown covers the general case beyond documents.
Where PocketLLM fits, and where it is headed
To be precise about the present tense: PocketLLM is pre-launch. There is an email waitlist and no App Store link yet, and the shipping focus is private, on-device chat with no account and zero telemetry on your conversations. Under the hood it pairs Apple's CoreML with llama.cpp, running GGUF-quantized models on the llama.cpp path.
On-device document handling with retrieval is a natural extension of that architecture. The same local runtime that answers a chat message can, in principle, index a document already on the phone and answer questions about it without touching the network. It is honest to call that a roadmap fit and dishonest to call it a live feature, so we will only claim it once it ships. If that is the tool you want, join the launch list and try it the day it is ready.
Frequently asked questions
What is private document AI?
Private document AI means pointing an AI model at your own files, such as a PDF, a long article, or a stack of notes, and getting a summary or an answer without that file leaving your device. Most mainstream tools that let you chat with a PDF upload the document to a server and run a large cloud model over it, which means a copy of your file now lives somewhere you do not control. Private document AI keeps the file, the model, and the computation all on the device, so nothing is transmitted or retained by a third party.
Can I summarize a PDF offline on my iPhone?
Technically yes, within limits. A small on-device model in the 1B to 3B range can summarize text extracted from a PDF while your iPhone is in airplane mode, because both the model and the computation run locally. The practical constraints are context length and memory: a phone-class model can only hold a few thousand tokens of text in its working context at once, so very long documents need to be broken into chunks and handled with retrieval rather than pasted in whole. Short and medium documents summarize well offline; book-length files need a retrieval step.
How does on-device AI read a document without uploading it?
The document is parsed into plain text on the device, split into small chunks, and each chunk is turned into a numeric embedding, again on the device. When you ask a question, the app embeds your question, finds the chunks whose embeddings are most similar, and feeds only those chunks plus your question to the local model. This is retrieval-augmented generation, or RAG. Because every step happens locally, the file never gets uploaded and no server ever sees its contents.
What are the limits of running document AI on a phone?
The two hard limits are model size and context length. Phones have limited RAM, and file size is not the same as memory need: a model also needs headroom for the KV cache (which grows with how much text you feed it), the runtime, and the operating system, and iOS caps how much memory any single app may use. That keeps comfortable on-device models in the 1B to 3B range. Context length is limited too, so instead of reading an entire long document at once, the app retrieves only the most relevant passages. Accuracy on very long or highly technical documents is lower than a frontier cloud model, so treat on-device summaries as a helpful first pass, not a final authority.
Does PocketLLM support document summarization yet?
Not yet. PocketLLM is pre-launch, with an email waitlist and no App Store link at the time of writing. The shipping focus is private, on-device chat that runs entirely on your device with no account and zero telemetry on your conversations. On-device document handling with retrieval is the direction described in this article and a natural fit for that architecture, but it is not a live feature today. Join the launch list to be notified as capabilities roll out.