Checked against the source · commit d28f1b1 · September 5, 2026

How OpenIntelligence runs.

Every mechanism in the app does one of two jobs. It keeps a promise, that your documents stay yours. Or it defends against a danger, that a language model invents. There are exactly two places where invention can enter: when the app rewrites your question, and when it writes the answer. Everything else is a calculator. Here is the whole machine in twelve steps, then the three modes that turn it up or down, the four places an answer is allowed to run, the nine gates, and the models by name. Nothing on this page is described from memory; each step names the function that runs it.

Twelve steps

Twelve is how this page groups the work, not a count the code keeps. The older documentation described one 29-step pipeline, six steps for reading a document and 23 for answering a question; the shipped code also runs stages that were never written down, and it branches by file type, quality mode, evidence and route, so no single number covers every path. What the code does enumerate is fifteen IngestionStage cases, seven retrieval trace stages and nine gates.

Ingestion a file becomes searchable, one document at a time

  1. 01

    A file is enqueued as a durable ticket and processed one at a time.

    Two large documents at once would double memory and heat for no gain on a phone, so the loop is serial and the parallelism lives inside a document: pages in parallel, embeddings in batches. Progress is a lease with a heartbeat, so a crash never strands an item as “processing” forever.

    RAGService.enqueueDocuments RAGService.runIngestionLoop addDocument

    15 stages, from queued to complete, including paused · files over 500 MB that cannot stream are refused before reading

  2. 02

    Its text is extracted by type.

    The cheap path lies: a PDF that “has text” may carry a broken OCR layer from years ago, and a two-column paper read straight across produces sentences that never existed. So the text layer is validated against a quick OCR pass, columns and reading order are reconstructed, and tables keep their headers. Scans are rendered at 360 DPI and read on the device. Audio is transcribed on the device in 600-second segments.

    DocumentProcessor.processDocument VNRecognizeTextRequest RecognizeDocumentsRequest StreamingXMLProcessor SFSpeechRecognizer

    PDF · Office (docx, xlsx, pptx) · CSV · images · audio · video · text and code

  3. 03

    The text is cut into chunks, and the count is checked by the real tokenizer.

    This step has its own history. A padding block once made the token counter return a constant, the guard could never fire, and more than half of every document was silently truncated while every log line read healthy. The counter now runs through the same tokenizer the model uses.

    SemanticChunker enforceTokenLimitOnChunks CoreMLSentenceEmbeddingProvider.countTokens

    at most 310 words per chunk · validated at 430 tokens · 80 tokens reserved for a contextual prefix · 50-word overlap

  4. 04

    Each chunk becomes a vector.

    A small model turns the chunk into 384 numbers, coordinates on a map where chunks about similar things land near each other. The model exports one vector per token, and averaging them correctly under the attention mask turned out to be load-bearing. The app can only tell Core ML which processors it is allowed to use; where the work lands is Core ML’s decision.

    CoreMLSentenceEmbeddingProvider.embed embedBatch DeviceCapabilityService.preferredComputeUnits

    MiniLM-L6-v2 · 384 dimensions by default · Core ML today, Core AI on the 27 operating systems · batches of 8 to 512 by device

  5. 05

    Chunks are written to two indexes, and summaries are derived on top.

    One index is for words, SQLite FTS5 with Porter stemming. The other is for meaning, a memory-mapped vector file a phone can search without loading it. A one-call document summary sits above them. The two indexes are why a keyword and a meaning can both find the same page.

    BNNSVectorDatabase.persist SQLiteFullTextService.storeChunks RAPTORSummaryRouter

    9 SQLite tables · _vectors.bin + _norms.bin written atomically · one summary call per document

Query a question becomes an answer you can inspect

  1. 06

    The question is profiled and planned before anything is searched.

    Intent, complexity, the quality mode, and whether the run is agentic are decided first. Short conversational questions are rewritten to stand alone, because the index was built from long specific text and “what about the other one?” finds nothing. This rewrite is one of the two generative stages, and it is off in Standard on purpose, because a hypothetical can poison an exact lookup.

    ChatScreen.sendMessage RAGService.query QueryRuntimeCoordinator.resolveContext QueryExecutionPath

    paths: standard agentic forcedAgentic plannerEscalated

  2. 07

    Vector search and keyword search run in parallel and are fused.

    Each arm over-fetches, then reciprocal rank fusion merges the two lists so a chunk that both arms like rises. On a library of a thousand vectors or more, with a Metal device present, the whole mapped buffer is handed to the GPU with no copy. That is the entire GPU story for search; the user’s GPU profile does not decide it.

    HybridSearchService.searchWithFTS5 BNNSVectorDatabase.search GPUComputeService.batchCosineSimilarityFlatBuffer

    RRF with k = 60 · weights 0.7 vector / 0.3 lexical · vector arm asks for 3× the target, keyword arm up to 60

  3. 08

    A cross-encoder reranks the shortlist; a floor and a diversity pass cut it; neighbours are added.

    The reranker reads the question and each passage together, which is expensive and precise, so it only sees the shortlist. A similarity floor drops weak candidates, maximal marginal relevance spreads the survivors across documents, and parent and sibling passages are pulled in so a clipped table row arrives with its header.

    RAGEngine.rerank rerankWithCrossEncoder RAGEngine.applyMMR filterBySimilarity

    cross-encoder/ms-marco-TinyBERT-L2-v2 · floor 0.28 / 0.25 / 0.20 and λ 0.60 / 0.55 / 0.50 by mode

  4. 09

    Evidence is packed under the real token budget, strongest first and last.

    The model’s limit is hard: overflow is a failure, not a warning, so the budget is paranoid. The best evidence goes at both ends because language models measurably underuse the middle of a long prompt, and there is a benchmark fixture that puts the answer in the middle to prove it.

    ContextPackingService FoundationModelTokenBudget

    4,096-token window · 3,200-token budget · 256 reserved · dropped chunks are recorded by ID, not lost silently

  5. 10

    A post-retrieval plan chooses abstain, deterministic, on-device, or Private Cloud Compute, and only then asks for consent.

    Routing happens after retrieval because before it the app does not know how big the evidence is or what exact text would leave the phone, so it cannot ask for meaningful consent. The plan always carries an on-device fallback. Today every plan resolves on the device; the cloud target becomes reachable with iOS and macOS 27.

    makePostRetrievalModelPlan ModelExecutionPlanner.makePlan CloudEvidenceMinimizer.makeEnvelope FoundationModelRoutePolicy.determineRoute

    targets: deterministic onDevice privateCloudCompute abstain

  6. 11

    The model streams a typed answer with citations; nine deterministic gates decide what survives.

    This is the second generative stage, and the gates exist because of it. Each claim is checked against the retrieved evidence: coverage, numbers, contradictions, grounding, quoted spans, completeness. When the evidence does not support an answer, the app says so and stops, instead of writing something plausible.

    LLMService FoundationModelSessionFactory.createSession LanguageModelSession.streamResponse VerificationGateService

    Apple’s on-device Foundation Model · temperature 0.4 / 0.4 / 0.3 · verification bar 0.50 / 0.60 / 0.80 by mode

  7. 12

    What comes back is inspectable: claims, byte-offset citations, the completed route, and a trace.

    An answer you cannot audit is a guess with good typography. Every citation resolves to a character range in a source you can open. The route badge reads from an execution receipt, which records what actually completed rather than what was requested. Configured capability is not executed capability, and the app knows the difference.

    StructuredAnswer ModelExecutionReceipt ResponseDetailsView RetrievalTraceCollector

    7 trace stages: vector lexical fusion boosted candidates rerank final

The three quality modes

Standard is one pass. Deep Think and Maximum loop. The same twelve steps run in every mode; the modes change how much is gathered, how strict the gates are, and whether the app goes back for more. Every value below is read from RAGQualityMode.swift, ConfidencePolicyService.swift, AgenticOrchestrator.swift, AgenticPolicyService.swift and DeviceCapabilityService.swift.

 StandardDeep ThinkMaximum
ShapeOne passSerial reasoning steps, then up to 8 reasoning sessions; may stop after 4 once the target is reachedUp to 50 sessions, scaled to the evidence pool at three chunks a session, never fewer than 8; no early stop before 8
Reasoning steps—By chip: 5 on A17 Pro, 8 on A18, 10 on A19, 12 to 32 on M-series by memory50
Stops when confidence reaches—By the same chip tiers: 0.85, 0.90, 0.92, 0.950.98, or when new sessions add nothing, or thermal stops it first
Results targeted303550
Similarity floor0.280.250.20
Diversity (λ)0.600.550.50
Temperature0.40.40.3
Verification bar0.500.600.80
Gate thresholds lifted by0+0.05+0.10
Abstains below0.350.450.55
Confidence calibrationDefaultConservativeConservative
Question rewrite and HyDEOff, by designOnOn
Query expansionsOffUp to 8Up to 12
Neighbouring chunks pulled in235
Contextual compressionOffOnOff, keeps full context
Conversation turns remembered51020
Specification boost1.2×1.3×1.5×
Verification loop after the chain—Only where the chip grants 8 or more steps; an A17 Pro skips itNever; the loop is the verification
Tools inside sessions—Allowed, six local toolsSwitched off, so the window cannot overflow
Research passes—Up to 5 iterations, 3 inside the verification loop, 180 seconds each; cooldown between steps 100 / 50 / 25 / 0 ms by chip tier

Deep Think is tuned by the chip, not by the mode: its step count and confidence target come from the device tier, which the Settings screen names. Maximum ignores the device profile and runs the unlimited one. Two numbers that look alike are not: Maximum stops its loop at 0.98 confidence, and its verification bar is 0.80, which the source notes replaced a 0.98 that could not be reached. Extractive questions, where the answer is a span to quote rather than a synthesis, halve the verification bar and lower the abstention floor by 0.05. On the free tier, Maximum is limited to three uses a day. In the picker these are Standard, Deep Think and Maximum; the chip that reads “Fastest” beside Standard is its speed label. Every planning and analysis call inside a loop is pinned to the device; only the final synthesis is ever eligible for the cloud.

Where the answer is allowed to run

Abstain

The evidence is judged insufficient. The app says the documents do not answer this, and stops.

Deterministic

A rule-based extractor can answer without a model at all. An exact lookup, a number in a table.

On-device

Apple’s Foundation Model on the phone, iPad or Mac. Today, every route resolves here.

Private Cloud Compute

Only if the capability snapshot allows it, the network is up, the app is in the foreground or consent is already granted, and either the evidence does not fit the local window or the query asks for cloud synthesis. Arrives with iOS and macOS 27. Every such plan carries an on-device fallback.

Consent is asked only after retrieval, once the planner has selected the cloud and built the minimised payload, so what you approve is the exact text that would leave the device. A refusal pins the run to the device. The reason for every routing decision is recorded on the receipt:

exactAnswerAvailableprivacyRequiredLocaluserRequiredLocaluserRequiredCloudinsufficientEvidencelocalContextFitslocalContextExceededcomplexSynthesispccUnavailablepccQuotaReachedconsentUnavailablenetworkUnavailablefallback

Private Cloud Compute, in Apple’s words

The cloud target is Apple’s, not the app’s. Apple documents the model the app would call as “a variant of Apple Foundation Models that runs on Private Cloud Compute,” available from the 27 operating systems, behind a managed entitlement granted to eligible developers, with a context size and a usage quota the app reads before it decides. Apple’s security documentation states that user data “stays on the PCC nodes that are processing the request only until the response is returned,” that it “is never available to Apple — even to staff with administrative access,” and that the device encrypts a request only to nodes whose attested software matches a public, append-only transparency log. What the app adds on top is its own: consent per request, the exact payload shown first, and an on-device fallback on every plan.

sources: PrivateCloudComputeLanguageModel · Apple Security Research: Private Cloud Compute · Accessing Private Cloud Compute

The nine gates

Run in order after generation, except the last, which is applied before synthesis. Confidence and fidelity are different questions: confidence is how sure the app is overall; fidelity is whether the text on screen matches the sources it cites. A confident answer with low fidelity is exactly the failure these exist to catch.

  1. ARetrieval confidence. Did the search find enough, by a margin?
  2. BEvidence coverage. Is every claim backed by a citation? Uncited claims are removed.
  3. CNumeric sanity. Do the numbers and units in the answer appear in the evidence?
  4. DContradiction sweep. Does the answer disagree with a source it cites?
  5. ESemantic grounding. The response is embedded and compared with its best source chunk. The one gate that costs an inference.
  6. FQuote faithfulness. Do quoted spans exist, character for character?
  7. GGeneration quality. Is the text well formed, or did the model degrade?
  8. HAnswer completeness. Was part of the question left unanswered, and is that listed?
  9. IDomain isolation. Applied first: scientific and clinical material is not mixed with unrelated sources.

thresholds: τ 0.40, raised to 0.55 for medical, legal, financial and safety topics · margin 0.03 · grounding 0.50 · a strict profile at 0.65 / 0.75 / 0.10 / 0.60

The models, by name

Reading a scan
VNRecognizeTextRequest, accurate mode with language correction, at 360 DPI. Layout and tables through RecognizeDocumentsRequest.
Transcribing audio
SFSpeechRecognizer, on the device, 600-second segments. The newer SpeechAnalyzer branch exists in the source and never compiles.
Embedding
MiniLM-L6-v2 through Core ML, 384 numbers per chunk. On the 27 operating systems the same model runs through Core AI and a saved Core ML default migrates itself.
Reranking
cross-encoder/ms-marco-TinyBERT-L2-v2, question and passage tokenised together to 512.
Writing the answer
Apple’s on-device Foundation Model, SystemLanguageModel.default, which Apple’s research describes as a roughly 3-billion-parameter dense transformer with KV-cache sharing and 2-bit quantization-aware training. There is no “advanced” on-device model in the SDK; the old preference for one now runs the default and corrects its own telemetry.
Writing it in the cloud
PrivateCloudComputeLanguageModel, behind the entitlement, on iOS and macOS 27, with consent per request. Not live today. Apple’s research describes that server model as a Parallel-Track Mixture-of-Experts transformer; the app never selects or sees an expert. It selects a target, and Apple runs the model.
Tools the model may call
Six, all local and typed: RetrieveCorpusEvidence InspectDocument CompareTopicAcrossDocuments GetLibraryOverview CountPattern SearchExactPattern. A counter bounds the loop.
A model you run yourself
Optionally, an OpenAI-compatible server on 127.0.0.1, such as llama.cpp or Ollama. Off unless you point the app at one.

In the source, not on the path

Honest documentation marks what is dead, reserved, or superseded, so nothing here is taught as current when it is not.

  • Bundled Core ML, GGUF and MLX generative backends. Superseded by the Foundation Models framework.
  • The 3B versus 20B model selector and the “Core Advanced” label. Apple’s SDK exposes no selectable tiers.
  • Six further tool types that are defined and never registered.
  • The single 29-step pipeline and the single recursive thought loop. Historical descriptions of an older shape.
  • HNSW indexing, model judges, a 32K cloud window. Historical.
  • RAPTOR levels two and three. Reserved names; one summary per document is what ships.
  • A neural extractive QA model. The protocol waits for a trained model.
  • Mixture-of-experts routing on the device. The old playground drew it, with invented code; nothing in the source does it. Experts exist only inside Apple’s server model.

What runs on which silicon

  • The GPU is used for one job: comparing your question against a library of at least 1,000 vectors, when a Metal device exists. Below that, Accelerate on the CPU.
  • The GPU profile decides other things: which processors Core ML may use for the embedding and reranking models, and whether the diversity matrix goes to Metal. Efficiency and Balanced keep those models on CPU plus Neural Engine; Performance and Maximum allow all three.
  • “On the Neural Engine” is a request, not a placement. Core ML decides. Core AI exposes no control at all.
  • Device tiers scale concurrency: agentic steps 3 to 32, embedding batches 8 to 512, PDF rendering 1 to 64 pages at once, capped by a per-page memory estimate.

How this page was checked

Every identifier above was found in the Swift tree at commit d28f1b1 on September 5, 2026. Every number was read from the source file that holds it. The repository’s own checkers, verify_doc_claims.py and verify_capabilities.py, pass at that commit. The long-form documents this page condenses are public: OpenIntelligence, edge to edge and the full system trace, with line anchors. Statements about Apple’s platform are quoted from Apple’s own pages: the 2025 foundation models report, the Private Cloud Compute security post, and the Foundation Models framework documentation.