PII-TRACE: Detecting Personal Data Before It Leaves the Device

Hybrid compute on Mac divides Perplexity Computer tasks between frontier models in the cloud and a local model on the Mac. Cloud agents handle research, reasoning, and planning, while the local model works with private files and sensitive information.

This boundary depends on detecting personally identifiable information (PII) before it leaves the device. A local privacy gate keeps sensitive content on the Mac, redacts detected private information, or requests approval before sending it to the cloud.

Detection becomes harder across long, multilingual conversations. The same identifier may appear several times across different turns. One missed mention can expose the information the system is meant to protect.

Introduction

Today, we are introducing PII-TRACE (Tracing Recurring PII Across Conversational Exchanges), a new benchmark for evaluating personally identifiable information (PII) detectors, and PII-Tracer, a compact 0.6B model for PII detection.

Cloud-first agents force users to balance context, intelligence, and privacy. More personal context can help an agent understand the user and produce better results. But providing this context often means sending private information to a remote service and may leak personal information. The information that makes an assistant useful may be exactly what a user most wants to keep private.

Hybrid AI eases this trade-off by dividing work between the user's device and cloud models. But that boundary only protects privacy if the device can recognize PII before text is sent to a remote model. Users shouldn’t have to inspect every message themselves. The device needs a local PII detector as its privacy gate. In long conversations, the same identifier can recur across turns, and one missed mention is enough for personal information to pass through.

Perplexity introduced a hybrid local-server inference orchestrator that decides what work should run on device and what work should go to agents in the cloud. The model, agent harness, conversations, and execution trajectories all reside on the user's machine. Tasks that require access to the outside world are invoked only when necessary and are gated by user permission. As a result, until the user approves, that content stays on the device.

PII-Tracer provides one local control signal for model routing by flagging spans predicted to contain PII. The application then enforces the routing policy. It keeps the relevant input local, redacts detected spans, or requests explicit approval before escalating to a cloud model. This makes routing more selective. Detected PII remains on the device while approved context still benefits from frontier cloud models.

Selective routing depends on consistent detection across the entire conversation. The same identifier can recur across turns, and a missed mention may still be transmitted.

Across 12 detectors, PII-Tracer records the highest character F1 and the highest consistent coverage of recurring identifiers. The benchmark explains why both results matter: on a long conversation, finding most PII is not the same as finding every copy of it.

PII Detection Faces New Challenges in Hybrid AI

PII detection is a long-standing security problem, and several benchmarks and detectors already exist. However, PII detection in hybrid AI settings introduces new challenges. In particular, detectors must identify PII in long, multi-turn conversations, where conversational context determines whether a string should be considered PII. For example, a name might identify the user in one conversation, refer to a public figure in another, or simply be a placeholder generated by an assistant. The surface form alone is insufficient to determine whether a string should be flagged as PII.

Existing PII detectors and benchmarks primarily target individual records. We find that detectors designed to process one record at a time perform poorly on long, complex conversations. Also, existing benchmarks generally don’t evaluate consistency across turns. As a result, strong performance on these benchmarks doesn’t necessarily translate to effective PII detection in long, multi-turn conversations.

PII-TRACE: A Benchmark for Deployment-Critical PII Detection

We designed PII-TRACE around three behaviors that matter when a PII detector is used on assistant conversations. The first is consistent coverage: when an identifier appears several times or crosses user and assistant turns, the detector needs to find every mention. The second is robustness to long context: conversations range from fewer than 1,000 to more than 100,000 characters, making it possible to measure what happens as the history grows. The third is handling multiple languages and mixed-format content: a conversation may switch languages and combine prose with code, tables, or structured records. PII-TRACE preserves these patterns by evaluating each full dialogue instead of splitting it into isolated records.

The benchmark measures performance at two complementary levels. At the identifier level, consistent detection asks the stricter question: did the detector cover every character in every mention of the same identifier? We report this score separately for identifiers with multiple mentions and for identifiers that repeat across turns. At the character level, precision measures how much of the text marked by a detector is labeled PII, recall measures how much labeled PII it finds, and F1 balances the two.

PII-TRACE contains 13,148 synthetic user-assistant conversations across 13 languages and 10 writing systems, with 37,431 identifier mentions labeled at the character level across nine PII types. A total of 41% of the conversations contain structured content. Among the 5,645 conversations with labeled PII, 63.8% include an identifier that appears more than once, and 28.7% include an identifier that appears across multiple turns.

Building a synthetic dataset from production conversations

PII-TRACE preserves the turn structure of source conversations without publishing the originals. The pipeline rewrites each turn and replaces every labeled identifier with a synthetic value.

First, multiple language models mark nine types of PII in production user-assistant conversations. A rule-based pass groups repeated identifiers of the same type under one entity ID. Each marked value is then replaced by a typed placeholder, leaving a template that retains the turn order, PII type, and links between mentions but none of the marked original values.

The system paraphrases each turn while keeping those placeholders intact, then inserts synthetic values that match the identifier's type and format. Repeated mentions receive the same value within a conversation, while different conversations use independently generated values. The final alignment pass recalculates every character offset.

Three automated gates check that replacements match the stored spans, repeated mentions use the same value, and marked source values are absent under a Presidio and regular-expression rescan. A stored offset must also recover the exact synthetic substring. Records that fail are regenerated or dropped. A second language model audits a sample, and humans review anything it flags as PII.

PII-Tracer: A Compact Detector for Local Use

PII-Tracer is a 0.6B bidirectional encoder adapted from a Qwen3 backbone. Privacy screening is different from text generation and requires finding relevant PII spans and returning their boundaries. As a result, PII-Tracer replaces Qwen3's causal mask with padding-aware bidirectional attention, so every token can draw on both earlier and later turns within a 4,096-token window.

For each token, the encoder produces a 1,024-dimensional representation. A linear tagging head produces scores for 37 possible labels: one special label (which we use O to denote) for text outside a PII span, plus four span-position labels for each of the nine PII types. In the BIOES scheme for Named Entity Recognition, B (Beginning), I (Inside), and E (End) mark a multi-token span, while S (Single) marks a one-token span. An auxiliary head also predicts whether the conversation contains sensitive material, such as health or religious information.

We train PII-Tracer for three epochs on roughly 714,000 training samples, combining multilingual assistant conversations with single-record examples. The shared bidirectional encoder has two training heads: a 37-class BIOES token head that detects and types PII spans, and a binary conversation-level head that predicts whether the conversation contains sensitive material. We train both heads jointly using $\mathcal{L}=1.5\mathcal{L}_{\mathrm{tag}}+0.3\mathcal{L}_{\mathrm{sens}}$. Here, $\mathcal{L}_{\mathrm{tag}}$ gives rare PII labels more weight so the frequent `O` label does not dominate, while $\mathcal{L}_{\mathrm{sens}}$ supplies the conversation-level training signal; the 1.5 and 0.3 coefficients keep span detection as the main task. This auxiliary signal reinforces context-aware detection: the model learns to judge a candidate span within the conversation rather than as an isolated string, helping reduce false positives with only a small tradeoff in recall.

At inference time, a constrained Viterbi decoder searches for the highest-scoring valid BIOES sequence instead of labeling each token independently. For example, B-private_person can continue with I-private_person or close with E-private_person, but not switch to I-private_email. The decoder maps the result back to exact character spans for redaction or local routing.

PII-Tracer Leads on Character F1 and Recurring PII

We compare the detectors at both the character and span levels. Character F1 evaluates detection character by character. Span-overlap F1 measures whether the detector identifies any part of a PII item, while span-containment F1 measures whether it captures the entire item.

PII-Tracer achieved the highest character F1 (0.629) among the 12 systems evaluated, and the second-highest span-overlap F1 and span-containment F1. Frontier models, namely GPT-5.6-sol and Claude Sonnet 5, achieved comparable overall performance. GPT-5.6-sol obtained higher scores on both span-level F1 metrics, but a lower character F1. However, these frontier models have hundreds of billions or even trillions of parameters and are closed-source models hosted in the cloud. As a result, they’re unsuitable for protecting sensitive local data in hybrid AI settings where unscreened text must remain local. In contrast, PII-Tracer delivers near-frontier span-level detection with only 0.6B parameters and can process unscreened text entirely locally. Other open-source PII detectors perform substantially worse than PII-Tracer.

Finding Every Recurring Mention

The consistency experiment applies a stricter test to the same predictions. The test set contains 899 identifiers that appear once and 959 that appear more than once; 790 of the recurring identifiers span multiple turns. The figure reports four mention-count buckets (one, two, three to five, and six to ten) and counts an identifier only when all of its labeled characters are found. It plots PII-Tracer with three selected baselines, while the aggregate table reports recurring and cross-turn scores for all twelve systems.

PII-Tracer stays ahead as repetition increases: its score moves from 0.917 for one mention to 0.873 for two, 0.796 for three to five, and 0.691 for six to ten. In the last bucket, GPT-5.6-sol reaches 0.464, while GLiNER2-PII and Claude Opus 4.8 reach 0.073 and 0.045. Across the complete evaluation, PII-Tracer finds every mention of 79.4% of recurring identifiers and 77.6% of cross-turn identifiers; GPT-5.6-sol reaches 57.0% and 55.1% on the same two measures.

Covering Long Conversations

We first group all 1,922 test conversations by character length and decode PII-Tracer with the 4,096-token window. The groups contain 167 conversations below 1,000 characters, 1,292 from 1,000 to 10,000, and 463 at 10,000 or longer.

Single-window recall is 0.975 below 1,000 characters and 0.955 from 1,000 to 10,000, but drops to 0.687 for conversations at or above 10,000 characters. Precision remains near 0.51 in the two longer groups, pointing to input coverage as the main issue.

To study the effect of input handling, we evaluate the same checkpoint with 50%-overlap sliding-window decoding. This raises overall character recall from 0.830 to 0.965 and multi-mention consistent detection from 0.794 to 0.954, without retraining.

Consistent Across Languages

PII-TRACE covers 13 languages; the language experiment reports a six-language slice spanning Latin, Cyrillic, and Hangul scripts: English, German, French, Italian, Russian, and Korean. We run the same 12 detectors on all test conversations in each language. Within each language, we pool predicted and gold characters and compute character F1.

PII-Tracer leads character F1 in four of six languages, German (0.735), French (0.633), Italian (0.676), and Russian (0.651), and is within 0.016 and 0.036 of the best results in English and Korean. In the companion analysis, it leads to consistent detection in all six language subsets, scoring 0.80–0.93. The per-language view shows gains beyond English.

Higher Character F1 Than the Privacy Filter on Five Standard Benchmarks

The final experiment moves outside PII-TRACE. We run PII-Tracer and the OpenAI Privacy Filter on the ai4privacy validation split (47,728 documents), Nemotron-PII test split (100,000), a fixed seed SPY set (8,688), the Gretel PII test split (5,000), and the TAB ECHR test split (127).

PII-Tracer has higher character F1 on every dataset: 0.950 versus 0.907 on ai4privacy, 0.847 versus 0.709 on Nemotron-PII, 0.585 versus 0.543 on SPY, 0.952 versus 0.895 on Gretel PII, and 0.594 versus 0.350 on TAB. TAB is the only benchmark in this group built from real, human-labeled text. There, PII-Tracer reaches 0.986 versus 0.982 precision and 0.425 versus 0.213 recall—twice the recall at essentially the same precision (details in the paper). The higher F1 therefore carries over from PII-TRACE conversations to all five conventional single-record benchmarks in this comparison.

Conclusion

Hybrid AI integrates the user's device into the inference stack, offering stronger privacy and lower cost. Frontier models in the cloud can handle research, reasoning, and planning, while local models work with files and personal data that should remain on the device. This division depends on recognizing sensitive information before any sensitive data goes to the cloud.

PII-Tracer is a compact model for detecting PII in multi-turn conversations, while PII-TRACE evaluates whether detectors can consistently identify recurring PII throughout a conversation.

Together, PII-TRACE and PII-Tracer provide a benchmark and reference model for advancing PII detection in multi-turn conversations and other long-context settings.

Agents are taking on longer tasks and working with more personal context. As a result, privacy controls should hold across the entire conversation, not just the initial input. Hybrid AI depends on reliable local detection to keep sensitive context on the device.

Read the technical report for details on the PII-TRACE benchmark, the PII-Tracer model, and our evaluation. PII-Tracer is publicly available on Hugging Face. We plan to release PII-TRACE soon.