How we engineer safer agents

To build safer AI agents, we need to design for both their capabilities and their failures. Even when carrying out legitimate tasks, and without any malicious instructions, agents may cross security boundaries as they try to overcome obstacles.

We believe this is a security engineering problem, and the industry should tackle it the way it tackled the internet worms of the early 2000s. The systems surrounding agents must limit what agents can access, detect unwanted behavior, and contain failures when they occur.

At Perplexity, we are actively collaborating with researchers and practitioners to advance security engineering for AI agents, and we apply multiple layers of defense across our models, agent harnesses, and infrastructure so that security does not depend on any single safeguard.

Meltdown behavior of AI agents

In July, AI agents broke into open-source AI platform Hugging Face's production infrastructure. During internal cybersecurity evaluation and training runs, OpenAI models discovered a way to exploit the Artifactory repository manager to establish a bulletin board for communicating with one another, enabling them to organize into swarms. Later, agents that were unable to make progress on tasks in a cybersecurity benchmark recruited other agents to help. These agents broke out of their evaluation environment and reached the internet. They also discovered ways to cheat by reverse-engineering answers to the benchmark tasks, but thought that the grading system would detect the cheating. To better understand how the benchmark evaluation worked to find ways to avoid being detected as cheating, they ultimately compromised Hugging Face systems (Hugging Face; OpenAI).

The incident is a clear example of a failure mode we described when we open-sourced Numbat in July: accidental meltdowns, a term coined by security researchers from Cornell Tech (arXiv). An agent pursuing a legitimate goal encounters friction, such as a missing file, a failed API call, an expired credential, or a permission denial, and goes looking for a workaround. In doing so, it may cross security boundaries through reconnaissance, privilege escalation, secret discovery, or data exfiltration. The agent itself effectively becomes the adversary, even though no malicious prompt or external attacker is involved.

Hugging Face was not an isolated case. More agent meltdown behaviors were discovered and reported in recent weeks. On May 27-28, agents trying to gather University of Iowa data from Data USA, a platform offering open access to US government data, ran into errors caused by a malformed query. They responded with probes, including SQL injection, cross-site scripting, template injection, path traversal, and command injection. The activity was later linked to OpenAI (SecurityWeek). On June 20-21, OpenAI agents trying to obtain per-person government costs for a category of medicines across local areas in Victoria from the Australian Institute of Health and Welfare (AIHW) were blocked from downloading a file. In response, agents downloaded the file, which was already public, from the AIHW pre-production server, circumventing the site’s anti-bot protections in the process (SecurityWeek).

The pattern is not limited to one lab. During cybersecurity tests, Anthropic models breached three companies after a mistake gave them open internet access, Google's Gemini guessed credentials to three websites, and a Meta model exploited a third-party service (Reuters).

Learning from history: The era of internet worms

These incidents are reminiscent of the era of internet worms about a quarter of a century ago. In the early 2000s, the internet grew far faster than the security practices around it. We saw a wave of viruses and worms, with consequences far more serious than what we are seeing now with AI agents in meltdown.

On May 4, 2000, people began receiving emails from familiar senders with the subject line "ILOVEYOU". Opening the attachment in the email resulted in running a script that mailed a copy of itself to everyone in the victim's Outlook address book. By 6 p.m. that day, the CERT Coordination Center had received reports involving more than 420,000 internet hosts (US GAO Report). The virus reportedly penetrated at least 14 federal agencies, including the Department of Defense, the CIA, NASA, and the House and the Senate (US GAO Report). It also shut down email servers at the Pentagon and the British Parliament (Wired). Insurance giant Lloyd's of London made an early estimate that the virus would cost over $15 billion in damages and lost productivity (US House Committee Hearing). Computer Economics later estimated $8.75 billion in damages worldwide (Information Week).

Worms that needed no human click came next. On July 19, 2001, the Code-Red worm infected more than 359,000 computers in less than 14 hours (CAIDA). On January 25, 2003, SQL Slammer doubled in size every 8.5 seconds and infected more than 90% of internet hosts vulnerable to it within 10 minutes (CAIDA), exploiting a hole that Microsoft had released a patch for six months earlier (CNET). In August 2003, the Blaster worm and the Sobig email worm appeared about a week apart (Microsoft, F-Secure).

At the time, it was reasonable to ask whether networked computers could be made safe at all. Six days after the ILOVEYOU virus, a US House subcommittee held a hearing on it. Many security experts called for legislation that would make software companies liable for damage caused by flaws in their products. Washington took a different path. The National Strategy to Secure Cyberspace stated that “federal regulation will not become a primary means of securing cyberspace.” It warned that broad mandates could create a “lowest-common-denominator approach to cybersecurity” (CISA).

The industry answered with engineering. Eleven days after ILOVEYOU, Microsoft announced an Outlook update that blocked executable attachments. In January 2002, the company made security a top priority (Seattle Times). Automatic patching, secure defaults, bug bounty programs, and coordinated disclosure became standard practice. In 2008, Chrome began running web content in a sandbox that "limit[s] what an attacker can do after exploiting a bug" (Chromium Blog). Encryption, once the exception, became the default. Security technologies, including multi-factor authentication, firewalls, zero trust, and security information and event management (SIEM), became widely used. Organizations worldwide are forecast to spend about $240 billion in 2026 on information security (Gartner). Worms never disappeared entirely, but the era of routine internet-wide outbreaks faded. At the same time, the internet, largely unencumbered by regulation, became a powerful engine of innovation and economic growth over the following two decades.

Security engineering for AI agents

Recent incidents have raised legitimate questions about agent governance. Existing security mechanisms were developed for pre-agent computing environments characterized by tightly scoped and largely deterministic software behavior. Access control mechanisms, in particular, are primarily designed around human users and often grant them broad privileges that are, in principle, sufficient to cause significant harm.

This design rests on several assumptions: most users act in good faith; malicious actors can be deterred through auditing and real-world consequences; and humans take actions, whether correct or erroneous, at a relatively limited rate. Applying these security mechanisms to agents requires accounting for differences between human and agent behavior. As we wrote in our response to NIST's request for information on AI agent security, AI agents challenge these assumptions. They can exercise privileges and take actions at machine speed, are not themselves deterred by real-world consequences, and may take actions that are technically permitted but that a typical human user would rarely take.

Answering those questions requires security engineering. We know the security principles for building systems that keep the risks under control when individual components misbehave. But we need to make sure that these principles are applied when engineering the environments for training, testing, and using AI agents.

The defense-in-depth security principle

Arguably the most important security principle that needs to be applied for securing AI agents is defense-in-depth, which advocates for a layered security architecture, where multiple, varied, and redundant defensive measures are employed to protect system resources. No single safeguard is perfect. People may fall for social-engineering attacks and reveal their login credentials, zero-day software vulnerabilities are discovered and exploited, and intrusion detection misses new attack variants. Defense-in-depth accepts that and stacks several independent layers of defense. Should one defensive layer be circumvented, other layers remain in place to counter or detect the intrusion.

The Hugging Face incident shows why defense-in-depth matters. OpenAI's retrospective reports that the automated safeguards used in its production systems, including harnesses, auto-review models, and safety classifiers, were absent from the evaluation environment. If its currently deployed chain-of-thought monitoring had been running on those evaluations, it would have caught the initial activity and paged the security team more than a day before the breach (OpenAI). The layers that could have caught the problem existed. They simply were not all in place at the same time.

Securing AI agents requires improving both the safety and security of AI models and the systems surrounding them. Work at the model layer matters, and it is paying off. Early LLMs were easily manipulated by injected instructions, while today’s models are substantially better at recognizing and refusing them. Model developers also train models for safer behavior and, as OpenAI's retrospective describes, monitor reasoning traces for signs of problematic behavior. But model-level defenses are only part of the solution. The systems in which agents are trained, tested, and deployed must also be engineered for security and containment. Model-level safety reduces the likelihood that an agent will go astray; system-level security limits the damage when it does.

Expand

Design rules for redundant safeguards

Defense-in-depth requires redundant safeguards. For agents, we advocate applying it with three design rules:

1. Independence. Each layer should fail for independent reasons. Two controls that share the same assumption can be beaten by the same attack. Monitors and enforcement points should sit outside the agent's reach.
2. Enforcement below the agent. At least one layer must be deterministic: conventional, verifiable code that blocks prohibited actions no matter what the model produces. A safeguard the agent can decline to invoke, or reconfigure, is not a safeguard.
3. Signals can only reduce authority. Detection should trigger responses that narrow what the agent can do, such as pausing, revoking credentials, cutting network access, or terminating the session. No risk signal should ever grant additional access.

We share these principles with our partners in the Open Secure AI Alliance, including NVIDIA, which has argued that the harness guides what an agent tries while the infrastructure controls what it can do (NVIDIA).

Defense-in-depth in Perplexity's products

Perplexity runs agents in four kinds of environments: in the cloud through our agent platform Computer, in the browser through Comet, on user devices through Portable Computer, and on employee laptops, where our engineers use third-party coding agents. Each environment has its own risks, and each gets several layers of protection.

In the cloud: Computer and SPACE

Every Computer task runs in SPACE, our sandbox platform, which treats the agent as untrusted by default. Each task gets its own microVM, credentials stay outside the sandbox and are injected only when needed, outbound traffic is controlled outside the guest, and snapshots let us roll back destructive changes (Making SPACE). Enterprise customers can also bring their own encryption keys. Revoking a key makes the protected data unreadable, and the agent can't override that decision.

We also test whether these layers hold. In Escaping SPACE: Part I, we gave nine frontier models root access inside a SPACE sandbox and asked them to break out. None escaped the VM, but some found ways around the network policy when partial network access was allowed. We fixed those gaps, confirmed the fixes in a re-test, and disclosed similar weaknesses we found in eight of 10 other sandbox platforms to their vendors.

In the browser: Comet and BrowseSafe

Comet has used a layered prompt-injection defense since launch: classifiers screen retrieved content, structured prompts mark it as untrusted, sensitive actions such as sending an email or placing an order require the user's confirmation, and users are told when something is blocked (Comet prompt-injection defenses). BrowseSafe, the open-source detection model behind the first layer, outperforms frontier models on our BrowseSafe-Bench benchmark, yet the same benchmark shows that no detector is fail-safe, which is why detection is never the only layer (BrowseSafe research). Computer inherits these defenses, which have been audited by Trail of Bits (Perplexity).

On the device: Portable Computer

Portable Computer is a local-first agent: the model, harness, conversation, and trajectory stay on the user's machine by default. A deterministic orchestrator, which is harness code rather than an LLM, runs the loop and enforces policy. The model proposes actions, and approved tool calls run in an OS-level sandbox that restricts processes, file paths, and network access. The sandbox is always on and fails closed: if it isn't available, the harness disables itself instead of running commands with the user's permissions. Actions that cross the device boundary, such as web search, connectors, and escalation to a stronger cloud advisor model, happen only when the user enables and approves them. Before an advisor call, a PII classifier flags sensitive information and the user sees what would be sent. The advisor receives only that approved context and returns text guidance, with no access to the device's files or tools.

On the endpoint: Numbat

Coding agents on developer laptops run for hours with access to source code, credentials, and production systems. Numbat, our open-source agent security suite, adds an independent layer that sits outside any single agent. It works with harnesses such as Claude Code, Codex, OpenCode, and Pi, blocks dangerous actions before they run, flags suspicious multi-step sequences such as reading a secret and then sending data out, and preserves session timelines for investigation. Numbat runs across thousands of Perplexity endpoints alongside Bumblebee, our open-source supply-chain scanner. Computer regularly reviews their findings and proposes improved detection rules, and a human approves every change before it ships.

A job for the whole industry

As AI agents become more capable and widely used, security engineering must remain central to their development and deployment. Researchers and engineers can build on established principles, use real-world findings to refine defenses, and test safeguards across the environments in which agents operate. Security is an ongoing engineering discipline that develops alongside the technology.

Security engineering for agents is a job for the whole industry. No single company controls the full stack an agent runs on. Model developers, harness builders, sandbox and runtime providers, cloud and hardware vendors, and enterprise security teams each own a layer. The safety of the whole depends on every layer holding when another one slips. That is why we build safeguards that work across agent harnesses and open-source our tools. It’s also why we disclose the weaknesses we find in other platforms and take part in efforts such as the Open Secure AI Alliance.

To work with the wider community on these challenges, Perplexity launched the Secure Intelligence Institute in March 2026 to study the security, trustworthiness, and practical defense of leading-edge AI systems (Perplexity). Perplexity is currently collaborating and/or funding academic researchers in Stanford, CMU, Duke, Columbia, Ohio State, and University of Virginia. We’re also conducting intramural research to advance the frontier of agent security research, translate those advances into concrete improvements in Perplexity's systems, and share knowledge and resources that help strengthen the broader AI ecosystem. Safer agents are something the industry can engineer. We intend to keep doing our part, and we invite others to build with us.