The first gate for a cybersecurity evaluation is evidence, collected inside the tool environment, that the AI agent cannot reach real systems. A prompt can say, “This is a simulation,” but it cannot close an egress route or block credential reads and public package publishing. Network and permission rules that the model cannot change must enforce the real boundary.
On July 30, 2026, Anthropic published its review of three incidents in cybersecurity evaluations. After reviewing 141,006 runs where Claude might have obtained internet access, Anthropic confirmed that Claude reached the real internet through third-party evaluation environments in six runs and gained unauthorized access to the production systems of three organizations. The post also says this is Anthropic's current understanding and will be updated if details change.
This guide covers evaluation ranges that should be isolated from real systems. It does not cover authorized red-team work on production systems. Stop the evaluation if tests from inside the tool environment cannot prove that the public internet, out-of-scope targets, public package registries, and unrelated credentials are unreachable.
Choose isolation based on what the task can change
Read-only analysis should not share an environment with a task that can change an external system. Start with the agent's capabilities, then choose the isolation level.
| Task capability | Minimum environment |
|---|---|
| Read already-redacted logs or code | No network or real credentials; inputs are read-only |
| Reproduce application vulnerabilities, run capture-the-flag (CTF) challenges, or use a shell | Use a disposable sandbox or virtual machine, connect only to the range, and provide no long-lived credentials |
| Test kernel, container-runtime, or escape capabilities | Use a separate virtual machine, microVM, or dedicated host isolated from office networks, production networks, and cloud management planes |
Treat any task that can install software, scan addresses, or modify files as high risk. Do not start if the team cannot name the allowed targets, available identities, environment owner, and stop authority.
What each of the three incidents exposed
Anthropic's review shows a direct mismatch: the system prompt said the environment had no internet access, but the third-party environment had a live egress path and the task did not limit where Claude could search for the scoring secret, or flag. Claude therefore treated real systems as resources inside the evaluation range.
Incident 1: A matching domain led Claude into production
The evaluation partner gave a fictional company a name that matched an active website domain, and the evaluation container could connect directly to the internet. Across four runs, Claude obtained application and infrastructure credentials and accessed a database containing several hundred rows of production data. Anthropic described this as the most serious impact of the three incidents.
Incident 2: A simulated package takeover became public malware
Claude published a malicious package to PyPI to complete the challenge. It remained publicly available for about an hour and was downloaded and run on 15 real systems. When a security company's scanner installed it, the malicious code exfiltrated credentials. Task instructions alone cannot govern public package installation or publishing, or installation from a version control system (VCS) URL.
Incident 3: The search for another target expanded to about 9,000 hosts
After failing to reach the fictional target, the model scanned about 9,000 addresses and compromised a company's internet-facing application. The latest internal research model running this task later stopped, but only after real-world impact had occurred.
The models reacted differently, but these incidents were not a controlled model comparison and do not prove that a newer model will stop on its own. Anthropic considers the root cause closer to a harness and operational failure than a model-alignment failure. The first engineering fixes should therefore address environment boundaries, monitoring, and third-party assurance.
Map what the agent can actually reach
Before launch, the evaluation owner must document and lock the following boundaries:
- Target boundary: Name the allowed targets, the flags inside the range, and the stop conditions. Use domains and addresses reserved for the evaluation instead of real organization identifiers.
- Execution boundary: Name the available tools, writable paths, and temporary identities. An outer control must reject every command or permission that is not listed.
- External dependencies: Open only the private package source, logging endpoint, and controlled inference channel required by the task. The public internet and cloud metadata endpoints remain unreachable.
The evaluation and security owners must approve these boundaries together. Keep the environment closed while any target, identity, or third-party configuration is still marked unknown.
Test the operations that must fail
A negative test deliberately checks that an out-of-scope operation fails. Run these tests in the same container or virtual machine that will execute the agent's tools, with the same network policy and tool permissions used for the actual evaluation. Looking at firewall settings from the host does not prove that the agent's real path is blocked.
| Test surface | Test | Passing result |
|---|---|---|
| Network egress | Connect to a harmless public endpoint controlled by the evaluator but absent from the allowlist; cover DNS, IPv4/IPv6, redirects, and proxy paths | An outer network policy rejects the connection and produces a searchable denial log |
| Internal addresses | Attempt to reach out-of-scope private networks, host services, and cloud metadata endpoints | The request is rejected before reaching the target, and the agent cannot modify the route |
| Packages and files | Attempt installation from and publication to a public registry, install code from a VCS URL, and write a startup file | Public registry paths are unreachable and persistent writes fail |
| Credentials and shutdown | Use a decoy credential to verify unrelated secrets are unreadable, then rehearse shutdown, disconnection, and token revocation | The read fails; no tool call or network connection appears after shutdown |
Record the environment and policy versions and associate every test with a log identifier. A test passes only when the expected denial occurs and the log can be found. A timeout, missing log, or broken test tool counts as a failure.
Put the boundary where the model cannot change it
Guidance published by the NVIDIA AI Red Team on July 30, 2026 says that prompt controls and LLM-as-a-judge reviewers can both be manipulated by an attacker. Access control, a hardened sandbox, default-deny egress, and secret isolation must be enforced by system controls outside the model.
Separate model inference from the tool environment. A runtime outside the sandbox should forward inference requests. Do not give the sandbox the model API key or a proxy that can reach arbitrary addresses. If the task needs public research, use a separate read-only session with restricted domains; the attack-simulation session connects only to the range.
Run code only in a disposable environment. Restrict mounts and processes, set processor and memory limits, and tighten the network boundary. A command allowlist is only an extra layer because ordinary commands such as pytest and npm install can execute arbitrary code indirectly. For ordinary local agents, also review the OpenShell and local-agent security checklist.
Keep long-lived secrets out of the tool environment. Environment variables, .env files, and credential caches used by command-line tools can all be read. When the task needs an identity, a token broker should issue a short-lived, low-privilege token for that task and revoke it when the task ends or an emergency stop is triggered. Use only decoy credentials when testing leakage paths.
Any attempt to expand network, file, or identity permissions should be rejected and logged by a control the model cannot modify. This is a passing condition to test repeatedly, not a permanent guarantee created by one configuration review.
Require the same evidence from third-party environments
High-risk evaluations cannot rely on reviewing model output afterward. The logs must let a reviewer reconstruct every operation that could affect an external system. Network allow and deny decisions and the agent's tool calls must be searchable; file and process changes and token use need timestamps. Redact logs before sharing them, restrict viewing and export permissions, and keep a tamper-resistant copy for security investigations.
Before launch, a third-party vendor must submit a network diagram and negative-test results. The work order should say who controls network policy, who retains the logs, and who can stop the run. The model provider, commissioning organization, and evaluation vendor are three distinct roles; “shared responsibility” is not a sufficient assignment.
- The commissioning organization and evaluation vendor must both be able to pause the evaluation immediately, with round-the-clock contacts and a defined notification deadline.
- After a configuration change, the vendor must rerun the affected tests. At the end of the evaluation, it must destroy the environment and temporary identities as agreed.
Verbal assurances, meeting notes, and configuration screenshots do not prove that isolation works. The commissioning organization should receive test results and denial logs with the same level of detail it requires from an internal environment.
Make the release decision from evidence
| Decision | Required evidence |
|---|---|
| Release | The boundaries are approved, negative tests produced denial logs, the shutdown drill succeeded, and third-party evidence is complete |
| Keep closed | A target or identity is still unknown, a test result cannot be traced, stop authority is unclear, or the vendor supplied only a configuration description |
Rerun affected negative tests after every configuration fix. A small evaluation, a short run, or a newer model is not an exception. Long-running autonomous work also needs the long-horizon safety gates.
Contain the impact first if the agent reaches a real system
If the agent enters a real system, do not let the model continue to explain itself or finish the task. Respond in this order:
- Stop tool calls and disconnect the network from outside the environment, then preserve the current state.
- Save the trajectory, network logs, file-system snapshot, and package-operation records, then revoke any short- or long-lived credential that may have been exposed.
- Have the security and incident-response owners determine whether data was read, written, or exfiltrated and whether persistence remains. Notify the relevant organizations under the response plan; do not reuse the environment until it is fixed and fully retested.
Anthropic's postmortem says its team began reviewing transcripts and stopped all cybersecurity evaluations on July 23, confirmed the three incidents on July 24, and began contacting the evaluation partner and three affected organizations on July 27. At the time of publication, Anthropic had reached two organizations and was still trying to contact the third. An evaluation plan should likewise name who can stop a run and set deadlines for internal escalation and external notification.
Do not read the postmortem as proof that a newer model will stop
Anthropic found no evidence that the models were pursuing goals of their own. They continued a CTF task under a false belief about the environment. That is not the same as choosing a real-world attack as an independent goal.
The incidents were not a controlled model comparison. The latest internal model stopped after recognizing that the target was real, but this observation still needs testing and cannot replace isolation or monitoring.
The postmortem also does not directly describe publicly available Claude products. The evaluated models ran without the standard safeguards used in public deployments and on dedicated infrastructure isolated from Anthropic's sensitive internal systems and customer data. A clear prompt still helps, but the environment must enforce the final boundary.
FAQ
Does a defensive evaluation still need network isolation?
Even a read-only offline task needs a basic boundary: the tool environment has no internet connection or real credentials and cannot write files or execute code. A task that scans, installs software, or connects to an external service must use default-deny egress. A defensive objective does not make the execution environment free of side effects.
Can the model decide whether a target is real instead of using network isolation?
No. Anthropic's review shows that a model can interpret a real system as part of the range. Target classification can trigger an alert, but it cannot replace a firewall, a target allowlist, a private package source, and permission controls.
What if the evaluation needs public information?
Separate research from attack simulation. The research session uses a read-only proxy and restricted domains without a shell, scanner, or package-publishing credential. The attack-simulation environment connects only to the range.
Can an LLM reviewer approve commands by itself?
No. The NVIDIA AI Red Team says prompts and LLM reviewers can be misled through social engineering, gradual escalation, or legitimate-looking workflows. Run commands in a hardened sandbox or virtual machine, with network, file, and permission rules enforced by software outside the model.
What should change first after an incident review?
First deny egress by default, then remove long-lived secrets the agent can read, and finally block public package publishing and code installation from VCS URLs. Run negative tests for all three controls from inside the evaluation environment and confirm that the denial appears in the logs.