10,000 Agents, $14,000, and a Quarter of Security Research in a Weekend

There is a number that should reorganize how you think about security budgets: a couple of days, ten thousand concurrent agents turned loose on an entire monorepo, and $14,000 in tokens — and out the other end, “several quarters worth of security research.” That framing comes from a recent conversation on Naval’s AI Industrial Revolution episode, and it’s the seed for this post. But the claim is no longer a thought experiment you have to take on faith. The receipts are now public, and they’re worth reading closely — both for what they prove and for where they quietly fall apart.

The thesis is simple. Red-teaming has always been gated by human hours. A principal pentester is expensive, scarce, and serial: they work one hypothesis at a time. Swap that constraint for compute and the economics invert. You stop rationing coverage. You stop testing the 25% of the attack surface you can afford and start testing all of it, continuously, because the marginal cost of another agent-hour is a rounding error against a salary.

The capability is real #

Start with the offensive side, because it’s the loudest. XBOW, an autonomous offensive-security platform, climbed to the #1 spot on HackerOne’s global leaderboard and submitted more than 1,060 validated vulnerabilities — out-producing every human researcher on the board. In its head-to-head benchmarks it matched a principal pentester’s 40-hour assessment in 28 minutes and cracked a cryptographic implementation in 17.5 minutes (XBOW). The 90-day run that put it on top broke down to 54 critical and 242 high-severity findings, including a previously unknown bug in Palo Alto’s GlobalProtect VPN that touched more than 2,000 hosts (XBOW). The architecture is exactly the shape the podcast describes: many short-lived agents, each with one narrow objective, coordinated by a persistent supervisor that holds the larger plan.

The defensive labs are running the same playbook. Anthropic’s Frontier Red Team, using Claude Opus 4.6, reported finding more than 500 previously unknown vulnerabilities in production open-source codebases — bugs that had survived years of expert review — by scanning whole repositories with multiple parallel agents stepping through source and data flows at once (Anthropic). OpenAI’s Aardvark agent (GPT-5-powered, multi-stage: build the threat model, scan commits against the full codebase, then sandbox-confirm exploitability) was credited with 10 CVEs discovered in open-source projects during early access (The Hacker News).

Academia lands in the same place from a different direction. The RepoAudit paper — an autonomous repository-level auditing agent, submitted and revised through 2025 — found 185 new bugs across high-profile codebases, 174 of which maintainers confirmed or fixed, at a precision of 78.43% on a 15-project benchmark. The cost-per-project line is the one to circle: $2.54 and 0.44 hours (arXiv). And the AgentFlow work pushed the scale further: a harness of 192 parallel explorers across 24 cloud nodes found 10 previously unknown vulnerabilities in Google Chrome — two of them critical sandbox escapes — over seven days on a codebase north of 35 million lines. Its standout finding is one builders should internalize: performance climbed from 35.2% to 84.3% across synthesis iterations, and harness architecture mattered more than which model you plugged in (arXiv).

That last point is the whole game. The leverage isn’t the model. It’s the factory you build around it.

The economics that make it inevitable #

The case for running thousands of agents isn’t capability — capability just makes it possible. It’s price. The ARTEMIS red-team agent ran at roughly $18 to $59 per hour depending on configuration, against a professional penetration tester’s average of $125,034 a year, and it still placed second out of ten in a head-to-head vulnerability-discovery competition with human experts (arXiv). On the cost engineering itself: ProjectDiscovery’s Neo platform averages around 26 LLM steps per security task — one task chewed through 67.5 million input tokens — and prompt-caching optimization cut LLM costs by 59%, dragging cache hit rates from 7% up to 84% (ProjectDiscovery). That’s the move that turns continuous full-coverage testing from an indulgence into a line item.

Why this matters now: enterprises today test only 20–25% of their applications because human pentesting is cost-prohibitive, against a traditional pentesting market of about $6 billion. And the offensive curve has already crossed over — AI-generated phishing went from underperforming human red teams by 18% in early 2023 to outperforming them by 24% by March 2025 (Menlo Ventures). The 75% of your attack surface you never test isn’t a budgeting choice anymore. It’s an unforced error.

Where it actually breaks #

Now the part the highlight reel skips. If you deploy this the way the marketing implies, you will hurt yourself.

False positives are real work, not a footnote. When Intruder’s engineering team used LLMs to write their Nuclei vulnerability-check templates, they found the generated templates “wasn’t using enough strong matchers to protect against false positives” — human experts had to add favicon-based matching the model consistently missed. Their conclusion was blunt: “expert input from our engineers is a must” (Intruder). Generation scales; the judgment that keeps the output trustworthy does not, at least not for free.

Hallucinations manufacture ghost vulnerabilities. Agents fabricate bugs that don’t exist and misread threat intelligence, and chasing those phantoms burns the exact SecOps capacity you bought the agents to free up (Help Net Security). A red-team report with a 30% confabulation rate isn’t a force multiplier — it’s a denial-of-service attack on your own triage queue.

Token cost is a ceiling, not just a punchline. $14,000 for one sprint reads as a steal against quarters of human labor, right up until you remember it’s a recurring cost that compounds across every team and every project. Enterprise AI spend is already outpacing the productivity it buys in some deployments — Uber reportedly burned through its 2026 AI coding budget in four months (Fortune). The economics invert in your favor only if you keep the harness efficient. Run it naively and the bill finds you.

The adversary gets the same multiplier. Every capability described here is dual-use by construction. The autonomous offensive platforms topping bug-bounty boards are, definitionally, autonomous attack platforms. Accelerate your defense with agents and you are also accelerating the threat environment you operate in — the floor for a competent attacker rises at exactly the rate yours does.

Judgment is still the gap. In that same head-to-head study, the agents dominated systematic enumeration and parallel exploitation but stumbled on GUI-driven web tasks and, more tellingly, on strategic pivoting after the first foothold (arXiv). That pivot — the creative lateral move from “I found a thing” to “here’s how I’d chain it into something that matters” — is precisely what separates a surface scan from a real red team. The agents give you breadth. They do not yet give you the attacker’s imagination.

What to actually do with this #

The honest read is that the 10,000-agent number is real and the marginal value of each agent is not uniform. The capability is here; the discipline is the bottleneck.

If you’re building, the shape of the work is clear. Spend your engineering on the harness, not the model — AgentFlow’s own data says the orchestration is where the gains live. Treat coverage as the thing you maximize now that it’s cheap, but put a human verification gate between agent output and anyone’s pager, because ghost vulnerabilities are how you train your team to ignore the real ones. Engineer your token spend like the recurring cost it is — caching is the difference between a sprint you can repeat weekly and one you do once for a press release. And keep your scarce humans pointed at the strategic pivot, the part the agents still can’t do.

The factory is real. So is the bill, the false-positive tax, and the adversary running the identical factory across the wire. The teams that win this won’t be the ones that spin up the most agents. They’ll be the ones who built the verification layer to trust what comes back.