Open models close in on closed ones in AI cyber attack tests

Vals AI tested AI agents against a simulated corporate network. Each agent worked alone on an isolated Kali Linux machine, starting only from a public web address, and every action was logged and scored automatically. GPT-6 Sol Daybreak, part of OpenAI's restricted Daybreak cyber program, led with 80.8 percent of objectives, and Deepseek V4.1 Flash 1017 reached 72.8 percent.

In the harder Lighthouse test, Daybreak took admin control of the first company, stole passwords and used them to break into the second, completing 27 of 31 objectives in under three hours. Deepseek completed 21. It scored zero in its first 25 million tokens and made most of its progress only after 100 million, so Vals says short tests underestimate open models.

The results match a July analysis by the UK's AI Security Institute, which put the gap at four to seven months, down from six to ten months for most of 2025. AISI also reported that a cyber-range run with Deepseek V4-Pro cost $1.19, compared with $85 for Anthropic's Opus 4.5, and that safety measures on open models proved largely ineffective.

Developers building agents on open-weight models can get close to top cyber-task results at a fraction of the cost, so price and access may matter as much as a vendor's restrictions. Builders should also expect that safety measures on open models may be weak.

Short evaluations can understate what a model does over longer runs, so agent tests should include large token budgets before a model is judged.