Vals finds DeepSeek trails GPT-6 Sol Daybreak by 8 points in simulated cyber test

Vals AI’s CyberOffense evaluation found that DeepSeek V4.1 Flash 1017 completed about 72.7% of objectives across two simulated enterprise cyber ranges, compared with about 80.9% for OpenAI’s GPT-6 Sol using a Daybreak configuration. The 8.2-percentage-point gap came from Vals’ method of weighting the two ranges equally.

The result put a model with downloadable weights within about eight percentage points of OpenAI’s approval-gated cyber access in this benchmark. But the test measured autonomous agents in isolated ranges, not attacks on real organizations. It does not show that either system compromised a real network or was used maliciously.

Vals ran each model through the open-source Pi agent on an isolated Kali Linux machine that could connect only to the model provider and the assigned range. Each attempt used a fresh environment and ended when the agent stopped, exhausted its token budget or reached 48 hours. Input, output, cached-input and reasoning tokens all counted toward a 500 million-token ceiling.

Nova is a simulated Windows network spanning several connected Active Directory domains. An agent starts with its own workstation and a public web application, then must exploit the application, elevate privileges, map the network, reuse credentials, move between machines and domains, and collect protected data. Nova contains 21 objectives.

Lighthouse has 31 objectives across two separate simulated corporate networks. Information obtained after compromising the first network provides the only route into the second. Later stages include simulated phishing, credential use, lateral movement, abuse of trust relationships and data extraction.

Each objective contained a hidden marker that became readable only after the agent reached that stage. Vals searched recorded command output for markers in a private answer key instead of awarding credit for an agent’s own claim of success. It then averaged each model’s completion share across Nova and Lighthouse, giving the ranges equal weight despite their different objective counts.

GPT-6 Sol with Daybreak averaged 15.7 of 21 Nova objectives across three runs and completed 27 of 31 Lighthouse objectives. DeepSeek averaged 16.3 Nova objectives across three runs and completed 21 Lighthouse objectives. Vals did not state a repeated-run average for Lighthouse or provide confidence intervals.

The models also differed in compute use. Vals said GPT-6 Sol with Daybreak reached 25 Lighthouse objectives within 100 million tokens. DeepSeek completed none in its first 25 million tokens and made most of its progress after 100 million, according to Vals, which estimated that DeepSeek needed two to four times as many tokens to reach a similar stage.

Vals put the cost at under $3 per DeepSeek run and about $22 per GPT-6 Sol Daybreak run. It did not publish an itemized token mix or pricing calculation, so those cost estimates could not be independently checked from the published material.

OpenAI’s Daybreak documentation describes GPT-6 Sol as a general-purpose model and Daybreak Blue as an approval-gated setting that reduces cyber-related refusals. The benchmark tested GPT-6 Sol with a Daybreak configuration; it did not compare DeepSeek with a separate model called Daybreak.

DeepSeek released V4.1 Flash 1017 on Oct. 17 and linked downloadable model weights. Its model card lists the repository and weights under the MIT License. In Vals’ results, no model completed Lighthouse’s final objectives, and only one run in the full evaluation reached Nova’s final stage.