CWE-bench v2 opens 100 public security-repair tasks

Collinear AI opened CWE-bench v2, a public set of 100 audit-and-patch tasks spanning 91 Common Weakness Enumeration types.

The release gives coding-agent developers an open way to test whether their systems can find and repair security weaknesses in real open-source repositories without breaking legitimate behavior. The tasks cover eight programming languages and use programmatic verifiers with documented binary scoring rules.

According to the public dataset card, each task provides a repository containing one weakness, an instruction, a Docker environment, a reference fix and a verifier. The language mix is 27 Go tasks, 15 Java, 15 JavaScript, 14 C++, 10 C, 10 Python, five Rust and four TypeScript.

A patch scores 1 only if it closes every supplied exploit input and preserves every legitimate test case. One failed check produces a 0; if the verifier cannot evaluate a trial, that trial is voided rather than counted as a failure. The benchmark is packaged as self-describing Harbor tasks, with registry versions pinned to exact repository commits. The documented full evaluation runs four attempts per task and averages the reward across 400 trials.

Access to the dataset is public but gated. Users must accept access conditions before downloading its files; the dataset card says approval is automatic and prohibits training evaluated models on the tasks or republishing solutions.

A November 6 changelog entry records fixes to two task verifiers so their reference solutions score 1 in plain Docker. Collinear AI says those fixes did not change task requirements or model scores; that result was not independently reproduced.

Collinear AI also says a separate held-out evaluation contains 150 tasks across 126 CWE types, bringing the combined benchmark to 250 tasks and 185 distinct CWEs. Because those tasks are not public, their composition, quality controls and status as unseen evaluation material could not be independently inspected. The company says every task passed seven quality checks and manual review, but no independent audit of that process was available in the opened sources.

CWE is MITRE’s community-developed list of software and hardware weakness types. The benchmark’s October research paper described an earlier private evaluation of 120 tasks spanning 73 CWE types and eight languages. In v2, the public set is new, while a separate evaluation remains held out.