Benchmarks
The right way to evaluate a security agent is to see whether it can actually find and exploit vulnerabilities end-to-end. Below are OpenHack's results on CVE-Bench — the leading agentic exploitation benchmark — plus how the harness compares to the previous generation of static analysis.
CVE-Bench
CVE-Bench (UIUC Kang Lab) gives an agent a live, vulnerable application in a sandbox and asks it to discover and exploit a real CVE end-to-end. We ran it 6 times in the one-day setting and once in the zero-day setting. OpenHack runs alongside the published leader on the leaderboard — using a fully open-source model.
Mean one-day pass@1 across 6 runs (best 35.0%)
Zero-day pass@1 (no CVE description)
Unique CVEs exploited across all runs
Agent receives the CVE description.
OpenHack range across 6 runs: 27.5% – 35.0%.
Agent gets no prior knowledge of the CVE.
Single validated run after fixing an NVD-description leak in our solver. The ~5pt gap to Opus 4.6 is within run-to-run variance.
We publish every run, not just the best one.
19 unique CVEs were exploited across all runs (of 40). 9 of those were exploited in every single run — the consistently-solvable core.
Same harness, same token volume, two different models. Kimi K2.5 numbers are what we actually paid. Opus 4.6 numbers apply Anthropic's published list price ($15/M input, $75/M output) to the same token counts, with a typical 85/15 input/output split.
Kimi K2.5 per full CVE-Bench pass (scan + eval)
Opus 4.6 estimated for the same pass
Cheaper, on equivalent token volume
Kimi K2.5 figures are what we actually paid. Opus 4.6 figures apply Anthropic's list price ($15/M input, $75/M output) to the same token volume, with a typical 85/15 input/output split. Bars are scaled to the Opus midpoint.
| Phase | Tokens | Kimi K2.5 | Opus 4.6 (est.) | Multiplier |
|---|---|---|---|---|
| Scan (40 CVEs) | 73.3M | $32.74 | $1,100–$1,800 | 34–55× |
| Eval (one full pass) | 16.3M | ~$8 | $250–$400 | 31–50× |
| Total per pass | 89.6M | ~$40 | $1,400–$2,200 | 35–55× |
| Full benchmark (7 passes) | ~627M | ~$230 | $9,800–$15,400 | 43–67× |
Frontier-model agents win these leaderboards because they have the raw capability. OpenHack's contribution is the harness: recon, hunting, validation, and sandboxed verification, all built to extract that same capability from open-source models. You get a leader-class result without the lock-in or the bill of a frontier API.
vs. legacy SAST
For comparison, we also benchmarked OpenHack against 768 known vulnerabilities across 17 open-source applications and ran the same codebases through Bearer, Semgrep, and Snyk Code. Static pattern matching can't see logic bugs — semantic agents can.
Out of 768 known vulnerabilities
Percentage of reported findings that are real
Benchmarked against 768 known vulnerabilities across 17 open-source applications.
CVE-Bench is the proving ground. Your codebase is the point. Connect your repos and run OpenHack against the things that actually matter to your business.