Skip to content
All posts

Introducing PentestBench

Introducing PentestBench: 17 web applications and 761 planted vulnerabilities for measuring how well OpenHack finds security issues.

Ananay Arora
7 min read
Introducing PentestBench, with the OpenHack logo and 17 applications, 761 planted vulnerabilities in white on black with a silver and gold glow
In this article

TL;DR

  • We’re launching PentestBench: 17 web applications and 761 planted vulnerabilities, including vulnerability-chain scenarios, to measure how well OpenHack finds security issues.
  • Our focus is finding and prioritizing web vulnerabilities, which we see as a largely unsolved problem.
  • OpenHack v2 with Mythos 5.2 scored 43.21% catalog F1, surpassing Claude Code Security and achieving the highest score among the six configurations shown. This result includes recovery of the proof pipeline.
  • The 17-app corpus is public, and the leaderboard is live. The ground-truth catalog remains private.

Introducing PentestBench

Today, we’re introducing PentestBench, a benchmark we built to measure how well OpenHack finds security issues across 17 web applications and 761 planted vulnerabilities.

We built it to benchmark OpenHack, the Always on AI Security Engineer we’re building. We need to know whether a new model or a change to our harness actually helps it find security issues in complete applications. PentestBench gives us a repeatable way to measure that progress.

PentestBench gives us a common set of applications and known vulnerabilities to measure against. Its first release compares six scanner configurations, with a public leaderboard showing detection coverage, report precision, and catalog F1.

Explore PentestBench and the leaderboard →

Why we built PentestBench

Benchmarks such as VulnBench already help evaluate vulnerability discovery. We built PentestBench to contribute more realistic web-application scenarios using modern frameworks, and to measure OpenHack against the kinds of applications we want it to secure.

That means testing how a tool follows authentication and authorization across routes, understands business logic, and connects vulnerabilities that become more damaging together. The corpus includes complete applications built with frameworks such as Next.js, SvelteKit, FastAPI, and Rails, so findings need to make sense in the context of the application.

Our focus is on finding and prioritizing web vulnerabilities, which we see as a largely unsolved problem. Finding a weakness is one part of the job. Understanding whether it is reachable, how it combines with other weaknesses, and what its impact means for the business determines what should be fixed first.

PentestBench starts with measuring vulnerability discovery across these scenarios. It gives us a repeatable baseline for improving OpenHack, with reachability and prioritization among the areas we want future evaluations to cover.

Built around complete web applications

Finding a vulnerability can require following a request through several parts of an application: a route handler, an authentication check, an authorization decision, and a database query. A function can look reasonable on its own while the way it connects to the rest of the application creates a security problem.

We wanted a benchmark that gives tools that broader context.

The first PentestBench corpus contains applications built with Next.js, Express, Fastify, SvelteKit, Django, Flask, FastAPI, Ruby on Rails, Laravel, Spring Boot, and Gin. The vulnerabilities are deliberately planted, so we have a reference catalog against which to evaluate findings.

Every evaluated tool receives the application source code. The reference vulnerability catalog stays sealed during evaluation. This first version focuses on white-box pentesting; black-box pentesting benchmarks are coming soon.

Vulnerabilities can combine into something worse

A security engineer needs to understand how one weakness changes the impact of another. PentestBench includes documented vulnerability-chain scenarios to test that reasoning in the context of an application.

For example, a weak session-signing key can enable an attacker to forge a session. If the application also trusts a role stored in that session, the weaknesses can combine into an authorization bypass. Cross-site scripting can also become session theft when session cookies are readable by JavaScript.

These relationships matter when investigating an application. The corpus includes them alongside individual vulnerabilities, and the leaderboard measures how many cataloged vulnerabilities each configuration finds.

Early results

We evaluated the following configurations across all 17 applications. These are the results published on November 16, 2026, ranked by catalog F1.

Catalog F1 by configuration
17 applications · 761 planted vulnerabilities
OpenHack v2 HarnessMythos 5.2
43.21%
Claude Code /security-reviewMythos 5.2 + Haiku
37.04%
Claude Code /security-reviewOpus 5.5
35.51%
Claude SecurityOpus 5.5 + Sonnet
34.91%
Claude SecurityMythos 5.2 + Sonnet
32.59%
Semgrep CE 1.185.0security-audit rules
4.69%
Results published November 16, 2026. Higher is better. All bars share a 0% to 50% scale. View the full leaderboard.
Scanner or harnessModel or configurationRecallPrecisionCatalog F1
OpenHack v2 HarnessMythos 5.236.40%53.17%43.21%
Claude Code /security-reviewMythos 5.2 + Haiku26.28%62.70%37.04%
Claude Code /security-reviewOpus 5.523.92%68.94%35.51%
Claude SecurityOpus 5.5 + Sonnet40.74%30.54%34.91%
Claude SecurityMythos 5.2 + Sonnet42.97%26.24%32.59%
Semgrep CE 1.185.0security-audit rules2.50%38.00%4.69%

OpenHack v2 Harness with Mythos 5.2 achieved the highest catalog F1 among the six configurations shown, finding 277 of the 761 planted vulnerabilities across 521 reports. Claude Security with Mythos 5.2 + Sonnet found the most planted vulnerabilities, while Claude Code /security-review with Opus 5.5 had the highest catalog precision.

The OpenHack score includes diagnostic recovery: failed verification attempts were retried after repairs to the proof environment, using the same frozen candidates and application source.

Those differences are why we show all three metrics. A tool can find more vulnerabilities while producing more reports that do not match the catalog. Another can produce a smaller set of reports with higher precision while missing more planted issues.

There is also substantial room to improve: even the configuration with the highest recall found fewer than half of the planted vulnerabilities.

We wanted to include the best models from OpenAI as well. We’re still awaiting Daybreak Red access from OpenAI to use GPT-6 Astra for this benchmark.

How to read the scores

  • Recall is the share of the 761 planted vulnerabilities a configuration found.
  • Precision is the share of its reports that matched a planted vulnerability.
  • Catalog F1 is the harmonic mean of recall and precision, balancing coverage with report precision.

Precision here is measured against the planted vulnerability catalog. A report that does not match that catalog does not contribute to the score; that alone does not establish whether it describes a real issue outside the catalog.

Each row represents a model and harness configuration working together. The results do not isolate the model’s contribution, and the tools produce different kinds of evidence. The leaderboard identifies those differences, from unverified static hits and reports to code review and runtime proof with review.

OpenHack maintains PentestBench and develops one of the evaluated harnesses. We’re publishing the results by configuration so readers can examine coverage, precision, and evidence alongside the overall ranking.

What comes next

We’ll launch and open-source the OpenHack v2 harness later this week. PentestBench provides the baseline for evaluating it, and we’ll share more about the harness at launch.

The first version of the corpus is now public. All 17 application targets are available in the PentestBench repository, with a manifest and checksum verification script. You can inspect the applications and use them for your own evaluations. The private ground-truth catalog is not included.

We also want to extend the benchmark to applications deployed in cloud environments, test vulnerability reachability, and evaluate the ability to identify false positives. Those are capabilities we need from an AI security engineer working on real software. Black-box evaluations will extend the benchmark beyond the source-code-access setting used in this release.

For now, PentestBench gives us a concrete starting point: the same applications, a known set of vulnerabilities, and a way to measure where models and harnesses improve and where they still fall short.

See the results at pentestbench.com →

PentestBenchOpenHackAI SecurityPentestingBenchmarks

Ananay Arora

Founder, OpenHack