This is Part 2 of our vibe coding security benchmark study. In Part 1, we compared how LLM-based security tools like ProjectDiscovery's Neo and Claude Code performed against traditional SAST and DAST scanners on AI-generated code. We found that LLM-based tools like Neo and Claude Code detected many high-value findings that traditional scanners missed. Between Neo and Claude Code, Neo produced more true positives and fewer false positives because it could validate hypotheses against a running application.

This post goes one layer deeper around the apps we built, how we classified and validated findings, detailed walkthroughs of what Neo caught and what Claude got wrong, what Neo missed and what it means for teams securing AI-generated code.

The three apps we built

We wanted to build three apps with different roles and business logic, so we chose domains that generally carry strong security expectations: banking, healthcare, and insurance. We also deliberately used three different AI coding tools and three different tech stacks to avoid over-fitting our results to a single platform or framework.

MedPortal is a healthcare patient portal with patient records, appointments, prescriptions, lab results, referrals, and audit logging. The application featured five distinct roles: Patient, Doctor, Nurse, Admin, Lab Technician. The domain demands HIPAA-grade access control on every record.

VaultBank is a digital banking platform with accounts, transfers, deposits, loans, disputes, multi-currency support, and two-person approval workflows. The application featured five distinct roles: Customer, Teller, Branch Manager, Compliance Officer, Admin. The domain demands financial integrity; money in must equal money out.

ClaimFlow is an insurance claims management system with policies, claims, document management, settlement calculations, automated triage, and a configurable workflow state machine. The application featured five distinct roles: Policyholder, Claims Adjuster, Agent/Broker, Underwriter, Admin. The domain demands strict workflow enforcement and data isolation between policyholders.

Generation process and prompts

Each app was generated using three sequential prompts: foundation, auth + RBAC and complex features.

Each app started from a fresh session. We did not go back and correct anything once the session began. Each prompt was run to completion before moving to the next.

All three apps were deployed with production-like configurations and 5 role-based test accounts per app. The full source code is available at vibe-coding

Here are the actual prompts we used. Each app got three prompts in sequence.

MedPortal prompts (Codex)

Prompt 1 — Foundation

Prompt 2 — Authentication, RBAC + Core Features

Prompt 3 — Complex Features

VaultBank prompts (Claude Code)

Prompt 1 — Foundation

Prompt 2 — Authentication, RBAC + Core Features

Prompt 3 — Complex Features

ClaimFlow prompts (Cursor)

Prompt 1 — Foundation

Prompt 2 — Authentication, RBAC + Core Features

Prompt 3 — Complex Features

How we classified and validated findings

Every finding from every scanner was manually classified by our security research team as either VALID (reproducible, impactful, present in generated code) or FALSE POSITIVE (unreachable, mitigated by framework or not exploitable).

Severity follows CVSS-aligned scoring:

What we found

We found vulnerabilities in all three AI-generated applications. After reviewing and deduplicating results across all scanners, we confirmed 74 unique true positives in total. These vulnerabilities clustered around recurring patterns: authentication without authorization, mass assignment through ORMs, and broken business logic (unlimited deposits, refunds exceeding transaction amounts, policyholders creating admin accounts). The output looks production-ready because AI tools are trained on production patterns, but they systematically miss authorization, business rules, and data isolation.

Here is how the total vulnerability counts broke down by app and by severity:

And here's how each scanner performed:

Neo detected 89% of all exploitable vulnerabilities, including 100% of Critical and High findings. Claude detected 55% missing 3 Critical and 5 High findings. Invicti found 10 valid issues, all Info-severity. Snyk did not surface any confirmed vulnerabilities.

Since publishing Part 1, Neo identified four additional vulnerabilities during follow-up testing, bringing the confirmed total to 74 and Critical+High findings to 21.

In the following sections, we'll walk through examples of how Neo detected vulnerabilities that other tools missed and how runtime validation helped it avoid false positives that tripped up code-only review.

A closer look: What Neo found that nobody else did

Neo discovered 24 valid vulnerabilities that other scanners missed, many of which were serious findings that would end up in a standard incident report. Here are four walkthroughs that show how Neo found them by combining code analysis with runtime validation.

Walkthrough 1: Dispute resolution allows arbitrary refund amounts (VaultBank, Critical)

In the VaultBank app, Neo discovered that a user who disputes a transaction can successfully request a refund for any amount larger than the original transaction. For example, a user can dispute a $10 transaction and request a $10,000 refund. Neo found that the endpoint in the app processed the refund without checking whether the refund amount matches the original transaction.

How Neo found it. Neo delegated the investigation work to specialized agents that pursued multi-step attack chains, pivoting between code analysis and live exploitation. For this finding:

In this Neo investigation, Neo read the code, identified the missing business rule that refunds shouldn’t exceed original transaction amount, built a proof-of-concept, and confirmed exploitability against the running application.

This is a type of business logic vulnerability that is easy for traditional scanners or LLM-based code review tools without runtime validation to miss. Code-review tools aren’t able to trace data flows from dispute submission through refund calculation, and DAST scanners would need to understand intended business logic to recognize the refund flaw.

Walkthrough 2: Deactivated user retains full application access (ClaimFlow, Critical)

In the ClaimFlow app, Neo discovered that when an admin deactivates a user account, the user's existing session keeps working. Every API endpoint still responded normally, as if the account was never deactivated.

How Neo found it. This required a multi-step, multi-role test sequence:

Neo traced the session validation logic and found that it checks whether a valid session token exists, but never verifies whether the associated user is still active. In an insurance platform, a deactivated user might be a terminated employee, a flagged fraud account, or a user who requested account deletion. If their session continues to function, every access control decision the admin made is void.

This is the kind of vulnerability that requires testing what happens after an administrative action, not just the action itself. No single-request scanner or static code review would attempt this sequence, because it requires understanding the relationship between user status and session validity across role boundaries.

Walkthrough 3: Systemic password hash exposure via Drizzle ORM relations (ClaimFlow, High)

In the ClaimFlow app, Neo discovered that every endpoint returning user data also leaked password hashes. The vulnerability wasn't in the application code itself, but rather in how the ORM loaded related data. ClaimFlow uses SHA-256 with a hardcoded static salt for password hashing, so the exposed hashes are trivially crackable.

How Neo found it. Neo observed password hashes appearing in API responses across multiple endpoints. Rather than flagging a single instance, Neo identified the systemic pattern: Drizzle ORM's default column selection. When the API loaded users via db.query.users.findMany() without specifying columns, the ORM returned every field in the table, including passwordHash. The API then serialized the entire object into the response.

The code looks correct. There's no explicit select passwordHash anywhere. The developer never asked for the hash the ORM included it by default. This is what makes it hard for code-review tools to catch: they analyze what the code explicitly does, but the vulnerability comes from what the ORM implicitly returns. Neo tested what the running API actually returned, and found data that shouldn't have been there.

Walkthrough 4: Manager can freeze accounts across branches (VaultBank, High)

In the VaultBank app, Neo discovered that branch managers could freeze or unfreeze accounts belonging to any branch, not just their own. VaultBank assigns accounts to branches and managers to branches, so a branch manager should only operate on accounts within their branch — but the freeze/unfreeze endpoint checked the manager's role without checking their branch assignment.

How Neo found it. Neo authenticated as a branch manager, issued a freeze request against an account belonging to a different branch, and confirmed the operation succeeded. The role check passed because the user was a manager, but no check enforced branch scope.

This is a business logic vulnerability that looks correct on the surface, as the code does verify the user has the right role. But verifying a role is not the same as verifying scope. Traditional scanners and code-review tools see a valid role check and move on. Neo understood that branch-scoped authorization was the expected behavior for a banking application and tested the boundary that should exist between branches.

Deep dives: Claude false positives that Neo disproved

Claude Code performed well in this benchmark, surfacing 41 valid findings across the three applications. It also produced 24 false positives, which is expected when analysis can't reach runtime to confirm exploitability. We investigated each one and these three examples illustrate why certain vulnerability classes are difficult to validate through code review alone.

False positive 1: Mass assignment on user profile update (VaultBank)

Claude flagged the VaultBank /me profile update endpoint as vulnerable to mass assignment. If exploitable, an attacker could send {"role": "admin"} in a profile update and escalate to full administrative access.

What Claude saw. The endpoint loops through request body fields and applies them directly to the user object with setattr():

This is a textbook mass assignment pattern. Claude was right to be suspicious: in ClaimFlow, an almost identical pattern (...updateData spread into a database update) was exploitable and rated High severity.

What Claude couldn't see. The protection is in the type signature, not the function body. FastAPI validates the request body against the UserUpdate Pydantic schema before the handler executes:

This schema whitelists exactly 5 safe fields. The User model has role, is_active, kyc_status, account_tier, hashed_password, and email, but none appear in UserUpdate. Pydantic silently drops anything not in the schema before the handler ever sees it.

Why this is hard for code review to resolve. To know this endpoint is safe, Claude would need to trace the UserUpdatetype annotation back to the Pydantic schema in a separate file, understand that FastAPI enforces schema validation before calling the handler (a framework-specific behavior), compare the schema fields against the full User model, and verify the Pydantic model doesn't have extra="allow". That's a four-step cross-file inference chain through framework internals.

How Neo resolved it. Neo sent one request PUT /me with {"role": "admin", "first_name": "test"}, and observed that first_name updated but role remained customer. Because Neo couldn’t validate the role escalation in runtime, it flagged the finding as a false positive and didn’t present it as a valid finding.

False positive 2: Timing-unsafe password comparison (ClaimFlow)

Claude flagged ClaimFlow's password verification as vulnerable to a timing attack because the === operator short-circuits on the first mismatched character, which in theory leaks information about how much of the hash matched.

What Claude saw. The password verification function:

This is a classic anti-pattern. Every security guide says to use constant-time comparison for credential checks. Claude flagged it because === returns faster when the first character differs than when only the last character differs, potentially allowing an attacker to reconstruct the hash one character at a time.

Why it's not exploitable in practice. ClaimFlow hashes passwords with SHA-256, which always produces a fixed 64-character hex string regardless of input. Both correct and incorrect passwords hash to exactly the same length. The timing difference between comparing two 64-character strings that differ at position 1 versus position 64 is nanoseconds. Over an HTTP connection, network jitter is orders of magnitude larger than any measurable timing signal. An attacker would need thousands of attempts with nanosecond-precision measurement to extract even one character, which is not feasible over a network.

Claude correctly identified an anti-pattern (and using constant-time comparison is best practice). But "violates a security coding guideline" and "exploitable vulnerability" are different things. The implementation details (fixed-length hash output) and the deployment context (HTTP network latency) make the theoretical attack impractical. This is the kind of finding where runtime context turns a code-review concern into a non-issue.

False positive 3: No CSRF protection (VaultBank)

Claude flagged VaultBank for having no CSRF (Cross-Site Request Forgery) protection on its API endpoints. If the app used cookie-based authentication, this would be a real issue: an attacker could craft a page that submits requests using the victim's session cookie.

What Claude missed. VaultBank uses JWT-based authentication via the Authorization header, not cookies. CSRF attacks rely on the browser automatically attaching credentials to cross-origin requests. With header-based auth, the browser doesn't attach anything automatically. The attack vector doesn't exist.

Neo's agents understand that CSRF requires cookie-based authentication to be exploitable and didn't flag it. Claude applied a generic rule ("no CSRF tokens = vulnerable") without checking the authentication mechanism.

What Neo missed

We also wanted to review what Neo missed to understand where it could be improved. Here are the eight findings that Neo didn't surface:

After reviewing the true positives Neo missed, we found they were predominantly low and informational severity findings, things like missing hardening controls and standard DAST observations. This outcome makes sense given that our prompt explicitly directed Neo to prioritize high and critical vulnerabilities. If we were to re-run this analysis, we'd expand the prompt to include a dedicated section for documenting lower-severity findings that Neo can validate along the way.

How Snyk and Invicti performed

Snyk and Invicti represent the leaders in non-LLM-based SAST and DAST testing. In this benchmark, they surfaced few valid findings. Snyk discovered 0 valid findings (and 5 false positives). Invicti produced 10 valid findings, but all Info severity detections such as missing security headers, outdated JavaScript libraries and email address disclosure. Invicti also generated 10 false positives.

We believe this is because AI-generated code generally doesn't fail in the ways these tools are designed to catch. The syntactic patterns that SAST rules match eval() calls, SQL string concatenation, hardcoded credentials in specific formats largely don't appear in LLM output. The serious problems cluster around authorization, workflows, and business logic: a deposit endpoint with no amount validation, an ORM query with no ownership filter, a role check that doesn't enforce scope. These are semantic vulnerabilities and signature-based scanning isn't built to reason about them.

Snyk's false positives illustrate this gap: it flagged hardcoded passwords that were actually demo seed data, an open redirect that wasn't exploitable and code injection patterns that couldn't be triggered. In each case, the tool matched a string format without understanding the surrounding context. Invicti's false positives followed a similar pattern; 5 Next.js CVEs flagged against MedPortal based on version fingerprinting where the vulnerable code paths weren't reachable in the deployed application.

These results suggest that as AI-generated code matures, the vulnerability surface may shift away from the patterns traditional scanners were designed to catch. The detection gap we observed is a mismatch between what these traditional scanning tools look for and where the vulnerabilities will actually appear in LLM-generated code.

What the detailed analysis revealed

The summary stats from Part 1 showed that Neo found more and Claude produced more noise. The walkthroughs in Part 2 tell a more specific story about why.

The vulnerabilities Neo found that others missed were a different class of problem entirely, not just harder versions of the same class: disputes that allow arbitrary refunds, deactivated sessions that keep working, branch-scoped permissions that don't enforce branch scope. These require understanding what the application should do, then testing whether it actually does. Code review, even AI-powered code review, produces hypotheses about these behaviors. Runtime validation resolves them.

The false positives tell the inverse story. Claude flagged patterns that would be dangerous in isolation (mass assignment, timing-unsafe comparison, missing CSRF tokens), but each one was neutralized by context outside the code itself: framework validation, deployment environment, authentication architecture. Recognizing these mitigations required either deep cross-file inference through framework internals or a single request to the running application. The latter is faster and more reliable.

This is the practical takeaway from the detailed analysis: code review generates hypotheses, traditional DAST tests endpoints, but neither reasons through whether the application's actual behavior violates its intended logic. Closing that gap requires both the ability to understand what the application should do and the ability to test what it actually does.

Open-sourcing the benchmark

ProjectDiscovery started as an open source company, and we've always believed that security research improves when the data is public. The 74 confirmed vulnerabilities we found likely isn't the ceiling. We're open-sourcing the apps and publishing a public table of confirmed issues so the community can validate, challenge, and extend this benchmark. If Neo missed something, we want to know.

We'd also like to see other tools run against the same apps. If you maintain a security scanner and want to benchmark against the same codebase and deployments, the repo has everything you need: source code, deployment configs, and the full findings table.

The goal isn't to declare a winner. It's to build a shared understanding of where different approaches work and where they fall short on AI-generated code. That conversation is more productive when the data is open.

Conclusion

We built this benchmark to understand where AI-generated code fails and which tools catch what. The walkthroughs suggest the difference is whether hypotheses get tested against reality, not the underlying model. When AI review can't generate that proof, the burden falls on security teams to manually reproduce each finding, and that triage cost becomes the new bottleneck.

The methodology and data are open for others to replicate, challenge, or extend.

What's next. To test whether Neo's approach holds up on production code, we also ran it against popular open source projects and have reported over 20 CVEs so far. We'll be publishing walkthroughs of those findings in upcoming posts.

If you want to see what automated runtime validation looks like on your own codebase, reach out for a demo!

Heading to RSA? Come visit us at booth S3131 to discuss these findings and test drive Neo for yourself. No canned demos, no video magic. Sign up here: https://projectdiscovery.io/events/rsac-2026