We Pointed Our Own PTaaS at OWASP Juice Shop
Oct 29, 2026
A real Helix PTaaS engagement, end to end: a gray-box web pentest against OWASP Juice Shop, run by open-weight models on self-hosted inference. 33 findings, every one reproduced by a second agent. Plus a side-by-side of two models, GLM-5.3-Flash and DeepSeek-V4.1-Flash, on the identical job.
The honest way to show what a penetration test does is to run one against something and show the result. So we did. We took Helix PTaaS and ordered a gray-box web assessment the way a customer would. The target was one we are free to attack as hard as we like: OWASP Juice Shop, the deliberately vulnerable app its author built for security training.
This post follows that engagement from the order to the report. Then it does something new for us: the same assessment, run twice on two different open-weight models, side by side.
One caveat up front. Juice Shop is built to be broken, and it ships its own scoreboard of planted bugs. That makes it a fair test of whether a tool finds things. It is a poor stand-in for a real production app, where the interesting findings are the ones nobody planted. Read the numbers below as a coverage check and a model comparison, not a league table against other vendors.
The setup: an order, not a script
Everything here went through the same portal a customer uses. We created an organisation, added two ordinary customer accounts to the Juice Shop instance, and ordered a web engagement with the gray box package. Gray box means authenticated testing. The fastest way to find broken access control is to log in as one user and try to read another user's data, so we used two accounts, two identities, on purpose.
Then the gate that matters. The portal will not provision an engagement until a customer admin has signed the rules of engagement, and the signature is bound to the exact host, the exclusions, and the test window. No agent sends a request before that signature exists. This is enforced in the backend, not the interface. Once it was signed, the portal provisioned a Helix project for the engagement and sealed the two account passwords into it as secrets the agents read by name.
How the test runs
Starting the run hands the job to one coordinator agent. It does not start attacking. It follows the same method our human-run engagements use:
- Recon, twice. An unauthenticated crawl maps the public surface, then an authenticated crawl maps everything reachable as each of the two accounts. On this run that produced 60 API entries and 44 single-page-app routes, plus a 53-row server-side role matrix across the REST API.
- Eight testing lanes, in parallel. Configuration, authentication, session, authorization and IDOR, business logic, injection and XSS, SSRF, and client-side each run as their own agent in their own sandbox, against a signed scope ceiling of exactly one origin and a per-lane request budget.
- Independent verification. The rule that separates a finding from a guess: the agent that files a candidate is never the one that confirms it. A separate verifier replays each candidate from scratch, with fresh sessions, and argues the strongest case against it. In the ledger this is a hard constraint, not a convention.
- Escalation chains. A final pass works over the verified findings and joins them into attack paths, so two low-severity issues that combine into account takeover get reported as the chain.
Throughout, testing is detect-only, every request passes a scope guard that blocks anything off the signed host, and secrets are masked before they reach the evidence ledger.
What it found
The run filed 44 candidates. The verifier reproduced 27 without qualification and flagged the rest for human review on impact context, with deterministic replay evidence attached to each. Zero were dropped as false positives. The published report carries 33 findings:
| Severity | Count | Examples |
|---|---|---|
| Critical | 2 | JWT alg=none accepted, including forged admin tokens; an endpoint that returns every user's password hashes and 2FA seeds to any logged-in customer |
| High | 9 | Cross-user reads of baskets, profiles, the user directory and order PDFs; boolean SQL injection in product search; SSRF with full response capture; DOM XSS; negative-quantity and expired-coupon logic flaws |
| Medium | 14 | Public metrics endpoint, directory listing, verbose stack traces, non-single-use checkout, admin config readable by any customer |
| Low | 6 | Permissive CORS, missing security headers, exposed Swagger UI |
| Info | 2 | Version and path disclosure |
The report reads like a human one because it is built from the same structure: a scope section, a per-lane methodology table with explicit negatives, one ticket per finding with replayable evidence and a severity justification, and a conclusions section. It names what it could not reach as honestly as what it could. Three surfaces came back marked untested rather than clean: the login rate-limiter's keying, the socket.io chat protocol past the handshake, and the external OAuth flow, which leaves the single-host scope. The escalation pass recorded 10 chains and proved 4 of them end to end.
That honesty about coverage is the part a scanner's output never has, and the part an auditor actually reads.
Two models, one job
The whole roster runs on an open-weight model on our own inference, coordinator and lanes alike. Nothing goes to a frontier API. So we ran the identical engagement twice, same target, same accounts, same instructions, changing only the model:
| GLM-5.3-Flash | DeepSeek-V4.1-Flash | |
|---|---|---|
| Approach | Orchestrated the full parallel roster plus an independent verifier | Same instructions; first pass capped by a sandbox-concurrency limit (since fixed) |
| Findings | 33 (2 Critical, 9 High, 14 Medium, 6 Low, 2 Info) | 9 (1 Critical, 2 High, 3 Medium, 3 Low) on the capped pass |
| Independently verified | 27, with 0 false positives | Yes, finder separated from verifier |
| Wall clock | About 3 hours | About 20 minutes on the capped pass |
Both models found the same signature vulnerabilities on the first pass: the SQL injection, the cross-user IDOR, the exposed file server, the authentication-detail leak. GLM-5.3-Flash fanned out eight lanes and a verifier and ground through 44 candidates over three hours. DeepSeek-V4.1-Flash surfaced the headline classes in twenty minutes and stopped there.
We first read that as a difference between the models. It was not. DeepSeek's run hit a sandbox-concurrency limit on our side. Only ten sandboxes could run at once, so most of the parallel lanes never got one, and the run fell back to a fast single pass over the obvious classes. That is an infrastructure limit, not a property of the model. We have since fixed it: runs now queue at the sandbox cap instead of being dropped, and the cap is higher.
So the only honest comparison is a clean rerun under the fixed infrastructure, which is what we are doing now. The question we actually want answered is the practical one: on the identical job, which model is better on cost, time, and accuracy? We will report those numbers here once both runs complete on equal footing. What the first pass already establishes is narrow but real: both models find the signature vulnerability classes; what is still open is how deep each one goes, and what it costs to get there.
What this shows, and what it doesn't
It shows that an open-weight model on self-hosted inference can run a real multi-agent web assessment end to end. It reproduced every finding it reported and wrote up the result, coverage gaps and all, in a form an engineer or an auditor can act on. The code, the traffic, and the findings never left infrastructure we control.
It does not show that we beat anyone. Juice Shop is designed to be found; a good scanner finds many of these too. The thing worth paying attention to is not the count but the discipline behind it: a second agent reproduces every finding before you see it, the report states what was not tested, and a human reviews and merges the fixes. That is the part that survives contact with a real production target, where no bug comes labelled.
If you want the same thing pointed at your own application, that is the Helix Cyber product, from web app and API assessments on up.
Appendix: the engagement, in portal terms
- Type, package, template: web, gray box, agile.
- Scope: one signed host (OWASP Juice Shop v20.3.0), authenticated as two customer accounts. The container host was out of scope.
- Authorization: rules of engagement signed before provisioning; detect-only; independent verification required for every finding.
- Delivery: one Helix project per engagement, agents in sandboxes, findings synced to the portal as tickets, report rendered from the structured records in technical, executive, and compliance-mapped forms.