# Can AI Penetration Testing Replace a Human Security Assessment? What ARTEX Tells Us About Accuracy, Limits and Buying Decisions

> Can AI penetration testing replace human assessments? Using public ARTEX attack reports: what automates well, how to ask about detection rates, scope and evidence.

- Published: 2026-11-03
- Author: 友田 陽大
- Tags: セキュリティ, 脆弱性診断, AI, 認証・認可, OWASP, 発注, B2B SaaS
- URL: https://tomodahinata.com/en/blog/ai-pentesting-artex-human-security-audit-comparison
- Category: Application-layer security
- Pillar guide: https://tomodahinata.com/en/blog/nextjs-supabase-application-security-guide

## Key points

- Think of AI penetration testing as a division of labor, not a replacement for human assessment. Automation is good at probing widely without rest; interpreting the spec, setting the permitted scope and taking responsibility remain human work.
- None of the public analyses from the five organizations the author read gives independently compared detection or false-positive rates. A figure like 'X% as accurate as a human' is not evidence; ask the vendor how they measured it.
- An authorization flaw can only be judged if you know what the response should have been, even when it returns 200, and that expectation lives in the spec. This is often described as hard for AI today, but giving the tool that expectation could widen what can be automated.
- The evidence here is reported attacker misuse. The author did not survey commercial AI assessment services you could actually buy. Because the author sells audits (a conflict of interest), the cases where you do not need one, and the free checks, come first.

---

## Conclusion: think "division of labor", not "replacement"

**The realistic view is that AI penetration testing does not replace human security assessment wholesale; the two are combined, each doing different work.** At the very least, that is as far as the public material lets us go.

- **What automation is thought to be good at:** probing the public surface widely, trying known weaknesses in turn, trying something else when an attempt fails, and doing all of this in parallel without rest.
- **What still needs people:** interpreting the spec ("who may touch which data"), understanding business logic, deciding how far testing may go, taking responsibility for the results, and producing evidence you can explain to management and customers.

This article is written so that CTOs, IT leads and procurement owners can judge **which checks their own product actually needs**.

**I sell paid security audits for web applications.** That is a conflict of interest, so I do three things: (1) I state **when free, do-it-yourself checks are enough**, (2) I do not exaggerate the danger of AI, and (3) I give no number I cannot back up.

### The limit of this article: ARTEX is not an option you can buy

First, the scope. The evidence here is **public reporting on attackers using ARTEX**. ARTEX is not a service you can order. The successes and failures reported tell us "what this technology can do"; they do not tell us "what an AI assessment vendor would find in your application". **I did not survey commercial AI assessment services or any vendor's products.** So I make no claim about which specific service is better.

## First, the terms

This article uses the following terms. (NIST SP 800-115 is a useful reference for thinking about testing and assessment, but where I draw the lines between terms is my own arrangement.)

| Term | What it does | Purpose |
|---|---|---|
| **Vulnerability scanner** | Automatically runs a predefined set of checks | Find known issues broadly, cheaply and by the same procedure each time |
| **Vulnerability assessment** | Enumerates weaknesses comprehensively (automated + manual) | Produce a list of issues and their priority |
| **Penetration test** | Tries to actually break in toward a goal (for example, viewing another company's data) | Confirm whether weaknesses chain into real damage |
| **AI penetration test** | An LLM agent decides the "next move" in the row above | Automate how the steps are put together |
| **Security audit** (what I offer) | Reads source code and configuration to check down to design gaps | Verify how "who may touch which data" is implemented, and give a fix plan |

How ARTEX works and the misuse that has been reported are covered in [What Is ARTEX? How AI-Driven Autonomous Penetration Testing Works](/blog/artex-ai-pentesting-architecture-guide). For costs and how to choose between assessment methods, see [Web application vulnerability assessment cost and pricing](/blog/web-application-vulnerability-assessment-cost-pricing-guide) and [Comparing vulnerability scanning tools](/blog/web-application-vulnerability-scanner-tools-comparison-zap-burp-semgrep-guide).

## What the public material shows AI can and cannot do

This is what I **could confirm** about ARTEX from the public analyses of five organizations I read (Google, Microsoft, AhnLab ASEC, JPCERT/CC and CrowdStrike).

**What it has done**

- **There is a case of reaching intrusion from a short instruction.** In ASEC's test, an instruction no more elaborate than "break into the target" was enough, with no human intervention, to get from a file-upload feature to execution of an arbitrary program, and, with ActiveMQ (CVE-2023-46604), to command execution on four internal hosts. Note that this was an **environment in which known weaknesses were deliberately placed**.
- **It is fast.** In GTIG's observations, individual runs came in bursts of 2 to 25 seconds, and for one specific actor GTIG tracks, going from the first compromised account to deep access normally took about 4 hours, and as little as 15 minutes.
- **It recovers from failure.** There is a recorded case where it read an error, trimmed its script, re-ran it and succeeded.
- **Records of business-function abuse.** GTIG documented abuse of a writable admin flag (`is_admin=true`), attempts to read other users' attachments, and mass email sent through a notification feature. The success rates, and how much of this it found autonomously, are unknown.

**What it has not done, or has done only in a limited way**

- **Success is not uniform.** Microsoft says most of the targeting did not succeed. In the same sentence, it also says Storm-0940 gained privileged cloud persistence in many organizations within a short time.
- **It is noisy.** GTIG assesses that, compared with a skilled human, the path to code execution is mostly trial and error and is far noisier from a forensic standpoint.
- **Agents may have interfered with each other.** GTIG recorded errors suggesting that multiple sub-agents may have overwritten one another (in the context of operations against the target).

**What the public material cannot confirm**

- **Detection and false-positive rates.** In the material I read, I found no figures that compare AI assessment's detection or false-positive rates against humans or scanners independently. (ASEC's test is two cases, not a rate.) **A figure such as "AI is X% as accurate as a human" cannot be used as evidence at present.** I have not exhaustively surveyed other public evaluations.
- **The share of unknown weaknesses, or of authorization and business-logic flaws, that it finds.** GTIG's records above are examples, not rates.
- **Whether AI makes assessments cheaper or faster.** I could not find public data.

## Checks that automate easily, and checks that do not

| Check | How easily it automates | Why |
|---|---|---|
| Public-surface discovery (subdomains, ports, services) | High | Can be enumerated mechanically; results are objective |
| Applying known vulnerabilities (CVEs) | High | Versions and known procedures are fixed |
| Exposed configuration/backup files, public admin panels | High | Found by trying file names and paths |
| Limits on authentication failures (lockout, etc.) | High | Try it and look at the result |
| Injection classes (SQLi, XSS, SSRF, etc.) | Medium to high | Patterns are well systematized, but context causes misses and false positives |
| **Authorization flaws (seeing another person's or tenant's data)** | Often described as hard today | What is correct is written only in the spec (see below) |
| **Business-logic flaws (bypassing approval, tampering with amounts, double use)** | Often described as hard | Requires understanding the business intent |
| Whether permission design across multiple roles and tenants is sound | Hard | Has to be checked against how the organization really works |

This table is **my own assessment as of November 2026**, drawn from observations in the public material and from general principles of testing; it is not backed by public data. As tools improve, the ratings may change. And as described below, **giving the tool the expectation (accounts and ownership relationships) could widen what can be automated.** Please also bear in mind that the audit I offer works in exactly this "hard" area, so this assessment carries a conflict of interest.

### Why authorization flaws are hard: the "expectation" exists only in the spec

The obstacle with an authorization flaw is that the response **looks normal**. Say a user of Company A specifies the ID of one of Company B's invoices and the content comes back with a 200. As HTTP, that is a successful, normal response. To judge it wrong, you need the **expectation** that "this user belongs to Company A and must not be able to see Company B's data". That expectation is written in the business spec, not in the code.

In the related article, [How should web apps prepare for ARTEX-style AI attacks?](/blog/artex-agentic-ai-web-application-security-defense), I laid out the actors (roles and tenants), the operations that should be allowed and denied, and the actual responses in a table, and verified 29 items. If you have an AI agent hunt for another company's data, the same holds: unless you give it at least two accounts and tell it whose data is whose, it cannot decide whether a boundary was crossed. OWASP API Security also asks, as a countermeasure for this class of flaw (BOLA), that you **write tests that evaluate authorization vulnerabilities**.

## How to ask about detection rate, false positives and reproducibility

If the numbers are unusable, ask about **how they were measured** instead.

| What to ask | Why it matters |
|---|---|
| How was the detection rate measured (against which targets, with how many known flaws planted)? | With no independent comparison available, the vendor's own method is the only lead |
| How are false positives handled (does a person verify before reporting)? | False positives consume the time of whoever has to check them |
| If you run it twice on the same target, are the results the same? | LLM judgment is non-deterministic. Low reproducibility makes re-verification after a fix hard |
| Does the report include reproduction steps for each finding? | You cannot tell whether you fixed something you cannot reproduce |
| Does it say what was tried and **not found**? | "No issues" and "not tested" are different things |
| Does the price include the human effort of verifying results and writing the report? | Even if automation gets cheaper, the verification cost remains |

The value of an assessment lies not only in how many issues it finds but also in the scope of "what was tried and what was not there". If the AI takes a different path each time, a record of what was tried (the execution log) becomes the basis for the report.

## Privileges, permitted scope and the risk of wrong operations

Automated assessment can cause **unintended side effects**. In the abuse GTIG recorded, ARTEX had an application's notification feature send test emails to more than 1,300 people inside the organization. For an attacker that was goal-directed behavior, but **if it happened during a legitimate assessment it would be an incident**.

If you use automated assessment (in-house or outsourced), settle at least the following in writing.

- **Ownership of the target and permission.** Only targets you own or have written permission to test. If the design touches external SaaS or shared infrastructure, check that provider's terms too.
- **Scope and prohibited actions.** What may be touched in production and what may not (sending email, payments, deleting or modifying data, heavy-load operations).
- **Hours of operation and the stop procedure.** Who stops it, and how, if something goes wrong.
- **Data handling.** If the design sends information obtained during testing (credentials, personal data) to an LLM provider's external API, confirm that this is acceptable under your contracts and confidentiality obligations.
- **Least privilege.** Narrow the privileges and tools given to the agent. What OWASP calls "Excessive Agency" in LLM applications (too much functionality, privilege or autonomy) applies to an assessment agent as well.

ASEC notes that ARTEX's design includes approval steps and a communication log. However, **I have not confirmed what the approval gate blocks by default, or whether it can be disabled.** Judge by the operating rules you set yourself and the procedures you can actually use to stop it, not by feature names.

## Preserving evidence, and the report

An assessment's deliverable is not just a list of issues; it is the evidence you show customers and auditors.

- Requests and responses that can be reproduced (in a form with secrets redacted).
- Dates and times of the test, the scope of the target, and the tools used and their versions.
- A record of items tried and not found, and the basis for each severity rating.
- Results of re-verification after fixes.

When the assessment includes AI, keeping the execution log (what was tried, in what order) fills the reproducibility gap.

## A realistic testing set-up for a small SaaS

It is not a choice between "outsource everything" and "do nothing". You can think in three tiers of frequency and depth.

| Tier | Frequency | What | Who |
|---|---|---|---|
| **Continuous** | On every change | Dependency updates, secret scanning, static analysis, **authorization regression tests in CI** | You (automated) |
| **Milestone** | Before launch, before a major change | An inventory of the public surface; broad checks with scanners and automated tools (in an environment you are permitted to test) | You + tools |
| **Independent review** | Decided by the size of the change and customer requirements (once a year is only one rough guide) | Check authorization, tenant isolation, admin paths and configuration from both design and code | A third-party human |

AI may be usable for the broad checks in the "milestone" tier. However, **a team with no "continuous" automated tests will get limited benefit from an AI assessment.** It has no foundation for fixing what is found and confirming that it is fixed.

**When you do not need an outside audit:** you have few users, hold no sensitive data and are pre-launch, and no customer or partner has asked for independent confirmation. Separately, if you do not yet have the "continuous" checks, building them yourself before any audit gives better cost-effectiveness.

**When an outside audit is worth considering:** you are putting a B2B SaaS with multiple tenants and roles into production; customers are asking for evidence of third-party verification; there are paths outside the ordinary screens, such as admin tools, operations tooling, batch jobs and external integrations; or you feel the limits of reviewing your own code (especially code built quickly with generative AI).

## What you can check yourself

Much of the self-check can be done for free. The concrete steps are in the "what you can check yourself" part of the related defense article above (how to confirm that another tenant's IDs are rejected, down to making the test fail under mutation; a free OSS scan; the 20-question self-check, and so on). For how to use assessment tools, see [Practical ZAP, SAST and DAST](/blog/web-application-vulnerability-assessment-owasp-zap-sast-dast-guide). When you take stock of your public surface, check that configuration files, backups, admin panels and update servers are not exposed (JPCERT/CC has reported attempts at exploitation in Japan).

## About the audit I offer

I describe only the current scope. Details and pricing are at [/security-audit](/security-audit) (I do not state amounts in this article because they go stale with updates).

- **Who it suits:** operators of SaaS and web applications around the time they start holding production customer data.
- **When it does not suit:** when you fall under "when you do not need an outside audit" above.
- **What I do not promise:** I do not say "completely safe". I report the issues found separately from the risk that remains.
- **A sample report:** [/vibe-coding/sample-report](/vibe-coding/sample-report) is a sample of the pre-launch light review (built around a fictional app, and the entry-level tier).
- **About how this article was written:** I have not obtained or run ARTEX and have not used it for assessments. This article rests on reading the public material and on verifying defenses on a fictional SaaS.

If you just want to talk first, there is also a free 30-minute technical consultation ([/contact](/contact)).

## Order of decisions

1. Know what you have: your public surface and the kinds of data you hold.
2. Build "continuous" automated checks and authorization regression tests.
3. Then consider whether to use AI or automation for broad checks. If you do, settle the permitted scope and the stop procedure first.
4. Decide whether you need an independent human review using the conditions above.

What sets the boundary to defend is the spec, and the one who understands that spec and takes responsibility for it is a person. Even in an age when AI can try things quickly, that order does not change.