Software projects now commonly run AI vulnerability discovery tools, also called harnesses, that wrap a large language model (LLM) around a series of fixed steps. The harnesses select files from a repository, send the code to the model to look for security flaws, and then collect and report any problems the model identifies. The model itself does not choose which files to open or decide which results to keep. Ordinary program code makes those decisions to control cost and reduce false alarms.[1]
In recent months, tools like this have helped identify serious flaws in widely used software packages that had existed for years. One example is a set of remote-code-execution vulnerabilities in PHP projects. Researchers using language-model assistance identified conditions under which an attacker who can supply input to a PHP application can cause the server to run arbitrary code. Similar assisted searches have produced reports of previously unreported issues in Python and Node.js packages. In each case, the model examined code that a human researcher also reviewed, and the resulting reports described reachable paths from an entry point to a dangerous operation.
These same tools can also return reports that list no findings. However, just because an AI tool indicates that it hasn’t found anything doesn’t mean that the code contains no exploitable vulnerability. It can mean the model examined the vulnerable code and failed to recognize the problem, that a deterministic step in the harness prevented the vulnerable code from reaching the model, or that the harness discarded a finding after the model produced it. The report a user receives is typically identical in all three cases. The exit status, the summary file, and any pull-request comment simply state that no problems were found.
In his paper, “Where AI Vulnerability Discovery Harnesses Go Quiet: Structural False Negatives and Hidden Vulnerabilities,” Gadi Evron documents his examination of several open-source AI vulnerability harnesses. He started with two that he maintains, OpenAnt and raptor, and expanded the research to include the source code of ten additional open-source harnesses, including tools released by OpenAI, Anthropic, Cisco, Visa, Capital One, Rivian, Vercel, and Trail of Bits. For each harness, he identified the stages that select files, parse them, build a call graph, invoke the model, filter results, and write the final report. He planted small files containing known reachable vulnerabilities and observed which stages removed the files or the findings.
Evron cataloged thirteen distinct points at which a reachable vulnerability can be dropped before the model analyzes the code or after the model has produced a finding. Six of the twelve harnesses dropped the same planted file before their model ever saw its contents, and five of those six produced no notice in any deliverable the user is expected to read. The drops fall into several groups.
One group concerns file selection. Several harnesses decide which files to open by checking the characters after the final period in the name. A file named “deploy” with no extension, or a file named “script.pyw”, receives no language assignment and is never passed to a parser. A pull-request scan that uses the same rule omits any changed file whose extension is absent from a fixed list. Another selection rule skips any path that contains the substring “log” or “script”, so a file at auth/login.py is excluded. A third rule treats the entire change as generated, and therefore ignorable, if any line contains the marker “@generated”. In each case, the harness never sends the file to the model, and the final report contains no finding and no notice that the file was omitted.
A second type of these drops occurs after a file has been read, when the harness tries to determine which parts of the code can be reached during normal execution. If a project defines its own function that happens to share a name with a standard function already provided by the language, the harness treats calls to that name as calls to the standard function. The project’s own function then appears to have no callers and is dropped from further analysis. If the number of values passed in a call does not exactly match the number the harness expects, it omits the connection even when the language allows a function to accept a variable number of values. Code that a framework starts automatically, such as a background task or a cloud-function handler, is absent from the fixed list of starting points the harness recognizes, so the code it invokes is treated as unreachable. In each case, the vulnerable statements remain in the repository, the model never analyzes them, and the report lists no finding.
A third group occurs after the model has already identified a problem. Some harnesses require the model’s reply to begin with the exact word “TRUE” or to be perfectly formatted data; any other text, including a reply that was cut off, is recorded as “no vulnerability”. An older stored record that marks a finding as fixed or as a false positive can take priority over a new report of the same issue, so the new report never appears in the output the user receives. A stored result that is matched only against the text previously sent to the model, and not against the current version of the harness, can return a prior “clean” verdict even after the harness itself has changed. A resume option can skip any repository that already has a results file, even if the code has since become vulnerable. In each case the model produced a finding, a later step discarded it, and the report the user sees lists no problem.
A fourth group of these drops comes from behavior the harness inherits from another tool it calls. When a harness runs Semgrep, a widely used static-analysis program, without turning off the option that honors inline comments, a comment containing “# nosemgrep” on the same line as a finding causes that finding to be omitted. The harness report does not mention that a suppression occurred. Similar comments recognized by a compiler-based analyzer produce the same outcome in an optional scan mode. In each case, the underlying tool found a problem, the harness discarded it because of a comment in the code, and the report the user receives lists no finding and no notice of the omission.
Evron also observed that an author who can commit code or open a pull request can deliberately place a vulnerability so that one or more of these rules will exclude it. Placing the vulnerable statements in a file with no extension caused six harnesses to ignore the file. Placing the same file under a test/directory, adding a suppression comment, or giving a custom function a built-in name each triggered additional drops. In several cases, a single change was sufficient to produce a clean report from more than one harness. The attacker does not need to alter the harness, its configuration, or its prompts. The attacker only needs to know the selection and filtering rules the harness source code implements.
These blind spots matter because the same class of tools is now used both to discover long-standing flaws and to assure that new changes are free of certain classes of flaws. When a harness silently omits reachable code, an organization that relies on the resulting clean report has not obtained the coverage the report claims. The same rules that produce accidental omissions also give an adversary who contributes code a reliable way to keep a chosen flaw outside the set of files or findings the model is asked to judge.
The research distinguishes a silent drop from a disclosed drop. A drop is disclosed if a deliverable the user is expected to read either names the excluded item or carries a skip or prune count that changed because of that item. Under this definition, several of the drops examined are silent: the report, the SARIF file (a standardized report format that many security tools create as an output), the pull-request comment, and the documented output files contain neither a name nor a tied count. A smaller total file count, or a line written only to standard error, does not meet the criteria.
Evron notes that many of the rules were written to describe typical code or typical model replies. A rule that “Python files end in .py” or that “a positive answer begins with TRUE” thereafter treats every non-matching file or reply as absent or as negative. The resulting harness therefore decides, rather than merely records, what to analyze.
The research includes three recommendations for engineering change. First, a harness that encounters a file, a call edge, or a model reply it cannot classify should keep the item in scope and record that the item was not fully analyzed, rather than treat it as clean. Second, every deliverable should name or count the items that were excluded, so a user can see the difference between “no vulnerability found” and “vulnerability not examined”. Third, selection and filtering rules should be written so that an author who knows the rules cannot reliably place reachable code outside the examined set; one concrete test is to plant the same vulnerability under each documented exclusion and verify that the report either surfaces the finding or explicitly records the exclusion.
The measurements Evron reports are limited to the twelve harnesses he examined at specific versions of their code and to the small files he planted to test each drop. He states that the same kinds of omissions can appear in any AI vulnerability discovery harness that chooses files by their name endings or by keywords in their paths, that connects function calls only when the number of values matches exactly, that reuses old results without checking whether the harness itself has changed, or that honors comments in the code that silence warnings. He also notes that published evaluations of these tools have measured how often they correctly flag real problems and how much they cost to run but have not measured how many reachable vulnerabilities are lost inside the harness steps themselves.
Organizations that adopt these tools therefore face a concrete coverage question. A clean report establishes only that the harness completed its run. It does not establish that every file the operating system would execute, every framework entry point, or every finding the model produced was retained. When the same tools are used to search for flaws in code already shipped, the same coverage limits apply to the search. The research provides a catalog of points where coverage is lost and a set of tests that can be applied to any new harness before its report is treated as evidence that the examined code is free of the vulnerability classes the harness claims to detect.
This article is shared at no charge for educational and informational purposes only.
Red Sky Alliance is a Cyber Threat Analysis and Intelligence Service organization. We provide indicators of compromise information (CTI) via a notification/Tier I analysis service (RedXray) or an analysis service (CTAC). For questions, comments, or assistance, please contact the office directly at 1-844-492-7225 or feedback@redskyalliance.com
- Reporting: https://www.redskyalliance.org/
- Website: https://www.redskyalliance.com/
- LinkedIn: https://www.linkedin.com/company/64265941
Weekly Cyber Intelligence Briefings:
REDSHORTS - Weekly Cyber Intelligence Briefing
https://attendee.gotowebinar.com/register/7855487668891299929
[1] https://six3ro.substack.com/p/when-automated-security-scanners
Comments