Fuzzing vs. AI: Comparing Fuzzilli and BigSleep in V8 Vulnerability Discovery
Project Zero’s recent research provides a valuable comparison of two advanced bug-finding methodologies: traditional fuzzing, exemplified by Fuzzilli, and the emerging capabilities of LLM-based systems, specifically BigSleep. Both tools are routinely deployed by Google to identify vulnerabilities within Chrome’s V8 JavaScript engine. This analysis highlights the distinct yet complementary strengths of each approach in discovering complex bugs in well-tested software, offering critical insights for vendors seeking to maximize pre-release security.
Technical Analysis of Automated Vulnerability Discovery Tools
The study, documented by Project Zero, focuses on Fuzzilli, a generative fuzzer specifically designed for JavaScript engines, and BigSleep, an agentic bug-finding system powered by Large Language Models (LLMs). These systems operate in production environments, with Fuzzilli running continuously to catch new code changes rapidly, and BigSleep performing daily reviews of new commits to generate proof-of-concept test cases. Both systems integrate with Clusterfuzz for initial crash triaging and bug reporting in Chrome’s issue tracker, targeting the highly optimized V8 engine.
The research examined data from April to September 2026, comparing the outputs of both systems. Initially, BigSleep reported a higher total number of bugs (441 vs. Fuzzilli’s 316). However, this included various configuration and integration issues. When the scope was narrowed to type:vulnerability bugs that were actionable—meaning they were not duplicates, “Works As Intended” (WAI), and received at least one fix commit—the numbers shifted. BigSleep identified 45 such vulnerabilities, while Fuzzilli found 37.
A key aspect of the comparison was the overlap in discovered vulnerabilities. Initial observations suggested a 25% overlap between Fuzzilli and BigSleep in the reported bugs. Further dedicated experiments, designed to estimate theoretical overlap in vulnerabilities reported since 2023, indicated a higher overlap of approximately 60%. This substantial overlap suggests that both systems can identify similar classes of vulnerabilities. However, the critical takeaway is that a significant portion of bugs were discovered exclusively by one system or the other. This demonstrates that neither approach fully supersedes the other in LLM-based vulnerability discovery in JavaScript engines.
The “shape” of bugs discovered uniquely by each system also varies. While the source doesn’t detail specific bug types, the implication is that fuzzers excel at exploring unexpected execution paths and edge cases through automated input generation, often finding crashes or memory corruption issues. LLMs, with their ability to “read, reason about, and write code,” might be more adept at identifying logical flaws, insecure patterns, or subtle semantic issues that traditional fuzzing might miss without specific grammars or heuristics. The complementary nature of these two approaches is therefore paramount for comprehensive security testing.
Actionable Recommendations for Maximizing Bug Finding
Given the distinct contributions of both fuzzing and LLM-based agentic analysis, security professionals should adopt a multi-faceted strategy for complementary approaches to bug finding in complex software like JavaScript engines.
- Integrate Both Methodologies: Vendors should not view fuzzing and LLM analysis as mutually exclusive or competitive. Instead, deploy both types of systems concurrently to maximize the breadth and depth of vulnerability discovery.
- Targeted Deployment: Leverage fuzzers for their strength in uncovering crashes, memory errors, and performance regressions through diverse input generation. Utilize LLM-based systems for their code comprehension and reasoning capabilities, which can identify logical flaws, insecure coding patterns, or subtle semantic bugs.
- Continuous Improvement: Regularly evaluate the performance and unique findings of each system. Invest in refining both fuzzing techniques and LLM prompts/models based on the types of vulnerabilities they uncover and miss.
- Prioritize Actionable Reports: Establish clear criteria for filtering and prioritizing bug reports, focusing on actionable vulnerabilities that pose real-world security impact. This includes filtering out duplicates, “Works As Intended” reports, and non-security-critical issues.
By combining the exploratory power of fuzzing with the analytical capabilities of LLMs, organizations can significantly enhance their ability to detect and remediate vulnerabilities before they are exploited in the wild, thereby protecting users more effectively.
Related: VOLTA Infostealer Leverages AI for Victim Prioritization, AI-Assisted Campaigns Target Latin American Organizations