Evaluating AI Models for Security Operations
Selecting a large language model (LLM) for security operations center (SOC) and digital forensics and incident response (DFIR) tasks involves a complex matrix of competing requirements. According to Cisco Talos, simply asking which model scored highest on a benchmark is insufficient and can lead to severe workflow inefficiencies or degraded investigative quality. Security leaders must balance model efficacy, analysis time, API costs, and result consistency.
To understand how different configurations perform under realistic conditions, researchers tested 66 model and reasoning combinations from Anthropic and OpenAI. The evaluation used a tool-assisted log-review task where reviewer personas analyzed a standardized dataset to determine whether logs were real or synthetically generated using standard Unix command-line tools.
Methodology and Testing Framework
The experiment utilized a synthetic telemetry corpus generated by EvidenceForge version 1.12.0. The corpus contained 80,054 simulated log records across 20 source formats, amounting to 48.0 megabytes of data. This included:
- Network telemetry from Zeek sensors (DNS, HTTP, TLS, connection logs)
- Perimeter security logs from Cisco ASA firewalls and Snort IDS
- Endpoint telemetry including Windows Security, Sysmon, Linux syslog, and shell history
- Application access logs and email artifacts
Each experimental panel consisted of four independently prompted reviewer personas: a Threat Hunter, a Detection Engineer, a Network Forensics Analyst, and a Host/Endpoint Detection and Response (EDR) Analyst. Reviewers utilized native agent harnesses such as Claude Code for Anthropic models and Codex for OpenAI models.
Key Metrics Measured
Rather than relying solely on raw analytical scores, the evaluation measured several operational dimensions:
- Cost: API-equivalent expenditure calculated from a frozen list-price rate card, factoring in failed attempts and guardrail refusals.
- Time: Total wall-clock time consumed across evaluations, including retries and provider-directed waits.
- Downside Score Consistency: The variance between the median panel score and the lowest individual score, highlighting unpredictable failures that could introduce false positives or false negatives into an investigation.
Key Findings from the Experiment
The research revealed that reasoning effort is not a universal quality dial. Increasing reasoning effort frequently drove up costs and execution times without guaranteeing a better analytical result, and in some instances, higher effort correlated with lower scores.
Furthermore, consistency emerged as a critical decision factor. A model condition with a strong median score could still produce occasional weak runs, presenting operational risks during high-stakes incident triage. Organizations building AI-assisted workflows must evaluate failure rates and processing latency alongside raw accuracy.
Recommendations for Defenders
Security teams integrating AI models into SOC and DFIR pipelines should adopt a structured evaluation approach rather than relying on vendor claims or generic benchmarks:
- Define Tolerable Failure Rates: Establish acceptable thresholds for cost, speed, and downside consistency that align with operational requirements.
- Test with Domain-Specific Tasks: Use realistic, repeatable telemetry datasets that mirror internal incident triage procedures.
- Monitor Reasoning Impact: Audit whether increased reasoning effort or higher-tier models yield measurable improvements for specific analytical tasks.
Related: Crime Script Analysis: Mapping Threat Workflows and AI Risks, Adversary AI Weaponization: A Data-Driven Analysis by Talos