# Evaluating LLMs for SOC Operations and Log Analysis

> Discover how Cisco Talos evaluated 66 model and reasoning combinations for SOC workflows, focusing on cost, speed, and consistency.

- Published: 2026-09-01T02:46:45.000Z
- Severity: info
- Category: Threat Intel
- Tags: Threat Intel, Artificial Intelligence, Incident Response
- Author: Runtime Rebel Intel
- Primary source: https://blog.talosintelligence.com/choose-your-fighter-balancing-competing-requirements-to-select-models-for-your-ai-soc/
- Canonical: https://runtimerebel.com/blog/evaluating-llms-for-soc-operations-and-log-analysis

## Key points

- Immediate impact: Security teams selecting large language models for operations must look beyond simple benchmark scores to avoid unpredictable performance drops.
- Affected systems: Security operations centers and digital forensics workflows utilizing OpenAI and Anthropic models for log analysis.
- Remediation: Establish repeatable evaluation frameworks prioritizing consistency, cost, and execution time alongside analytical efficacy.

## Evaluating [AI](/glossary#ai) Models for Security Operations

Selecting a large language model ([LLM](/glossary#jailbreak-llm)) for security operations center ([SOC](/glossary#soc)) and [digital forensics](/glossary#digital-forensics) and incident response (DFIR) tasks involves a complex matrix of competing requirements. According to [Cisco Talos](https://blog.talosintelligence.com/choose-your-fighter-balancing-competing-requirements-to-select-models-for-your-ai-soc/), simply asking which model scored highest on a benchmark is insufficient and can lead to severe workflow inefficiencies or degraded investigative quality. Security leaders must balance model efficacy, analysis time, [API](/glossary#api) costs, and result consistency.

To understand how different configurations perform under realistic conditions, researchers tested 66 model and reasoning combinations from Anthropic and OpenAI. The evaluation used a tool-assisted log-review task where reviewer personas analyzed a standardized dataset to determine whether logs were real or synthetically generated using standard Unix command-line tools.

## Methodology and Testing Framework

The experiment utilized a synthetic telemetry corpus generated by EvidenceForge version 1.12.0. The corpus contained 80,054 simulated log records across 20 source formats, amounting to 48.0 megabytes of data. This included:

* Network telemetry from Zeek sensors (DNS, HTTP, [TLS](/glossary#transport-layer-security-tls), connection logs)
* Perimeter security logs from Cisco ASA firewalls and Snort [IDS](/glossary#ids)
* [Endpoint](/glossary#endpoint) telemetry including Windows Security, Sysmon, Linux syslog, and shell history
* Application access logs and email artifacts

Each experimental panel consisted of four independently prompted reviewer personas: a Threat Hunter, a Detection Engineer, a Network Forensics Analyst, and a Host/Endpoint Detection and Response ([EDR](/glossary#edr)) Analyst. Reviewers utilized native agent harnesses such as Claude Code for Anthropic models and Codex for OpenAI models.

### Key Metrics Measured

Rather than relying solely on raw analytical scores, the evaluation measured several operational dimensions:

* **Cost:** API-equivalent expenditure calculated from a frozen list-price rate card, factoring in failed attempts and guardrail refusals.
* **Time:** Total wall-clock time consumed across evaluations, including retries and provider-directed waits.
* **Downside Score Consistency:** The variance between the median panel score and the lowest individual score, highlighting unpredictable failures that could introduce false positives or false negatives into an investigation.

## Key Findings from the Experiment

The research revealed that reasoning effort is not a universal quality dial. Increasing reasoning effort frequently drove up costs and execution times without guaranteeing a better analytical result, and in some instances, higher effort correlated with lower scores.

Furthermore, consistency emerged as a critical decision factor. A model condition with a strong median score could still produce occasional weak runs, presenting operational risks during high-stakes incident triage. Organizations building AI-assisted workflows must evaluate failure rates and processing latency alongside raw accuracy.

## Recommendations for Defenders

Security teams integrating AI models into SOC and DFIR pipelines should adopt a structured evaluation approach rather than relying on vendor claims or generic benchmarks:

* **Define Tolerable Failure Rates:** Establish acceptable thresholds for cost, speed, and downside consistency that align with operational requirements.
* **Test with Domain-Specific Tasks:** Use realistic, repeatable telemetry datasets that mirror internal incident triage procedures.
* **Monitor Reasoning Impact:** Audit whether increased reasoning effort or higher-tier models yield measurable improvements for specific analytical tasks.

**Related:** [Crime Script Analysis: Mapping Threat Workflows and AI Risks](/blog/crime-script-analysis-mapping-threat-workflows-and-ai-risks), [Adversary AI Weaponization: A Data-Driven Analysis by Talos](/blog/adversary-ai-weaponization-a-data-driven-analysis-by-talos)

---

AI-generated analysis from the primary source above; not human-reviewed before publication — verify anything operational against the original (https://runtimerebel.com/editorial). Quote with attribution and a link to the canonical URL: https://runtimerebel.com/blog/evaluating-llms-for-soc-operations-and-log-analysis