Choose your fighter: Balancing competing requirements to select models for your AI SOC
Cisco Talos conducted a systematic evaluation of 66 model and reasoning combinations from OpenAI and Anthropic on a log-review task simulating incident triage, analyzing tradeoffs between accuracy, cost, time, and consistency. Using 80,054 synthetic log records across 20 formats and a Pareto frontier analysis, the study found no single optimal model—instead, different reasoning effort levels produce unpredictable quality-vs-cost curves, with higher effort not always improving results and sometimes increasing inconsistency. The research establishes a repeatable methodology for SOC/DFIR teams to select models based on their specific workflow constraints: teams must balance analysis quality, inference time, API costs, and downside consistency rather than chasing the highest-scoring option. For example, the top-scoring model took 33 minutes and cost $55 per analysis, while lower-ranked options achieved acceptable results in seconds for under $5, making them preferable for high-volume triage workflows despite lower absolute accuracy.