Automating AI Hacking with Tree of Attack
Tree of Attacks with Pruning (TAP) automates LLM jailbreak generation by using one LLM to iteratively propose attack prompts, a target LLM to respond, and an evaluator LLM to score success—building a tree of candidates and pruning weak branches. The approach combines tree-of-thoughts exploration with the PAIR algorithm to achieve >80% jailbreak success rates on state-of-the-art models with <20 queries per attack. Special Token Injection attacks are particularly effective: adversaries embed architecture-specific control tokens (e.g., OpenAI’s <|im_start|>, Llama’s [INST], Mistral/DeepSeek variants) to manipulate role interpretation and bypass safety guardrails. The authors released parley-ng, an improved attack framework with interactive visualization and proof-of-concept exploits for manual testing against LLMs.