Augustus vs CyberSecEval

Augustus is a free, self-hosted Go scanner for security professionals assessing large language models. It runs adversarial probes and produces vulnerability reports, with more than 210 probes across 47 attack categories, including jailbreaks, prompt injection, data extraction, safety benchmarks, and agent attacks. It supports 28 provider categories and 43 generator variants, including hosted providers, Ollama, and custom REST endpoints. More than 90 detectors cover approaches such as pattern matching, LLM-as-a-judge, HarmJudge, Perspective API, and unsafe-content detection; seven prompt transformations include encoding, paraphrase, poetry, low-resource-language translation, and case changes. Results can be written as tables, JSON, JSONL, or HTML. The project lists concurrent scanning, rate limits, retries, and timeout handling. Augustus compiles to a portable single binary and can be installed with Go or built from source; the README requires Go 1.27.0 or later. It is Apache 2.0 licensed. The project is intended for authorized security testing, and users are told to test only systems they own or have permission to assess. Provider API use may require credentials.

CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its tests measure model responses to cyberattack requests and whether borderline benign requests are incorrectly refused. Secure-code checks look for insecure suggestions in both instruction and autocomplete settings. Prompt-injection tests cover textual, multilingual textual, and visual attacks; a judge LLM evaluates whether injected instructions succeeded. The code-interpreter benchmark evaluates compliance with malicious prompts and sorts judged responses into three categories: extremely malicious, potentially malicious, or non-malicious. AutoPatchBench measures patch-generation agents on fuzzed C and C++ vulnerabilities, while CyberSOCEval, developed with CrowdStrike, covers malware analysis and threat-intelligence reasoning. The guide lists API support for OpenAI, Anyscale, and Together, and explains adding custom providers for self-hosted models. Setup uses a Python virtual environment, installed requirements, and a Python command-line module. The suite is free under the MIT license for research and commercial use. AutoPatch needs substantial resources: the page recommends at least 80 CPUs and 8 TB of storage.