Topics
Browse posts by category and tag — every topic we cover, with the latest pieces under each.
Tags
- #methodology 7
- #benchmark 6
- #evaluation 4
- #jailbreak 3
- #prompt-injection 3
- #attack-success-rate 2
- #llm-security 2
- #red-team 2
- #red-teaming 2
- #reproducibility 2
- #advbench 1
- #agents 1
- #ai-agents 1
- #benchmarks 1
- #classifier 1
- #detection 1
- #eval 1
- #eval-harness 1
- #false-positive-rate 1
- #garak 1
- #harmbench 1
- #jailbreakbench 1
- #llm-quality 1
- #llm-scanner 1
- #meta 1
- #production-llm 1
- #pyrit 1
- #refusal-rate 1
- #robustness 1
- #safety 1
- #security-testing 1
- #tools 1
Categories
methodology 8 posts
- Designing a Reproducible AI-Security Eval HarnessA reproducible AI-security evaluation is an engineering artifact, not a notebook. Here's the harness design — separation of corpus, target, judge, and report — that lets a stranger re-run your number.
- Measuring Prompt-Injection Robustness in Tool-Using AgentsPrompt-injection robustness for an agent is not a single number — it is utility-under-attack against targeted attack success. Here's how AgentDojo and InjecAgent measure it and what the metrics actually mean.
- Comparing LLM Safety Benchmarks: AdvBench, HarmBench, JailbreakBenchAdvBench, HarmBench, and JailbreakBench are not interchangeable, and treating them as one undermines every comparison built on top. Here's what each measures and when to use which.
- Red-Team Eval Methodology: Pairing Attack Success Rate With Refusal RateAn LLM red-team evaluation that reports attack success rate without reporting refusal rate is half a measurement. Here's the paired methodology that makes the two numbers mean something together.
- Benchmarking LLM Jailbreak Resistance: Attack Success Rate Done RightAttack success rate is the headline metric for jailbreak resistance, and almost everyone computes it in a way that isn't comparable across runs. Here's how to define and report ASR so the number survives a re-run.
- Reproducible LLM Scanner Benchmarks: What Everyone Forgets to PinAn LLM security scanner benchmark that isn't pinned to a model version, a seed, and a corpus hash isn't reproducible. Here's the full list of what to pin and why.