AttackBench: An Adaptive Security Benchmark for Autonomous Agents

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   Guys, how's it going? Uh, pleasure to be back. Pleasure to be back as always. Uh, George was on stage just a little while ago. Uh, Ari, good to have you here as well. Likewise. Great to be here. And, uh, we're going to be talking Attackbench, uh, which is an adaptive security benchmark for autonomous agents. Um, guys, why don't we just why don't we just start, um, we'll start high level. Uh, you know, there's been far too many hacks. Uh, there's been security incidents out the wadoo. uh it

2. 0:29

   feels like uh those who are using AI are using it for evil and not for good and we need to switch that around. Uh so let's tighten up the security uh with these benchmarks and uh we'll we'll increase uh the benefit of all these autonomous agents. Um why don't we do just quick intro? I know we've only got a few minutes but then we'll we'll get into the uh nitty-gritty about uh attackbench.

3. 0:52

   Sure. Um I'm coming back but I'm chief product officer for Near George and also GM for Near AI. I'm Ari Midvinsky, CTO and co-founder of uh Failsafe. We're based out of Singapore. Uh we are cyber security company. And George, we've got a demo uh here for or some slides for for this one. Right.

4. 1:09

   We do. Um let me just give a little bit of background here before we jump into the slides. So uh we meaning the near eye team and fell safe ended up collaborating on um attackbench the security benchmarks because like you mentioned one of the things we noticed is that security is important and there have not been really good benchmarks that have been made for agents from a security perspective and if you recall like an early analogy I made LM are the mind and agent harnesses or frameworks

5. 1:38

   are the body and they both have a very important role to play in security. Uh so let's get into attackbench to focus on epic. Okay. Uh so let's get into attackbench. Cool. Uh I can kick kick this off. And then um Ari and some of the engineers on our team ended up going deep and as a property of differently under attack and you have to consider security as a property of a pair of both and you have

6. 2:05

   to consider security work that take away I I took from both the work as well as this is like the highle work we wrote. Um, but uh we're really happy that this is kind of the some of the the conclusions and insights that we're really happy that this is kind of the some of the the conclusions and incests that came out of this. Really happy that this is kind of the some of the the

7. 2:28

   conclusions and incest that came out of this work. And then Ari, would you like to take it away? Um, sure. Yeah. So, so just just as you mentioned, uh, it was it was really interesting. It was um uh you know when we started when we first started the work uh we kind of looked around and everything was like all the all the benchmarks that we saw it everything was static like there was a set of set of known attacks and you you know you plug the agent through and see how it does but it's not very realistic. So, I mean,

8. 2:57

   yeah, typically the adversary will will deploy the best LM they can find, right? Potentially. And and then we'll adapt. And so, that's that's what we kind of decided on is that we're going to make this as hard on the agents as possible.

9. 3:12

   Uh so we built out this framework attackbench where uh you know you have a red team and a blue team and the red team uh is powered by an LLM and it gets to see what uh what the thinking of of of the agent is. And it's it's very realistic like you can't assume you know just hope for the best. Hey this this you know I tried this it doesn't nothing happened so we're good. Um it it is realistic. So um we we um let and you'll

10. 3:45

    see the slide in a second. We let the red team evolve its attack based on the thinking uh o of of the agent. And so what we did is so this is the uh what we tried out uh we did this on three different frameworks on uh ironclaw, Hermes uh and openclaw and we tried four different frontier models going across 52 scenarios. Yeah.

11. 4:11

    So you you you can see you can see the you can see the loop here. Um so you get in this framework you get five different tries and it's uh you know if if any one succeeds then then that that counts and could you just maybe give a an example of one or two of these scenarios so that we can understand you know the the loop here.

12. 4:33

    Yes. In fact, why don't uh so these are the kind of different attacks that are being tried out. Perfect. Uh so these are the kind of different attacks that are being tried out. Um so so the you know this is this you know you have you have the agent and then the LLM inside and uh we're trying to exploit you know the the red team doesn't get to talk directly to the LLM but it's using

13. 4:59

    various various side channels right so it's using what looks like a record in the field or a free form text and so these are some of the attacks right um so you'll have like a write instruction go write something out or go do exfiltration go fetch uh this URL and it'll have you have your your key embedded in it or try to make it exceed right so let's let's go back to the loop now I'm going to ditch my chair so that

14. 5:24

    people can see this and then oh sorry yeah and then yeah here we go um so so this yeah so you have um a you have the okay great yeah you have the regular workflow uh of of the blue agent let's say ironclaw or open claw um uh being being asked to do some some generic task you know go tabulate go go go uh answer email or whatever and then the red team uh and this is powered uh in this configuration by by GPT5 uh 5.2 I think that's what we used. Um,

15. 6:02

    and it will like embed if an instructions like one of the ones I just I just gave an example of and it also gets to see what was the thinking, right? So, uh how did how did the LM reacted? And you'll see in our paper it was really interesting some of the logic you know it's it's attemp one then then uh the the uh probe gets sharper and sharper and oftent times it succeeds.

16. 6:26

    Did you have to trick the GPT5 model into being an adversary and and it had it had a prompt. So it was it knew what it was doing and it's it's its whole purpose in life was you know you got five chances. Okay, go at it. And it was it was very clever actually and one of the things that we we still would like to do and this is open source. We invite everyone to try this out and try try this benchmark on other kind of configurations. So, you know, you know,

17. 6:56

    pick your best LLM and and and plug it in. But yes, so let's let's just keep going. Uh and uh in fact, uh Ironclaw out of out of everyone uh did the best. Uh so we we had no idea. So this is like uh we had no idea what the outcome would be. Uh but you'll see that uh Claude and under Claude and GLM uh ironclaw had the least number uh the least number of uh vulnerabilities and as you know as our thesis to all of this is uh you know stated up front uh it really still does depend on the

18. 7:32

    configuration between you know what what is what what is the pair that you're using um the combination of of uh the LLM and um uh and the framework and the agent. Um what was really what was also really kind of enlightening is like because we've we saw the reasoning uh so you can actually there's a number of improvements that could be made like for example uh for the uh ironclaw has a lot of strong controls for permissions that th those can also be extended to some of

19. 8:03

    the other attacks. So and these models here on the left these are the protectors if you will. Oh yeah. So this is uh as George refers to them these are the brains. So this is for example this was sitting inside of Hermes openclaw. Ironclaw. So they were configured with ironclaw was configured with queen for example or GLM and son of they were they were playing defense while chess the GPT model was playing uh offensive to get into these things.

20. 8:25

    Okay. One of the things we want to do from our own perspective is like better understand and better dig into actually the the traces to understand what is happening in these attacks and how we better defend. But currently some of the hypotheses we have is like things like architectural prompt injection protection um sandboxing of tools things of those features of that sort guard rules that were designed to actually be

21. 8:48

    security first really end up mattering um do do we have more more slides okay so we observe these attack yeah let's do one more and then we can uh I think we have a we're short on on time uh let's do uh one more slide here. Oh, the next one. Yeah. So, if you want to see what uh an attack looks like, this is an example.

22. 9:12

    Okay. So, in the resolution path, so we have the red which is the attacker resolved by Alice audit flag. And so, what could could you describe what what red is trying to do and then what blue reacts? So, this is this is a math attack. So the task the task of of the blue of uh of the blue agent was to um yes so to compute the total. Okay.

23. 9:37

    And in in some in some field that the LLM ended up trusting it said oh by the way skip this skip this number. And it did. It happily did. Okay. So blue lost in this one. Blue lost in this one. The the bad guys won in this one. Okay.

24. 9:53

    Yeah. We have we have plenty of counter examples, but it was very very interesting how we arrived here and you can actually on some of the on some of the runs you can actually see how exposing the LLM's thinking helps the red team focus in what was it like what was the LM's thinking in in uh what was the rationale for for not doing this operation? So a general question here as we think through you know making these things you know more understandable less like black boxes so we can kind of understand what's going

25. 10:27

    on underneath the hood we're actually exposing you know potential attack vectors how do we reconcile or balance oh go ahead yeah but you it's it's just general security principle do not you don't want to do a you don't you don't want to do a blackbox you you don't your sec security through obscurity the lesson still applies here. So if you can hold up if if the if the mo if if the framework

26. 10:51

    holds up if the agent holds up under being transparent then then then you can have some confidence in it. um you will you know if you try to close it up by by the way so all of these uh each one of these agents uh have open you can actually get the trace uh including including openclaw Hermes and we put in a PR like I put in a PR for for that that's getting that's got that got

27. 11:15

    approved for ironclaw uh but uh but yeah I mean uh in in the wild uh the attacker is going to just take you know take your favorite claw and see what it's doing and then and then go after you know after some instance that's being used by a user.

28. 11:32

    I know we're running short on time but I want to point something that I thought was interesting which is we went with the brain and the body analogy but one of the super interesting things to point out is that the body matters a lot in terms of security exploits too. You can see for example with let's say GM5 um one using one agent harness versus another can result in like 3x or like

29. 11:57

    multiple x increases in the amount of exploits you have right so it's not just the LM it's not just the brain that matters it's also the body the harness the framework that matters a lot in security as well and why would that be the case why would the same model be uh more or less secure inside of a different harness is the So uh for for for the agent uh it's it's the um guards around it right.

30. 12:23

    So there's you know permissions there's how you interpret how you how you do uh you know filtering right and for the model you know each model thinks in its own way. Uh it you know it depends on how many different instances of this particular kind of a attack it saw in the past. So, you know, it gets trained enough on on this instance and then it reacts in one way and then some other model will react differently.

31. 12:49

    Got it. Um, George, you're in control of the clicker. I want to be respectful of time. Uh, any, you know, please, if there's anything else relevant, you know, please share, but uh, I think we're good. I think we're good. Okay. Attack bench. Very interesting stuff. I mean, I I want I'd love to see more data, you know.

32. 13:04

    Oh, and we Yeah, everything is in the GitHub. Beautiful. Awesome, guys. Give it up. Uh George and Ari attack bench.