Benchmarking Agent Systems: Safety, Reliability and Trust

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   Guys, uh, last but certainly not least, uh, we're going to be talking about safety, reliability, trust. Um, you know, obviously you guys just heard from George and Ari about Attackbench. Um, I I think that's the right place to start.

2. 0:13

   Uh, Stephen, David, let's get some reactions. Um, can't see David all the way over there. So, I'm right here. There he is. Uh, David, uh, why don't we start with you? Uh, what do you what do you think? Attackbench. Maybe we'll just do a quick intro and then uh we'll get into Attackbench and like benchmarking uh in general.

3. 0:30

   Well, I'm embarrassed to say you're making an assumption that I was fully paying attention. I was having some conversations. So, I'll I'll pin it over to Stephen. Yeah. You guys are doing uh you know, red team and blue team against each other, pitting AI against it, also to do benchmarking and find which ones are kind of the best combination of agents uh to do the defense and what they can find, exploit. So, very familiar with that. We're doing that right now at

4. 0:52

   Halbornne with um you know doing kind of the the red team side mostly which is what we focus on. Benchmarking is extremely extremely important. Uh you have to have like a a ground truth data set uh to to measure against but that's very hard to do in security because you don't really know there is no ground truth in security because there always could be more vulnerabilities. If you guys saw the anthropic uh video around that

5. 1:17

   20-year-old operating system they found like a critical vulnerability. it's been there even though it's been pentested. So what is the ground truth? So that's one thing is like relying on a truth data set um you know from the tagger perspective and another thing is uh expectations around what the models can do because it's your AI is fuzzy and fuzzy out so you could do the same kind

6. 1:37

   of like script there but you may get different results each time as we see you know ask chat GBT the same thing you know again and you'll get a different answer sometimes so the for things as complicated as security expect the same different results. Yeah.

7. 1:51

   Um, okay. So, when we're thinking about this in general, maybe uh you know, uh this is just an evolving thing. Security principles thing, it's always evolving. It's fuzzy in uh are tried and true. But like you said, AI is a very comp complicated thing. It's always evolving.

8. 2:06

   It's fuzzy in, fuzzy out. Um you know, George Ari, maybe we could just reiterate some of the findings, some of the you know, core key takeaways. uh and then we can build on this when we we start to make some benchmarks around safety and reliability in general.

9. 2:20

   The thing I would add is I go back to it was really interesting that there was such a big delta between um different agent frameworks using the same model. Right? And one of the next steps we have to do is is go through the trace and try to understand that better. Try to understand why. Right? We know that there are very important guard rules that matter. Things like using sandboxes, things like you know prompt injection protection. But one of the things we want to try to figure out is like which of the guard rows have the

10. 2:48

    highest ROI of actually reducing security instances while minimizing like any disruption potentially to a user experience. Right? That that was my main takeaway and kind of like a big next step I took from this. Yeah. Just just to add to that, I think the um the setup is really important.

11. 3:06

    You want to make it adversarial. You want to you want to make it a fair a really a fair test. uh don't go for like the oh you know we'll just give it a bunch of static files and see how it does you know just set up an environment and that's what we try to do here that is realistic this is what an attacker does uh so this is your worst scenario not not the best case one guess and and you know you dodge the bullet I I have a question related to it and I

12. 3:30

    know you're moderating but for them when you guys did it uh do you use like sub agents or anything at all because I've I've heard mixed feelings about having like one master model that uses this framework and kind of does all of it or delegating out to other sub agents to smaller tasks like you know scanning versus like exploitation and code writing. How did you guys approach that?

13. 3:49

    So on on the red team side it was uh you know you get to configure the you get to config configure the LLM and it gets max of of five tries uh and it gets to see the uh you know it can basically do whatever it wants to do in you know in in its loop. Um but um yeah, I mean what we'd like to see is uh you know additional tests with other other LMS configured. Uh I think that'll be really interesting to see. And then the and then the uh the blue team it's whatever whatever ironclaw whatever openclaw does

14. 4:20

    to accomplish the task. So it can be sub agents or whatever. Um is are we approaching a world because um you know another takeaway from that was that these these are pair right there. there's not one model that does it best, you know, do do we approach a more pragmatic world where maybe we do get like some composability amongst harnesses sometime almost like a harness aggregator if you will you know we have something over the top and it'll you

15. 4:48

    know have an agent complex around openclaw ironclaw and it'll start to it'll start to segment you know certain tasks that could be potentially vulnerable to to certain prompt injections uh and then use or choose a harness in a model accordingly.

16. 5:06

    That's an interesting question. Um I think it may be difficult to do that for the foreseeable future because there's so many sort of moving parts of the architecture that may not work together if you try to essentially do like a a harness on top of multiple different harnesses. It's kind of what you're saying, right?

17. 5:25

    Um, it's probably better to just build the specific harness with the best security practices that you can and with the best features that you can and not rely on this potentially moving mix of different features and different security features.

18. 5:41

    That's I was going to say I agree with you because they're coming out so fast the models is like by the time you finally okay you pick the best one the best mixture another new another new one is released there that's like better you have to start it all over again.

19. 5:54

    Yeah, I mean they're constantly leaprogging frogging each other. But I think that again what what we try to do in attackbench is having you know the idea is that this is an ongoing thing. So as these models come out plug them in plug them into attackbench and see what happens. Uh because you know today you think that you know you plugged every hole you thought of everything tomorrow it changes right. So yeah.

20. 6:16

    Yeah. Yeah. I I I also think it's it's use case specific as well because there are certain harnesses that would be good for one use case and there are certain harnesses that I wouldn't even think twice about you know using because I think the answer really comes down to software architecture in some respect when you're using the inference like that should be like the little like the smallest amount of the AI that you're using everything else that you're doing is effectively somewhat deterministic in some respect if you're building out an

21. 6:40

    agent and you're having it doing certain things all of that should be deterministic I think too many people are thinking through AI security of let me just give it instructions, let me do this or let me segregate the keys and move it over here. It really has to be thought through. The answer is more architectural and not let me build the most secure harness. It's more about infrastructure as opposed to like harness specific. When when I'm building um you know this type of solutions for

22. 7:05

    like a large institution, it's not a let me go and grab a harness. It's really thinking through I'm grabbing data, I'm putting it into a data lake. building this particular function and broken out into seven different types of agents. AM policies fully restrictive. It's the answer is a lot more complicated than just let me find that one harness and just deploy it.

23. 7:24

    And so you know George you guys are building a harness at Near AI. Um how does this inform your thinking? What do you what what do you contribute to the road map as a result of what what you know some of these some of these find?

24. 7:36

    Um, one of the most immediate things literally we're having already come to our all hands tomorrow to present some of the findings because just taking the traces of like what happened studying it and then making sure to build build the learnings into ironclaw going forward is really important. I also think just bringing awareness to this is so important like most of the the the

25. 7:59

    conversation people have about different agent frameworks are about the capabilities not around the security risks but if you're actually using agent frameworks to do that as well um we we heard a lot from George privacy matter a lot as well um we we heard a lot from George and Ari so I'd love to just get a better idea of you know where attackbench is right now and how that can inform these things I'd

26. 8:22

    love to just get a better understanding of how you know sovereign and howorn are thinking about you know building these secure frameworks um and then ultimately you know how how these can all work together. Yeah. Uh so definitely you've been thinking about it for a while because you know how one of our you know core capabilities is doing you know security testing and code reviews smart contract

27. 8:41

    audits pen testing and obviously a lot of that now is uh becoming empowered by these models and you know people can can build it and and test it themselves. So I think that what we're shifting to uh is is much like you guys are building the harnesses and hooks is using the the expertise and and building around the models to one benchmark do the research for it that can be provided and it's almost like prompt engineering mixed with MCP creation mixed with you know

28. 9:07

    data flow of how you construct you know the the execution whether it's attack path defense path or even you know does have to be you know security just you know advisory recommendations uh so the way you ask things and the way you put the data together for us like the customer experience is you're putting all in one package rather than just singular uh you know requests that and

29. 9:30

    get results from that too because um you know the overall security is what really matters not you that's we've taken that approach since the beginning and it's still the same it's not about like do you have a vulnerability in your code or you know layer one or smart contract it's the mix of things it's the custody it's the infrastructure like Dave had said it's the code it's the people and all of that has to be kind of looked at, you know, in comprehensive uh, you know, full view.

30. 9:54

    Yeah. I mean, nothing really to add for what Stephen said. You know, there's nothing like specifically a sovereign that we're doing as far as like a benchmark. But again, the proper way to think about it is just with any other architecture that you're building is to really think through what is the specific use case, think through the threat model, what type of access, you know, to what type of data, and just make your decision from there. It's not hard. AI security isn't something that we haven't seen before. It's securing code. Um I think the problem with what's

31. 10:17

    happening is this world is allowing people who shouldn't be doing certain things to do things that they weren't able to do prior which is causing a lot of security you know issues. So you know securing code thinking through architectural security thinking through you know um how to air gap certain decisions into enclaves like those are things that we've done before and we're going to do in the future. So there's nothing new here, but I think what's happening is people are deploying things

32. 10:43

    that they I don't want to say shouldn't be doing, but it's making things easier for people, but then they're just completely exposing their entire lives. So yeah, and I think we have the exposure still around like blind reliance like people even I feel sometimes like almost you trust the the AI you're working with too much and it's not about like not thinking for yourself. It's that you know it does it so quickly and so fast that you don't

33. 11:05

    you know you know challenge it sometimes uh and you can have like the take it back as a judge to do it challenge it for you but at what point do we really take it back and say like all right is what we're doing does it make sense and can we trust the results uh or the defense or whatever it is for for all that fundamentally yeah I mean perfect point is you know you're not going to have one employee that's going to write your code do the security scan approve deploy manage your infrastructure but that's how

34. 11:30

    everybody's thinking about AI agent today and and the answer isn't simply just well four different sub agents that's not how you have to think through things again identify the pieces in your architecture that can be deterministic and not done through the agent that is the proper security model I have actually a question for my fellow panelists if you'll humor me um which is you know a lot of when you're when you're writing code often times for every thousand lines of code I think it's what like 10 to 20 bugs per 10,000 lines of code um crypto as an industry

35. 11:56

    is built upon the premise that essentially code is law right given the amount of bugs that exists and connected to the fact that code is law in crypto and the amount of money at stake. What do you guys think that will mean for uh this period of time in the crypto industry going forward for related to AI in that the matter? I think that well lines of code you can't really tie value to that. Um that's actually even something you know applies to us like a lot of security reviewers will look at lines of code and the more

36. 12:26

    lines they are the more you charge somebody for. I I've never believed in that model since the beginning. I think it's you know complexity value at risk is more important for that. It's you know how you know that that's really the way you apply to it. If you have something that is high impact, highly complex uh that has a lot of stakes that should be treated much uh much more secured and much more valuable that you

37. 12:50

    should be uh you know taken serious with our time and and you know when you look at charging somebody for that or if you put a stake inside of something it's um you're the more exposure you give it the more uh risk you're taking on. So I think it's I think we're now going into a world where it's not about quantity and it's not about time either because things can happen uh very quickly. You could you look at something that's maybe

38. 13:15

    a thousand lines of code more and it's just seconds difference of what the AI can do for like a review on that. So I think that you have to think about um you know more than just quantity or time. It's more about results driven.

39. 13:26

    that's uh you know results driven and the results are usually around what kind of impact or what kind of uh you know complexity around the ecosystem around the crypto um is involved with it. That's kind of like the you're looking at not just one thing that but you're looking at the the concentration of assets and infrastructure that surround that and integrate with it.

40. 13:49

    Um I I have a question around uh the use case, right? If we're if we're starting from, you know, you know, here's the use case. Um, and then uh threat detect the use case, you know, see see what kind of threats are are potentially going to penetrate this thing. H how, you know, we're obviously at consensus. You know, a lot of onchain digital asset teams here, but, you know, these two worlds have started to collide with AI uh AI and digital assets have started to collide. Do you foresee a lot

41. 14:20

    of the the research and and security testing and things that you guys have done, you know, start start to make its way out of the digital assets world and make its way into the pure AI world and then what sort of use cases do you forecast for you know the near AI harness and you know the things that Hobborn is working on? How do you see these things ripple the ripple effects into the broader landscape here?

42. 14:45

    Yeah, I'll give you my two cents. I think, you know, I I it's funny that you bring it up like that, how much is going to get out. I think I'm hoping it's the other way that things get in. I'm seeing AI security much more mature outside of digital assets. So, it's I see the complete opposite of what you just said, funny enough. So, I'm hoping some of those principles make its way here.

43. 15:04

    Uh, and just just to just make it concrete, um, our experience at Failafe has been that look, I mean, we we've done uh threat modeling like I've done threat modeling for for like decades. Uh I spent like eight and a half years at Microsoft and it was it was like it was a religion there. Um not not as much as other companies I've been with. Uh threat modeling directly applies to web

44. 15:28

    3 to web two u as well as to agents. Um and and I I really mean like you know traditional you know uh what are the security boundaries what's you know the data flow diagram uh what are the invariants and having um LLMs built that out for you I mean this is something that we we did uh with swarm it works it works quite well I mean there's the LLM still has creative freedom um to explore different uh you know to use its

45. 15:58

    knowledge base but when you give it this kind of framework uh it it produces better results. I predict predict there's going to be a lot of security engineering hiring at crypto web3 firms in the next few years and I also predict our friends that you know end up doing a web 3 flavor of security or are experts at that will get a lot of business from a lot of protocols. Um, I also think what will be super interesting as well is that if you think about like what these agents are becoming, one way I think about these

46. 16:28

    agents is you can have general assistant agents that help you with general things like emails and calendars etc. You can have vertical specific agents, things like Harvey Loro for legal. Maybe there's a vertical specific version of a security agent for crypto in the web3 world. And you have horizontal agents that do engineering like cloud code or Devon or customer experiences like you know something like uh Sierra or Decagon

47. 16:51

    right I think the ones that probably I think all three will impact crypto for the foreseeable future. The general assistant is going to make everyone more productive. The vertical specific crypto agents may end up impacting yield and security and a couple other things. And like all the other industries, I think crypto will crypto and web3 will adopt a lot of the functional experts to help make their engineering more efficient, sales and marketing more efficient, etc.

48. 17:17

    as well. Yeah. So I I agree with all what all three of them said. Um so another different perspective on how like the security aspect may spill out into the real world or vice versa. Uh something we're not thinking about or you know really have a framework around is privacy and policy and also liability for that. So I see a lot of people building these agents that are doing crypto trading and you know trade on my

49. 17:41

    behalf or do arbitrage automatically for me or uh you know open call say hey like you know liquidate my portfolio and put it into here take it out it's like you're giving one you're giving trust you know to access the to do all that um but when it when it goes wrong because again fuzzy in fuzzy out or maybe it something goes off the rails who who's fault is that and like you know who's to blame for you the agent the LLM devel

50. 18:06

    developer the the host that's using that agent uh and how much access do you give it you know as far as to the the keys of the collateral and I think that you know when it comes to we see even like outside crypto like IP rights um in music you know there's like these new apps like that are making these amazing songs and like there's artists that are like now wanting to say hey they're like training on my song but it's like okay

51. 18:29

    well you know before we had AI when every time you listen to music growing up you're you're being trained on that song too in a way. So it's like it's a weird you know psychological way of looking at you know what's what's the difference between all that and I think I think that's going to be a very weird world pretty soon on how we can do it outside of the tech just like law privacy and compliance around that. What are some of the the things that you're most excited about, you know, a as we

52. 18:52

    tighten up, you know, the security around these things? Are there uh you know, use cases or uh customer propositions? I I'd imagine, you know, in near and and how in particular, you're you know, traditionally a crypto kind of security company, but you're now expanding outwards. What are you guys most excited about this this new world?

53. 19:11

    I something that you just said got me excited about like the whole vertical focus like a security framework with with AI like because that's that's what I see too and that's what we're building is that you know taking our expertise and making that into a framework and just delivering what we do now but faster and you know more consistently for for customers. So I'm excited that you that you said that. I I also want to just inject a little excitement in into

54. 19:34

    this because I personally think people aren't talking enough about the fact that a lot of these technologies we're working on I think will usher in an era of abundance never before seen. And I think some of the biggest drivers of this will include math and scientific research right that people don't talk that much about. I have friends who help people cure incurable cancers in like the last six months. Like what a time to

55. 19:58

    be alive. If you if you if any of us if any of us that deserves a round of applause. We're curing cancer. If any of us get cancer in our lifetime, I bet you we will be able to find a personalized way of treating cancer. That's amazing.

56. 20:12

    Can I I just want to dig into this a little. I mean, how did they do it? Like what what did they do? How were they able to cure cancer with You guys can look this up on GitLub uh uh on on Twitter. like a a friend uh worked with uh I'm trying to figure out how much I want to dox. Okay, I I'll just um I'll just share maybe some broad outlines.

57. 20:31

    So, um people have been able to use things like Alphafold, Gemini, uh Chacht and Gro to actually design personalized mRNA vaccines to target specific cancers. And people have been able to um there's a Twitter story of like a a person who the GitLab founder who cured his osteo saroma bone cancer using this approach and there was a dog in Australia whose owner helped cure his dog's cancer using this approach as well.

58. 21:06

    Speaking of uh doxing and identifying things, I heard that AI was able to figure out who Satoshi was from reversing like emails and stuff too. Who is he? Go look go look online. Um yeah no it's incredible I mean uh obviously the the medical you know in math science invention uh rapid pace of acceleration um Ari David any anything to add here in terms of the use cases that you're most excited about what do

59. 21:36

    you think you know after we tighten up the security practices here uh what what comes uh in the era of abundance here with all these agents um just just to add I mean I think dem democrat izing tech, democratizing uh capability to have your own engineering team to do things that you know was like you know need tremendous resources in the past. I think that's that's one of the the key things.

60. 21:58

    Yeah, for me like I was talking to Stephen about this before like I feel like I'm wearing an Iron Man suit. I can do like a 100x I'm more productive. I can teach myself things that I've never you know not that I wouldn't have been able to teach myself before but like I can learn something in an hour that would have normally taken me two or three weeks which means I can explore projects that I wouldn't have had time

61. 22:15

    to. So it's given me actually back time to do more things. Yeah, same unlocking that too. I think you know we people have a vision, people want to achieve something, people have an idea, you know, if you that entrepreneurial spirit or inspiration, whatever it is is still there. We're just able to get to it much faster. And I think you are having the age of abundance, but we've been in like the age of impatience for a long time with things. I mean, things are just happen faster and everybody wants faster faster. So now, you know, we we're

62. 22:44

    getting that too. So, it's kind of satisfying in a way. Last thing I got for you guys is just I I just want to hear the f your favorite tool, your favorite AI tool. George is going to say ironclaw. We already know, but maybe maybe has another one. And and just one tool that you think it's it's ready, everyone should use it, your favorite one. It makes you the most productive. Just just one tool as we as we finish up here and we get uh we get ready for our It has nothing to do with security. It's Suno. I like Okay.

63. 23:09

    It's a music a music a music making tool because I do music production. Like at first I had a crisis internally. I'm like oh my god my creative work is not going to I'm not going to be able to make song. Now I'm like using it as like another advanced instrument in a way. So okay very cool.

64. 23:23

    I'll choose a non-ironclaw example. So uh love love ironclaw. I also have been using notebook o a lot. Um, and my favorite thing to do is like have like a complicated paper or something like maybe a math paper or a paper from from Hugging Face Daily Papers and I'll drop it into Notebook LM and I'll create a podcast on it and I'll like listen to it while running the gym or something. So that's been like really really helpful.

65. 23:48

    I got addicted to Opus in VS in VS Code. Uh, it's just super productive. Yeah. I mean nothing terribly exciting. It's for me it's cloud code and building my own harness and running my own agent. Epic. Awesome guys. Thank you so much. That's a wrap.