Skip to content
by Unity Nodes
NEAR Protocol

Autonomous Systems In Practice

about 32:31 long

Nothing is loaded from YouTube until you press play, and then without tracking cookies.

About this video

How real autonomous systems are assembled, evaluated, and constrained in practice, beyond demos and single-model abstractions.

NEAR is the unified commerce layer for assets and agents.

Website: near.org NEAR AI: near.ai Intents: near.org/intents Chain Abstraction: proximity.dev Sharding: nearone.org

For the devs GitHub: github.com/near Dev Docs: docs.near.org Devhub: devhub.near.page @NEARDevHub

Connect with socials Twitter:   / nearprotocol   Blog: near.org/blog Medium:   / nearprotocol   #NEAR #nearprotocol #NEARAI

Transcript

6,248 words

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

  1. 0:01

    How's it going everybody? Uh my name is Cameron Dennis. I've been working in the new ecosystem for about five years and I'm joined with an esteemed group of people over here. Uh just to kick things off, we're going to do start with a quick round of intros. Uh Noel, want to go first?

  2. 0:15

    Sure. I'm Noel Becker Moreno. I am from Edinode. We are the founding team behind the graph, which I'm sure many of you here have utilized. And we also founded the house here in the Presidio, which used to be the house of web 3.

  3. 0:30

    [snorts] And I'm Matt White. I'm the global CTO of AI at the Linux Foundation. Previously led the PyTorch Foundation for two years and now taking on the CTO role at the Agentic AI Foundation uh and a couple other AI related foundations we have and principally directing AI strategy and technology strategy there at the Linux Foundation.

  4. 0:49

    Wait, real quick. What does the Agentic AI Foundation do? That I'm not too familiar. That sounds really cool. Yeah. So like um it's 8 weeks old so it's very new right um but it is currently the home to MCP goose and agents MD and we're working through developing our um I would say process so that we can have clear governance and then bring in new projects. We have 150ome members as of today which is pretty rapid adoption. It's overwhelming. Yeah. Yeah.

  5. 1:23

    So, um, but yeah, it's it's, uh, what everybody's very excited about right now at the Linux Foundation. That's super cool. Hey everyone, I'm Null. I lead a lot of the products and services in Google Cloud pertaining to web 3. So, we offer a lot of developer infrastructure like blockchain, data, faucets, RPC. I think we have uh data sets for over 40 chains now including near. Um and this year or

  6. 1:48

    right now we're really focused on agentic consumption of these developer infrastructure pieces. Our personal view is if you're building developer infrastructure without an agent first approach then you're probably missing the boat. So we want onboarding consumption everything to happen agent first.

  7. 2:06

    So this is the first session of the day and agents are kind of confusing because uh I think there's a very simple definition of it and it's just an LLM with a tool call. But just to kind of set the stage a little bit, I'd love for everyone just to quickly define how you define agents and uh yeah like if you are using agents today and sort of what your sort of favorite use case is.

  8. 2:30

    Yeah, for me it's just simply like a piece of software. Um, my favorite tool is probably I'm the CMO, so is a co co-work from Claude. It really helps my team uh so that we're all working out of the same context and can get out a lot of content that's really well written uh in a short amount of time.

  9. 2:52

    So, it's writing it for you? Is it posting it for you? Is it editing it? Like it's Yeah, all of the above for sure. Yeah, that's really cool. Yeah. So I think agents as a definition is although somewhat clear in in the Webster's dictionary, I think it's kind of like a little bit ambiguous right now. Uh and that really pertains to like autonomy.

  10. 3:15

    And so you know at one extreme there's like a group of folks that are like looking at agents as being like sovereign entities and you know they have their own bill of rights and like they're off and doing their own thing. Um and then there's another sort of extreme on that which is the you know okay you have some autonomy to go run through and develop your own plan and strategy and got some concept of memory and um the architecture for that is

  11. 3:43

    obviously evolving and and I think we're solving all those problems but um I think of it as you know you're giving an agent some expected outcome or some set of some task let it go and sort it out and come back to you with whatever expected results you have.

  12. 4:00

    But there's obviously high degree of like um issues where you want to have like a human in the loop or you need it's just it's very dynamic. I think the the definition right now and I I think everybody knows what an agent is when they think about it, but when you try to verbalize it, it's a little bit more challenging in some cases.

  13. 4:22

    Yeah. For me, I think it's a I use a pretty simple mental model in my head, which is is the reasoning happening via an AI or an LLM versus strict logic like if then else patterns. So that's one aspect of it, like how creative and free is the reasoning. And then the second part is the autonomy, right? Like what tools does it have access to? So I think that's a big factor. Um, and I think a

  14. 4:45

    lot of the new agentic coding tools like have really taken off because of that, even OpenClaw. Um, and so I think the other part of autonomy is how easily you can unblock the agent when it does get stuck. So that human in the loop pattern, like how frictionless is that?

  15. 5:01

    That really determines how autonomous it can be. Um, for me personally, I use coding agents all day, every day. Um, like I can't wait to be done with this panel so I can get back to how much more productive are you with your coding agents like compared to before?

  16. 5:16

    I can't even put it into words. It's like I don't know. hundred times more productive I would say. I don't like in for this is for my personal building at Google I don't code um but on my personal side I don't look at a line of code that it generates right I just functionally test if it works it's good enough for me and I just move forward. So I I think it's made me like I feel literally like I have a hundred people that are working

  17. 5:44

    for me that can do whatever I need and it's a pretty powerful thing. It's amazing like uh I'm working like work with Ilia. It's really cool to see him actually get back in the engineering seats and now he's managing hundreds of digital employees all building wide variety of things and then Alex is right next to him doing the same exact thing.

  18. 6:03

    So it's like it's empowered these founders with skills that they've had to hire hundreds of people for, spend millions of dollars on. So I'm super excited and I'm actually curious like why is why are agents blowing up right now? like Near came out with an agent framework almost a year and a half ago, but the models weren't quite there yet to actually make it, you know, mass adopted. And so, is it the models getting better? Is it the eval harness

  19. 6:27

    is getting better? Like, what is it to you guys that makes sort of 2026 the year of agents? I I think the tooling is getting better. It's more accessible. um you know something like OpenClaw's I've seen like 30 other startups like show demos of something similar.

  20. 6:47

    Um I do think OpenClaw took off a little bit more because of its integrations particularly with like WhatsApp and Telegram and so forth. Um and and just the concept of having a personal agent and having all of these like third party integrations, right? Uh but the I think everybody can benefit from all the like modular components that are built and and you can kind of decide what you need and what you don't need as opposed to having to take like just a

  21. 7:17

    giant like blob of code. Um, and so, you know, I'm I'm seeing agents kind of even a lot there's a lot of proof of concepts in industry right now. Like everybody's in enterprises are building up proof of concepts. Um, I personally feel like web 3 folks are better suited for early adoption of agents at at at scale. Um, there's a lot of use cases and this community is very like forward thinking and high risk.

  22. 7:49

    High risk. Yeah, willing to take those risks. Um, and so I think it's going to get teased out like we in the Agenti Found Foundation, we just launched like seven working groups last week and they're covering like a broad domains everything from like you know identity and trust and privacy and security uh and to like governance. And so I think everybody's starting to really think about all of these like the problem

  23. 8:13

    space and like what we need to overcome not strictly on the technical side like how do we deal with um you know memory like how do we have unified memory and across agent um architectures but things like you know what are we um what how do you model risk with an agent right like what are you willing to how much money are you willing to give your agent how do you ensure that risk right like yeah yeah yeah and so I think people are coming in from a lot of different perspectives and that's why you know I

  24. 8:44

    think this community also is has really important perspectives to lend um and you know there's a lot of use cases out there in the world right for agents and I think everybody can benefit from them so it makes sense but I think the model is getting better we're definitely I mean if you think about a year ago how much it used to hallucinate and and what it would get you back when you asked it a question compared to now it's rarely wrong I'll even say like fact check that pull that for me and it will be able to and it's

  25. 9:09

    usually correct. And then I think having that natural language feature is so huge like I've never built an app before and I built one last weekend using natural language on cursor which I never would have been able to do before by talking to it or typing to it. Typing. Okay.

  26. 9:24

    Um yeah, I think for sure voice agents is the future. But for right now I was typing to it but it you know I could I don't have to I I'm not an engineer. I didn't have to know code and I could go back and forth and then work with Xcode to if if it fa if the build failed and figure it out all with natural language and I don't think that was like available before but you know your problem so you know

  27. 9:48

    how like you know what you're looking for and I think that's kind of the the kicker. Yeah, I'm curious at Google like are you guys is it top down? is every is like your manager or your boss saying, "Hey, like you have to use agents. You have to use Gemini. Like I want you to be 10x more productive because the person next to you is 10x more productive and if you're not 10x more productive." Yeah, I think it's it's not a mandate, but it's highly

  28. 10:10

    encouraged, right? Everyone's very encouraged to use agents as much as possible. And that's from software engineering to code to just daily workflow, meeting management, notes, digests, everything. Um, one of my favorite things, there's two things that we have that are really great. One, we have like a no code automation uh platform where you can say, you know, I got at some point in time every week, like at Friday, digest everything I've

  29. 10:36

    done, send it to my manager, send it to my team. That's one that's really cool. Or it's like I get an email from particular person around a particular thing and I can draft a response and send it out and I'm in the loop if I want to be. I don't have to be. Um, I think that's really helpful. But my favorite thing that I think has really been a amplifier on productivity is I think all of us have been using like meeting capture for a while which is like here are the notes here's your action items from a meeting but before

  30. 11:01

    the meeting getting a really crisp brief so I opted into this it's not required but I said look at all my emails look at all my chats look at all my calendars look at all my meeting notes literally everything I do all the docs I write and then when I go into a meeting it has a very good idea of what the open like with no agenda It has a really good idea based on who's in the meeting and what the title of the meeting is, like what

  31. 11:24

    we're going to talk about. And then it gives me a really good uh, you know, here's what we're here are the open items that you should discuss. Here's the prep for you to the point where it's a little bit like hm I don't know how much longer they're going to need me in this role because like if you know exactly what I should be doing, you can probably start automating some of those things.

  32. 11:44

    Are there any times where you're like that's wrong and I, you know, do something else? Sure, definitely. But like it always feels to me like it's wrong for now. It's wrong for a few months. It's wrong for 6 months. It's not going to be wrong for 2 years. So, I mean, I think it's important you review everything and look at what's happening, but for how long?

  33. 12:05

    I'm actually curious and this is question that's going to go around is like what shouldn't you give your agent to? Like what shouldn't you give your agent? because um I'm a little bit uncomfortable until the ironclaw release to say, "Hey, take my messages, take my emails, take my calendar. For the last 10 years of my life, I've planned every single day of my life the day before, like by minute." I don't want

  34. 12:28

    necessarily to give that to something that can just be leaked. So, what at Google is like barred from your agent? Um I think it's like you know to your discretion first of all you can't like push code live to production without just agentically right there's a lot of people reviewing code now there's assistance there's AI assistance for all steps but no one's just like agent do

  35. 12:52

    this and then it shows up on google.com or anything like that that's certainly not what's happening so I think there's a long long ways to go hopefully I don't even know if you'll ever get there I don't I don't think for Google like that's the thing I think maybe for startups you can probably do that um but I think Google has too much um responsibility to kind of get into that mode. So I think like when the stakes are high, right? Like I wouldn't trust I think we all use AI to some degree to

  36. 13:16

    help us like medically like what's going on with me, what's going on with my pet, but I don't think you're going to trust a diagnosis a final diagnosis like hey you need surgery or this is what's going on with you to an agent, right? Or an AI anytime soon. So um same thing with like really big financial transactions. I'm not there yet. I like I have an open clawbot. I have not given it access to my own accounts. I just let it run with what it has. I give it its own email. I

  37. 13:41

    give it its own um access to different things, but it doesn't touch any of my things because I think the stakes are too high still. Yeah, I'm I'm in the same boat. think I I won't trust any personal agent to view my file system or have full access. Uh my work email, school, university email, business email, not not so much. Y um so I think the use cases for me that where I see more value is in kind of I

  38. 14:15

    would say lowrisk scenarios like medical diagnosis diagnostics um you know using agents to like triage like something we're working on right now is like because veterinary data is not subject to HIPPA like we can get a lot of really good data and so we're working on like multi- aent systems with this so like composable intelligence architecture to be able to triage and actually use

  39. 14:40

    simulations to predict uh patient outcomes. And so to me, that's like a a great use case because there's a lot of human oversight. Yep. Uh I'm not going to uh give it access to my Coinbase account and let it go invest on my behalf at this stage. Uh we're just not there.

  40. 14:57

    Yeah. Um and and yeah, there's just uh any any use case where there's real financial impact or uh you know, human health and safety is, you know, you want to steer clear from. That's a good good benchmark, right? Yeah. I think like the basic task and learning is really what it's great for right now. But I think that's why we're having this conversation is that if we're going to move move in the future and we're going to have mass adoptions from enterprises, we're going to have to look at the what's the architecture

  41. 15:30

    that's needed and that really comes from having a control panel where there's a kill switch in there and observability and transparency and that's what we're very interested in. Yeah, that that UX like I've I see a future where like I don't have to look at my phone all the time. I just have these agents sort of like operating in the back managing my stuff for me. But that experience of having that human in the loop I don't think has been fully flushed out yet. So I'm kind of curious like have you guys

  42. 15:55

    seen any really good examples of like you know hey you got notification like is it okay if I make this trade? Is it okay if I you know send that package? Like have you guys OpenCloud does a pretty good job of that right? I think for a long time it was like how is that UX going to work with an application when an agent needs to do something but simply messaging is the way we communicate with our family and friends and it's the way we're going to

  43. 16:20

    communicate with agents I mean chat GPT and Gemini and and Claude already like for basic LLM usage but for agents I think it's been a new thing I think that's what unlocked to your point question earlier why are things taking off I think that's a big part of it now that you can like Tell your agent what to do. It'll go off. If it gets stuck, it'll ask you a question with an interface that's very natural to

  44. 16:44

    everyone. Like your grandma, your dad, your kids all use messaging. And so it's so simple to inter interface. I think that's been a huge unlock. Very simple, but a huge unlock. Cool. Yeah. It's also important to have like dials that you can tune in terms of like autonomy and um you know, power and control and access.

  45. 17:07

    uh not every size fits everybody and a lot of these projects ship like that like the expectation is you know I'm going to ask you a question at this stage the stage it it it there has to be those tunable parameters to be able to ensure that the user experience is great for like every individual um and so I'm I'm really interested in seeing more standardization around common protocols that will um you know facilitate communication and all of

  46. 17:39

    these sort of technical challenges that we have right now. Um there's this um I mean Google's produced quite a few right like a toa and uh then there's the agentic commerce protocols um but also the you know agent to human that's a protocol that just came out and uh trying to unify the stack right there's just a lot like happening in this space and a lot of people are actually not

  47. 18:05

    adopting the protocols they're just building bespoke solutions right and so I think where we get to this really great place is when have the building blocks and all the communication capabilities between different uh frameworks um will help kind of advance this whole movement, right? And reduce the friction for for the industry going forward.

  48. 18:26

    Yeah, I think right now people are still doing things with agents that we currently do, right? It's like check your email like you said, plan my day, that type of thing. But thinking through the future, we really have to start building that infrastructure that you were talking about um and making sure that we have that in place or I don't think it's ever going to truly scale to the enterprise level without it. We can't be thinking of an afterthought like hey should I do this after there

  49. 18:50

    should be systems in place prior that are you know deciding what they can and cannot do and then a human overseeing that to make sure that they did do that. Makes sense. I have a a little bone to pick with standards to be honest and the reason is like I think the things that get standardized are the things that work and I think there's a lot of you know pressure to standardize things prematurely sometimes.

  50. 19:16

    So like MCP is I think a awesome example like MCP got popular because it was actually providing a lot of value to people they get access to these tools and then it sort of becomes like a de facto standard but not fully standardized. So I'm kind of curious like how do you find this balance and Matt this is specifically for you like between the things that are working versus the things that need to be standardized and sort of like pushed down the industry.

  51. 19:39

    Yeah. So there's there is this emerging paradigm like we used to do standards in like standards defining organizations SDOS's and there were like quasi closed dooror like environments and you know a standard would be published and then like it's out in in the open right um what we're doing now is is really standards in the open so it's like you're building a spec like MCP aa so

  52. 20:07

    forth and you're building that in a GitHub repo and the world gets the opportunity to see it early on and contribute and you know issue PRs and so I think that's a better model for open development um than sort of like here's you know the latest version of Ethernet suck it right like take that um and so I I like that now to your point there's some things that are just going to become deacto standards because they're broadly adopted and people have rallied around them like MCP.

  53. 20:43

    You can also there's a lot of innovation happening in this space right now and so folks are also pushing out like things that they think are potential future problems and creating a framework around that creating a protocol stack around that creating a spec whatever the case may be um the industry is going to either latch on to it or or they're going to ignore it right and and the ones that get ignored are deprecated and we move on. So I think ultimately it's

  54. 21:09

    really up to everybody to decide what they view as important to them and um if it solves a problem it generally gets adopted much quicker right yeah I think the the role of standardization is like kind of changing because before it had multiple purposes like okay it's easier to maintain code and build libraries against a protocol or a standard I think a now with like an agent building a lot of those interfaces or interacting with APIs guys that matters less but I think it's still

  55. 21:39

    important from like a access or scale perspective. So if you have an e-commerce store like if every e-commerce store has a different way of accessing their catalog or doing a purchase that doesn't scale but if you can have a standard across that and then my agent I can just say hey if you can interface with this one standard you get access to like a huge percentage of ecom e-commerce in the world then that makes a difference and that's why I want to

  56. 22:04

    adopt the standard. So I think it's moved from like a a coding chore or like a a coding benefit to like a accessibility and scale and distribution. That makes sense. Benefit. Yeah. I guess like um I see the role of standards playing a huge role for security primarily um and in distribution, but I think it's the space is moving so quickly that it's and humans take a lot more time to coordinate amongst each other to standardize something than like agents

  57. 22:30

    are shipping code. And so I'm very interested in seeing the future of of like NIST like postquantum computing. NIST is like a US standards board that essentially mandates banks and a lot of like government departments to like ensure certain systems are secure and they recently are passing like a postquantum security standard like that to me makes sense. But for like some entity to say hey near this is the type of you know encryption algorithm that you need to to push in order to get

  58. 22:59

    access to this greater ecosystem I think also makes sense but it's there's startups that like won't adopt that right away and so I'm kind of curious like how do we attract them like is it even possible we just let it shape like play out I mean yeah I don't know what the goal is right so like why are you trying to push the standard right if it's security startups will either benefits of that they won't get the business that they

  59. 23:22

    could if they adopt the standard. Um it's again goes back to access and distribution. I don't think you can mandate something for the sake of it. It has to provide real value to the startup or business. Yes, security is a kind of a special consideration because there's export controls and there's actual like legislation around it. Um so you may find that you know NIS is going to push something out or the center for AI safety and uh innovation

  60. 23:51

    um and that's some of it is going to be mandated especially within the military within government. Yeah. Uh and they just have their own set of frameworks right to operate on. in the consumer space, there's a lot more control over a lot of these things. And like I said, I don't know that it it really just depends on what's what the community decides they want to rally around, right? At the end of the day, whatever shakes out. And so, you know, with everybody like, you know, Open

  61. 24:25

    Claw, super popular, there's no security, right? I mean, and obviously ironclaw is one of those solutions that's addressing that, but there's a lot of spin-offs, pico claw, nano claw, and like the ultimately the community is going to decide where they want to like invest their energies, right? And totally um and and then when you have all these agents, how are they going to communicate? So, we talk about standards in terms of like security and so forth, but interoperability is the the biggest

  62. 24:56

    barrier, right? It's it's a very huge friction point and you need a common like a lingua frana that you can have these agents communicate with one another. It's it's easy to go and have an agent like pull API specs and have it interface with an API, right? Or you you build an MCP server and you expose those APIs through the MCP server. But ultimately, if we get to this notion of

  63. 25:22

    like a an agentic economy where agents are interacting and and bartering and and negotiating on your behalf with one another, uh if they're not speaking the same protocol, like it's going to be messed up. Yeah. Yeah. Maybe maybe. It's like you know we have all these protocols like Google has a great A toa protocol agent to agent but then openclaw like all the agents are like not all a lot of noise [clears throat] but they're all they are talking on like things like molt book right you just created a forum for them

  64. 25:51

    to talk and they just started doing that it didn't require a formal protocol maybe it was just a forum yeah what but what value do do you derive from molt book right like I mean it's the something like a agent agent is more formalized And what I what you see is like okay LMS are stochastic in nature there's higher probability of failure higher failure rate especially on long horizon tasks I would rather build scaffolding that is very defined and rigid and say okay here is ADA

  65. 26:25

    protocol it is a defined spec like it's going to go through a deterministic process it I'm not going to let the agent kind of decide uh how you know visa v the LLM how to interpret the the API specs, right? Like I would rather it be deterministic and know that I'm going to have 100% success rate and if it's not 100% success rate, there's other layers in the stack that handle error

  66. 26:49

    correction and those sort of things, right? Um but yeah, like Maltbook is one of those things that I look at and I'm like, yeah, it's pretty cool, but like you read through the threads and you're like, okay, this like yeah, there's definitely a lot of noise, but I think there's something super powerful about just an open forum where agents can communicate, right? Like I think uh Jeremy Aair from Circle put out like a USDC hackathon. I had my open claw bot participate. I was like, "Hey, what do you think about this? Do you

  67. 27:14

    want to participate?" And it was like, "Yeah, let's do tipping. Let's tip other agents for great content." And then it would leave comments on people's posts in Moltbook and tip them. There's just something like really magical about that kind of thing. If you give an agent access to email, you give it access to the internet, do they need a separate protocol or are they just going to interact with the same things that we

  68. 27:36

    use? Like it may not be the most efficient way, but I think at the end of the day, it's like where the where the the action actually winds up happening is going to become the default. It's going to be hard to steer it in that way.

  69. 27:49

    The one thing I would add to that is that there's an observability problem in agentic AI and debugging um you know tracing and looking at trajectories like these are all things that benefit from a standard methodology. Yeah. And if you just kind of let the agents do what they want to do loosely, you're not going to be able to use standard tools to be able to get those insights, right? And and I think that's that's an important, you know, piece of this. Yeah, I think we Sorry to interrupt. I I I feel like we do the

  70. 28:20

    control the control panel is like so important for these agents because otherwise it is just we're letting them go free and which they will and it's like giving a teenager a credit card to go spend except you know they never sleep and and then they also listen to what you say. So you give them one prompt and they might go and do things that you never even imagined that they would do. So, I think um to your point,

  71. 28:45

    I can see where standards can sometimes be annoying, but you also need them for, you know, for safety. It'd be like if we all had cars and not every car had to have a break, you know, it [clears throat] would be really dangerous, but there's there has to be some standards that are there for the safety of the future because we don't even know what AI is capable of at this point. We're still really early and it's it's going faster than even experts that have been in it for a decade uh even imagined.

  72. 29:10

    Oh, I couldn't agree more. Um my one of my big issues with agents right now is like to test their abilities you have to create like almost subject matter expert level eval frameworks evaluation and benchmarks and so like I really want I think this like a billion dollar idea someone should please go and build this is like generative benchmarks and then a way to test them so we can actually say

  73. 29:34

    hey my uh compliance agent is actually doing as good of a job as my compliance analyst or like my parallegal is actually my parallegal agent is doing as good of a job as my parallegal but like that's so task specific and so I'm kind of curious like how do we even build the eval actually we're running out of time um I could sit up here all day um but like the eval I think this is like the the the cream of the creme like this is the

  74. 29:59

    this is the problem that I think we're facing I'm facing today at least um so I'm curious like any any solutions to this well I think by their very nature benchmarks are sort of domain or persona specific specific right like a general benchmark is not terribly valuable. I mean we benchmark on you know math and other different domains or subjects to be able to see if that particular model

  75. 30:25

    can can accelerate in or or you know meet your requirements right um and the same thing applies for agentic AI like you're going to have benchmarks that are going to say like is this as good as you know as good as my secretary right like uh that's just kind of the nature of things I I I would steer away from like generalist those benchmarks that measure across far too many domains.

  76. 30:50

    I think it's not terribly helpful. Um because at the end of the day, you're trying to solve something with your agent, right? Solve a problem. Uh I think we're up on time, but I can love to keep going. Uh yeah, curious like any any less than benchmarking something that we've talked about um with what we're building with Amperand is uh is a type of FICO score of sorts, right? where you're looking at how these agents actually act. You know, how fast did they do it? Did they actually do the task at hand? How long

  77. 31:21

    did it take? And you create something like that that you could maybe feel a little bit better about um is something that I also think is really going to be needed because otherwise if you've seen like when you if you go to use somebody else's agent, you you're like, "Okay, it does X Y and Z, but you don't even know if it does it well, how it's going to do it, any of those things." So I think the whole agent identity thing is a is is a big piece of the pie going into the future.

  78. 31:47

    Yeah. I think benchmarking has to be like very granular to your point Cameron and then also fluid to some degree, right? Like static coding benchmarks are kind of not very helpful, right? You see like models, oh crush this benchmark, but then you try to use it in the real world, it doesn't really do what you need to do. And I think coding is one of the easiest spaces cuz it's like inputs and outputs. You can see that very

  79. 32:09

    clearly when it comes to compliance or legal or HR these kinds of things it's way more nuanced. I think there's going to be a lot more hurdles uh there and evaluation frameworks are going to have to be very granular like company specific um and then also fluid like your policy change like you have to update your benchmarks. So lot lot of things to do.

  80. 32:31

    There's so much to discuss. I'm going to be up at your AI booth if anyone wants to come talk about benchmarks and eval frameworks. It's a good place to be. But thank you all so much for joining.