Shade Agents Truly Autonomous, Non Custodial, Multichain Agents - Kendall Cole | Proximity Labs

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   all right hey everyone my name is Kendall I uh work on a company called proximity that does a lot of research and development for the near ecosystem just last week we had a very exciting release we are calling shade agents so the tldr right there on the screen these are truly autonomous and we really do mean that non-custodial multi-chain agents so uh agents are very hot right now mainly outside of crypto but even

2. 0:26

   inside of crypto um these are just a few of the many many project s that are launching uh different types of Agents um on the web two side of things things things are moving very very quickly uh probably some of the most exciting things you've seen is like the operator from open AI that can actually control your screen uh we have anthropic that has uh a similar type of product like they've developed an entire standard for

3. 0:49

   how agents can interoperate together uh but for the most part they don't uh complete many actions for you because they really do struggle with Authentication payments in particular and really anything that involves confidential information so that's the web two side of things on the web three side of things most of these projects here are super exciting but there I would classify them as like Frameworks

4. 1:12

   in the case of Arc uh Eliza Etc uh we have virtuals up there they're also a very cool project but similar kind of a framework and then a Launchpad and then we have things like uh a BT which has done an amazing job of of think generating a ton of traction it's like probably accomodating M share more than just about any other entity on crypto Twitter but it's it's you know it's mainly it's a it's a Yap bot is how I describe it I think uh frasa in particular that's some with the green hair is probably one of the more interesting experiments they were also

5. 1:45

   kind of just the interacting with the agent trying to talk to it but you could actually try to extract money from it which is kind of like a fun way to turn prompt injection into into a game um so basically the the tldr and the way agents are today they can uh get information really well um they can kind of get you to the point of taking an action but generally they're not very good at actually taking that action and

6. 2:09

   there's a few reasons why um so I mean I think this is a there's a couple of things that uh that you need in order to have a truly autonomous agent I think the most important end result is that they can actually complete any workflow for you end to end which includes payments includes actually going out into the world or the internet or wherever it is and doing something um but uh yeah we're going to focus this

7. 2:32

   mainly on how we can build autonomous agents in the web 3 space and so these are some of the criteria from how you actually create a an autonomous agent like what you sort of need uh the first one is probably something people don't think a lot about but we actually think it's one of the most critical pieces which is this idea of decentralized Key Management what we mean by this is that agents are going to need some set of keys that they can interact with but

8. 2:56

   those keys actually need to be able to be interacted with by multiple in instances of the agent and the reason for this is that if you have a key that's too closely embedded into like where an agent is actually running what that means is that if that agent fails for any reason that key is lost and thus any funds or any authentication that was dependent on that key is also lost in crypto in particular this is kind of the most fundamental piece right is that

9. 3:20

   like basically not your keys not your crypto so if you don't actually have a persistent access to the key and authenticated access to the key then everything that was a result of that key which is literally everything you doing in crypto is lost so we actually think that first piece is incredibly critical unfortunately that's something we've been building in near for quite some time now uh with the product we call chain signatures getting into some of these other ideas here so uh we also

10. 3:43

    think it's very important that you know exactly what your agent is supposed to be doing and one of the easiest ways to do this if if you can is you verify the code that's running you and ideally you even verify the models that are going to be used to kind of perform some sort of action um so what we've been doing is working on ways that we can actually provide these guarantees as much as possible um and then getting into some other very important aspects but uh you know kind

11. 4:09

    of I guess further field is like you do want uptime guarantees uh the uh neari Team launched a really interesting paper called proof of response it's basically designed to solve this problem we've been investing very heavily in this idea of trusted execution environments that are very good at verification but they don't actually guarantee you uptime for any reason and that's what proof of response is is designed to resolve um and then some other things that I think are going to become best practices but are still a bit nent and new is this

12. 4:34

    idea of using smart contracts more as like guard rails for these models and these agents rather than as like the where you actually code the agent itself I think realistically given that agents are require a lot of offchain compute they require a lot of access to offchain dat data and State uh you actually want to use Smart contracts more to keep them on track and make sure like hey this agent is proposing to take this action

13. 4:58

    it's proposing to deposit my money into a well I want to know that that like a is a good not a good app it's like somewhere that I want my money to be sent to like I I that basically we we think smart contracts can be used uh very creatively to actually keep these agents doing the things they're supposed to be doing and then last but not least most of these agents when they're acting on behalf of a single user they should be kept as private as possible at least

14. 5:22

    the actions that they're taking I mean the second you touch a blockchain that's mitigated a bit but like the decision process that goes into making that uh that action your personal data that might be a part of that decision-making process all that should be kept private and so what we did is like sought out to create a system that that can give you those assurances so uh yeah basically what we call them is shade agents um we are convinced that these are currently

15. 5:45

    the first truly autonomous agents because they actually hit each of those points and we'll go through exactly how it works so here you can see a diagram of basically the shade agent stack there's actually two components that matter the most here um so the first and you see this is like in that kind of trusted execution environment Banner is this worker agent this is basically what you think of as like an agent this is what most people are building when

16. 6:09

    they're building an agent usually it uh accesses some or like there's some wakeup condition okay agent's going to run whatever that is like someone requested agent go do this for me then there's uh some access to some data some State maybe that's information about the user maybe that's information about like uh the price of a cryptocurrency because it's going to decide to make a trade or not uh and then finally there's like

17. 6:34

    some llm or model doesn't have to be an LM could be any sort of model where there's going to be basically like a decision-making process that'll be resolved and then finally once that decision is sort of finalized and the agent's like okay I want to go I want to take this action then that action needs to be sent for execution somewhere so that's the worker agent that's running inside of this trusted execution environment meaning that the actions that are taking inside of this environment are only known Within

18. 6:59

    environment there's no way to exfiltrate or no easy way to exfiltrate the data from here and also means that this code cannot be changed without basically wiping the whole environment and thus we'll talk about in a second why like basically removing any of the information that's in there the second major component is the Smart contract so we're using the smart contract for basically three main things as you can see there the first is to actually register this worker agent what we mean by register is that this smart contract

19. 7:25

    is going to require that this worker agent actually submits basically a proof of the code that it's running this does this using kind of a primitive of the of a trusted execution environment which is called an attestation quote so basically what will happen is that the the the environment itself in this case like let's say an Intel CPU running this this special environment will actually generate a quote that shows the exact

20. 7:48

    current state of this environment so uh in our case the most important thing it's going to be showing is the code hash of what's actually running in there and then all that is basically it's signed by a key that's embedded within the hardware itself and then that key is basically attested to by Intel in this case who's saying yes that's actually one of our CPUs that we sold we verified that like it should be you the Integrity should be intact and so the smart

21. 8:13

    contract will actually take that quote it'll confirm the signatures from Intel valid it'll confirm the signature from the hardware itself is valid and then it'll actually extract that code hash and check that it matches the code hash that the developers of the smart contract the deployer of the smart contract set as like this is the this is valid worker code and what this means is that anyone can actually spin up this

22. 8:35

    worker agent on their own you know their own assuming they have the right device their own device submit this quote to the the smart contract and then all of a sudden they're now a registered instance of a worker agent it's important because what this means is that if any of these worker agents go offline anybody else can spin up a new one which gives you the resilience that's important for like actual autonomy um and then finally what this smart contract is going to do at

23. 8:58

    least in what we think is a best practice is if uh if you take advantage of the feature we have which is chain signatures which tldr is that a smart contract can sign transactions basically we've given smart contracts Keys via a decentralized multiparty computation Network then the smart what the the worker agent will do is actually request to send a transaction uh on behalf of this whole system the smart contractor will verify that the worker agent is

24. 9:25

    registered verify that the transaction the requesting is one that makes sense that fits some some rules uh and then they'll actually go out sign the transaction using the smart contract's own key and then finally send that to the network and we'll actually walk through the code that shows how that works in a second um so yeah here's kind of another flow of how the process works so it's like step one deploy the smart

25. 9:47

    contract step two anyone can deploy a worker agent uh and there's a couple of of like helper steps that we've created you have to each worker agent will have a near account because ultimately it's sending near request to the smart contract on near in order to like send these transactions to the destination chain uh so there's like a funding step then you also check like is this worker agent registered because if it's not you

26. 10:11

    can call the smart contract but you'll get rejected every time so you have to send you have to complete that registration step and then finally you can actually then make requests to the smart contract to send transactions and it'll approve these signatures and then actually uh you know complete these actions for you um so how does this different from from currenty implement mentation so it's become in Vogue I guess to run crypto agents in a trusted

27. 10:35

    execution environment because those are seen as at least at least better security than running it on just like you know some some random Hardware somewhere uh the problem is that for the most part these uh T's they they have a key that's embedded in them and that's good for security because that means that you know someone else doesn't have that private key and they're not misrepresenting that this only the te can basically perform actions using this key uh the problem though is that if

28. 10:59

    anyone is able to pull the plug on this physical Hardware that key is lost forever and that's important that's a really important security feature but it does mean that you can't actually trust that any of the funds you send to this key are always going to be there which is like an insanely it makes zero sense like that's not we don't think that's like viable at all I would not recommend you send your funds to any of these kind of Agents uh and so what we set out to fix with the

29. 11:22

    shade agent is that the worker agents are basically jointly able to use the same set of keys so the keys are persistent they'll always exist uh but the only way you can actually request these keys to sign and do something is if you register and you prove that you're running the right code in the T that's kind of the tldr of this whole system um so here yeah we kind of show like how you can basically deploy one of these worker agents it's actually pretty

30. 11:47

    simple uh if you use a product called follow Cloud they um they provide a really really nice way to deploy these agents so you can basically build your application your worker agent using Docker that generate a unique hash um that is basically like the the hash of the code that you're actually running uh and then all you have to do is basically uh create this Docker Docker composed.

31. 12:11

    yaml and and deploy that using follow cloud and your worker agent will be up and running um and then here you can see actually the code of this is running in kind of our example worker agent so what's happening here is we're generating a key within that trusted execution environment that will only ever exist in this instance um and the the point of that is that if this worker agent goes Offline that key is lost

32. 12:34

    which is an important part of like the security properties of this thing but you can then just spin up a new instance you'll generate a new key that's fine you just have to re-register that's all that means um and then here you can see where're actually uh getting the code hash of this creating this this attestation quote and then calling that register worker function on the smart contract basically proving that within this instance of this worker agent we're actually running the correct code

33. 13:00

    and uh yeah here so we we we built like basically a reference smart contract that actually implements some of these methods uh so we've got two basically helper methods one is this uh this owner just basically checking like is there who who's the owner of this smart contract who can update this this code hash in a best practice this should either be immutable you should just not actually have an owner meaning the code has can't be changed or you know if you

34. 13:24

    want your system to be able to be developed over time this should be either like a dow or I guess like you know worst case like something like a multi- um and then we also have this kind of you you see there that like method a that's basically uh requiring that the CER of that method which will ultimately result in a transaction is running with a given code hash and so one way you can actually use this is you

35. 13:47

    can have different agents that perform different functions they'll each have different code hashes but then you can have like basically maybe there's like three different types of Agents you want to run in one system they each have different code hashes and you'll have different methods that will pertain to those different workers so one worker might be able to like rebalance funds across lending protocols one worker might be able to uh make trades one worker might be able to update the whole

36. 14:11

    system uh and they can each basically call it be be authorized to call different functions on the smart contract um yeah here's kind of just some other another way to sort of view that cool so we're going to go through all that in more detail in uh just a minute here um but let's zoom out a little bit and talk about like what can you build like why is this interesting so four major areas that we've identified and I mean frankly this is

37. 14:35

    probably just the beginning um I think actually one of the most interesting use cases for agents is taking a look at the defi apps that exist today recognizing that just about all of them have a critical component of humans in the loop and thinking about like where can you actually remove some of those humans in the loop or what new types of applications could you build if you could have some sort of intelligent

38. 14:59

    decision making that could happen very rapidly which is like one of the clearest advantages of something like AI versus you know having to like have this kind of slow necessarily solar process of having humans in the loop so it's like that's one really interesting way to think about like imagine you could just replace a dow with some agents and what would that enable you to build I think another way to think about what you can do here is that because of the fact that we're running a lot of the core logic of this system in something

39. 15:24

    like a trusted execution environment that can itself have sort of verified access to the internet meaning it can call offchain data providers um and I mean at least know that the uh that that it's going to consistently call that route you still have to trust that that you know that that like URL is going to serve the right data but um but basically this enables you to more flexibly utilize offchain data uh without having to like get an oracle

40. 15:48

    spun up for specifically that purpose and think about and then and then think about if you can act in the soft chain data but then also uh utilize any type of model in order to make decision so we have a couple of of example ideas there uh we think like you know a very basic one is just a lending Optimizer right where you can delegate effectively your funds to this agent uh it'll then look for the best rates on a set of lending

41. 16:13

    markets and it'll rebalance your money across these over time as it notices that things are changing there um and then last but not least I think like you can actually create interesting markets for basically anything that's digital we've seen some experiments already where people are like letting you pay for a single use of an API key so I mean a basic one could be open AI itself right like you want to send a request like a prompt to open a open ai's API

42. 16:38

    and then get back some sort of response you can actually create a protocol where you can basically delegate anyone could delegate a single call and then have you pay them for that single call uh which could get really interesting another area that we find pretty fun and I'll actually zoom out and show you exactly how this works today is this idea of a headless UI so because these agent can be tracking interactions basically anywhere and then

43. 17:02

    respond in a verified way you can actually remove the idea of uh creating a new UI and like let people for instance uh interact with your application entirely through something like Twitter um so let's actually show how this can work because we built an example all right so uh we built this agent called shade agent 007 basically what it does is it listens to any sort of request for its name it then analyzes the body of the tweet and determines is this Throwing Shade is this like a negative tweet is this saying something

44. 17:37

    positive uh and then it determines which token it's going to send to the user and so right now we have the user including their account in there but you can actually even even get rid of that um so I'm going to send this tweet I'm actually not sure whether it's going to uh frame this as H base or shade but uh let's find out and it takes about a minute to respond uh but yeah so basically the idea here is that um you can have any sort of existing interface where users are able

45. 18:11

    to interact either directly with these agents or even just make any sort of comment um these agents can then see okay cool there was a request that was basically effectively came to me they're like this user is trying to is trying to do something and then it can perform an action on their behalf uh so you could even have like uh an entire wallet that's built into x.com where you just could say like send X tokens to this address and then the agent would be able

46. 18:37

    to respond and actually like take that action and actually do something um let's see if it's and then last but not least is this idea of web markets um and so for this one we actually worked with uh with uh the Uma team to come up with a uh initially very basic demo but one that I think is pretty interesting that can um basically uh uh create user generated markets uh and these markets can be agent settled they could be agent

47. 19:04

    resolved they could be agent traded uh and they can access any sort of public API or private uh or private web data uh so we think this opens up like a lot of opportunities to uh basically trade based on digital information and um digital like yeah basically digital information all right let's check back in on our bot and see if we've gotten a response yet there we go go cool so it actually sent me these base tokens and this is going to be on hyper liquid so

48. 19:35

    let me um yeah we'll come back to that one during the workshop but basically I'll show you guys that I actually received those tokens on the hyper liquid main net so basically what it did is this agent looked at this tweet determined that this was a positive tweet I guess it's harping more on the age of AI execution uh than on my comment about AI slop uh and then it basically decided to send me these tokens to this address on hyper liquid

49. 19:59

    uh which is pretty cool so I welcome anybody to try it out if you tweet at shade agent007 includes your address there and then say whatever whatever you kind of want and uh it should respond to you and actually send you some some real tokens they're technically soulbound tokens they cannot be transferred um but it's still pretty fun you will be able to see them in your metamask wallet um all right so now we can actually just go ahead and hop into uh a specific example

50. 20:24

    so um we will be walking through this template but and if you want to follow along I recommend you go to near ai. a shade and then you can actually just scroll down just a little bit and you can see uh the template itself and then we'll go ahead and walk through this code but first I'm going to show you guys uh what we built uh that uses the Uma optimistic Oracle um so what's cool

51. 20:51

    here is like you can actually see this you all like this this uh this entire UI is being served from within this trusted execution environment generally that's a bad idea because like you basically want to minimize any sort of access to this environment that you can ideally you have this environment only calling into the smart contract and that's one way rather than two-way otherwise you do open up some attack vectors but for the purposes of this we figured we'd make it

52. 21:14

    a bit simpler and actually have a UI that's still being served from here so what we're going to do here is we're actually going to take some of the resolution criteria from a recently settled Market on poly market so we'll take this one this is from uh what is this here it's the most passing yards on Championship Sunday so this is resolved about a month ago um but never too late to kind of check out what's happening so

53. 21:37

    we'll go ahead and post this in here and then I'm going to click submit and then so we can see what's happening I'm going to pull up the logs here um all right let me check this environment let's do this cool so you can see here uh it's like checking a couple things like my balance checking the key making sure that I'm actually authenticated to call this method then here you can see it's actually this is the input of okay this

54. 22:02

    is the the market that we're aiming to resolve and then you can see uh here it's basically has now like authorized that I can call this transaction uh that I'm I'm able to actually send this this signature it's generated the signature here this is going to be the address on uh ethereum aoia that it's sending to and then bam it looks like we're were're sent so where we can actually check this

55. 22:25

    is on the um Uma test that page itself to see like did this actually send and let's see it looks like this is ours right here yeah this is our uh Market will resolve according to whoever had the most passing yards so we've actually now basically from from here yeah we think we have a success so from here we basically said okay this is the market that we're looking to resolve we entered that in what happened then is that we actually walk through this here

56. 23:02

    let's walk through what happened so this question came in here well this is like us kind of making sure that we know which address to call uh so basically this question came in this is this question string then we passed it to perplexity basically asked perplexity like who had the most passing yards on Championship Sunday perplexity then search the internet used its LMS basically took probably I guess their

57. 23:25

    like Google data plus their LM and determine who had the most passing yard and then we actually went and we like gener like this in this is all in the worker agent we generated this uh request to actually send to the contract saying like hey we've determined we want to take this action we're going to prove we're a verified agent where we have the right to basically call this smart contract so here you can see us actually constructing the transaction um and yeah so we're going to call this assert truth

58. 23:50

    with defaults on the Uma actual um contract we're going to do this on spolia um and then here we actually put the transaction together so we've got the address the chain ID uh the actual like transaction call itself we generate this unsigned transaction request and then this is where we actually call the near contract is with this like complete ethereum transaction so what we're

59. 24:13

    actually going to do is we're going to send this transaction request this unsigned transaction to the near smart contract um we can actually look at the near smart contract here so we have this get signature function that's going to be being called with this uh this payload which is that unsigned transaction and then it's actually going to like basically call the chain signatures contract which will then then give us the signature back and then once

60. 24:37

    we have that signature back we can con reconstruct the transaction like complete the transaction and then send it to the sepolia network so we basically completed this Loop of like worker agent that's going to be responding to these requests that's running in this trusted execution environment it registers with the near smart contract says I'm running this code has like this is like this is verified code this is only running with

61. 25:01

    this in this uh in this trust execution environment the nearest Smart contract white list that agent so that that agent can then send arbitrary transaction requests and then what we just did is we sent that arbitrary transaction request of actually calling out to the Uma contract um so that's basically the flow of what actually happens so now let's dive a little bit deeper into the structure of this repo so that you guys

62. 25:25

    can kind of do this yourself um so as we said before there's those two main components there's that worker agent and that smart contract we'll look at the Smart contract first because it's like a bit simpler um so basically in this contract we've got just a couple of files here so primarily we've got this like this live file um so you can see here we have an owner ID we'll have these approved code hashes and this is what initially the owner ID can

63. 25:49

    basically add to and again I best practice this is either immutable or managed by something like a dow and then we'll have these worker agents these like register worker agents they can be registered by ID and they'll also have like a mapping to a code hash um and you see here yeah we'll just initialize it and then like the most important functions even more so than the require owner is this require worker so this require worker function basically what this is going to do is this is going to

64. 26:14

    tag a specific function as can only be called by a registered worker agent uh and so what it's going to do is like check the account ID of anyone who calls this and then determine like are they basically in the approved list and then like look at their code hash and say okay this worker agent is running this code hash does this match one of the wh listed code hashes um and then here you can actually see the registration step itself so this is what the worker agent

65. 26:38

    will call when it's first getting set up and uh you can see here like we're actually picking a piece aart the collateral they sent this is basically that attestation quote from The Trusted execution environment and then we're going to actually call out to this collateral contract and run this verified check which is basically looking at all of its signatures and making sure that they're valid from both the Hardware uh the hardware itself and

66. 27:01

    from in this case Intel since we're using Intel tvx so that's the contract so basically like for the most part you guys can Fork this contract uh use it as is the only thing you have to do is add additional methods here like I I showed before on this um on this Uma bot is we had that kind of like like basically the the the one to get the ecdsa signature that's all you have to add and then you're pretty much of and running and

67. 27:25

    then we can look at basically this worker agent which which is effectively the rest of the codebase so we set it up as like a nextjs app to make it really easy to handle this registration step cuz like it is a bit involved you have to generate this quote you have to generate a key and then you have to actually call the near contract and you've like fund your account first it's possible to do all of this through the CLI but we just wanted to make it simple so we put together this nextjs app um so

68. 27:49

    you you don't actually have to change too much I would say like you can mostly keep that part out of the box what I recommend changing is if we go to the pages here you're going to want to add routes to this API folder because basically right now it's just set up this is just basically the registration logic so it's not actually doing anything but uh if you add say like another call here that does actually does something interesting and you add

69. 28:13

    basically a file to this uh this API folder then you can actually like perform an action and so if we go look at my Uma example here then I can show you the action that I added so we'll go ahead and open up my code here so you see uh here like I added this Uma file here I also added to this index I added this uh handle question submit that's basically that form field that I was using and you can see that what happens when someone enters some data into that form field is I call this

70. 28:46

    Uma API route and so here is where we're actually doing the kind of interesting stuff so this main function here this is what's called when you call that umao route so the first thing we're going to do right we're going to generate that um the address and so what we're doing here is we're actually determining what address we want to call from on the evm site and so this is basically like the way chain signatures works is each smart

71. 29:09

    contract basically has a determining address it can request uh transactions to be signed for and so the inputs to that there's basically a master public key of the MPC Network itself we have that in the docs it's super simple to get um they'll be the actual account ID that you're calling from so this is basically your account ID of your near smart contract a path which is just an arbitrary string the reason for this

72. 29:34

    path is that you can basically control n number of addresses on a destination chain and then finally like the chain itself like what type of chain in this case it's an evm chain so we're going to include that here but uh you can also do this for Bitcoin very soon you'll be able to this for salana and basically every other chain um and then yeah we basically you know take the question string that was being asked and and do what we were talking about before calling perplexity uh getting the result

73. 29:57

    from perplexity constructing a transaction and then actually sending that transaction first to be signed like verified and signed from the near smart contract and then finally to actually be relayed to ethereum and so the important thing to remember here is that you can actually do this for basically any chain right now every any ecdsa chain any chain that supports ecdsa signatures uh within the next few weeks you'll be able to do this for any chain that supports

74. 30:21

    eddsa signatures which at that point means you can pretty much call any chain um so yeah that's that's that's pretty much the uh the structure of this project so um yeah we'll probably have a little bit of time for questions um but I just want to make sure you guys have this link so again just the uh this is the link to go to to basically get the docs of how to build a shade agent and

75. 30:46

    then get redirected to that repository so that's going to be this near. a shade route