How to Create and Host your AI Agent in a Trusted Execution Environment TEE

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   hi everyone thank you so much for coming today um so I'm Pier I work with neari as a software engineer and today I want to quickly uh show you how to build stuff on tees especially agents so I'm curious how many of you are uh Builders and have ever worked okay and uh how many of you have actually used some te stuff before okay so uh yeah first of all I'm going just going to quickly go through where we come from and where we are going and then I'm going to do some demoing and

2. 0:33

   more like life coding um the more the funnier part so um AI systems right now are kind of fing us so all the answers that you get back from these uh centralized paries are are you don't really know how it's generated it's uh they have a lot of ads embedded into them so they have a lot of biases and then whatever you put into their centralized companies you don't actually know what they're doing with your data so it's like complete boxes and all this data that you share with them is a big security risk because like all this data

3. 1:05

   is like very important and it contains a lot of like super sensitive data and it's just waiting to be to be leaked because hackers are working days and night to try to get hands on this data and sell it to the highest bider but um yeah so and another thing is like op has been failing so one of the examples is stability AI they were unable to pay for their GPU uh bill this last year and uh

4. 1:30

   it's because you know everything AI costs is like super expensive and you have to find a good business model that works and all these problems with the centralized parties created like huge trust issues and um so a lot of countries restricted access to this centralized uh to these AI systems and as we know now like all these AI systems are like uh very good for GDP growth and uh so restricting individuals or companies to access this uh um the

5. 2:00

   Services is like very bad and um yeah so I just want to to say why it matters for all of us so right now we are sharing stuff with this um with these companies and like centralized parties that are very sensitive to us it could be like I don't know some therapy session that you have with your AI agent or stuff like this and you don't want these conversations to end up uh with people that you don't actually know so you put

6. 2:24

   a lot of trust on them and it's it's not very sustainable for the long term and um so to fix this we are uh going towards a user owned AI approach and so basically the idea is that you give access to any one uh to this a platform and you can actually verify and audit everything that's happening there and so then you don't have to put uh Trust on these centralized parties you can actually like do the research on your on your on your own or even the community

7. 2:50

   can actually check what's happening and um so yeah all this uh gives you access to your uh you get back the control of your over your data and if you decide to let let's say um share some data you you get rewarded if you share with some some parties in the ecosystem you get rewarded for sharing this data and then like other things that you can do as a community you can like build a model and

8. 3:13

   then whenever somebody is using this model you get rewarded for it it's not like just you just lose your data and uh um and so yeah if if you lose your data right now that to up an AI and stuff like this you don't get anything out of it but so we are really going towards like a an AI is optimized for optimized for user well-being and economic success for everyone I'm just going to Qui quickly go through the the stack that we

9. 3:36

   have so uh first of all we have the agent Cloud so basically the the place where you come and host any of your AI agents so it could be neari agents or it could be like El Eliza agents L linkchain agents whatever and all these agents uh then like get access to a whole Marketplace you can actually interact with different agents and um so whenever your agent is being used by other people in the ecosystem you get uh

10. 4:01

    Revenue out of it because you can say okay my agent requires you to pay one usdc for the service I'm offering and then so that's how you get you generate Revenue um to create these agents we are offering this near agent SDK which basically the IDE is that you can write in python or typescript any kind of agents and uh it could be like very complex agent to very simple one we're going to build a simple one afterwards

11. 4:25

    um but yeah so to on top of this we use the near protocol chain to do all these microtransactions so whenever you know you transact between two agents that requires payment and stuff like this this is all happening on here we have a bunch of payment channels and stuff like this and on top of it we also offer chain abstraction which is the idea that you can transact on any blockchain um and so then like you can do stuff on

12. 4:48

    Sana on bitcoin from your AI agents and so yeah so on one side we have uh agenting payment so it's what I what I mentioned uh agent Discovery so here we have like already 400s 400 like agents so we have to have a discovery layer because uh the assistant so basically is the idea that you have you speak to one particular agent and then so this a this assistant is not able to do everything so you need to discover agents that are experts in specific fields and then it's going to ask this

13. 5:20

    these sub agents please do this for me I'm attaching one usdc for for the service so this is the idea and then you have youro data and shed memory so this is where uh you want to really have a personalized assistant that knows you and you can actually like trust it because you verify the whole stack and you know it's secure and we have agent hosting so it's basically where you send your agent and so all this runs inside a

14. 5:44

    trust execution environment um so trusted execution environments is basically a a piece of Hardware that allows you to like you don't have to trust the hosting provider that they're going to that they are not going to access your your data like there there is no way for them to actually access it and you can verify that they're uh that they're running the appropriate software that you expect expect them to be running so we have this for CPU base so

15. 6:09

    it's typically like a agents in our case and also for gpus so we use gpus so GPU is for any kind of model training model fine tuning and model inference and uh so all these we we run them on called So-Cal CVM so confidential virtual machines so it's just it's just an abstraction on top of the just a way for us to run to run anything inside these tees so for our use case what does this te magic enables us to do so one very cool thing is the fully

16. 6:40

    encrypted models so basically you can train your model fully inside a te so you almost have no overhead so our friends at F did did a study and you we like a research and we see that the performance impact when you use these TS is from 8% to down to 1% for these bigger models because they spend more time computing so the downside of encrypting everything is is not that that that big um and then you have a monetized model um sorry monetized

17. 7:09

    confidential inference so basically if you offer access to your Hardware to do any kind of uh computation and um computation for completions on this from these LMS you get rewarded as a service provider and then you have verifiable agents so you can as a user you can check that the agents being running in this te is actually what you expect it to be the version that you expect to be running and so you can really fully

18. 7:33

    verify everything so you don't have to trust anymore you can actually verify so the demo I'm going to go through is uh first I'm going to demonstrate a bit of a private inference so it's just the idea that you can uh discuss with one of these llms in a fully uh private way so nobody can see what really happened um so for this we use a VM and uh so we have um we are working with Sal Network on this and um

19. 7:58

    we are working on this private LK uh which is what what I'm going to be demoing today and on the other side we're going to be doing private a private engine so we're going to create an agent hosted inside te and then do some very simple competions with it just to show you how how it works so now comes the the fun part so I hope it's going to work let's see um so first of all the private inference uh side of things for

20. 8:27

    example here on inside the T we have uh Lama 3.1 I hope you can read uh we have Lama 3.1 8B running and uh we are just going to ask for simple completion does he lik near protocol and um yeah so we have so this specific te is not running with https so we don't have uh the whole verification setup it's just um so this is a developer preview the other the other things that we have in T is are actually fully uh with TLS and https so you can actually nobody has access to what you are really

21. 9:06

    saying to thiss but in that case since it's only HTTP anybody can actually spoof on the network and see uh what the question and what the answer back from the from the LM is but it's this is like minor issue it's going to be fixed anytime soon so I'm just going to ask for stud this completion so here it is uh okay so uh we saw that we send the the request to this CVM on the port 8,000 for chat completions and then it answered that it loves near protocol um

22. 9:48

    so yeah and then you can actually get the quotes So the quotes in this cvms is basically what uh because you have to to have a way to check that it's running the software that we expect it to be running and so so how to do this you basically get a quote out of it and so for This Server this one so we expose uh attestation SL report uh endpoint and when you call this basically it's going to uh get some measurements from both uh the CPU and

23. 10:19

    the GPU running inside the te and so here for example you have the Intel quote which is a bunch of random stuff it's like X and then we have a verification process that actually text this and um checks that it matches the expected values so for for example for the Intel quotes uh we're going to go a little bit deeper afterwards but you can actually decompose all this and check that the

24. 10:42

    version that is being run is the appropriate one and then we have the exact same for the Nvidia side of things because you have to actually check both um to make sure it's secure so this is it for the private inference part of things so if you want to give it a try already you can uh we only have a 3.18 running right now on the machine uh for storage reason I wasn't able to run a bigger one um but it's totally capable

25. 11:06

    of running this way bigger models so now we going to create like a neari agent so to do this we have this very convenient CLI so you do neari agent create it's going to ask for for a name of the agent so you e Denver emo for example a description uh instructions so it's just going to be like a this is perfect helpful ous assistant oh my God uh so now we are just creating it and we can actually go and check how it

26. 11:55

    looks like to once you created an agent so you can go and update it and change and do very way more advanced agents so if I have time I will show you like more advanced agents and how they actually work and can call like vector stores and stuff like this but here we can see the the instructions we told the the agent to be doing and so it looks correct I'm not I'm not going to do something more

27. 12:20

    advanced for now but then we just going to upload it so we're going to upload this version so this is uh being sent to the to the CDM again so to this te um so oh yeah I have this it's 7 megabytes it's not the correct size just a sec it's because of my Pi okay so I'm uploading it again perfect so now it's uploaded so here we can see it's sent again to the CVM ne.

28. 12:53

    so here we are using TLS that's why you see all these warnings because the certificate is generated inside the te and it's not like signed by a counterparty so usually you should not be able you should not trust these certificates but here in our case uh since we do um some process I hope I will have time to show you but basically when uh you will get the quote from the from the te we put the public key of the SSR certificate inside the some hash and

29. 13:19

    then on the client side you recalculate this hash and if it matches you know that the public key that you are discussing with on the https side of things you know it's the like it's actually secure and you're speaking to the correct server so now we have this uh agent upload in inside our te and now we're going to want to discuss with it so just one very simple uh how are you very simple completion um so this is this goes to the this API running inside the CVM and then this is going to ask

30. 13:57

    the VM instance that we saw before is also a CVM inside a te so all of them so they don't trust anyone any each one of them so all of them are doing like uh these te attestations every time and uh yeah so that's that's how you they ensure that it's actually everything is secure and private and only you can can see the result so here we got back the the answer we saw that we saw the like he's taking into account is Denver it was in instructions so it's yeah so it's

31. 14:28

    working um so now I'm just going to show you more how it works so because here we are using the the CLI so everything is hidden um but so here we get the the quote from the from the CVM which is the The Hub we call it so it's the basically the big API that we use to to do all these AI agents interactions and um agent hting and everything so we get to a code we do the testation checks that it matches the Cor correct version of the software um so in our case for

32. 15:02

    example I noted before but for our version of The Hub you expect to get uh from the I need to run it before so if we run this python script so here we have way more logs uh so here we see the quote again that is huge and here we have the verification results so this is the from the attestation so it's actually checking that the the t is is is correct so we you should see uh the correct Ash for the

33. 15:36

    for this version of the The Hub that you audited before that you checked it's running it's behaving the way that you be that you expect it to behave and so you know that this ash is uh is correct or is it okay so here we can see uh it got the hashack so you know it's the diversion of uh The Hub that you actually checked before and you know it's um it's genuine and not like selling your data to anyone else or stuff like this um and then so we have a fully the API is fully

34. 16:09

    compatible with open AI so you can create a thread then you add you add a message to the thread and then basically you you run an agent on the given thread so in in this example I'm running this what bits Rock so it's just a game it's like a rock paper scissors but more advanced with LMS so basically you are fighting with the llm to know uh what can first is to be the Rock and then so

35. 16:32

    in this example for example I puted lava so here I say lava and then the LM is going to answer back I hope it's readable uh you want lava melts Rock and then it's going to ask you like what beats lava so you have to find the next um material that's going to beat lava and so then you can just go on with the thread and just play with this game it's a little useless game just for fun so we start the run and then we just wait for the completion so to actually we are

36. 17:02

    checking that the run is um is finished and once it's finished like um it's cool we just uh print the response back from the agent um so now with the little time I have I'm just going to show you more the kind of agents that we we have uh so this is one I did in Bangkok for redacted uh basically it was the idea of you create an agent with the the vector store contains all of IAS tweets and basically you you ask ilas ilas Vector

37. 17:34

    store you ask him for financial advice not Financial advice and you hope that it gives you the next big mem coin to to buy and um so here I run it before um so what community coin because what if you ask for mcoin it's never going to tell you anything it's going to say I'm not giving Financial advice but like here for example uh it would speak about black dragon long um all these all these different uh meme coins on here

38. 18:02

    and uh so how we do this so here we just it's out coded just for de moing purposes we out code the vector store ID and then uh on the the agent side you can just do uh on so EnV is just from the S oh sorry better um so the EnV so on is just uh from our SDK so we offered this query Vector store function you put the the vector store ID and then the query from the user and then it's just going to like find back all the most relevant

39. 18:38

    documents so Tweets in that case um tweets from Ilia that are good to that are mostly speaking about mem coins so this is one uh another one so this is what bits Rock from from the de I already showed you it this is Sound Stage so this is the one uh is one of the most advanced agent that we have um so all these agents you can call it from the assistant so for example for sound Sage if you say I want to buy so it's an agent specialized in

40. 19:09

    buying uh headphones so if you if you tell him um I want to buy like I know some grided green headphone to your assistant is going to go to the Discovery layer and find an agent that is capable of doing this for you so in that case for the demo I'm just directly on the on the sound sound stage agent um but so we can go here and ask him uh I want to buy uh cheap headphones for example then we just hope that the completion is going to be fast don't have much time but uh so then how it

41. 19:42

    works I have some I have a diagram so can you see this so we have the chat. uh UI that you see right now and then this discusses with the assistant so in that case I went directly to the sunset agent so between these two uh you have the discovery layer that basically then um communicates between one another and then this agent is communicating to the sound Sage mCP server so it's just an API that is uh available to buy stuff

42. 20:13

    from Amazon so the sound Sage agent uh gets the intent that you want to do so here in that case it's to buy like some some headphon some cheap headphones and then it's going to discuss with this API so it's uh another API because it contains some private API keys and uh you don't really want to share it with anyone and um so then this just discusses with Amazon and then you can do all the all

43. 20:38

    the payments and stuff from there uh so it's still okay so here it found back some uh some headphones from Amazon you can see the price uh all these things so let's say we want to buy this super cheap one I don't have any usdc on this account so we won't be able to actually finish the the whole process um but it's it's quite slow the completion so this is something that uh we need to improve

44. 21:08

    on but yeah so once once you want to buy it then it's going to ask you for the shipping info and then you you fill all this and you you're going to receive your headphones at some point uh in the next few days um and so yeah that's uh everything I have actually I have something else so K from the team she this yesterday it was asked quite a lot so basically it allows you uh to um embed that the chat box anywhere on any website so if you want to have like a a

45. 21:39

    near like just an assistant embedded into your web page you can um just like use this iframe for example like this and then you specify the agent that you want to be discussing with and um so it could be an agent that's linked to your like a vector store of all your documentation about your project or stuff like this and then this this agent is going to be able to answer like any kind of questions for your project and

46. 22:03

    uh so yeah I have so this is one one example here so here you see it's an a frame uh inside some documentation so inside the N documentations and then you can ask whatever question it has a as Vector story it has all the documentation that we have on nii and so you can ask whatever question so that was it thank you so much for listening