Optimizing NEAR AI with RAG From Theory to Deployment

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   um so yeah hey my name is Josh I am working with near Dev um we do developer relations and a whole bunch of other things for the near ecosystem uh today I'm going to be talking about optimizing your near AI agents uh using something called rag or retrieval augmented Generation Um and you may say why why do I need to uh use this rig and uh it kind of helps a lot of the llm shortcomings uh part of that is preventing hallucinations these AI aren't really giving you the right information all the

2. 0:31

   time um and it's going to help prevent that from happening um it's also going to show solve um short context Windows there's only so many tokens of chunks of data that you can give an llm or a model during a prompt and um you may have a larger amount of data that you want it to parse through to feed it um to this model um but what I'm really uh keen on is it's a key component for user privacy um and that's kind of near theme is you know user owned internet user owned Ai

3. 1:03

   and all those um core values that we try to really move towards um it's a great article by uh pine cone they're um a uh Vector store solution uh but it talks about yeah why rag must be a part of your data privacy plan and um some cool you know tidbits out of there is it really enables control of your data um you control the data that's provided to the llm at Generation Um and the data does not need to be stored in these models you can have it separate you don't have to feed it and train it and

4. 1:34

   and trust that all these things are going to you know hold your data securely um you know and it's shared as a per request basis you don't have to again have it on there all the time and then it allows for quick removal from the system so like earlier we're were talking about tees and trusted execution environments you know the ability to give it some information run something on it prove that it wasn't exposed at all in any way and then shut it down and

5. 1:57

   wipe it out it's yeah just to totally awesome uh and puts me at ease using some of these uh AI systems um so how it works is kind of like an overview um you basically have you know you as a user or an AI agent even could submit a query um you have this retriever um system that's in place there's a whole bunch of different types of ways that you can do this but it's kind of a simplification of it you have a retriever and a vector store that

6. 2:23

   stores all this data it finds the most relevant content and then it brings it back up to the retriever gives the query uh plus the original query plus the context this data that it found to the llm that then will give you back the information to the N user so let's do a demo um recorded a demo we will show how to quickly make an AI agent on near AI uh simply run near AI agent create it'll walk you through this um prompted tutorial you just enter

7. 2:54

   an agent name you enter um the U description of your agent and then the initial prom so however you want to have it initially set up and later we can um modify that information later it says hey is this is what you want to do proceed and then it gives you a bunch of um uh options it shows you where your code was stored and then some useful commands about how to run it locally up upload it to the agent Hub or run this

8. 3:17

   agent from your local registry so what we're going to do is we're going to copy this little nugget and we are going to copy pasty uh copy pasta in our terminal and then we're going to ask it a question um so what types of access keys does near protocol have which is a really cool feature about near um and it gives us this answer which is kind of interesting um full access Keys yep function access keys but this thing uh

9. 3:40

   it's not a thing view function access Keys it hallucinated it doesn't that's not a thing we don't have to sign for transactions view calls are free um so it said oh we have these Keys you're like all right that's kind of interesting um let's ask it another question how many keys can an account actually have um and it thinks about it and it also hallucinates again it gives you 128 which is also not correct at all

10. 4:04

    um it thinks that you only can have 128 Keys it's a lot of keys but it's actually not the answer um so we'll go back to it and say you know are you sure like again what are the the types of keys uh that that you can have on near and it's like oh yeah I forgot there's uh these other things as well there's like a function called permission access key and a method called permission ACC key um which again do not exist at all

11. 4:27

    uh it's just making it up and I think it's like deriving it from this permission access key thing that it thinks uh it's about um and then I start messing with it I'm like are you sure those are all the keys are you sure there's not more types and it's like oh yeah I forgot there's actually all of these types you have these storage access keys for reading storage and Standalone and Method call and yeah so it just keeps going and going and it's hallucinating it doesn't really

12. 4:52

    understand and to its credit it's trying to figure it out it says it apologizes for its responses I tell it's it's okay AI you're fine um but yeah so it's trying to figure out what exactly it you know the information is based on what it knows from this open source model um so what we can do is we'll exit out of there and we're going to run into um we're open up an IDE and we're going to check out the code for this agent um and

13. 5:17

    all agents are real simple um I just opened up my registry I have my different accounts in there I'll go under my near aid. near account and I'll open up this agent that I just created it's two files you have your agent python file metadata. Json file and this is basically the how you configure your agent um you can see it's the Llama instruct is our model and then Max tokens it's kind of important talk about

14. 5:41

    that later um but yeah this is basic the code for your agent super super simple and it just did our initial prompt so what I'm going to do is I'm going to go to docs. near. I'm going to check out a new tutorial that we just pushed out was this documentation rag um and in there um it'll get take a step byep for all this stuff I'm going through it quite quick because I have limited time um but yeah what we need to do is we need to

15. 6:05

    give context to this model we need to tell it hey this is information so this is um two documents from official doc.org about the protocol um and we're just going to give you some commands so you can um download that easily to your local um your local agent and give it some context um so we're going to grab these markdown files uh I'm just going to change directory real quick um go

16. 6:28

    into near AI uh dev. near my username and my agent and the version number just going to download these two files real quick and you can see I have now these markdown files that are both access keys and CLI uh commands the CLI is also part of the tutorial that we're not going through with this um uh update or this demonstration but yeah the other thing we're going to do is we're going to

17. 6:50

    update our agent with our code is to load those files you can see in the very top um um here's two files take a look at those and what we're going to do is use those files to um uh with our query from the user um and then use that to to um answer the question um and often times I forget what agent I named it so if you just run a agent in uh neari agent interactive gives you a list of all the ones that you've created

18. 7:21

    I'm going to create number three and then I'm ask the question again how many access keys are there there in near um take a second to uh respond and it did it correctly this time now it's got actual data to do it there's two types uh there's full access and function call it's cool that like there are a lot of uses that you can do with function call you can like really scope them down to a whole bunch of different things and I think that's what the model was trying to figure out it's like actually with this function call you can create

19. 7:46

    different types of keys but ultimately there's only two um so yeah we'll um yeah we um went over yeah basic full and function um we'll say hey how many keys can uh and count have um and I believe it'll it'll it'll spit out um you can have multiple but there are two main types so it's still not hallucinating on us it's still telling us yes these are the right ones it's just these two um and then like okay well what's the limit

20. 8:13

    for the number of keys that I can actually add to an account and see if it'll throw me something off like the other one did and um yeah it tells us yeah there's actually no limit theoretically There's No Limit um you might run into I don't know like try to do like a million keys on an account I don't know I haven't really tested it but theoretically you could do almost near infinite amount of accounts so yeah um basically that's kind of a simple way

21. 8:35

    of how rag works but if I go back to My Demo slide we skipped one part and that's this Vector store basically what we did is we just took these markdown files and gave it to the llm and gave it with the prompt um that doesn't scale there's a big problem with that and you can only have X amount of data that you can feed the the model um when you're asking it a question so what we have to do is use something called a vector store the vector store what it does is

22. 9:00

    it takes your raw documents it like chunks them up it embeds them in these Vector points uh and they all become these numbers and then we store them in a vector database so the chunking is basically just going through your document it's grabbing this these sections and doing a little bit of overlap so that way um there's a little bit of context with it and you can adjust all these settings um within your uh within your program um and then it plots it out and it's like all right we

23. 9:24

    have all these array points here are all the D here all the markdown files or whatever files that you're pointing out and then it's like all right here's the question how can I create an account and how you know what's closest to that answer and then it grabs about a few of them and then it returns it back to that individual um so yeah basically let's go through real quick on how that part

24. 9:47

    Works um basically all we're going to do is go back to our tutorial we're going to go to Vector Stores um scroll down a little bit again goes through all this a little bit more in depth if you want to learn more um and we're going to grab the entire docs data set from n.org so grab the entire directory now um which we couldn't have done before because it would have ran out of um uh um it would have received a Max token limit and would have errored out don't need agent thatp or I'm sorry

25. 10:17

    we don't need our two docks um anymore because now what we're going to do is upload it into a vector file also we have the vector file code for you so we'll just create a new file vector. piy add these um add this little boiler coat um code and yeah basically what we're doing here is we're just running this um Vector store it'll start grabbing each one of these files that are in this data set it's about like a 100 or so maybe

26. 10:43

    more um and it's chunking and automatically embedding them for you you can do the C you can also customize this um however you'd like we can show we'll show you that in the tutorial if you go to doc.ai um but yeah basically what it's using is like an open AI compatible client it's not actually open AI it's just the client that we use um it's actually uploading it to the near AI platform um and yeah it's just looking through these entire um markdown files

27. 11:09

    and it's grabbing all the information that we have there um and then in each file taking in uploading it and then we're storing it in in these chunks and then now we're going to have this ID and that's really key is that now you have this Vector store ID that you can now use with your query when you're trying to get your information from so we'll grab that little ID we're going to go to our rid agent um and yeah we just shipped this uh earlier today we're actually embedding our agents now in

28. 11:35

    documentation and this is an example of a live implementation of it want to get some ducks out on how to you can embed your own agent in there uh as well um but yeah we'll just go ahead and uh update our agent. P file to now use this Vector store and we're going to copy past the this snippet code in here we're going to go through and um just update the vector store ID so I'll grab that ID

29. 11:59

    that gave us earlier and then we'll just copy and paste it over our our own Vector store ID um but this part is pretty easy it's it's just like you know here's your query and then you have your vector results you just go and you know you query the the the vector store and then you can see here we have six so like give me six chunks that are as close as possible to the answer now we'll take that send it over to the

30. 12:24

    model and it will uh answer the question hopefully um in in the best way possible um yep and it just returns an answer so it's all pretty simple you know python script it's not really too crazy I think anyone honestly could probably figure it out even if you're not a developer I'd be happy to talk to anyone that's not a Dev that wants to try this out after um after but yeah um we're going to go

31. 12:48

    ahead and now run our agent um and now we can ask it more stuff we can say hey what languages dis near support for smart contracts and now it knows more information um it knows that hey there's actually rust and JavaScript that are to main smart contract sdks we're actually working on a python one right now um it compiles it to web assembly knows some more information on that and it's actually giv us some boiler PL code um to get started building smart contracts on our platform and then it gives um

32. 13:13

    some links to the SDK documentation um below so yeah that's kind of like a speedrun obviously through how to use rig how to implement it how to you know build these agents and um yeah as you can see see we used that Vector store we got the content and we following uh this diagram and I guess you know I have just a couple minutes left and what I would say is you know near in a sense we're kind of moving from one you know people like oh you know what are you guys working on what are you focusing on at

33. 13:45

    the end of the day we're just trying to build a better internet and build one that is user own and basically the internet Belongs to You belongs to the people that actually use the internet um not to a couple of Corporations um you should be in control control of your data you should be in control of your assets you should be in control of your identity no one should own that but you uh and you should be in control of your choices um it's one thing that you know I think the internet was awesome and

34. 14:11

    kind of veered off course and I feel we really need to make a course Direction back into this realm um so yeah and that's kind of why I'm here and continue to contribute to this platform so anyway Let's Build a Better web together come join come talk to us try it out for yourself scan this takes you to doc.ai come talk to me after this talk and um yeah that's pretty much it uh thank you

35. 14:36

    very much