{"id":"49lX4MTUSIQ","title":"Private Inference on NEAR AI: TEE vs. Proxied Models Explained","description":"NEAR AI runs some models inside TEEs and proxies others to providers like OpenAI, Anthropic and Gemini. Both work with the same API key, but they give you very different privacy guarantees. Proxied requests strip your identity and still leave your content readable by the provider. TEE-hosted models run inside an attested enclave where nobody, NEAR included, can see your prompts or outputs.\n\nThis video covers what that difference means for builders. You’ll see where routing quietly breaks the privacy guarantee, why you should build a model allow list and require an attestation report before trusting any model, and how end-to-end encryption keeps the gateway from reading your requests. I close with a real example from Mockingbird, a health app I’m building, showing how notes, embeddings, reranking and on-device vector storage stay protected at every step. Embeddings get the same protection as the original text, because research shows you can reconstruct the source text from them.\n\nChapters\n0:00 Intro\n0:53 Proxied vs. TEE hosting on NEAR AI\n3:19 Routing: where the guarantee breaks\n4:33 Build an allow list and require attestation\n6:04 Can the gateway read your data?\n8:33 What actually leaves the phone\n10:03 Example: memory in a health app\n13:15 Why embeddings leak\n14:51 Recap: where each piece runs\n\nGet started at : near.ai","channel_name":"NEAR DevHub","published_at":"2026-10-23T00:00:00.000Z","thumbnail_url":"https://img.youtube.com/vi/49lX4MTUSIQ/hqdefault.jpg","transcript_text":"Hey folks, if you've been following along with our builder series, we've done a lot of different videos around private inference and what it means to build on top of it. But I think it's time to go a little bit backwards and really go over what it means to actually use private inference and what it actually means and how it all works. More specifically, because we're using near AI for all of this, I want to talk about what the difference is when you're using different models on near AI. This is an important concept I think for builders and just users to understand the difference between what is something that is truly private, something that is anonymous and so forth. So let's go through this and let me kind of paint the whole picture for you about this because I think it will really inform how you actually make decisions when you're building and it will loop back into some of the videos I've done around like logic and things like that for how you route different information and how you keep different things private. So let me start with this concept. So Nearai provides both proxied and TE uh hosting. And what that means is that you're going to have two different types of setup for different models. And depending on what model it is is going to can show you like what is actually happening with that data and what level of privacy exists with it. The nice thing is that is when you build with something like with near AAI, it is one API key. You don't have to have things separate across the different models or across the different types. It's all still one setup. So, as I mentioned, there's two different models. And I did a whole video about how it's easy to replace the base URL if you're using OpenAI uh specs that you know you can just change that out to be near. It follows the same API pattern. So, all that is the same. You don't have to change your code depending on the model necessarily. What you do need to pay attention to is difference between what's TEH hosted and what's proxied. So right now on Nearai things like OpenAI, Anthropic, Gemini, other things like that are all proxied. And what that means is that your data is anonymous but it is still readable by those providers. The T guarantees are not there. There's no private enclave. So just as you are chatting with anything else, it goes through. What it does though strip is some of that identifying information where there's no username attached to it. Uh it does there's no like IP address, there's no anything like that that's being passed through. You are anonymized except of what you actually put in there. So if you put private information in that request, it is now no longer truly private and you still could put pieces together about all that type of stuff. T hosted is one that is actually private because it's running inside a private enclave and so no one can see except that process of what was actually requested. The providers don't see that there's addistation around that so that you can see that nobody including near will see your prompts or your outputs. two very different outcomes and this is going to be tied to two different very kinds of models because again one is going to be open source that's on TE hosted and the ones that are considered more of the frontier models that are coming out of these larger labs are likely to be proxied if you have an app and I'll use the example of one of the apps I'm working on called mockingb bird and I have my near AI gateway so I might have stuff that is run in a tested TE maybe that's handling the routing things like that and then that routing may put things into different places and I might stay in a TE hosted or I might go to an upstream provider as soon as I get out of a TE in any form. So even if you're simple as like triage is not in a TE you have then lost that guarantee. So if I triage with T that's great because okay now I'm protected there but as soon as I then moved out of that for whatever reason maybe I needed just a different type of model I now have lost that guarantee this is even in your AI's model docs uh the TE privacy and verifi verifiability guarantees do not extend to the upstream provider meaning again once you leave out of that TE and you make a request to another provider even if it's within in the near system, you have lost that guarantee because you are now having to send data that is no longer in an enclave. One of the things that I recommend doing if you're building something that really needs to stay private and you but you do want to be able to leverage routing and use different models is actually build an allow list and actually make sure you know that which ones are going to be attested, which ones are not and make sure that anytime that things are routed outside of that that they're following the pattern that you expect them to. If you expect total TE protection through those enclaves, then you need to make sure that you're only using models that do that. So build that allow list. Again, if you're trying to show improve privacy, you should always have an addestation in some place around that as well. The catalog flag, you know, so if you pull that model list and it says, \"Yeah, this isn't a tested.\" Obviously, there can be errors where somebody makes a mistake and they actually mark something as attested and it's not. uh things can actually again can go can go wrong and so you know you just want to make sure that when you say something is private in that enclave that you actually have the addestation as well. One of the things that you can add again as a an additional layer of protection and I would recommend is that if a model cannot prove where it's been run you cannot get that addestation report then you can't use it. So the addestation report needs to come first and then you can say that again understand the difference there. Otherwise if anything's proxied. Um it's the same as again if you're comfortable sending stuff to just one of the providers anyways here though we're just stripping some information out of it. But again depending on what you send them it still could be identifiable from there. So hopefully naturally the next question for you becomes well what about the gateways right? So Near AI has a gateway that is still connecting to all of this. can't that read it. So, Near AI cloud does say that there's end toend encryption that allows basically no one to see what is happening here. You are fully protected and that's an important aspect. You need do need to be careful that you know someone who is offering private inference through addistation is great but can that gateway do the logs still see stuff? Typically encryption ends at the gateway. So using TLS over the internet, it would end at the gateway because the gateway would then have to decrypt it and then have it in plain text and hand it off to the model trans the inference. So now you have an exposure lane where now that for a brief period of time uh it is in plain text and that gateway technically could then see and do whatever with it. And that's usually because if someone sets up a gateway like this uh the encryption is happening you know at purely the gateway. What you can have though and what you have here at Near AI is your phone actually is encrypting to the model's public key and then you can pass it over to the gateway. The gateway still has it in cipher text and it's routing from there in the uh encrypted state and then from there the that cipher text is then decrypted by the model. it's run and then the encry encrypts it back replying to your key and then your phone can then decipher that from there. And what's nice about all of this process here is that now there is no exposure lane going in either direction. Uh the way the model key comes through is through a verified report. This is something that's built into the API of all of that. And so you can go through and you can get that and then you can verify every candidate against it. And then you have your now public key to do that. The catch here though is the key is only as good as your addestation check. So if you're not doing the addestation check, then your key is no good and you can't consider it valid. So you need to be doing that part as well. You can't just skip over that. If you're going to say that I'm using a TE and I'm going to do things in a protective enclave, I need to have the addestation check as well. Uh for those who are more curious about what kind of cryptography is in here, here you go. Uh the easiest way to do this though is let's just let the SDK handle it. the SDK from near AI uh manages all of this and so you can just use that and then follow those directions and then go from there. What actually leaves the phone when you make a request and it's really straightforward. There's a model ID, there's maybe some headers that go along with it. Uh then there is the cipher text which is the message content which is what the includes the prompt anything else that you're proposing to the model. This is in hex only the model CE that can then open it because again we have now encrypted it to go all the way through. It's important to understand what end encryption actually leaves visible. Uh metadata is still there. They can still see your actual API gateway can see your API key, the model ID and the request size and timing. Uh there is obviously tool definitions that might come through. There's response API. Uh but it's also important to know that there is no plain text fallback. So if a model doesn't have an end then path um then the request actually just fails instead of going out unencrypted. So that's a really important part of using this SDK as well. The important lesson here is that you need to always be verifying the key and then encrypting your stuff to it. So the addestation is part of this. So this is new to people who maybe are used to just developing um in a more traditional website kind of backend setup where maybe you're just encrypting for that server and going back and forth. you actually have to still do this add a stationation as part of that to make sure that what is being what I'm interacting with is what it says it is and that there is no change to that. So with all this in mind about privacy and how this is all being managed using private inference, let me use a practical example that I've personally used inside a health app that I've been working on that shows you kind of the benefits and how it's important to use a TE the right way and that you're not just leaving things up to chance. Let's apply this to using a note and have that go through what it means to go through model. So every time you save a note uh inside the app means you're saving some piece of information to use later. Uh it does make an embedding call to to rank it and to organize it. When you ask a question, we'll also make an embedding call and then rerank it. So how do we make sure that all that stuff is protected all the way through and where does that data stay? And this is a really important part of how you make architectural decisions in using AI inference or any kind of other data. So you see here that there's two different paths through memory. Uh to save a note, a note comes up. The TE will then do the embedding and then you will store it on the phone as a vector. If you ask a question, it will do the same thing. The information will come in and then you now have some new information that is associated with whatever you asked. The TE can then rerank it and then now the model can now answer from there. whole point here is that it's either the phone or the TE. So nothing memory related and how any of that's stored or used or whatever is not in a protected area. The phone itself you assume is protected by the user in how that's being done and then using it to use the other side of it. If you were then to include some other system to do the ranking like I have seen examples where people maybe use a like flash model from one of the major providers like let's say like a Gemini flash model to do some of this ranking you know or to do some triaging or whatever else that kind of eliminates the point of using a private inference because now you're exposing this in in other ways. So you need to be really conscious about like each step whether it's asking the actual question whether it's a memory process whether it is triaging whatever it might be how is that data being handled in my example here I prefer to keep the vectors on the phone because again I feel the phone in this scenario is the most protected place it is the the end it is the user's own device that they are in charge of there's no outside you know input or output put from that. So for these type of things, let's keep them there. Depending on what the information is and what is being stored and how it's being stored, uh everything needs to have a relevant score, right? Right? And so as we start to build out this memory and we start to build out like what is going to be relevant to use later on, what is the you know app need to remember basically uh that needs to be reranked and so again that reranking needs to be done in a protected way because we are passing memories that are protected that that should that are supposed to be private to that user. So we want to leverage the model to maximize that, but we need to make sure that we're actually again protecting it all the way through. From a pure memory standpoint, the reason why this all matters is because 92% of the 32 token inputs that were rebuilt exactly from the embeddings we saw in a previous study. Um it was really important that like everything matched pretty closely. So you want to make sure that when you are using the embeddings that that these vectors are set up and that you're leveraging it. But again those vectors have important information. So how you are then storing it is important. So the text embeddings basically are the same as like having the actual text input because it is basically storing these memories. So the same method that pulled full names out of a database of clinical notes then it's same idea here. um you're storing vectors like notes that they came from. So like you this is I again this is I think what's a little bit new for de some developers of like you have these different layers of information that are not just straight input output right it is data being processed at different times is processed to be used maybe later all these different things and you need to be very thoughtful about if I'm trying to protect particular data am I truly doing it in all the different stages and where are potential leak leaks here. If I was managing my vectors for my memory at a different in a different manner, I have now exposed potential potent personal information because I was not being proactive. So that's just kind of like this is a good example for a builder, especially if you're new to building with private inference, the things that you should really be aware of and pay attention to. Here's a general overview of just kind of what I was mentioning here. If you're betting a note or a question for the memory, so if you're running it on an AIT, you have that protection of the tested gateway uh end to end all the way through. If you have a vector and keyword index, so for for that memory, keep it on the device, keep it on the phone. Uh that protection now is encryption at rest and you have that protection also too of it being on that user's particular device. It is not stored in the cloud. It is not is only being used when it needs to be used from there. reranking again that needs to go through the TE because again you have protected information there uh in the example that I just showed of that previous test from 2000 uh 2023 is that you have now a set situation where private information can be retrieved out of these uh vectors and so we want to make sure that we are protecting that all the way through and just as a reminder without the end to end encryption the gateway can handle your note in plain text and even if it's inside its and tested TE. So there's just things to just be aware of all the way through of just again the architecture is super important through all this. Moral of the story of all of this example here is that when you're using embeddings, you need to they need to be in a TE because if you then have that and you create these vectors for that memory, those vectors need to be guarded like the original pieces of information that came from. So if I am inputting again the example here I'm using is like health like clinical notes and then we're pulling information out of there and then we want to make sure that we're storing that information then in for the user and we have these vectors we need to make sure that it's being stored and protected just like anything else because that exposes private information. So we have N for our embed embedding and reranking and then keeping things on the phone which is basically that index right those vectors we want to encrypt that at rest. So now we have a full suite here of information being processed handled reprocessed in a protected manner where nothing is exposed nothing can be leaked our data is private all the way through. Building with AI is a new paradigm for a lot of people and we're going to keep investigating some of these different setups and hopefully this helps you work on your own stuff too. It's watching.","transcript_synced_at":"2026-11-03T03:45:59.667Z","segments":[{"start_seconds":0,"text":"Hey folks, if you've been following along with our builder series, we've done a lot of different videos around private inference and what it means to build on top of it. But I think it's","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=0s"},{"start_seconds":6,"text":"time to go a little bit backwards and really go over what it means to actually use private inference and what it actually means and how it all works.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=6s"},{"start_seconds":14,"text":"More specifically, because we're using near AI for all of this, I want to talk about what the difference is when you're using different models on near AI. This","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=14s"},{"start_seconds":22,"text":"is an important concept I think for builders and just users to understand the difference between what is something that is truly private, something that is","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=22s"},{"start_seconds":30,"text":"anonymous and so forth. So let's go through this and let me kind of paint the whole picture for you about this because I think it will really inform","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=30s"},{"start_seconds":39,"text":"how you actually make decisions when you're building and it will loop back into some of the videos I've done around like logic and things like that for how","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=39s"},{"start_seconds":47,"text":"you route different information and how you keep different things private. So let me start with this concept. So","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=47s"},{"start_seconds":53,"text":"Nearai provides both proxied and TE uh hosting. And what that means is that you're going to have two different types","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=53s"},{"start_seconds":62,"text":"of setup for different models. And depending on what model it is is going to can show you like what is actually","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=62s"},{"start_seconds":70,"text":"happening with that data and what level of privacy exists with it. The nice thing is that is when you build with something like with near AAI, it is one","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=70s"},{"start_seconds":79,"text":"API key. You don't have to have things separate across the different models or across the different types. It's all still one setup. So, as I mentioned,","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=79s"},{"start_seconds":88,"text":"there's two different models. And I did a whole video about how it's easy to replace the base URL if you're using OpenAI uh specs that you know you can","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=88s"},{"start_seconds":95,"text":"just change that out to be near. It follows the same API pattern. So, all that is the same. You don't have to change your code depending on the model","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=95s"},{"start_seconds":104,"text":"necessarily. What you do need to pay attention to is difference between what's TEH hosted and what's proxied. So","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=104s"},{"start_seconds":111,"text":"right now on Nearai things like OpenAI, Anthropic, Gemini, other things like that are all proxied. And what that","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=111s"},{"start_seconds":118,"text":"means is that your data is anonymous but it is still readable by those providers.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=118s"},{"start_seconds":125,"text":"The T guarantees are not there. There's no private enclave. So just as you are chatting with anything else, it goes","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=125s"},{"start_seconds":133,"text":"through. What it does though strip is some of that identifying information where there's no username attached to it. Uh it does there's no like IP","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=133s"},{"start_seconds":140,"text":"address, there's no anything like that that's being passed through. You are anonymized except of what you actually put in there. So if you put private","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=140s"},{"start_seconds":148,"text":"information in that request, it is now no longer truly private and you still","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=148s"},{"start_seconds":155,"text":"could put pieces together about all that type of stuff. T hosted is one that is actually private because it's running","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=155s"},{"start_seconds":163,"text":"inside a private enclave and so no one can see except that process of what was","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=163s"},{"start_seconds":170,"text":"actually requested. The providers don't see that there's addistation around that so that you can see that nobody","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=170s"},{"start_seconds":177,"text":"including near will see your prompts or your outputs. two very different outcomes and this is going to be tied to","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=177s"},{"start_seconds":184,"text":"two different very kinds of models because again one is going to be open source that's on TE hosted and the ones","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=184s"},{"start_seconds":191,"text":"that are considered more of the frontier models that are coming out of these larger labs are likely to be proxied if","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=191s"},{"start_seconds":199,"text":"you have an app and I'll use the example of one of the apps I'm working on called mockingb bird and I have my near AI gateway so I might have stuff that is","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=199s"},{"start_seconds":208,"text":"run in a tested TE maybe that's handling the routing things like that and then that routing may put things into","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=208s"},{"start_seconds":217,"text":"different places and I might stay in a TE hosted or I might go to an upstream provider as soon as I get out of a TE in","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=217s"},{"start_seconds":226,"text":"any form. So even if you're simple as like triage is not in a TE you have then","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=226s"},{"start_seconds":233,"text":"lost that guarantee. So if I triage with T that's great because okay now I'm protected there but as soon as I then","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=233s"},{"start_seconds":242,"text":"moved out of that for whatever reason maybe I needed just a different type of model I now have lost that guarantee","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=242s"},{"start_seconds":249,"text":"this is even in your AI's model docs uh the TE privacy and verifi verifiability guarantees do not extend to the upstream","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=249s"},{"start_seconds":257,"text":"provider meaning again once you leave out of that TE and you make a request to another provider even if it's within in","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=257s"},{"start_seconds":264,"text":"the near system, you have lost that guarantee because you are now having to send data that is no longer in an enclave. One of the things that I","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=264s"},{"start_seconds":273,"text":"recommend doing if you're building something that really needs to stay private and you but you do want to be able to leverage routing and use different models is actually build an","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=273s"},{"start_seconds":281,"text":"allow list and actually make sure you know that which ones are going to be attested, which ones are not and make sure that anytime that things are routed","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=281s"},{"start_seconds":289,"text":"outside of that that they're following the pattern that you expect them to. If you expect total TE protection through those enclaves, then you need to make","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=289s"},{"start_seconds":298,"text":"sure that you're only using models that do that. So build that allow list.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=298s"},{"start_seconds":302,"text":"Again, if you're trying to show improve privacy, you should always have an addestation in some place around that as","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=302s"},{"start_seconds":310,"text":"well. The catalog flag, you know, so if you pull that model list and it says, \"Yeah, this isn't a tested.\" Obviously, there can be errors where somebody makes","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=310s"},{"start_seconds":318,"text":"a mistake and they actually mark something as attested and it's not. uh things can actually again can go can go","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=318s"},{"start_seconds":325,"text":"wrong and so you know you just want to make sure that when you say something is private in that enclave that you actually have the addestation as well.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=325s"},{"start_seconds":333,"text":"One of the things that you can add again as a an additional layer of protection and I would recommend is that if a model","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=333s"},{"start_seconds":340,"text":"cannot prove where it's been run you cannot get that addestation report then you can't use it. So the addestation","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=340s"},{"start_seconds":347,"text":"report needs to come first and then you can say that again understand the difference there.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=347s"},{"start_seconds":353,"text":"Otherwise if anything's proxied. Um it's the same as again if you're comfortable sending stuff to just one of the providers anyways here though we're just stripping some information out of it.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=353s"},{"start_seconds":364,"text":"But again depending on what you send them it still could be identifiable from there. So hopefully naturally the next question for you becomes well what about","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=364s"},{"start_seconds":371,"text":"the gateways right? So Near AI has a gateway that is still connecting to all of this. can't that read it. So, Near AI","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=371s"},{"start_seconds":378,"text":"cloud does say that there's end toend encryption that allows basically no one","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=378s"},{"start_seconds":384,"text":"to see what is happening here. You are fully protected and that's an important aspect. You need do need to be careful","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=384s"},{"start_seconds":391,"text":"that you know someone who is offering private inference through addistation is great but can that gateway do the logs","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=391s"},{"start_seconds":400,"text":"still see stuff? Typically encryption ends at the gateway. So using TLS over the internet, it would end at the gateway because the gateway would then","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=400s"},{"start_seconds":408,"text":"have to decrypt it and then have it in plain text and hand it off to the model trans the inference. So now you have an","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=408s"},{"start_seconds":415,"text":"exposure lane where now that for a brief period of time uh it is in plain text and that gateway technically could then","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=415s"},{"start_seconds":422,"text":"see and do whatever with it. And that's usually because if someone sets up a gateway like this uh the encryption is","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=422s"},{"start_seconds":429,"text":"happening you know at purely the gateway. What you can have though and what you have here at Near AI is your","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=429s"},{"start_seconds":437,"text":"phone actually is encrypting to the model's public key and then you can pass it over to the gateway. The gateway still has it in cipher text and it's","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=437s"},{"start_seconds":446,"text":"routing from there in the uh encrypted state and then from there the that cipher text is then decrypted by the","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=446s"},{"start_seconds":455,"text":"model. it's run and then the encry encrypts it back replying to your key and then your phone can then decipher that from there. And what's nice about","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=455s"},{"start_seconds":464,"text":"all of this process here is that now there is no exposure lane going in either direction. Uh the way the model key comes through is through a verified","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=464s"},{"start_seconds":473,"text":"report. This is something that's built into the API of all of that. And so you can go through and you can get that and then you can verify every candidate","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=473s"},{"start_seconds":480,"text":"against it. And then you have your now public key to do that.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=480s"},{"start_seconds":485,"text":"The catch here though is the key is only as good as your addestation check. So if you're not doing the addestation check, then your key is no good and you can't","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=485s"},{"start_seconds":493,"text":"consider it valid. So you need to be doing that part as well. You can't just skip over that. If you're going to say that I'm using a TE and I'm going to do things in a protective enclave, I need to have the addestation check as well.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=493s"},{"start_seconds":504,"text":"Uh for those who are more curious about what kind of cryptography is in here, here you go. Uh the easiest way to do this though is let's just let the SDK handle it. the SDK from near AI uh","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=504s"},{"start_seconds":513,"text":"manages all of this and so you can just use that and then follow those directions and then go from there. What actually leaves the phone when you make","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=513s"},{"start_seconds":520,"text":"a request and it's really straightforward. There's a model ID, there's maybe some headers that go along with it. Uh then there is the cipher","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=520s"},{"start_seconds":529,"text":"text which is the message content which is what the includes the prompt anything else that you're proposing to the model.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=529s"},{"start_seconds":535,"text":"This is in hex only the model CE that can then open it because again we have now encrypted it to go all the way through. It's important to understand what end encryption actually leaves visible. Uh metadata is still there.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=535s"},{"start_seconds":546,"text":"They can still see your actual API gateway can see your API key, the model ID and the request size and timing. Uh there is obviously tool definitions that","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=546s"},{"start_seconds":555,"text":"might come through. There's response API. Uh but it's also important to know that there is no plain text fallback. So if a model doesn't have an end then path","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=555s"},{"start_seconds":564,"text":"um then the request actually just fails instead of going out unencrypted. So that's a really important part of using this SDK as well. The important lesson here is that you need to always be","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=564s"},{"start_seconds":571,"text":"verifying the key and then encrypting your stuff to it. So the addestation is part of this. So this is new to people who maybe are used to just developing um","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=571s"},{"start_seconds":579,"text":"in a more traditional website kind of backend setup where maybe you're just encrypting for that server and going back and forth. you actually have to","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=579s"},{"start_seconds":588,"text":"still do this add a stationation as part of that to make sure that what is being what I'm interacting with is what it","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=588s"},{"start_seconds":595,"text":"says it is and that there is no change to that. So with all this in mind about privacy and how this is all being managed using private inference, let me","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=595s"},{"start_seconds":602,"text":"use a practical example that I've personally used inside a health app that I've been working on that shows you kind of the benefits and how it's important","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=602s"},{"start_seconds":610,"text":"to use a TE the right way and that you're not just leaving things up to chance. Let's apply this to using a note and have that go through what it means","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=610s"},{"start_seconds":618,"text":"to go through model. So every time you save a note uh inside the app means you're saving some piece of information to use later. Uh it does make an embedding call to to rank it and to","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=618s"},{"start_seconds":627,"text":"organize it. When you ask a question, we'll also make an embedding call and then rerank it. So how do we make sure that all that stuff is protected all the","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=627s"},{"start_seconds":635,"text":"way through and where does that data stay? And this is a really important part of how you make architectural decisions in using AI inference or any","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=635s"},{"start_seconds":643,"text":"kind of other data. So you see here that there's two different paths through memory. Uh to save a note, a note comes up. The TE will then do the embedding","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=643s"},{"start_seconds":652,"text":"and then you will store it on the phone as a vector. If you ask a question, it will do the same thing. The information","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=652s"},{"start_seconds":659,"text":"will come in and then you now have some new information that is associated with whatever you asked. The TE can then","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=659s"},{"start_seconds":666,"text":"rerank it and then now the model can now answer from there. whole point here is that it's either the phone or the TE. So","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=666s"},{"start_seconds":675,"text":"nothing memory related and how any of that's stored or used or whatever is not","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=675s"},{"start_seconds":681,"text":"in a protected area. The phone itself you assume is protected by the user in how that's being done and then using it","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=681s"},{"start_seconds":691,"text":"to use the other side of it. If you were then to include some other system to do the ranking like I have seen examples where people maybe use a like flash","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=691s"},{"start_seconds":700,"text":"model from one of the major providers like let's say like a Gemini flash model to do some of this ranking you know or","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=700s"},{"start_seconds":708,"text":"to do some triaging or whatever else that kind of eliminates the point of using a private inference because now you're exposing this in in other ways.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=708s"},{"start_seconds":719,"text":"So you need to be really conscious about like each step whether it's asking the actual question whether it's a memory process whether it is triaging whatever","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=719s"},{"start_seconds":727,"text":"it might be how is that data being handled in my example here I prefer to keep the vectors on the phone because","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=727s"},{"start_seconds":735,"text":"again I feel the phone in this scenario is the most protected place it is the the end it is the user's own device that","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=735s"},{"start_seconds":743,"text":"they are in charge of there's no outside you know input or output put from that.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=743s"},{"start_seconds":749,"text":"So for these type of things, let's keep them there. Depending on what the information is and what is being stored and how it's being stored, uh everything needs to have a relevant score, right?","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=749s"},{"start_seconds":758,"text":"Right? And so as we start to build out this memory and we start to build out like what is going to be relevant to use later on, what is the you know app need","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=758s"},{"start_seconds":767,"text":"to remember basically uh that needs to be reranked and so again that reranking needs to be done in a protected way","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=767s"},{"start_seconds":775,"text":"because we are passing memories that are protected that that should that are supposed to be private to that user. So","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=775s"},{"start_seconds":784,"text":"we want to leverage the model to maximize that, but we need to make sure that we're actually again protecting it all the way through. From a pure memory","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=784s"},{"start_seconds":791,"text":"standpoint, the reason why this all matters is because 92% of the 32 token inputs that were rebuilt exactly from","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=791s"},{"start_seconds":799,"text":"the embeddings we saw in a previous study. Um it was really important that like everything matched pretty closely.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=799s"},{"start_seconds":806,"text":"So you want to make sure that when you are using the embeddings that that these vectors are set up and that you're leveraging it. But again those vectors","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=806s"},{"start_seconds":814,"text":"have important information. So how you are then storing it is important. So the text embeddings basically are the same","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=814s"},{"start_seconds":822,"text":"as like having the actual text input because it is basically storing these memories. So the same method that pulled","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=822s"},{"start_seconds":829,"text":"full names out of a database of clinical notes then it's same idea here. um you're storing vectors like notes that","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=829s"},{"start_seconds":836,"text":"they came from. So like you this is I again this is I think what's a little bit new for de some developers of","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=836s"},{"start_seconds":845,"text":"like you have these different layers of information that are not just straight input output right it is data being processed at different times is","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=845s"},{"start_seconds":854,"text":"processed to be used maybe later all these different things and you need to be very thoughtful about if I'm trying to protect particular data am I truly","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=854s"},{"start_seconds":863,"text":"doing it in all the different stages and where are potential leak leaks here. If I was managing my vectors for my memory","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=863s"},{"start_seconds":872,"text":"at a different in a different manner, I have now exposed potential potent personal information because I was not","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=872s"},{"start_seconds":880,"text":"being proactive. So that's just kind of like this is a good example for a builder, especially if you're new to building with private inference, the things that you should really be aware","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=880s"},{"start_seconds":889,"text":"of and pay attention to. Here's a general overview of just kind of what I was mentioning here. If you're betting a note or a question for the memory, so if","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=889s"},{"start_seconds":897,"text":"you're running it on an AIT, you have that protection of the tested gateway uh end to end all the way through. If you have a vector and keyword index, so for","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=897s"},{"start_seconds":906,"text":"for that memory, keep it on the device, keep it on the phone. Uh that protection now is encryption at rest and you have that protection also too of it being on","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=906s"},{"start_seconds":914,"text":"that user's particular device. It is not stored in the cloud. It is not is only being used when it needs to be used from","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=914s"},{"start_seconds":921,"text":"there. reranking again that needs to go through the TE because again you have protected information there uh in the example that I just showed of that","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=921s"},{"start_seconds":930,"text":"previous test from 2000 uh 2023 is that you have now a set","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=930s"},{"start_seconds":937,"text":"situation where private information can be retrieved out of these uh vectors and so we want to make sure that we are protecting that all the way through and","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=937s"},{"start_seconds":945,"text":"just as a reminder without the end to end encryption the gateway can handle your note in plain text and even if it's inside its and tested TE. So there's","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=945s"},{"start_seconds":954,"text":"just things to just be aware of all the way through of just again the architecture is super important through all this. Moral of the story of all of this example here is that when you're","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=954s"},{"start_seconds":962,"text":"using embeddings, you need to they need to be in a TE because if you then have that and you create these vectors for that memory, those vectors need to be","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=962s"},{"start_seconds":971,"text":"guarded like the original pieces of information that came from. So if I am inputting again the example here I'm using is like health like clinical notes","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=971s"},{"start_seconds":980,"text":"and then we're pulling information out of there and then we want to make sure that we're storing that information then in for the user and we have these vectors we need to make sure that it's","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=980s"},{"start_seconds":988,"text":"being stored and protected just like anything else because that exposes private information. So we have N for","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=988s"},{"start_seconds":996,"text":"our embed embedding and reranking and then keeping things on the phone which is basically that index right those vectors we want to encrypt that at rest.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=996s"},{"start_seconds":1004,"text":"So now we have a full suite here of information being processed handled reprocessed","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=1004s"},{"start_seconds":1012,"text":"in a protected manner where nothing is exposed nothing can be leaked our data is private all the way through. Building with AI is a new paradigm for a lot of","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=1012s"},{"start_seconds":1021,"text":"people and we're going to keep investigating some of these different setups and hopefully this helps you work on your own stuff too. It's watching.","url":"https://www.youtube.com/watch?v=49lX4MTUSIQ&t=1021s"}],"youtube_url":"https://youtube.com/watch?v=49lX4MTUSIQ","url":"https://near-pedia.xyz/videos/49lX4MTUSIQ"}