Switch From OpenAI to Private Inference in One Line
Nothing is loaded from YouTube until you press play, and then without tracking cookies.
About this video
If you're already building on the OpenAI API and you've been wondering what it would take to move to private inference, the answer is smaller than you think. NEAR AI's cloud API uses the same setup OpenAI does, so migrating is a base URL change. One line, and you're running.
Same request, different road. Instead of your users' data traveling in plain text through a provider, it goes down a private lane to a model hosted in a trusted execution environment, sealed, with an attestation report you can pull for proof.
The clearest place to start is triage. Rating the privacy needs of a request, working out the topic, deciding which model should handle it. You want a cheap fast model for easy work rather than paying top rates for everything. Triage also has to straddle high latency tolerance and high sensitivity at the same time, and a TEE-hosted model handles that combination well. You don't know what your user is about to type, which is the whole argument for protecting it.
Then the part I want you to actually hear. Know what this covers and what it doesn't.
The guarantee attaches to the model on that specific path. Not your app. Not your product. Not your company. Models you're proxying through for anonymity aren't protected by the TEE. And if your app logs the request before it leaves, that data is exposed and the protection is on you, not the enclave. Be careful how you word your privacy claims, because a TEE-based triage report doesn't license a blanket statement.
I also open up the attestation receipt: the hardware root, a quote signature proving this is genuine confidential computing rather than something merely claiming to be, chip measurements, freshness so you know it isn't a cached report, and reproducibility. One dependency remains. If a hardware vendor is wrong about their own silicon, that's a break in the chain. Trust gets smaller and more auditable. It doesn't vanish.
Chapters 0:00 Already on OpenAI and thinking about private inference 0:20 The good news: it works out of the box 0:44 The migration, one base URL 1:01 Same request, different road 1:23 Why triage is the obvious first use case 2:02 Latency tolerance versus sensitivity 2:18 What the switch actually covers 2:52 Proxied models and logging are not protected 3:15 Don't make blanket claims about your app 3:47 What's inside the attestation receipt 4:23 The break in the chain: hardware vendors 4:39 Run the benchmarks and judge the trade-off
Get started at cloud.near.ai
#PrivateInference #TEE #ConfidentialComputing #OpenAI #NEARAI #AIEngineering
Transcript
1,069 wordsCaptions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.
- 0:02
Have you already started experimenting with AI and see how you can integrate it into your products? Maybe building a new product on top of something. Well, let's say you're already using Open AI, but you want to move to private inference.
- 0:11
And the benefits of moving to private are really great for both you and your users. So, how do you go about doing that? Is it going to lot add a lot of extra work? Well, the great news is that it doesn't. Uh today out of the box you can start building with private inference using something like near AAI that is going to take you literally two seconds to make a switch on how it's
- 0:36
actually set up making those calls because it's leveraging the same type of setup that OpenAI does today. Let me show you. So if I were to change and everything my migration from open AI's APIs to nearest cloud APIs which are again are using a private inference system using tees is really easy. It's this one line switching the base URL and then we're off and running. The beauty of this is that it is the same request
- 1:01
down a different road. Instead of having all that data in plain text through a provider like OpenAI, we can now send it through a private lane using a TE hosted model. that trusted execution environment model is sealed and we can actually have an addestation report from that. You would then get a response out of there as well. It's the same pathways. So for building on anything
- 1:23
like for myself even uh one of the most obvious places to use a private inference system in an application is doing any kind of triage work. By triage, we mean that like how do we rate the privacy needs of this or how do we start to determine what kind of topic this is or what is being said here. And the importance here is that maybe we want to send it to different models, right? Because some models are more
- 1:47
expensive than others. So, we don't always want to use the most expensive model for really easy things. So, a triage model really is great. And this is something that sending through a private inference makes a ton of sense because you don't know what's being said by the end user. So preserving that privacy is really important. And you can see when we talk about like you know where this all fits in from a high
- 2:09
tolerance for latency versus also s sensitivity a triage lane kind of has to straddle both of those and we can actually solve that pretty well using a TE based model. Now it is important to understand of what this actually is and isn't.
- 2:27
So that one line change to a private instance gives you something again like having that TE hosted model. This scope change allows for a couple different things. The specific TE hosted model name is still on the screen which is great. You can still actually send all your requests on that path everything through that session and you have an addestation that you can pull in. It's important to know what's not covered. So models that are being just proxied
- 2:52
through so that you can just have some anonymous request to it are not protected by TE. If your app is logging stuff before the request leaves, that's not protected because you are still exposing that information. Uh so that's an important piece to understand is that that data protection now is on you because you have now logged that separately outside of that. And again, you can't just make some blanket claim just because you're using some, you know, TE based triage report. So, just be careful of how you pose what your
- 3:23
actual app is doing. Uh it's important that anyone who is using this if you're an end user or if you again if you're their builder, this guarantee is about the model on a particular path, not the entire app and not the entire product or company. So it is locked to those requests only, not to anything else that's happening inside the app. If you saw my video on trust execution environments, then you would have seen this already. But if not, let me just show you actually what's in the receipt.
- 3:52
Uh when you actually make the add astation report about using private inferences. So you get some different things. You get what the hardware route is. Uh you get a quote signature coming back that showing that it's it is not only confidential computing, but it's genuine confidential computing. uh you also will get you know measurements of the chip how fresh that is so you know that it is actually coming from you it's not something that's just cash as a
- 4:14
report and it is reproducible so that's a really important piece to show that this is proof that it's not just being made up as we go we are dependent on something here though it is important to be aware of there is a break in the chain and that is the hardware vendors themselves uh if they are wrong about their own silicon then that that is something that they have to be aware of trust gets smaller and I'm more audible but does not disappear. So in this one simple line change, you can get up and
- 4:39
building and see right away the difference in performance and difference of actually privacy and security and all that stuff. And it's a really easy switch to make. Now I would encourage you if you are building a product using AI to actually go and try this and switch this out and see how it works.
- 4:53
Look at the benchmarks. See if it still fits into your needs. And I think you'll see the trade-offs maybe of even a little bit of latency or a little bit different type of answer is worth that trade-off of actually being protected for your users's data. Especially if you're dealing with sensitive information like health, financial stuff, anything that would something that people would not want to get out or be exposed to is a really important
- 5:16
piece here. It's really easy to get started. You can go to cloud.near.ai and start today.