Building Private AI

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   We're going to welcome Sergey, Luke, and Sabrina uh to the stage. It's a pleasure to have all of you. Uh why don't we do a quick round of intros just to start. Uh I think a couple of you guys have been here before. Uh Sabrina, it's your first time on the show. Welcome. Welcome to the rest of you as well. Sabrina, why don't we start with you? Quick intro and we'll go down the line here. Absolutely. Thank you so much, Robbie.

2. 0:21

   It's a pleasure to be here. Uh hey everyone. I'm Sabrina, head of API at Venice and developer experience 2. So basically making sure everybody get the best experience and and has a great time using our API. I'm based in Brazil and yeah I've been in Venice for about one year a little bit over an year now.

3. 0:41

   Fantastic. Uh Luke, why don't you uh do a quick intro next? I think you've been on before and then we'll go to Sergey. Yeah, I'm Luke Muls. I'm the VP of business operations at Brave. Uh and you know we have the browser, search engine, AI, API, all the good stuff.

4. 0:57

   Very cool. Very cool. Sergey, how's it going? Hello. Um, I'm actually first time here too. So, nice to meet you everyone. Um, my name is Sergey. I'm the head of product at Near AI. So, I focused on everything AI uh within the Near ecosystem and uh the private inference is one of the focuses that we're going to be talking about. Uh I've been with Near for uh a little bit over a year and I'm based in San Francisco.

5. 1:22

   Very cool. Very cool. And uh I think that there are deep interworkings between you know all the all everyone on the stage here Near working with Venice Near working with uh uh Brave and then maybe even Venice and Brave. So I I'd love just to kind of get a highlevel understanding of how we've kind of arrived here. Um you know model routing as it stands is sort of still a very early question. I think we've seen this in a couple different ways. You know Venice does this in a private way. Near's also

6. 1:53

   got its near AI component and then Brave obviously has you know privacy elements to it as well more so in the browser perspective. Why don't we start on the Venice side. Sabrina, maybe you can give us an understanding of, you know, in short, how the Venice product and and specifically the API works today and how you've been able to implement privacy into something that again, you know, especially evidenced with, you know,

7. 2:18

   this this millennium math problem in the last week, you know, seems like this is extraordinarily important. Maybe you can tell us how Venice solves for this. Absolutely. So, um, Venice has a strong privacy philosophy in mind. So, this obviously translate to the API. We started out as only a web application as a chat but now you know we grown into having this place where we offer every

8. 2:41

   single model out there and we try to offer it as with the best privacy guarantees that we can. So um in the API you're going to find every private model in a zero day retention way. So basically um KK3 uh GLM models every single one and you're also going to find anonymized models when you're talking to closed source models. So we try to make them all available at the same place. As for example, you can also talk to fable. Uh you can also talk to GPT Astra, but

9. 3:12

   those uh are going to be anonymized through us. So basically the end provider never actually sees that you are the one talking um to those models. But of course there's um many things that comes to place when you're talking about model routing when there's so many things that come into place um when you're talking to a model that has that privacy guarantee and you cannot break

10. 3:36

    that contract. So always something that we need to to keep in mind um but just the products tries to bring intelligence and bring AI um as best as we can uh to everyone in the private way. Yeah. Uh, Sergey, I want to hop hop over to you and kind of compare and contrast a little bit. Sabrina just gave us a great intro to how Venice's model routing works and how they do it in a confidential way. How similar or different is Near AI's cloud when you're thinking about model routing and how you're doing this privately?

11. 4:08

    If we talk about uh private inference specifically and uh serving models, it approaches are actually very similar. Uh, one thing I wanted to highlight is like the overall vision of uh near AI. We believe in userowned AI. Uh this is where because the world is moving to AI more and more. Uh we believe that the owner of the AI, owner of the data should be the user, not the corporation that serves the model, the not the

12. 4:36

    corporation that builds models, not anyone but user. And privacy is a key component of that. So we uh built the inference engine so that the user has full control of their data. uh similar to some extent to Venice as well. We have uh private models that we host ourselves and serve um to the end users. Uh those models deployed behind the TE.

13. 5:02

    No one has access to that. Uh not even us developers at near can see um the input output of the models. Uh we serve them directly to the users. We serve them through partners as brave for example. We'll talk about that in a second. Uh and then we have the we call them incognita models or anonymize models basically similar concept to Venice. Um and it's great to see that we across the board at least within this panel understand the important of privacy and how that leads to the user owned AI.

14. 5:38

    Absolutely. And Luke, I mean, I I'd love just an understanding, you know, how you guys approach this both at the browser perspective, but also you've got Brave Leo and, you know, this sits next to, you know, some pretty uh sensitive data sometimes in terms of browsing history, in terms of history, login, passwords, all these things. How how do you guys manage that and make sure that you keep LEO segmented from an existing browser profile?

15. 6:05

    Yeah. No, it's great. I mean, it's like one of the one of the powerful things about having the uh browser, having the AI built into the browser is having access to that context, but also kind of making sure that it's isolated uh and and that we're guardrailing that as much as we can is is obviously like a super important piece. I mean, what we're in a interesting category too because we have like 126 million uh monthly active users

16. 6:28

    and so we've been working with Near since like November of last year. here. I think we rolled out um private inference in in our nightly channel which is very developer focused and and uh where a lot of people are being very picky about which models they're using but like as you get to the broader segments uh you know you see a lot more auto you know selection being key. So right now we've actually got that in our release channel um with GLM 5.1 using Near's um private AI like inference and

17. 6:58

    so um we're we're marketing that out and seeing how users take to it and move to it and and all of that. But um you know with everything we're doing it's user first and and trying to kind of even where it's not fully private. um trying to sanitize it as much as possible. Uh whether it's proxying things or or you know making sure that that you know brave our whole ethos is that we don't want to know who you are you know but we want to just kind of get out of the way.

18. 7:24

    Got it. Okay. And so the way that the the the routing works here and and you kind of point it to like the auto mode which you know I'm typically in as well like you know you kind of put it through your LLM of choice but then maybe it will pick the level of sophistication that it needs to answer your question correctly or effectively. And so as the model is kind of passing it to what it thinks is the correct level of sophistication, you're trying to maintain privacy when

19. 7:58

    it's kind of going and sending that data or that prompt along. And as I understand it here, correct me if I'm wrong, that's what you, you know, near AI is for is that when it goes to route this prompt to another model with a different level of sophistication, it can do so privately via Near's AI.

20. 8:18

    Yeah. Like ideally, every model is doing what Near's doing, right? Like you have you have I mean like in an ideal world like it's all private. Um and I think that's kind of where it needs to go. But right now, just think about it as like the the uh Leo will pick the right tool for the right job, right? Like and and so ideally all those tools are private for every job. Um we're not quite there yet, but we've made an important step in getting this out there to our release

21. 8:44

    channel um for for all of our monthly active users to to start using uh and and yeah, kind of like see how how we even message that, right? like and uh and and and actually marketing that to people too because I think a big problem with the privacy story in AI is that there's there are only a few players that are actually doing it. Um and that people aren't even really aware or

22. 9:07

    people might make assumptions that things are private that aren't necessarily private. Yeah. And I think that's what I think people are now waking up to, right? Even if you know it was just open AI and it was a matter of hey you know there's this you know pretty significant thing that everyone not everyone but you know some some very smart people were working on okay now we are kind of realizing or waking up to the fact that you know open AI might have been spying or reading or

23. 9:36

    training itself based on these things. I mean question to all you guys just feel free like can you can you provide any clarity here because I think there is still some misconceptions like even just in the last few days you know we're starting to wake up to the fact that maybe open AAI is listening for lack of a better term like are they or is it still kind of just on like a hit or miss

24. 10:00

    basis yeah Sergey what what what's your what's your read on this well I don't know the answer to this question uh I have some speculations and uh I think there are a lot of speculations uh in the world as well and this is where I think our approach is important. If you protect your data not on the policy level but on the encryption level this is what we do like we deploy models within the TE no one has access to it you don't need to trust anyone you can

25. 10:30

    actually validate that no one has access to your data through the attestation that's the right approach uh I believe uh and then in terms of what open AI and that's uh the recent story about the math uh problem um yeah there's so many opinions it's hard to tell which one is true. there is like the discussion that did they use uh the data did they create synthetic data from the all the conversations that um

26. 11:01

    are happening um we don't know uh but I prefer to live in the world where you don't need to ask those questions where everything is private and secure um and routing is a big problem problem uh about that too like the routing also has to happen in a private environment We're working towards that solution like when you decide which model to use that model that makes the decision should also ideally write uh run in a private

27. 11:32

    environment so that the data again is not exposed to anyone and the cost of error also is high like oh if you route it to wrong model then your data is exposed you don't want to have that happen all these problems they pile up and um yeah require a thorough solution.

28. 11:49

    Sure. Yeah. Uh Sabrina over to you. I mean you know Venice was at least in the market a huge benefactor of you know what happened with this potential you know breach in privacy. What what what was your reaction internally at Venice?

29. 12:08

    Um no surprise really. uh it feels like at least you know it has been a little clear that these big companies they they do benefit from from the conversations and I I feel like they have been open to it to some extent as well. So um it's it's honestly not the first story we've heard about something like this happening. I think we have we've all also seen some other cases where um you

30. 12:33

    know big companies reach out to to other folks on on conversations that were had within the platform. So there is of course some level of monitoring happening and I feel like um the big thing here is really that the user sometimes isn't aware um of the contract they're making when talking to these models through these platforms. Right?

31. 12:54

    So it is assumed that that the user has the information and and it is aware of what can happen. Um, but it's it's just not the case. And this is why we prefer just defaulting to private um, and letting the user make that decision when they want to use a closed source model more consciously um, than to have that be the default of every conversation.

32. 13:18

    Yeah. Absolutely. And so, you know, Lou, come coming full circle here. I think you know we've got different levels of granularity or different levels of confidentiality when we're considering how these models are sharing prompts both you know how they're routing within their own sort of closed environment but then in the case of some more of these open environments how these things are getting passed back and forth where are

33. 13:44

    you current and I think Brave Leo is kind of a good example here where are some of the the the stop gaps like the gaps internally at Brave Leo where you know I I think Near and Venice both have taken an approach where they've kind of made privacy a non-negotiable like hey if you want to use us for AI you're going to do so in a private way Brave Leo still kind of has a choice open why

34. 14:08

    is that the right case I take it you know significant user base you want to have optionality but how what is the migration plan as you go to a you know kind of uh privacy choice to a privacy by default regime team inside of Brave.

35. 14:24

    Yeah. I mean, I think it's balancing demand for new models with what's available and like, you know, when we can uh kind of self-host um and and sanitize as much as possible, we do. Um and so I think it's a matter of time where and market maturity like as things get people become more aware of the need for uh these private defaults, which everything should be defaulting that way

36. 14:48

    anyway. But I mean just just thinking back on the last you know set of questions and responses it's all just deja vu like the same thing happened with adtech and with uh with monetizing the web where you know before we know it we looked around everybody's racing um you know running with scissors to basically scale monetization and users don't realize how much of their information they're giving away. Well, now you not only like AI is essentially

37. 15:12

    kickstanding the entire economy and um you've got these companies that are at a break pace like just running very fast and um and so you know that's why it's so important that people on the frontier are innovating at this level with privacy because if you're not doing this it's just going to be a disaster when you think about what AI can the potential there and and and all of the um you know the vulnerabilities that are

38. 15:35

    out there with the web and and also just from a user perspective and a B2B perspective Yeah, and that you you preempted my next question actually that I think the last question as we kind of come to a close here. Uh question for you Sergey you know on on the B2B side on the enterprise side of things you guys have you know flexible and gen general tech that can apply to both the retail perspective that we're seeing you know adopted on Venice and across the board

39. 16:00

    at Near AI but then also in an enterprise perspective how you know there are sensitive business secrets and information that is proprietary that enterprises want to protect via a private model. How are you seeing adoption progress on the enterprise side of things and how how do you see it keep pace with the retail side of things? What are you seeing at Near?

40. 16:24

    We constantly interview people uh and ask them what's important for them in the AI and if you ask that question out promptly it's like quality of the models cost and then there's like additional factors ease of use and rarely the privacy comes up right out of the box.

41. 16:44

    uh especially like say last year beginning of this year that was not on people's radar and now we see uh more and more stories where data was leaked there's like 100 millions uh conversations was leaked from one of the websites uh then there's this um kind of controversy around open AI people starting starting paying more and more attention to privacy um I agree with Luke there's a lot uh in education in uh user behavior change that still needs to happen. Uh, and uh, I hope that happens

42. 17:20

    soon because the AI adoption is going so fast and um, the more contacts AI has, the more powerful it is. And I want to share this context with the AI, but I'm like concerned who's going to own this data, who's going to own this AI at the end. and the privacy I think that's the only approach that uh make sure that uh those AI agents they represent me uh not the company that build them if if uh it's done in a proper and private man manner. So to answer your question

43. 18:01

    people start recognizing it more and more and it becomes a bigger topic of conversations and privacy goes up and up and up on this core values. Fantastic guys. Thank you so much. Um I think this was a very relevant conversation uh based on everything that we're seeing in the AI space and how there is this suspicion and this growing suspicion of an invasion of privacy. Um couldn't do it without you guys building and uh fighting the good fight out there for private AI and model routing. Um and

44. 18:33

    so we're going to keep things moving along. Thank you so much. We'll see you all again soon. Thank you. Thank you. Thank you. Cheers.