Trusted AI in Production - Marvin Tong + Jerry Yu

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:09

   Thank you for coming. So yeah, today we have this panel about like TE's in production and how do we use this to provide like AI private AI inference. So um I'm Pierre. I work with Nei. Jerry works for Intel and Marvin for FA. So I guess we can all do like a very quick intros. Marvin, do you want to start?

2. 0:29

   Yeah. Uh, thanks for having me. Um, hi guys. Uh, this is Marvin um, from FA Network. We've been building uh, for confidential comput for for a while ago. Uh, we started seven years ago and uh, now they call it confidential AI. So glad to be here.

3. 0:48

   Thank you. Uh, this is Jerry from Intel. Um we build a attestation service called Intel trust authority which provided the independent attestation for the confidential computing environment. Uh so think of it as a proof that your for example your AI system security claims are real. Yeah.

4. 1:13

   Thanks. Yeah. So I'm Pierre. Um I work with Near AI where we've been doing like a lot of private inference and a bunch of other stuff. Uh but yeah, that's a very quick TLDDR. So uh my first question would be for you Jerry. Um confidential computing is pretty recent but it's becoming more and more like uh trendy and important uh the more like AI is growing and so I just wanted you to explain us how it works,

5. 1:41

   what it is and like the chain of trust what we call chain of trust. Yeah sure. So as we probably know the confidential computing is basically a set of technologies to ensure the data uh you know in use right so um uh the most essential one is really the hardware um enforce the in-memory encryption of the whole runtime right then the host OS the hypervisor cannot uh see the data actually yeah right but the encryption

6. 2:16

   alone is not uh enough. So we must establish the trust in the components running in the confidential computing environment. Right? So for example I needed to know what firmware was using to load the OS the boot loader what the boot loader was used to load the OS uh and ultimately what exact workload is running in the environment. So the solution to this is to build a chain of measurement right every piece running in the environment uh we call it you know

7. 2:49

   chain of trust. So the chain of trust begins uh with uh uh some root of trust which is usually built into the hardware like some private key you know uh from the chip manufacturer which can be used to sign the chain of measurement which will be verified later by anyone right third party. So the public key for that party is usually open. Then you know anyone can use that to verify the whole chain of the measurement. Uh you know uh

8. 3:23

   that's the kind of the uh basic uh idea. So um for example when the system is start the the you know firmware you know the uh will measure the workload before hand over the control to the workload. Then the boot load will measure the OS the same thing you know finally you know the workload will be measured as well.

9. 3:47

   One important thing is you know all those measurements are anchored to the hardware you know based uh temper proof extend only registers and you know extend only uh means you know those hardware registers cannot be returned directly too right um so it has to be you know cry cryptographically extended based on its current value and the new measurement. So with that you can see you know this creates a an kind of

10. 4:20

    irreversible you know chain of measurements. Uh so um later you know we can verify so you can see that you know uh uh kind of a chain of the trust is the foundation to build the verifiable uh computing environment. Yeah. Yeah. Yeah. Thanks. Um so like as we know like these big cloud providers such as like Google, like AWS and Asia and many others. So they've been running in this very very like already safe and

11. 4:53

    secure environments kind of. But like more and more they're going towards tees. So what's what we're talking about today. So even these um these companies these cloud providers they need some some form to uh actually um for their users for external parties to actually verify what's happening inside their clouds.

12. 5:14

    And so I know Intel has been working on this ITA um like service which basically allows anyone to verify and like verify this signatures coming out of this like u secure enclaves. So I'm curious if you can just explain us why this is needed like why ITA exists and why like GCP and AWS relies on this. Why don't they just sign everything themselves and just return this back to the users? Why do you need like a right I think that's a good question. So

13. 5:46

    uh you're right you know all the major uh cloud provider they all have you know great secure technologies for sure and they also have well definfined trust model uh of their you know confidential computing offerings right they even provide their own attestation service like you know Google cloud they have a cloud Google cloud attestation service and Microsoft Azure has their own Azure attestation service but I think you know There is a fundamental challenge for

14. 6:17

    their customers you know especially the regulated industry like uh healthcare, finance, government. They their compliance uh framework requires the u uh you know separation of duties principles which mandate uh the infrastructure provider and the uh verification provider have to be independent. Right? So think of it like uh like uh financial auditing right the companies uh don't do the uh auditing

15. 6:50

    themselves right because the regulation and the shareholders will you know require them this has to be done by neutral third party so it's the same thing for confidential computing a testation service here plays kind of you know independent auditor role for the confidential computing environment So actually you know um the major CSP like Microsoft, Google they all recognize this is a strong requirement from their customers. So Intel uh we

16. 7:24

    have a very close co-engineering collaboration with them since the very beginning we start you know this kind of confidential computing this kind of thing. So uh today you know if you look at their confidential uh computing service offering release announcement you will see you know uh both Google Microsoft they will highlight and feature you know Intel trust authority as the independent

17. 7:56

    attestation service for their customer who are seeking the operator independent attestation. I think this is exactly a a good example to show that the industry uh is moving uh to a mature you know verifiable confidential computing which you know all the enterprises with you know security governance requirement can uh actually adopt uh you know um uh actually not just the regulated

18. 8:29

    industry. I think uh any you know enterprise uh or business which has the security uh governance requirement uh would benefit from this uh separation of duties principle because the independent uh attestation neutral third party attestation will give you the a stronger security posture I think that's the reason why we you know uh Microsoft the you know Google they would like to actually collaborated with us to in integrate inter trust authority you know kind of set up party authentication

19. 9:06

    service into their service infrastructure. I see. Yeah. Nice. Thanks. So Marin coming to you. So um as J explained so we have this whole like chain of trust and how T are kind of working. Um so there is this myth that like TE's are very hard to work with and quite unsafe. So the first what everybody heard about is like Intel SGX.

20. 9:28

    It was very hard to work with and uh yeah there was some yeah security faders and I'm curious how now at Fala you make it easier for any developers around the world and users to actually leverage this and how to verify all the workloads and everything.

21. 9:44

    Great question. Uh thank you. Um uh uh actually like 7 years ago when we just start a company the original idea is pretty simple like we see there's a lot of like um criticals happen in a computer world for example like um in crypto space you know when people using uh for example Binance to trade they always wondering is Bance trying to stole my money or stole my idea and uh in other space too like people just

22. 10:16

    don't trust the service provider naturally. That's also reason why Jerry just bring up like Google Cloud, AWS, OpenAI, they're also using TE to provide some kind of guarantee for the trust. So that's the very original of uh FA's idea. And uh when we dab deeper we realize back then there's only SGX uh actually a very uh uh uh have solid security foundation and achievable uh accessibility for the hardwares and uh

23. 10:49

    then we realized okay SGX is good but it's not that developer friendly like you have to uh for if you are building a application there's so many things you need to concern about And the learning curve is very kind of um hard in the first maybe a couple months. You have to learn too many stuff. And uh so the initial idea for FA 7 years ago is that we want to provide this kind of computer

24. 11:20

    trust technology with very userfriendly uh journey especially for developers and uh basically all of the application builders. And that's where we start. And we spent maybe two or three years in SGX world trying to use you know uh open source uh uh projects like raine you know this and that to uh make it a very friend there's a one uh product we shipped before called fat contract something like smart contract but some

25. 11:51

    also like serverless functions um that you can easily run serverless function into the server and you don't need to consider about like how to how do I maintain the no container no dev op works you can just uh just write program write javascript deploy it and the program can automatically run it for you but uh that feature like that product didn't make a hit eventually you realize okay uh there's there's a there's a

26. 12:23

    there's some a missing part of the PMF between what we can shape from the foundation of trust computer to what developers need right now. So during the last three years, we make some changes. First of all, we uh try to very be focused on containerized programs. As long as developer can doize their program, we can help them to uh make oneclick deployment in a trusted computer very easily in TE. And secondly, we kind of be very handy with

27. 12:58

    our clients, learn their business, learn what's their KPI and how they want to suit up their architecture to make sure uh whatever solution we ship to them, they can uh um easily achieve the value they are expecting from confidential compute or confidential AI. And also we uh make running AI models in GPUs super easy uh with uh encrypted memory and cryptographic proofs. So that's uh how our journey you know to to to to help

28. 13:31

    make making the confidential compute and confidential AI achievable accessible for everyone. Yeah. Thanks. I'm quite curious. So, uh, you started on Intel SGX and, um, I'm I'm so you mentioned like there were some limitations with SGX and I'm curious how did Fala knew something like TDX would happen at some point because like it was Yeah. Yeah. Because TDX is a is a great uh uh progress happened in confidential

29. 14:01

    compute world in the last two or three years. First of all, because you know uh you need to understand the the the history of how uh security hardware involve probably Jerry is a history here. He know everything about each milestone. But based on what I learned about in the very beginning, SGX is mostly built for local computer application. something like maybe a I don't know just a a programs running in local computer not just a natively or

30. 14:34

    natively born for cloud native use cases. So uh it make it very hard if you want to uh build something based on SGX but serve for cloud-based ideas. Uh but then uh TDX came out. TDX is pretty uh good uh for cloud native applications.

31. 14:54

    So it's very kind of friendly for Docker Knight virtual machines and even Kubernetes. So I guess uh that's a big difference. Yeah. Yeah. And and Jerry about so this migration from SGX which is like you know very as Marvin said it's like mostly local stuff and like you have your process that might use some things from SGX to move all the way to like TDX which now allows us to run these full

32. 15:19

    Docker containers. So these uh these VNMS these big AI models in there like was it something that happened naturally at Intel that you guys like foreseen that it would be possible to run like these big workloads inside or I'm just curious if if you have the the answer like what this the story the history of from SGX to TDX like was it something natural or Yeah, I think you know what Marvin just

33. 15:46

    uh mentioned it's really about uh how easy we can uh adopt the confidential computing right so for SSGX technology you know the workload has to be technology aware which means the workload needs to be somehow to customize to to leverage this uh particular technology but for TDX you know you you workload you don't need to do anything right so you just run as is because the the whole environment is running in uh you know

34. 16:20

    TDX uh hardware uh enforced uh you know trusted domain so workload itself is not necessary to do any change I think that's the uh the point you know yeah lower the bar to adopt a confidential computing right yeah and do you still see like SGX being used by customers or now everybody is using TDX like uh I think you know uh it's a uh for a different purpose. You know, if you want S SGX is still active, right?

35. 16:56

    If you want to have a small, you know, uh uh trusted uh computing base, you know, it's still the uh you know, great technology to use, right? Makes sense. Okay. Um yeah so to come back more on the the the AI side of things. Um so now Nvidia recently released I mean recently I think it's like two or three years ago I'm not sure. Yeah.

36. 17:23

    Um their Nvidia confidential computing uh like implementation which basically does the same as Intel wanted to do. I mean the same putting very high level. So Intel did like all this T stuff on the CPU side for years and years and now Nvidia wanted to do the same for their GPUs. So the two can work in a fully like confidential manner. Yeah. And so Nvidia released this yeah two three years ago and um so I know Intel ITA the

37. 17:50

    service you're you're working on allows you like users or developers to verify at the stations. So so these signatures coming out of like both uh the CPU side and the GPU side. Can you tell us a little bit about how this works? like is there like any specificities of Nvidia's CC implementation compared to right you know um you actually just describing the uh the whole uh computing environment it's not just a CPU we also have a GPU right so when you needed to

38. 18:23

    establish the trust of au uh confidential computing environment uh you need a you need needed to uh make sure the CPU is genuine like Intel platform, right? Uh you also needed to guarantee that the GPU is a GPU supporting confidential computing or private computing, right? So that's exactly the concept of composite attestation. So basically in the attestation uh request, you need to provide both the evidence for the CPU T and also the evidence for the GPU. You also somehow you know

39. 19:03

    needed to provide the evidence that they are bound to each other. It's not something separate right. So that's the composite attestation. Intel trust authority definitely support the composite attestation which means uh in the single attestation request uh it include both CPU and GPU evidence. We can verify both and release the attestation result. At this point I would like to mention that we are uh actually closely collaborating with

40. 19:35

    Nvidia, Microsoft you know to uh you know define the uh standardized attestation result profile together. Mhm. So which means you know um the the claim profile are unified uh you know which uh can u be accepted uh by different you know confidential computing uh service.

41. 20:00

    So I yeah that's what we are doing. Nice. And um so historically Nvidia's conf started happening on the oper architecture which was I mean it was already quite performant like GPU cards but now the the new version that's coming out it's called like the blackwell architecture.

42. 20:21

    It's like at least twice as better per GPU cards compared to the the hopper architecture. And also there is this big promise on the T side of things. So let let me go back. So the there is this limitations right now with hoppers um when you use Nvidia CC that you cannot do like uh any kind of model training across different hosts um because the the conial computing only works on one specific host. you cannot do like a cluster of host of nodes and there is big promise

43. 20:52

    from Nvidia um with black whales that would allow people to do like cross nodes um AI training and I just wanted Marvin if you can tell us a little bit about about this like is it something that you guys are looking at or yeah um thanks uh for asking I actually TLDDR is uh the most frequent questions we received from our clients bas basically two group uh one is um what's the trade-off for using TE on AI right

44. 21:23

    like can you run what size of model can you host uh the answer is all of the size model uh as long as you can name it we can run it there's a the tra the confidential uh compute performance trade-off is only within 5% in a hardware perspective but like most of the business we are running with right now is for confidential AI inference instead of confidential AI training. We know that anthropic is doing a lot of

45. 21:56

    works in the training. But uh to enable training you need to match massive amount of CL GPU clusters within the same security assumption and for that doing that uh you need to do multiple nodes management or Kubernetes for the GPU clusters and of course the the latest of black well architecture like um maybe B200 B300 support that we haven't practiced a lot by ourself but we know that it's doable by other companies and uh since our uh mission uh

46. 22:35

    for the team is to democratize you know the u uh the technology holding by Apple or anthropic to every developer in the world. So uh this is our next step to bring Kubernetes based you know GPU cluster management with a better security with better privacy to everyone. Yeah. And yeah, it's it's totally doable.

47. 23:00

    Nice. Yeah, putting Kubernetes in is tough. Yeah, we are. Yeah, trying to do it. Um and I mean this quite a very technical quite technical question but um can you tell us um if if you know like what are this like um change in the trust boundaries when you go from this hopper technology which is like you have encryption within one node when you go to like this whole like when you have a cluster of black whales. Do you know if Nvidia is taking

48. 23:31

    any like security changes or it's not public? I'm not sure. I think Jerry can uh share some but based on our team's research and then acknowledge I think the trust assumption is very similar. Um because as Jerry mentioned, you don't just verify the secure GPUs, you also need the CPUs. The difference between let's say a Kubernetes mapping to a single node is only based on how many machines you are verifying. Um and are they physically right in the same location or they're

49. 24:10

    just a cross uh cross clusters? So it quite depends on the situation. Yeah, I see. But the trust assumption didn't change too much based on our understanding. Yeah, thanks. Okay, thank you guys. Thanks for time. Yeah, great. Thank you.