NEAR AI Office Hours w Phala

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:02

   okay hello everyone and welcome to near office hours uh today we have uh fall network with us uh do you guys want to introduce yourself and very briefly say what F network is all about and uh yeah yeah sure um hi everyone uh this is Marvin I'm the co-founder and CEO of f Network and uh uh uh probably like h and make your uh own intro very short uh what we are doing is to provide uh offchain High perform uh security competition for uh web3 uh no matter uh

2. 0:43

   you are running like blockchains or smart contct decentralized applications AI agent AI models uh you can use FAS te Service as a co-processor uh you know for verifiable confidential uh compute hey uh H uh co-founder and CTO from F network uh we have been working on uh te for already five six years so that's quite a journey uh and uh yeah uh super happy to be here uh because you know uh both near and F network is working on uh very you know Cutting Edge

3. 1:25

   stuff yeah so so it's interesting right because te have been around for a while right like the first version says GX has been like it's been at least a decade right MH but it's it's only picking up now uh so uh for for people who don't know yet what the te is can you guys like explain it in the in layman terms what te is why it is why it is useful yeah I think I can take it so um when I explain to people what is te and

4. 2:00

   especially for people who uh don't have any uh uh understanding before uh I will take uh zero knowledge proof as a example as a as a comparison it's kind of like uh zero knowledge proof where uh you can run some uh execution of uh of the blockchain uh and uh you can verify what is the program and the correctness of the execution uh but uh it has a lot of in uh interesting property compared with uh ZK first is it's super super

5. 2:31

   efficient like when run in ZK you're going to spend say thousands uh times of the overhead so if you spend $1 to run uh to uh uh uh say run some machine learning models then you have to spend $1,000 US dollar uh for the same amount of the workload but uh approved in uh DK so this is one big difference another difference is that uh uh unlike uh most of the uh users uh people's understanding uh they can only give you the Privacy if you run everything

6. 3:05

   locally but if you give your execution to somewhere else for example you find a ZK proof Cloud then uh uh you don't have the privacy because you still need to give all the inputs the execution uh uh uh uh uh uh states to some someone else so uh but te is another story te um can give you all the pro uh privacy preserving uh prop properties so um essentially te is a hardware based uh

7. 3:36

   solution it allow you to put some software or some uh program into the hardware uh area so in this area is a secure area any anyone else including uh even The Operators of this Hardware or the ad administrator or the operating system cannot access the data inside this blackbox so so uh by putting your program in this blackbox nobody else can manipulate how it is executed it was just a uh exactly executed as how your

8. 4:10

   program is designed just uh very similar to Smart contract and uh nobody else can see the data running inside so people cannot steal data from the te combine those two properties as well as the provability then you have privacy preserving you have integrity uh uh uh guarantee and you can also prove that and being verified either onchain or offchain uh and this yeah still a little bit similar to DK but uh in terms of performance properties and uh how you

9. 4:43

   develop the program running in the te those are the big difference and uh to me uh although you know ZK has very uh uh you know strong security uh guarantee but on the other side if we don't need that high level of the security guarantee then we can get huge gain on the performance and the Privacy yeah yeah so I think so so you compare to Z proof right so Z proof gives you the guarantee that a particular output is in fact a result of

10. 5:19

    a particular computation on particular input right and so te can also give you that uh but then another comparison to like you know the world of um of cryptography would be uh fully homomorphic encryption right because with fully homomorphic encryption I can I can have someone else run the computation uh right but I will know that they neither saw the inputs nor did they see any intermediate States nor do they know the output which I do right and like if you combine fully H morphic encryption and Z

11. 5:51

    knowledge proves zero knowledge proofs you will get something that has similar guarantees to tees right but both of those today are quite slow right and te works at the effectively at the speed of computation right there is overhead is either almost zero already or getting very close to zero right some uh uh Benchmark and for both for cput and GPU they are the the overhead is almost less than 5% yeah

12. 6:25

    yeah and I think for for Blackwell gpus it's it's going to be even lower right yeah I think so yeah I I can also give some like uh uh additional comment on the like from the market perspective because Hong is providing from uh know technology or stack perspective right point of views so from uh what user need and I would say uh in web stre te can have like uh some uh extra benefits

13. 6:55

    actually I have a uh slide for it but uh uh yeah but uh TDR is uh uh uh we have a we have a matrix called uh uh you know trust list Roi which means uh which trust list level you can reach for the competition uh compar is the uh potential like Financial cost and time cost for development so of course we all know that like blockchains like Bitcoin ethereum uh near you know have extremely

14. 7:29

    High uh trust list level and uh but if you want to reach out into that Benchmark the competition trade-off is kind of very high too zkp is kind of match that uh layer and uh the a good part of te is that probably like using te won't get you let's say 100 scores of trust list level but uh it can easily help you get like let's say 80 scores in some way uh using just a centralized you know web 2 te can help you get maybe

15. 8:03

    probably 70 scores but to reach there the cost is extremely low so it means that using te for trust dis use cases is kind of have a very high uh trust RA in the meanwhile another benefit I want to mention is the web to Friendly so uh te as a tag is super web to Friendly so uh ALU mentioned right like running AI models inside te is already very possible and so does many other use cases for now like uh uh you know if you

16. 8:40

    as long as you can WRA up the program into a Docker you know you can run it into a t and get some proofs so the web to Friendly part means that how much development uh cost or cost the development uh uh uh cost you need to uh uh give away uh to get on board in into web stre for example even the most easier uh smart contract solid smart contract or some uh virtu machines uh for ZK right the developers still need to rebuild their uh original web to

17. 9:14

    program and rewrite most of the code base uh to uh onboard into website but for Te is super smoothy migration that's another benefit is super web to Friendly yeah so so I think it's it's worth um uh being a little more specific right why why you would say 180 not 100 right so I think there are three main ways in which te could be uh less secure than Alternatives the first one is that you do need to trust the hardware

18. 9:46

    manufacturer right so like Intel or Nvidia in this case th though I think people people don't often look at it this way but you have to trust hardware and manufacturer no matter what even if you don't use the te right what what happens on your like you know when you buy a laptop and you start using it right if if the manufacturer of the laptop put something into the bias uh right it's uh you know then it doesn't matter if you run fully H moric

19. 10:11

    encryption or ZK knowledge cryptography on that machine that machine is spying on you right then the second yeah the second is that there are some uh in the past in the previous generation of Intel te technology people did find side Channel attacks and whatnot right so like there's all is a chance that the new generation the TDX will eventually be you know there will be some uh some some way of like you know using like a cash uh or something like that uh to find to like K into what

20. 10:45

    happens inside and I guess the third one and the third one is the most interesting and I think this is where I personally find uh this stack which we need you know I guess we will talk about in a second very helpful is that te techn is notoriously complex right like like if you try to uh to use it from first principles right first of all documentation is extremely lacking right like if you if you Google Intel TDX you

21. 11:10

    will get 10 links of marketing material from Intel and then only like you know in the second page you will finally find any technical documentation right it's even worse with Nvidia right because the technology is relative the young right and I think the uh the beauty here is that with the uh with the with this stack right you can abstract away from the complex uh from the complex complexity of TDX and and and and use it with a with a very simple abstraction so so I

22. 11:43

    guess do you guys want to like you know talk a little bit about D and which exact obstruction it provides to people yeah yeah I I think this is a really uh interesting topic uh because uh when using te usually uh there are a lot of bit FS uh it doesn't mean if you use a t then you're all good you still need to take care of of a lot of things for example uh you run a program you have the input then uh if the user give you the input how can you make sure the

23. 12:14

    input you get from the user is not manipulated or uh if it's uh involve privacy uh is it uh encrypted end to endend encrypted so this is one problem like protecting the inputs uh they it also apply to the output so if it the program give out some output how can we make sure uh the user can trust the output of this program and also say uh pretty similar to Smart contract uh ethereum can be very secure or like a near uh it's very secure it's the very

24. 12:48

    fundamental layer but it doesn't stop people to write a very buggy a smart contract you can have a smart contract with the back door uh it can still screw out your users so there are lot of PS so um in especially uh when using te we're dealing not only with the Integrity this is us usually the case of zkp you just want to make sure you have a proof so that uh as long as you get the proof

25. 13:17

    then the execution should be corrected but in uh te usually we also uh imply that we're also dealing with privacy so privacy is a super super hard problem to uh solve because uh even if everything look very good but uh at some point if the Privacy is leaked you even don't realize okay someone has already got the Privacy well it's even not uh only it's even not only the user privacy sometimes it's the private keys so it even can

26. 13:48

    cause someone to lose money so inherently we're dealing with a harder problem so that's one I would say one of the biggest uh source of the complex so to doal all those complexity in including say proving the hardware is secure the input is secure output is secure and the program running inside is exactly the program people you uh expect to uh run then uh we need a bunch of things including uh say the uh uh remote testation this is so-called te proof you

27. 14:25

    can get a proof so that you can use this proof Pro to establish uh to well easier word is to ensure all the components in this pipeline is good but only the attestation is not enough we still need to have multiple components to make sure multiple areas are secure enough so um uh I do think it's very important to have a uh very standardized uh uh uh security best practice and everyone should follow this practice and of course following the practice is very hard so that's why like we try to put all the best practices

28. 15:06

    into the open source project that's called the dstc and on on top of that we have the near private Mr SDK so uh yeah by giving you know uh the the SDK that minimize the people to deal minimize the effort for people to deal with the Privacy then people can achieve some high level of the uh uh security best practices so this is the start Point yeah so so the way I like to think of this stack is so so te is uh I think it takes quite a while to fully

29. 15:42

    understand the security model of the te right and it's it's one thing for me as a developer to understand it when I build it but the user who uses my product right they need to fully understand this security model because otherwise they do not uh uh like you know if there's five if there five steps that is ensure the privacy and any one of them you know missing means that all the data is public right the user need to fully understand all the steps right and uh the the uh I think the the way I

30. 16:12

    think of this stack is that it removes it it simplifies significantly the mental model right the DC effectively and correct me guys if I'm wrong but the way I think of DC is that uh there's you know like the developer publishes a effective so allows you to run Docker containers inside of the MH CVM so CVM stands for confidential virtual machines right so so so now as a developer I publish a doer container right and uh the the user will establish

31. 16:43

    a TLS connection to the to the U with the API exposed by the docker container right and then the user can be certain that that TLS connection was established in fact with the KE and moreover uh not just any code running inside of the key but the very Docker container that the user has access to right and so then as a user I say well if I manage to establish this TLS connection then a I can be certain the code handling it is this very Docker container right so if I can inspected

32. 17:15

    and if I don't see any you know back doors then I can be certain that my data is handled with the with the code I trust right and then I can be certain that it's running inside of the te right so nether the docker container create nor the hardware operator can can sneak on me you know like can look at my data right and uh right and and so then uh it's much easier to Design Systems running on top of the key right because now the design like effectively it all

33. 17:46

    boils down to the API exposed right so it's an inference server we say hey it's a VM running right you have this open AI compatible API and and you know that it's running inside of the te with a Docker container so your uh you know your connection is encrypted by the kls the code running inside of the key is protected by the key guarantees so the Privacy at this point is implied right

34. 18:10

    so it's a I I like this uh like it simplifies the the mental model for the user right and and then it makes it much easier to design new systems that take advantage of the te because instead of thinking in terms of uh you know like Concepts native to the te which are you know new for users and not well documented they think in terms of the docker containers and TLS connections things that are very well understood yeah essentially uh we want

35. 18:40

    to make everything as easy as accessible to even the non-technical user as possible so uh the one point you mention is really really good is about the uh the TRS uh uh connection uh because usually when people thinking about te proof uh they only think about okay there is a te and my program is running in the te that's all but uh actually um another very important property is that uh when you start talking with this program you need to ensure the program

36. 19:11

    you're talking to is exactly the program you want to talk it's like you get a say from some random people then you want this say to be the right say if you want to like jump into some meme coin right uh very silly uh uh like uh uh analogy um and here uh we borrow idea from ipfs so uh when you are uh uh accessing some content ffs host a static content like a uh website web page uh image or you know

37. 19:43

    video uh and every content has a hash so that is the ipfs CID the hash of the of the file so you just use the hash to access the file so it ensure you know you can always check this file if you hash it you get this hash so pretty straightforward so we borrow this idea that's why like you can have some kind of a hash of your program and you access this URL then you talk to this program

38. 20:14

    yep yeah and so I guess it still has some caveats right so like for example when I look at the docker container right like a good example of a Docker container which is not a good Docker container is a Docker container in which SSH access is not disabled right s access is enabled that means that the operator can SSH to it uh and do nasty things to to my data right so it's still it still requires quite a bit

39. 20:37

    of uh uh diligence in looking at the at the docker container making sure that it's uh um you know secure and it's it's in fact only you know interfacing with me via the API and not in any other way with anybody else but but it still simplifies it quite significantly yeah uh that that's right I'm thinking uh like now it's pretty early but in the long run probably uh people are going to develop similar uh ecosystem or uh I mean uh like a habit

40. 21:11

    of other blockchain ecosystem so you have a program so you need to open source it otherwise nobody know what is running inside you can have a easily have a back door nobody uh can easily detect it and the other hand uh uh we also want people to AUD the code like when using defi project I will not use it if they don't have audit report and at least uh having some people auditing it can give me some kind of uh you know

41. 21:41

    security Fe secure feeling uh so way better than uh nobody else taking uh taking into the codes uh and uh then the problem uh is uh we also need to ensure uh the Auditors can really audit the code and we can link the codes back to the proof so this is actually turn out to be a little bit trickier thing uh because uh uh how people do that in a smart contract is usually you have the code and the the there are a bunch of configurations you apply this

42. 22:16

    configuration to the codes and it will compile to the uh smart contract and if you can do that I can also do the same so everyone can repeat the same U uh the same uh process to get the exact same uh output the compounded smart contract because compounded smart contract is it's just a blob of banner you cannot really see what is inside so having a way to map the source code to the banner is very important we call this a

43. 22:46

    reproducible build or sometimes people call that deterministic build uh and this is uh not uh enabled by default for most of the build system so uh this is also remain a challenge uh but uh I know there are already a bunch of good people working on that say NX uh is a project that can allow people to build uh uh most of uh yeah a lot of things deterministically uh and uh there are also some other ways for example if you uh run the uh JavaScript or python um

44. 23:22

    code uh then uh you don't really compile the code to the banner you just copy and paste all the code bases into the uh Docker image then you can just uh open the docker image to check if those file exactly match the one in the GitHub rapo so those are also the things I think uh U is a challenge but I do think uh people can uh find a really good solution in the very uh near future yeah yeah so the another interesting uh thing

45. 23:56

    to to mention here is that so I was talking yesterday to a friend of mine who is working on a uh like he doesn't work in web three at all they work on like Edge you know deploying databases on the edge right and and one of the problems they have is that you know now that all the databases are in the cloud right effectively Enterprises need to trust those providers you know database providers Etc with their data and

46. 24:22

    Enterprises they really don't like trusting other companies with data right and so I mentioned the is and the person was you know like they were in denial that this technology could exist because if it existed then all the problems would be solved right so so so I think it's a uh we look at the very early stages of you know that technology you know getting widespread not only in the web not only in the web3 space but in the web two as well right I think it's but the web3 will be a catalyst for that I think for that

47. 24:53

    technology to become you know like for all the tooling to appear for all the so so I'm curious to hear what what what use cases you guys see as the like obviously private inference that's something that everybody's very excited about but what other use cases are going to be the you know you're most excited about yeah uh most excited this is a good question uh to me I do think a lot of use cases are really exciting uh I don't have like a ranking for all of

48. 25:25

    them just name a few in the web stre and probably Marvin can also add some uh uh add up some something later so uh first of all uh like from the purely uh web to very deep web web Street side uh uh one thing uh that is pretty pretty interesting is uh NPC so uh you know NPC Bridges NPC wallets and a lot of NPC use cases and it turns out uh because of the limitation of the NPC algorithm usually you can only have a few notes or like a

49. 25:58

    TENS of node you cannot have 100 or even thousands of node so this is a limited uh degree of uh decentralization so with limited node how can we ensure uh the network is still uh resilient uh enough so uh a way to do that is you just put the NPC node into a te then you know if you want to compromise the NPC node you need first you need a few NPC operators to collude together and at the same time you also need to break the te so the two things are completely different so you

50. 26:35

    get another layer of security so people call this defensing deps uh and another I think a very interesting use case is uh well from web uh uh privacy inference is one thing right in the uh pipeline or I mean in the supply chain of uh machine learning you first you need to have data and and then you need to train model then uh inference and the inference result is consumed by the AI agents especially recent days AI agents is

51. 27:08

    pretty popular in web stre so in this whole pipeline we need the both privacy and integrity and also say if we want to uh monetize the model if like we we we put a lot of resources to train open source model and then people just take the model and use it for free without giving back any value to the community this is a problem right but if we can put all the models uh the the data sit the data data prep preparation s uh and the training and the inference all in

52. 27:41

    the te then you know we can protect this model way and the model is owned not by just one person but multiple persons uh all contributors it can be like token holder of a uh you know machine learning model uh Dow or something like that then this can uh make a lot of things that uh doesn't work in the past even in web to uh that can work in webay uh and then once we have the model we can have the AI agent to consume the model uh and AI agent also needs te to ensure uh this is

53. 28:15

    real AI it's not uh human controlled AI so like uh they cannot manipulate how AI interact or uh uh run to provide a service to the user so in this process I think it's also very interesting because uh clearly the other Solutions cannot help a lot uh because you know AI is already really data intensive uh we cannot spend say SS uh times of more uh money just to train a model or run an inference it's already very expensive so te is enabler for those Technologies yeah yeah I want to

54. 28:56

    emphasize you mentioned it but uh but I just want to reemphasize that the for the model inside of the KE you can achieve an interesting uh property where uh the model would be in a state where everybody can get it from anybody else who already has it right so like if I have it inside of my te right and it probably wouldn't be the model right the model probably would be on the dis it just would be encrypted right what I

55. 29:20

    have in the key is the decryption key right then we can have the API that allows anyone else running the same Docker container to get the decryption keep from me right so so effectively the model is fully decentralized nobody can prevent anyone else from getting it right but nobody can use it for free right the only way to use it inside of the KE is by opening a payment channel right and doing the INF right so so that

56. 29:43

    gives you the best of of Both Worlds in the sense that everybody can get the model you can even expose the API to fine tune it right so everybody can get it everybody can Tinker with it but nobody can use it without paying the original developer exactly just like the first ever you know personal owned model just like the first time I know how job boox work right I can own some digital space to storage my uh photos and other drives but in this time like you can actually

57. 30:16

    own yourself a model you can customize it you can import your database into it fing it uh but it's the ownership is yourself and uh if you want to share to others that's your job but uh you could also keep it for yourself I think this is the uh most exciting moment for us y yeah and I think another very interesting application is that um like like you know today so first of all today nobody publishes the training Loop

58. 30:44

    right training Loops of the Kno housee right but I think uh even if someone did publish a training Loop that is no guarantee that the model is actually the result of the training Loop right so like you know if a model is trained by uh like let's hypothesize there is a country that brainwashes their population I don't know if any such country exists but let's assume for a second it does right and they publish a

59. 31:07

    model you would expect that model to be to have the biases right and even if they publish code and the code has none of that logic there's no guarantee that that code indeed was created was something in fact that created the model but with Te You can actually design your training Loop to be running inside of the te for practically no overhead right like with the with b100s I think that is something that Nvidia is striving for to have almost zero overhead right but then you can

60. 31:37

    have the testation at the end that says hey as a matter of fact these weights are the output of an orchestrated effort of many many machines running this very code on this very data right and so so now you you know you can inspect the the pipeline and say well you know there's no biases in fact in the model right and so it's safe to use so so I think this is a very exciting application use as

61. 32:02

    well yeah oh on top of that I do have an interesting idea not sure if anyone else has explored before so uh in uh you know science uh area researcher area uh one problem of the research uh field is that people want to make sure all the research is reproducible but sometimes like for a lab you still don't want to just give out everything uh to the others because you still want to take some Advantage so what people do right

62. 32:33

    now is they write the paper but maybe in the paper they hi some of the tricks when they uh work on the the the paper or like when doing the model training they have some specific uh you know charry picked uh uh input data or some some tricks but uh then it means that if they cannot open source everything nobody else can reproduce this uh research result so probably another thing uh people can do with uh this model is that they can uh reproduce they

63. 33:07

    they can make all the things happen in a te and the te can give a proof uh also the proof of uh the satisfiction of some property in the training process or the data preparation process so uh yeah I think this can also uh be a really value add up to uh the research field because it can remove a lot of like a uh uh uh academic uh you know uh uh fraud and uh can generally make this Fields much

64. 33:38

    easier to uh make progress without worrying about people stealing uh other state other statea or ideas yep yeah I don't know if it helps against cherry picking I can still cherry pick the best inputs and and give you the proof that they run inside of the te but it definitely helps with the like I'm pretty sure that half of the papers published have completely made up or inflated results right so that that is

65. 34:02

    fully soled right like you know you just have you know and you know a proof that that like you know we did have a model that gave that that good of result on The Benchmark right it can prove the existence but not the yeah people can still charry pick data but at least say if I train a model and is said this model have a very high accur accuracy and if the test uh set is open source then uh this can ensure at least you have a model that can match this

66. 34:33

    accuracy yeah yeah it's not necessarily the one you talk about but yeah it's it's at least yeah you can you can definitely you can definitely get like uh it's a spectrum right on on on one end you can publish whole the whole code and the full proof that this code is in fact giving you this result right or like you can say hey this is the code but I also have a set of hyper parameters I'm not going to tell you and with those hyper parameters you get this good good result right yeah right you

67. 34:58

    can have like you know or like at the very far end you can say hey I have something and it's a complete Black Box right I I claim that this is what I say in the paper right but it's a black box I'm not going to show any results but I will still show you that they have something that gives this result on The Benchmark right at least you know that I didn't completely make them up so it's a

68. 35:21

    yeah yeah so te at least gives you tools to be able to be on that Spectrum right because today without the te the only thing you can have is either publish all the code for people to be able to to run it themselves and get the same results right like this is the only way to have any reproducibility yeah yeah yeah okay so we're running out of time but I guess um is is there anything else that you think is particularly exciting that people should know about tees or fall Network or what's

69. 35:52

    coming yeah to my pers I want also want to introduce like several kind of direction we are uh working very hard uh into uh uh both uh from AI besides the uh you know private or user owned model uh we are also very into you know like the browser based uh operator for AI agent I think uh open I present this couple months ago right and uh uh basically nobody really using it every day but I think the specific is probably

70. 36:28

    the perform or the efficiency is not still like the best right but uh we can see how it unlock a a lot of potential ideas and use cases because based on VA uh the agent can read any website and access you know to work for you uh with your personal account with your approval of course uh uh without preprogramming so they you don't need to like integrate apis to have that so this kind of uh uh browser based uh agent adist definitely

71. 37:03

    will unlock a lot of use kit but of course none of us want to dedicate our accounts to openi because I don't want to you know insert my metamask private key into the cloud browser of open AI definitely not so we need some uh security environment to run it I I believe this is a potential uh uh you know collaboration or interesting use case uh from near Ai and uh we are happy to provide the te uh powering this

72. 37:36

    another Direction uh uh is the collective intelligence this means that you know you see how deep see you know change the efficiency of uh you know breakthrough of these uh benchmarks by uh you know using thee mixer of experts in a really smart uh way uh you know change some incentivization know reward incentivization mechanism for these multiple models and change the way how they rank the incentivization layers so I we think uh collective intelligence

73. 38:09

    can definitely bring a breakthrough for intelligence even these a very very low cost I think uh this could be a next direction for multiple AI agent uh uh you know ideas and I think uh uh by using te we can build something cool around it for the um you know for multiple agent based collective intelligence so um yeah this is super possible uh but mostly most exciting moment for us will be will there a second deep seek level of breakthrough

74. 38:43

    from web stre I think is possible for all of us you know include near and other teams uh because you know the the team behind deeps after all is a is is a a Quantum trading uh company back sub company right I be using a very limited resources and make some break through I think this could happen in the crypto space to yeah yeah certainly awesome okay cool okay uh thanks a lot guys for coming and uh for

75. 39:15

    the great discussion uh yeah and excited for um what's coming next thank you Alex uh glad to chat with you yeah thank you thank you for hosting thank you uh yeah all to be here