Whiteboard Session with Illia Polosukhin & Alex Skidanov

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:04

   time blockchain flows near the one we close to won't stop us now together we me toine the game when near we'll find the flame empowered Minds riseing so use your own AI we'll explore hold the future hand in hand in this Tech driven Wonderland okay welcome back to the AI office hours and uh Alex and I here in person at as UI offsite we've been cooking some really cool stuff we're going to give you some Alpha now and

2. 1:04

   then you obviously learn more at redacted uh in a week so Alex let's dive in let's do it okay problem there's a problem uh AI is taking over the world everything AI happens inside of large profit seeking corporations all our data goes into their databases and data sets we like we're repeating the same mistake we've done with social networks the same mistake that we've done with the search engines we need to stop it now um for

3. 1:36

   that we need a plan and we will we will lay out the plan right now um shall we well maybe the other part is uh we you know kind of as a space we believe in open source but the challenge has been that open source doesn't have inherent mization approach right and even more there is actually a a um fact that like if you build something in open source any of the labs can take it and use it and it doesn't work the other way right you have this asymmetry that is clearly

4. 2:12

   uh benefiting any closed lab and uh undermining open source research like case in point elastic search and like dozens of others yeah okay um so let's say that uh you know like GPT 5 is happening any moment now right so so gp4 is no longer the Battleground even though you know nothing really caught up with gp4 in open source but GPT 5 you know will appear any moment now and then clo 4 or whatever version they will

5. 2:44

   choose 4.5 probably um so let's say that we believe that to compete with that we need to train a 1.4 trillion parameter model 1.4 trillion parameter model that is is we we did some numbers with IIA today and we concluded the training 1.4 trillion parameter model will cost approximately 150 million dollar right it's a massive amount of money uh and uh there is a chance there is a chance that we can do it that once we can pull enough resources to train a model of that size for the benefit of

6. 3:21

   humanity right but the the race doesn't stop there so year a year from then or two years from then open a entropic of new models open source does not right we stop Forever at 1.4t the next model cost $300 million to train the next one is 1 billion we have no chance to catch up so so the first problem we need to solve is how do we train a model in such a way that it's economically viable right so people do not just you know donate money to train it people actually invest money into training it and then you know in

7. 3:52

   some way or another it returns Returns on this investment in such a way that it can reinvest to the bigger bigger models at the same time obviously we do want uh the properties that a you can use the model without your data leaving you know without it going anywhere provably right not not just like you know someone said hey I'm a corgi trust me I'm not going to use your data right and we want to make sure that you know if I want to have the model locally I need to be able to get it locally that's problem number

8. 4:23

   one right problem number one economically viable problem number two is e Eco would be a coer problem number two is um consider lama 405b lama 405b presumably is almost the same size as the best models today from proprietary providers but it's not as good right I don't think anyone would say that Lama 45b is as good as GPT 4 so what we learned from that we learn that it's not that easy it's not just

9. 5:02

   the parameters right there's something else right and in order like like if we train if we get the money and we train the model and we can monetize it it's still probably not going to be competitive right so we need a way to create a competitive a competitive model competitive and open a has you know the best research is they go to open AI uh and they have in a single place they have all the data all the models all the resources all the experiment results from the past uh all the

10. 5:38

    expertise in a single place right in in the open first of all it's much harder to get the same number of the same quality of people concentrated in one place uh but second of all even if we bring them they don't have they don't have access to to that so yeah let's talk about Solutions let's talk about Solutions and begin with let's begin with an interesting one how do we make how do we train a model in such a way that uh everybody doesn't

11. 6:08

    just use it for free right but it is but it is privacy preserving and available to everyone uh this is where we use the magic of a technology called trusted execution environments so on uh modern processors uh like generation 5 Intel Zer um there's a technology called TDX that effectively allows you to do something that sounds like magic it allows you to run it allows me to run code on Ilia machine even if I don't trust IIA at all whatsoever in a way

12. 6:42

    that I know first of all the code that is running is in fact code that I think is running and two IIA cannot see anything all he knows is that something is running but he cannot see the code he cannot see the data uh he cannot see anything so um the way it works is uh like in the in the processor itself there is a you know volt and you know which allows you to uh you know which runs like there's a barrier between that Vault and and the external processor so it doesn't see anything that happens inside and then

13. 7:16

    within that there's a key that Intel knows right and so it can sign and attest to a lot of things and then Intel will tell you that the testation is correct so it's not quite as magical as it is because we have to trust info but but this is way better than trusting random people running servers on the internet right so TDX and you're trusting Intel to effectively print chips yes

14. 7:42

    and already yes yeah you're not trusting them with like running you know passing data Etc and we're already trusting them indeed that they going to make chips that are following specification actually doing the computation we need right and so TDX is for processors and starting with h100s the latest not latest anymore right one of the latest gpus from Nvidia they have the same

15. 8:03

    thing right meaning that uh if your processor is TDX enabled and your GPU is at least h100 your GPU computation can can have the same properties right so you can do that immediately allows you to do private inference right so like Ilia can spin up a private inference server and I will send you know uh I will send a request and I will be able to know before I send the request I would be able to verify that IL will not be able to spy on it and after I send it

16. 8:28

    you know I get the response throughout I know that as long as Nvidia and Intel do not conspire with Ilia uh Ilia has never seen my data which is great right this is something that on itself is already value even even even if even before we go yeah so effectively you can a verify that Bob runs a uh TDX enabled image that is exactly the specification that you require and for example the specification is a uh AI inference model

17. 9:04

    and you can actually get a key with which you can encrypt uh the data that only TDX can decrypt uh run inference and respond to you back with encrypted with your uh with your key meaning Bob only sees you know request response with uh uh effectively attestations and encrypted data going in and out right so how how do we use this technology to train our 1.4 trillion parameter model in the way that it is U economically

18. 9:37

    viable what we can do is we can um we can train the model fully in tees uh te stands for trusted execution environment so those volts are they call tees so we can train the model entirely in tees so at every step every parameter update every forward backward pass everything happens inside the te so at no point any of the participants of the training see the actual weights right so at the end of the training what you get is is is quite interesting you get a model for

19. 10:09

    which you know the code that was used to train it and you know the data you know all the data preparation pipelines could all be open source and moreover you can prove again as as long as you trust Intel that the model you have is in fact produced by that code using that data something you cannot do about you know with now none of the models today have that property uh and then that model it it is encrypted nobody has ever seen the weights the only way to get the weights

20. 10:39

    is to use a specially crafted virtual machine that is running inside of a te and that specially crafted virtual machine is guaranteed to give you the response to your prompt for as long as you establish a payment channel on year with it uh and pay a little bit of like a tiny uh amount of money right but uh you know tiny amount of money accumulates very fast if a very large number of people

21. 11:03

    uses the model so as long as it is actually very good 1.4t that's a very viable approach right so it's it's a model that has the properties that um you know how it was trained you know exactly the code if there's any bias you can verify you can tra track it down um you can have it locally you can spin up your own machine with the last generation processor and the h100s and you can pull it like you know the the VMS just have a protocol where one te will pass it to another right so as long as you know anyone who

22. 11:34

    has it you can get it to yourself or I mean it can be encrypted in De centralized storage and you can pull it from there and only your te can decrypt it well te needs to talk to another te to get the key yeah but yeah the model the model the encrypted model is somewhere the key is only to communicate to to pass the master key uh and um yeah but at the same time there is no way to use that model without actually paying for the usage so that is

23. 12:07

    a it effectively solves all the problems right it it it creates a way an economically viable way for the community to build Frontier AI models you know to to fund them to build them to uh to monetize them and by default fascinatingly by default it it has private inference and more moreover you can build it in such a way that there there's no way to use it without private inference so it's a uh you know so like there's no way if it's built properly there's no way for someone to shoot themselves in the food right whenever

24. 12:39

    they use that model the inference is always by Design private you can also obviously fine tune it privately you can find unit on private data and so new kind of derived versions can be private you can also guard let's say you find tunit on your data you can say that only you effectively can use it or you can share and monetize on top effectively you know when that model charge there is like you know set of kind of distributions that happen part to the

25. 13:10

    hardware provider part to the people who build the original base model and then Part Two Fine tuned it and you can keep stacking it right and so it's effectively like you program kind of for every inference you can program how much uh of different parties will receive payouts Okay cool so that being said uh if we don't have the same research as open Ai and entropic the model is still not going to be great we will we will spend 100 million to train

26. 13:44

    a model it will be it will have all the properties just described but it's not going to be good so how do we make it good uh effectively the idea would be to uh uh to build first of all we want to build a Central but decentralized Place uh where all of that is pulled together right like one place where uh you have access to compute to code um to to data uh to agents like effectively a large H Hub uh

27. 14:15

    which stores you know like like you go there you publish it's like decentralized hiding place right it's a single Hub uh where you can publish resources you can find resources you can with the help of tees you can also monetize resources but that's let's not get there just yet right and then we would bring for every model we would we would iter go from small models to the large models and at every stage we would say we we would have a competition right we will bring a lot of people we will have resources

28. 14:41

    allocated for it uh and you know the researchers show up uh and they compete their goal is to create there's a benchmark uh that is uh that is that is used to measure their progress right and uh you know and and you begin with with a smaller model like let's say we begin right now in our little calculations uh we use the size of5 billion parameters I'm pretty sure you cannot see it there .5 billion

29. 15:12

    parameters right so it's a very small model which has a property that you can train it um uh on one machine on one machine in one day to to an amount of data which is very reasonable for a .5 billion parameter model right and so uh with such a size we can have we can have a like with a with a reasonably sized cluster we can have a competition with a very large number of participants to

30. 15:36

    whom we can provide compute right so they um so that's a first step and then at every consecutive step the model gets bigger and bigger right and like right now we we we are considering six seven steps to get to 1.4t uh but at every Point only the top K participants Advan to the next round so the number of participants is smaller the number of compute that we have is bigger and bigger we're going to be accumulating

31. 15:59

    more and more compute as the model gets bigger uh so it's a you know at every step it's viable for that sort of participants to come up with the a great model and and when we uh when we say great model what we mean is data preparation primarily data preparation I think the biggest differentiator between the model quality model qu models qualities is how well you know the data is duplicated how well you know low

32. 16:24

    quality data is removed how well you know what sort of data you augment M your you know internet dump and gith GitHub dump with Etc but then also yeah hyper parameter tuning so like you know learning rates affected quite a lot in other things uh Optimizer tuning uh in parameter initialization so there's there's plenty of things you can do uh that that will affect like gradient clipping and yeah uh and starting go ahead yeah I

33. 16:57

    mean I think the interesting part here here as a researcher who participate in this competition you have a fixed budget of compute and so now if you can write a more optimized code you now have a leg up because you can actually process more benchmarks so this naturally also uh uh invites people to optimize and kind of contribute to have a more performance code as well as as models get bigger you can you will have access to more machines and uh including in the kind of

34. 17:27

    distributed cluster they not interconnected with uh kind of hyperconnectivity and so you can also like competition naturally starts to uh El incentivize figuring out how to do distributed training decentralized training right yeah up to certain size of the model up to anything that is trainable on 4,000 gpus let's say give or take uh it is still relatively easy to get a cluster of that size right but for for 1.4 trillion parameter model the clust size is getting you know to like

35. 18:01

    you know High tens low hundreds of thousands of gpus meaning that you know like El and couple others have this cluster but for us it would be relatively challenging right so our options is either to well go to elen or uh or pull together multiple or pull together multiple clusters right and find a way to train across clusters and that is uh I think you know like the research is happening there are

36. 18:25

    multiple companies in the ecosystem that work towards that and uh either they will figure it out and we will just work with them or or we will also start you know in the same format like this competition LS competition format LS very well to figuring it out right and then uh at some point U I guess we mentioned it multiple times already but we have a active project on near that

37. 18:48

    you know where we explore data notation right so once we move to fine tuning uh today you know I it's not very well known how those big on which data sets those big models are fine tuned right so like gp4 clae llama we know they are fine tuned on something we don't know on what uh but u in a in a decentralized manner there's a very good chance that we can we can create very good data sets for fine tuning uh very quickly right so similarly to how uh in the pre-training

38. 19:21

    step researchers will have access to data to data uh sorry to Computer Resources uh in a fine tuning step in in the fine tuning step they will just have credits to launch arbitrary data anotation tasks and there will be a pool of people ready to to pick them up right and so you know like at the fine tuning step now you can um you know compute wise it's a very cheap job like find tuning a model you you find tune on a

39. 19:46

    fraction of the of the data but that data now needs to be extremely high quality this is where data annotation potentially is or like synthetic data is potentially way more important than filtering down massive data sets right so it's uh or or you can have this directly in with your model for our lhf where you have humans feedback directly into the model and so yeah so the process is effectively there's competition going the 05b is ongoing

40. 20:13

    competition there's finalist decided based on the kind of finalist submissions the production run is started for that size of the model because you want to train for longer for kind of uh produce and then fine tuning starts uh from that as well yep and then somewhere along the way we figure out um uh similarly we can uh like in this competition format we can effectively ah I'm I'm being ambushed here okay um okay that we will need to cut

41. 20:51

    out from from the final version in the same in the same competition format we can figure out a lot of other things such as multimodal uh voice I guess voice is multimodal um as we mentioned uh training over slow networks um potentially um and oh yeah like like Chain of Thought things like you know like open a strawberry again that would be an interesting competition and uh that's actually something we're working on right now where um we we will be launching relatively soon one of the

42. 21:26

    competitions where uh the goal is to you know Sol solve interesting math problems using Chain of Thought or some sort of reinforce reinforcement learning so it's a yeah there's other types of competitions kind of on on a side that argument the main push for the Frontier Model y so that's the that's the plan that's the plan uh and we started executing on it relatively recently but full speed uh there's going to be more

43. 21:57

    Alpha at um which happens in 4 days six days nov 10th yeah November 10th cool what else do we want to cover here well I think the maybe interesting questions I think the this part I mean we talked about like data prep curriculum learning back at the first whiteboard um I think the interesting question to kind of highlight a little bit of the existing work on decentralized training and uh yeah yeah okay so decentralized

44. 22:36

    training first of all there are existing efforts right on top of my head I know of noose noose yeah yeah noose and Prime intellect uh the former I don't think their approach is is known this is only a technical report uh but the latter is building on top of a technology called doco uh which is a uh Deep Mind paper uh in so I guess interestingly Google for some reason is interested in a training over slow Network I guess there clusters are there clusters all over the

45. 23:08

    place I mean they have potentially hundreds of data centers yeah uh and uh we had a reading group with Deco we have a video we will publish it soon so we will have it's like a one hour discussion uh diving pretty deep uh the paper the paper itself is relatively short but there's a lot of experiments and those experiments have caveats so that's an interesting video stay tuned uh and then on top so Deco Deco is very straightforward they effectively just change the optimization process ever so

46. 23:39

    slightly so that you train the model across many different clusters but it still has a requirement that every cluster can fit the model right so effectively you cannot have if your model doesn't fit on a single box it means you cannot just have a you know like a network of disjoint machines steal every node in the network would have to be a small cluster with infinite band uh but the uh the follow-up paper from Deep Mind called deaco yeah so maybe just effectively

47. 24:10

    have replicas of the model that are um updating on different parts of the data set and then there's a process that sends their accumulated updates to a parameter server which runs a little bit different optimization process to update parameters and then now kind of new parameters are being sent back so an interesting thing in doco is that they show that if you only keep one replica but still use dco instead of just training

48. 24:46

    using Adam they see better results than just training using Adam which is slightly suspicious but uh it's an interesting one of the interesting things in the experiment section in the paper so this is example by the way of right now the the way research is done when you're observing outside you get effectively a paper which is a snapshot of work you get some benchmarking that happened on their experiments and you kind of need to decode whatever the parameters and what are the different tradeoffs that they made the idea was

49. 25:18

    kind of open competitions especially on this uh effectively infrastructure that allows to uh validate that exactly this code and exactly on this data was run you can you know try to reproduce this experiments you can actually show uh what what is the differences and people can go and actually look at it and fork and modify and build on up and so that's really right now the lack of this

50. 25:44

    infrastructure is hindering open source development as well as you know reading papers like you cannot reproduce these experiments effectively even if you had compute and if you don't have compute it's even higher problem for any anyone who's trying to do re search in in this area and I guess there are papers with open source with open with open source code uh but it's not as simple like the beauty of this Hub that was on the

51. 26:10

    previous page is that the way person submits their job is like you know they say near I submit right and it goes to the worker and and the worker you know has a very specific way of pulling resources meaning that for someone to reproduce it all they have to do is to say near I submit right and they know that you know it will go to the worker be able to train so so so yeah like iterating on experiments is very simple right because once you work an experiment you know

52. 26:34

    that you know that in the it's current state works right so like if you while you know if it's an open source repo you need to reproduce exactly the setup in which they were running it you need to pull the data in exactly the way they expect expecting it to be pulled Etc figure out the dependencies libraries hide cool yeah I know yeah and uh yeah so so there's a followup paper from Deep Mind called deaco which tries to solve the problem with models not fitting on a single machine

53. 27:05

    and that one is way more involved it's not just you know a small change in the optimization step and the experiment section is a um if you try if you like try to extract Apples to Apples comparisons you will notice that de paku is not doing as great which is probably sort of something you would expect when you um because it's only using a subset of parameters on every pass so but it is it is a great forward first step right and like from here again in in a form of

54. 27:34

    a competition we can push it forward and hopefully figure out a way to train a massive model in a in a decentralized manner though I I guess it wouldn't be as important for us that you know it fits on a single machine I guess if people are spinning up machines with h100s and uh you know Gen 5 CPUs they probably can can have small clusters that's a that's not an unreasonable assumption right if you go to GPU list AI there's plenty of those small clusters it's just that you know having 100,000 of them in one place is a

55. 28:06

    challenge right but yeah I guess like having like tens of them in one place is quite reasonable yeah and for 1.4t you effectively need something like 15 machines with 8 gpus each yeah to fit so yeah you kind of the minimum size of a cluster is that yeah yeah so deaco might not be necessary right like like we we can it can be just you know like the idea would be you know the participants

56. 28:30

    of the training have 16 16 node clusters of which we there's plenty right now you know like Lambda um exibits a lot of them will will give you clusters of that size cool what else is interesting well let's talk a little more about about the Hub so oh yeah let's talk about uh this interesting idea so so I I kind of U discarded this idea of the not discarded but brushed off brushed off the idea of

57. 29:17

    the U monetization right but um a lot of data is you know people have proprietary data that they would like to monetize quite quite a lot of it and it's it's a very hard thing to do because once your data once someone else has your data they can U you know they they have it right they can replicate it they can use it for free like it's fully right now it's fully reliant on on licenses and people you know owning licenses uh but

58. 29:48

    te change it quite drastically right so you can have if you already have a place where all the data all the code leaves right you might as well and that place happens to be on a blockchain where where money is a native primitive native primitive yeah you might as well go and uh introduce monetization where you say hey this is my data set and you can use it to train your models but you can only do it on the TDX you know data set is

59. 30:11

    encrypted and only inside the te right and and and every time you do that you have to pay me right and if you use a subset of it you pay me you know like the same same thing with the payment channel right so suddenly you can monetize your data and be quite certain that your data is not going anywhere uh same thing applies to potentially smaller models right so it's one thing where we train uh like you know the

60. 30:36

    model on the path to 1.4t people might have some other ideas specific to particular use cases where they train a model based on their proprietary data Maybe proprietary ideas right they can publish it and say hey guess you can use my model in a very same way you know establish a payment Channel with a key I get paid every time you use it but you can even have it on your machine you know as as long as as I get every time

61. 30:58

    someone uses it uh it's great right so that effect allows you to have a marketplace around all the data models agents um yeah so the benefit the TDX allows to run uh potentially arbitrary code so your agent usually is some program that has some AI element LM element and has potentially State machine and lots of kind of execution so this itself can also run inside so you have private execution over potentially private code of the agent although there's like set of assumptions that you

62. 31:38

    need to apply or limited to accessing internet for that and this allows for example to have marketplace where you say hey I you know I want somebody to do my taxes there's multiple agents that are bidding on this you select one you upload all your tax documents which you would never never give to any you know particular party and now they're able to run interpret generate tax return and uh kind of return it to you and neither people who wrote the code for this agent

63. 32:10

    people who trained this LM or where it executed were actually able to access the data that was in Flight yep and you know because compute is part of the of the of this Cent how do we call it a hub right yeah part of this Hub uh you don't need to have any infrastructure locally to do any of that right you just show up there and say Hey I want to use the data potentially to you know fun a model or like use a fun

64. 32:36

    model all of that happens you know in a in a click of over a mouse button no need to no need to have infrastructure locally or if you have infrastructure locally you know you don't have to trust even with the you know like in some cases you literally want your machine to be offline separated from the internet even that use case work despite despite the fact it's a payment Channel you can structure it in a way that the

65. 32:58

    payment channel is pre-created with slightly different parameterization uh and then you know you just put it on like a you know a flash drive or a q code go to your Vault yeah or like on a QR code yeah go to your Vault enable it and for as long as the payment channel is not depleted you can use it completely offline so that's a yes I think I mean that's a really cool feature because yeah like the payment channel that you establish here you

66. 33:22

    effectively prove that you paid that you deposited some money on a blockchain and so now GDX can verify that and this can be verified offline but importantly it uh it to close a payment Channel you need to request a kind of closing message from GDX and that allows to as well uh to kind of to prevent double spends uh in in this payment channel so so generally with the payment channel that that's actually an interesting technical thing uh not not particularly

67. 33:52

    rocket science but is interesting right because usually if you uh and your party want to have a payment Channel you don't trust your counterparty right so you need a protocol where you can like generally the way payment Channel ends is that uh you know we both sent to each other a message saying we're ready to close the payment Channel and then any of us can close it by providing this transaction but because we don't trust

68. 34:15

    our counterparty you know we cannot expect them to send us that message so there's always a protocol for payment channels where I go on the blockchain and say I want to close it now my counterparty you know is not cooperating uh and if I do that I might be the malicious actor I might be trying to close the payment channel in a state that is stale uh right where the more there's a more recent State uh where I

69. 34:38

    paid more money right and so in this case my counterparty must be able to go on chain and say you know like like there should be a grace period where the counterparty have to be able to challenge so you know that natural doesn't work if the TDX is offline right if it's offline and I started a payment Channel I paid it for a lot of inference and then I go back to the blockchain and say hey the payment channel is actually closed at the very beginning I never

70. 35:01

    paid anything right and TDX leaving in isolation cannot do anything about it right but the thing is that we absolutely do trust the TDX right so as long as it doesn't crash as long as it is online we know it will sign the message uh if we ask it to do so right so we can change the parameters of the payment channel to say that we cannot uh close it using you know a message which is is not authorized by the counterparty right and so this way

71. 35:31

    um like a fact the the thing inside the TDX will say I either need to be online uh when we're doing inference right I need to be able to check that like at least as of right now you haven't tried to close the payment channel in this case you can open it with an ability to close without a closing message like a standard payment channel uh and and you would be able to you know even if the TDX crashes you still can't close the channel you know it will not be able to challenge you or uh you can you can you

72. 35:57

    can go to the one that is offline it's like I'm offline but the payment Channel you're trying to use does not allow you to close it without me authorizing it so I will gladly use the payment channel right and so you know that that the te will close the payment channel for you if you ask it to do that uh and the te knows that you will not be able to close it without its authorization right so then it can be completely offline and

73. 36:20

    your your only risk is you know if it crashes if it gets wiped out but then you know so so that's that's a tradeoff right you choose how much you want to put in the payment Channel understanding the risks but it's a it's a viable viable solution and for those who are coming from the eyes side the payment channel here is effectively paying for credits for using the inference and so

74. 36:42

    uh it just in this case like you can I like if you have your own your own gpus right you don't need to pay the cost of gpus you just pay whatever the cost of the model you run and if you are running on the kind of shared Cloud uh then you know you're paying for the cost and that presumably will be still dominating the price most of the time uh it also this this system also kind of uh motivates people to optimize inference code as well because now you

75. 37:12

    have kind of unified infrastructure that everybody can push and uh and benefit from yeah there's a Marketplace if if your if your machine is if your influence is cheaper you can you can have a lower price naturally uh and so yeah so so then you are able to effectively just buy some credits for any machine and uh use them and just kind of routes you for like per request or per batch of requests uh to machine or mini cluster that actually can does inflence on them

76. 37:46

    Y and the payment channels also you know a great thing to have for agents to pay each other that's a that's also A Primitive that will be available to to people and I I think one important thing to mention here is that uh it's not necessarily our goal to go and build all of those components that's a lot of work right but these there are there are for each of them there are already people building them within the ecosystem right so like you know exhibits is building compute and there's

77. 38:18

    no reason not just to have you know like that is just exhibits right and so for every like thanks to the blooming ecosystem Within the on the intersection of artificial intelligence and near uh most of those things are already happening right and so you know the Hub will be just a way a place where they all come together come together and effectively creating this uh evolutionary pressure to yeah keep keep improving keep

78. 38:50

    improving in the coordinated way yeah I think one one of the interesting yeah meta points is that there's a lot of excitement in a lot of people doing different parts of the work but indeed it's like right now everybody needs to collaborate peer-to-peer right like I'm I'm you know working on agents uh I need some computes so I go to exhibits I need some you know new model or I want to run some experiments to fine tune I need to find somebody who can do that and so you

79. 39:15

    have kind of lots of overheads like coordinating this effort and again this is this is how as centralized companies are effectively able to move fast is because like this is all in one repo you know all computers in one Cloud that they use and everybody's in the same slack and so you're able to you know move way faster with this so we're trying to really bring a lot of the same uh kind of principles but allow to combine all of the resources of existing

80. 39:46

    companies as well as existing talent in the open source ecosystem by matching them with the resources like like a good example is uh so so so one one part of the that works already very well is the inference right so we have an inference router where if I want like you know I need a completion from llama 405b it's not as important for me who is doing the completion as long as it's the right model right so we have a router

81. 40:11

    right that can go to fireworks which is uh web you know Web Two that's how we call the things Web Two inference provider we have hyperbolic which is web three right and uh you know for me what matters is that it's cheap and it's correct right so so inference router will will will pick the best route for me and you know inference is a it's very hard to fully win the market for inference right like I'm pretty sure hypothetical hyperbolic would prefer

82. 40:35

    that everybody just goes to them and there's no router right but because inference is a competitive market it's great now right that uh you know developers don't need to integrate with 100 uh places right but but also you know hypothetical hyperbolic as long as they provide the best quality for the best price they will be naturally picked you know by the router or by people in the config so it's a um you know it's it's a win-win for

83. 40:59

    everybody you know easier for developers companies capture bigger share of people as long as they good um ET right so like inference I guess inference is here as well you know like inference is one part that that that you know of the Hub that at this point works very well and is integrated with the you know Downstream providers all right well I mean this is a lot already yeah yeah I think that was that was a good chank of information so um yeah I mean there's obviously a lot more going to be coming

84. 41:41

    out as as we develop this and as well as a lot of projects in the ecosystem developing things the I kind of some other interesting things obviously is that agents himself are um kind of through this model can have set of effectively Secrets right this can be private key this can be um API keys where they can act uh on either their own behalf or on behalf of the users so you can either delegate them some permissions or you can have a fully autonomous agents right which actually

85. 42:14

    are not indeed like behaving on behalf of any user right this is effectively a smart contract like agent uh that can act on its own behalf and now transact um uh it can raise its own money and pay for its own inference in the future yeah not not only by providing my secret not only I know that you know the owner of the machine will not see it I also know that the agent that is using it is actually the agent I needed to be right so that's an interesting property that or like you know if I'm relying on a

86. 42:47

    particular response I know that it was an agent that they thought it was uh it was supposed to be right so T te is also enabling quite a few use cases around agents so say all in all you know T ke is a magic yeah I mean important like this TD like the previous generation of Intel tees were very limiting and kind of had a very limited use case and had issues and had a lot of issues this is a new generation that has uh it's much

87. 43:20

    better yeah a way more kind of generic framework which allows even to run the full operating system inside it not but it's it's like the way you're supposed to use it yeah and same for h100s effectively the first uh gpus that supports working directly with TDX yeah um I think the other maybe uh returning back to the economic viability right viability viability uh the interesting thing here

88. 43:50

    is uh effectively you can imagine a few different models of pulling res Computer Resources for this um and you can have effectively per model token where you're raising funds as well and through that then you know obviously it's a token so you can actually um resell it before it starts operating and uh making Revenue but then and you know this revenue is distributed to the uh token holders and so that is all kind of been done through payment

89. 44:34

    channels repaying back to the uh to the Token holders so effectively creates kind of an autonomous way of pulling resources uh starting a training run and then with that paramet encrypted parameters then repaying back as they used for each million to in effectively the cost that was set by the Dow and the cost can actually go down right over time as this model maybe become less viable and then over time it can be open sourc open that's viable it just there's

90. 45:08

    a better model now yeah better model so it makes sense to kind of discount the price uh and then also like over time potentially can open up parameters as well if if that's a like token holders uh wish which would require effectively re-encrypting data um or scripting data through the GDX yep yeah I think that's uh that's a lot of alpha already that's a lot of alpha yeah I think I think

91. 45:40

    that's so so that's uh U yeah it's hard to put in words how excited we are about this this direction so uh uh yeah the the future is pretty bright uh it's almost it's near and it's yeah and and it's definitely near yeah I think yeah the the the kind of the thought we've been trying to figure out is like given the user ownership right given how do we have this guarantees that we want and

92. 46:10

    privacy we want you need some Machinery to ensure that and you need a way to pull kind of talent and resources and this is actually all the pieces coming together kind of enabling this and uh really moving us forward so yeah really excited and uh more to come at reacted yeah