{"id":"r0pweLNP21w","title":"NEAR AI Office Hours #27 w Prime Intellect - Find compute. Train models. Co-own intelligence","description":"Illia & Alex discuss the latest AI research topics and unpack some NEAR.AI focus areas.\r\n\r\nGuest\r\nJohannes, Co-Founder\r\nhttps://x.com/johannes_hage\r\n\r\nPrime Intellect https://x.com/PrimeIntellect\n\nIn this episode of NEAR AI Office Hours, advancing decentralized AI model training. Learn about distributed low-communication training (DLOC), scalable compute aggregation, and the challenges of building open-source AI models. Discover how Prime Intellect is reshaping the way we approach artificial intelligence.\n\nTakeaways:\nDecentralized AI Training: Prime Intellect is pioneering decentralized training methods, making cutting-edge AI accessible beyond large corporations. Their distributed low-communication training (DLOC) reduces communication overhead during training.\n\nEfficient Resource Use: They aggregate global compute resources, tapping into underutilized GPUs to train large models at reduced costs while maintaining scalability and reliability.\n\nOpen Source AGI: Prime Intellect's vision is to create an open-source, decentralized artificial general intelligence (AGI) ecosystem, fostering community collaboration and reducing monopolization in AI development.\n\nChallenges and Innovations: Key challenges include memory limitations on nodes, scaling contributors, and achieving reliable decentralized frameworks. They’re addressing these with asynchronous training and advanced techniques like quantization.\n\nFuture Plans: The upcoming Intellect 2 aims to scale model sizes, leverage advanced optimization methods, and potentially introduce heterogenous compute setups, marking significant steps toward decentralizing AI on a larger scale.\n\n\r\n0:00 - 0:10: Introduction and opening theme\n0:11 - 1:27: Overview of Prime Intellect and its mission\n1:28 - 3:20: Aggregating compute resources for AI model training\n3:21 - 6:05: Challenges of trust and reliability in decentralized compute\n6:06 - 9:55: Distributed training frameworks and DLOC innovations\n9:56 - 13:10: Scaling models and data efficiency in decentralized setups\n13:11 - 16:00: Data composition and benchmarks for Intellect 1\n16:01 - 18:10: Future plans for Intellect 2 and scaling challenges\n18:11 - 23:10: Overcoming training hurdles, including fault tolerance\n23:11 - 29:10: Decentralization in AI and pipeline parallelism\n29:11 - 35:00: Monetization and the role of open-source in AI development\n35:01 - 41:00: Trusted execution environments and hardware considerations\n41:01 - 45:00: Vision for decentralized AI and community collaboration\n45:01 - End: Closing remarks and call to action for engagement\r\nNEAR is the Blockchain for AI\r\nWebsite: https://near.org/\r\nNEAR AI: https://near.ai/\r\nIntents: https://near.org/intents\r\nChain Abstraction: https://www.proximity.dev/\r\nSharding: https://nearone.org/\r\n\r\nFor the devs\r\nGitHub: https://github.com/near \r\nDev Docs: https://docs.near.org/\r\nDevhub: https://devhub.near.page/ @NEARDevHub\r\n\r\nConnect with socials\r\nTwitter:   / nearprotocol  \r\nBlog: https://near.org/blog/\r\nMedium:   / nearprotocol  \r\n#NEAR #nearprotocol #NEARAI","channel_name":"NEAR Protocol","published_at":"2024-12-03T00:00:00.000Z","thumbnail_url":"https://img.youtube.com/vi/r0pweLNP21w/hqdefault.jpg","transcript_text":"blockchain flows near the one we're close to won't stop us now together we'll redefine the game when near we'll find the flame empowered Minds rise and S use your own AI we'll explore hold the future hand in hand in this tax driven Wonder okay hello everyone uh it's a a new episode of near office hours uh today with us is johanes from Prime intellect um I uh I learned about prime prime intellect last October when you guys started training intellect one uh and mostly what I knew was that you're training a large model using Theo but I've been reading up in the last couple days leading to these office hours and I don't know I don't know how we going to squeeze all of that into one hour that's a a lot of exciting things uh that U there are to discuss do you want to give a quick intro on on yourself and the prime intellect yeah for sure happy to do it and yeah nice to meet you Alex um so yeah I'm johanes one of the co-founders of Prim and uh yeah my background is more on like uh J side so I've been um yeah turning Foundation models that a yeah Foundation model that appear in Europe since end of 2021 and um yeah basically what we do at Prime intellect is we aggregate Global Computer Resources from all providers all data centers and have them yeah all in a single platform as well as um yeah working on um yeah training distributed across all those different clusters so we are making like um yeah working on fundamental research to make disted tring work in scale and um yeah obviously had the intellect one launch where we actually scaled up our like research that's been we've been doing since uh end of last year basically uh to the 10 billion parameter size and uh yeah from here on um on a scale it up further obviously and uh yeah generally are interested in all types of uh research directions which are important to make open source AGI a reality so not only on The Descent training side which will be one um yeah important part to make like last compute resources available obviously but also um all other research directions from like large scare synthetic data generation to um yeah obviously not just training a pre-t trend model but also maybe going more into this test time computer action yeah okay so I I suggest we begin with the lowest level and go up uh uh you know climing up the master plan right so so the first step was to to pull the compute and and that's something that many companies do right like there's a cash Network exhibits hyperbolic so my understanding is that Prime intellect is sort of orthogonal to all of them right you're not trying to compete with them uh you're trying to take advantage of all of them I saw hyperbolic was participating uh in training of intellect one right so so how do you see uh the role of prime intellect here and the how does it work with all of those yeah on the on the aggregation side I would say um we are a bit on a on a layer on top in the sense right just by being more like a computer exchange right like having all those providers on the platform basically and letting people like tap into the cheapest compute resources through it right um so we have a bunch of like of the centralized like new Cloud providers on the platform which have like varying uh like availability right uh like often it's been a joke over the last couple years right that lamb Labs didn't have like the h100s available to train U AI models or something like this right but somebody else might have it right and uh yeah that's the approach we've been uh going for there like having more of like a computer exchange basically you can always tap into the cheapest computer resources and yeah obviously uh um yeah we we aggregate all of the providers right from like the repb two players to also more like an aash for example is on our platform we have an easy like interface to to make use of it and uh yeah just give more access and cheaper compute to um Source developers right and so my understanding is that uh the the interface with the provider right is that they give you an SSH access right your SSH to the machine do whatever you want right and there's a but uh like naturally if you think about it on a very high scale right it's uh there's certain trust that needs to be between you and the provider right that they will not you know give you inferior Hardware pretending it to be Superior Hardware and whatnot so so right now there's no attempt to fix that problem right like we just asse that if they give us a machine with 8 h100s then it's a machine with 8 h100s right yeah in a sense right now we rely a bit on the provider in a sense right obviously we make it easier for people to to know which one is a trust Prov trustful provider in a sense having Community Cloud offerings and a secure Cloud offering um the other thing is uh yeah um it's not only about like trust uh or like trust's computer in that sense right it's also like reliability of a provider right um especially if you're an AI researcher and want to scale up things um yeah to more than an h100 node for example there is uh yeah a lot of considerations basically that some of the providers fail way more often than the other ones right so uh there's been some interesting reports of that in the in the Lama 3 paper and so on right um that uh yeah how reliable different providers are right um so uh yeah I think that's another component right those two reliability of the provider as well as uh yeah can you trust the compute resources it actually giving you an h100 right so um yeah yeah yeah okay and so like the next step of the master plan was to build Prime I guess it's called right u i don't it's a Transformer implementation right an Optimus Prim is a Transformer I don't know if that's intentional or not um but you know where the name Prime intellect is coming from no I don't know where is it from um there's a nice sci-fi book about it basically um about like an ASI um that's called Prim um this where the originally name is coming from yeah yeah I I'm reading ungodly amount of Science Fiction but I haven't heard of that one yet so that's a that's a I'll be sure to read it is it a good book um ah I'm GNA get cancel for um it's uh it's definitely interesting in a sense I don't agree with the whole conclusion obviously there but uh um it also Paints the interesting ASI future where everybody has everything everybody is Immortal in a sense but they also don't have anything to do in a post I would yeah it's not a yes it's yeah but uh so I saw so so there was a paper right you first published the open decal paper and then there was a technical report later I I guess very recently right uh and technical report has more things right there's fsdp and whatnot so I guess let's dive into all of those so let's begin with dilco right uh can you give us like a quick overview of what Theo is and why it is needed for sure to to give like a timeline there basically so um end of last year it's a it's a work by Google deep mine called distributed low communication training um which is basically a technique uh they've published for um yeah doing disability training with relatively low communication requirements so um they have yeah they train their models on different islands of devices basically and then usually how you have to do it in your distributed training setup you have to Sy after every step in your training basically so every step you have to send quite a bit of data basically of the gradients uh in like a an all reduced fashion and um in this uh paper they basically show they could get the same performance out of it by only syncing every 500 steps and yeah are training independently for about 500 steps on each of those machines and then they sink between those and uh yeah do an auto optimization Loop basically to uh yeah still achieve the same performance as you would get in your centralized training setup and um yeah maybe to to to go to open the Loco basically for for the timeline there b uh so we started reproducing that uh deep mine work at the beginning of the year so um yeah at uh yeah I think in April or May or something like this we we launched open de loo which was the first replication basically of that work and we scaled their work to the 1 billion parameter size because in their paper they only showed it it was working until 400 million parameter model sizes so we scaled it up to the 1B size and um yeah then from there on obviously did more research on the on the algorithm side of things obviously so um yeah we we kind of realized you can very much also um yeah quantize those pseudo gradients so usually your gradients are in uh uh 32-bit Precision um when you tried to send them and we got them down to 8bit basically by comp quantizing those and was still working as well as you would send the 32 bit uh um gradients and um yeah that was one of the things on the algorithmic side as well as also on the on the framework side so we we switch basically to our own um yeah decentralized training framework uh called Prime um where we basically build a stack on top of pytorch uh for for tolerant trade so um in that sense uh yeah note can drop out on the training and the training is still continuing basically um it yeah has a lot of great features in a sense have to to go into detail on some of the things um like for example you can join with a note obviously during in the middle of the training one right um then it peer-to-peer downloads the weights from the the other note and the network basically right and uh yes able to to contribute then to the next outer step basically with zero gradients at the beginning and then it actually starts contributing right um so I think the the for tolerant part as well as the onboarding offboarding dynamically um is like yeah one of the the main features of it as well as uh pretty good efficiency but yeah happy to to go into more detail with questions so so we did U we did here in San Francisco reading group on the Loco so I had to dive very deep into all their results we did the Loco and there's a second one um gaco right so so I I guess we can we can talk about that a little later but for the Loco generally I was a little disappointed with the uh baselines they chose to report against I I think in many cases the Baseline I would want to see was missing right so in this regard you only had two baselines of which one I think was the the most meaningful one right which is a training in data parallel fashion with the same number of replicas right but but I think Deep Mind paper had like the basic report against that one but everything else was against the single replica which is a very weird Bas like obviously it performs worse it's a right right but I I think a very interesting question is so so in your report you show that uh yeah the uh the G is slightly worse right than uh than data parallel but the question is what would be the uh what would be the size of the uh like what should be the badge size for the Loco to precisely mat match for for replicas of data parallel it's a good question right and uh how how I think about it is like obviously uh if you want to match then the the data parallel Baseline like in our report maybe to to go back to that uh we we match the data parallel Baseline um uh at a later stage in training basically right so right now it seems like the convergence that's the initial phas uh with a lot of steps basically is a bit worse right so um you're saying eventually they converge with the same number of exactly oh that's great so that's one of the things um as well as uh yeah it is all like a bit of um yeah depending on the uh number of in steps you already choose right like 500 inner steps are quite a huge number especially if we do in eight quantization on it right we can get down the um communication requirements by 2000 right um and for for a lot of the things we we didn't need that right for our inter one run for example right we uh we're fine with 100 steps basically doing in eight quantization so we have a reduction of 400 basically and we're running uh pretty efficiently across the whole world right um between all those nodes so pretty comparable to to centralized training right um so um yeah obviously unfortunately don't have a baseline for that one if anybody want to sponsor it happy to to run a baseline experiment but it specific care it it gets too expensive in a sense right um yeah so I have couple technical questions maybe they answer it in the in the report I didn't read it very carefully but one is so so the actual number you use for intellect one is 500 right the number of steps no 100 in our steps so 100 in steps with our quation technique so we have 400 uh X reduction basically in 20 and the and the optimizer use the same as in paper like Adam Adam internally n of momentum exactly we didn't want to like take risk basically on that front but yeah but but you don't have like in the original paper they train uh in a like a single replica for the first several steps you don't do that right that's a good question like we didn't do that no we we started uh uh yeah randomly initialized from uh uh from scratch basically right um we've been uh obviously in in contact with different folks have been doing stuff on their Loco of a himself right and uh he was also showing it it works also if you just start from a scratch uh not sure why he did it in the paper to uh yeah start from the 24K steps uh checkpoint do Las I I think in the paper they show that it doesn't matter so their point was more like hey if you only do one replica you save some compute but uh I don't know if the complexity of this extra step you know justifies uh no not using Compu for 24,000 steps which is negligible anyway okay so uh the U let's dive a little deeper into into intellect one so so I guess generally the um something so so so we discussed uh with Ilia a a lot the idea of training a model in a decentralized fashion uh and uh there's a couple interesting things one is that uh training model is easier than training a good model right and uh I I I I wonder how you approached uh choosing like I I presume you didn't do a lot of you know like grid search and what not for hyperparameters right so how did you approach using the the particular model architecture hyper parameters and data composition right so like for example I see that you use 55% of where did I see it yeah uh so fine web edu right so so I guess one interesting question is what is fine web EDU and you know how did you approach you know choosing that data composition yeah uh happy to share uh about the data composition we had help from from huging face they were interested in obviously what we are doing here right and uh yeah uh propos this like kind of data set we are going for right now um and yeah find U is one of the latest data sets uh yeah at least uh in terms of like benchmarks performance and stuff like this one of the best data sets out there um for um yeah doing pre-training basically right as as well as we we put a lot of other data sets into the mix just uh to have more a bit more variety right we had the stack in there which like a coding data set right we had a open rep math in day so more like a a um yeah math data set right um as well as let me guess dclm or something like this I think it's from Apple um yeah some of the other uh yeah uh open source data sets that came out so there's been a lot of progress actually in the open source world to yeah just create better pre-training data sets um that wasn't really around like a couple years ago when we all were training on on the pile by Uther yeah so I I don't know St V2 but I was I was playing with st V1 I remember St V1 was just individual files so you wouldn't get much in terms of uh uh you know code in context I don't know if V2 is solving it or if it's uh similar and fin has no code right um probably not as much no well very little um yeah I I was going through fine web with the filter that would say hey only show me the things that have line break space space which I think is a you know relatively simple filter for code and uh like I went through the first 2% and I didn't find any I think that's sufficiently good indication probably doesn't I mean they probably intentionally filtered it out as a out of not a Target use case I guess makes sense yeah um I may to share a bit more there right on the on the data set side as well as obviously the intellect one model it was the the first release to basically show it's possible Right to to train efficiently uh a 10 billion parameter model around the world right obviously we want to uh make it a decent model right we we Tred on a a decent data set obviously we trained on a full like one trillion tokens right obviously the current llama series and uh a lot of the other models basically out there in the open source right now are trained on a lot more tokens right um they get better performance we just want to show hey you train on one trillion tokens in a decentralized fashion such a model and you get a model that you would expect to to be um you of this quality in a sense right um so that was the plan for intellect one but obviously for for intellect two we we have a lot more planned there decentralized training needs to show that can actually train a Frontier Model obviously right and uh yeah that's uh yeah the the moment like all the like more like engineering challenges are currently solved right we we had a lot of like issues with for Toler and the training obviously right um where uh we we fix stuff in production in a sense right to uh fix all the failer modes for for tolerant see where where was the failing right and then at the end of The Turning one I think we got a pretty decent stack now of uh yeah people being able to to onboard to the training one as well as like offboard yeah and and it's like architecture Wise It's just uh it's like Lama right like Gago RMS Norm exactly we we took the Llama three architecture and uh yeah just added a couple more layers basically to to the Llama 38b one same tokenizer um so it didn't really innovate there but also very excited obviously on all the uh yeah architecture improvements but yeah for for the initial one didn't want to take the risk to uh uh yeah that it could be failing because of those kind of reasons to to change the different architecture yeah but and and it's not sure of experts it's a it's not yeah yeah so I'm looking at the table with the comparison right so it's a and uh specifically intellect one versus llamas I I think llamas is the most interesting Benchmark at least because you know actually don't know some of the other models I don't know maybe some of them are more exciting but llama is a I mean it's an industry standard more or less uh and so on certain benchmark I would say my my vision of what I see is that on most benchmarks it's a little worse than 7B right which I guess is still pretty good for the first model uh but uh interestingly on grade school math it is outperforming even the 13 B Lama which is uh it could be I guess because of 5% data mix but it's interesting right so it's already outperforming on some benchmarks yeah for sure um there could be a reason uh find web U I think has a lot of like great uh um yeah more educational data in there so would expect it to be good at uh great school Marth in a sense right so uh that could be one of the reasons uh too um but in general we were pretty happy with the with the benchmarks results that um yeah was scaling uh at least to that size in a sense right all the other models we compared that to on the Lama 2 Series at least right we're trained on like two trillion tokens right so um that's that's a good point right I guess if Prime intellect was train for twice as long that would be right but but even like I guess you know La llama has a you know like 60 people centralized cluster it's uh like I think for the first model even even if it underperforming not because of the smaller number of tokens it's still an outstanding result and and a pretty good prove that that the approach is viable which is uh which I think is quite important I think generally it's it's it's my it's my very big uh I don't know if I'm fully convinced uh that scale will continue right but if it does continue uh you know the the fact that uh only you know three entities in the world can train can train a one trillion parameter model is not is not where we want to be as a society right if uh so so I think it's a you know the effort to to make it work right now is uh I mean the best time was last year uh but the second best time is right now exactly yeah so something else uh which is interesting that I wanted to ask is uh I spent quite a bit of time last year training large language models uh and one problem I was fighting with which uh uh there's very little information online about fighting it it's mentioned as an existing problem but not much in terms of a solution spikes so I wonder if you observed spikes and how you dealt with them if you did yeah honestly we we didn't have to uh uh yeah let me think about it but I don't think we had any like um yellow Spike basically in the current tr1 right um which was interesting like that's one of the things you need to test in like a large scale train1 right it could have very much happened that Delo is uh not as good in terms of uh yeah training stability in a sense right that it could have like especially in the end of training basically have l l spikes right um where yeah you just have to restart from a checkpoint and hope that it is not happening again in a sense right um we we obviously we integrated some research on top of actually the Lama 3 architecture with the set loss uh that one of our researchers a huge fan on for fof for like observing how um yeah stable the training is in a sense right um where you can basically identify lost spikes earlier than they would happen basically um so we implemented that one looked all good uh didn't uh yeah look like it's going to have a lost Spike right but yeah yeah I guess the reason I I thought of a loss spike is that there is like a little spike in the training but now that I look at the picture that's because the data composition has changed oh yeah that's a good point um at the the end of training so basically maybe to to share there a bit um what we went with with the learning rate scheduler so um usually when you train your model you have to Define in the beginning how many tokens are you training on um so you have a a specific learning rate schedu that is just uh um un leing the learning rate to zero at the end of the training R and like a um yeah Co and curve and uh what we did in the training one from like research from also I guess from hugging phase but also other uh um yeah organizations uh yeah released this over the last couple months that they've been training models like this is basically to have a constant learning rate after warm-up phase where you can basically then decide at which point do you actually want to do um yeah in an needing phase to actually get the model yeah a need the learning f rate to zero but you you basically have the option to train on as many tokens as you want right and um yeah we obviously publish the checkpoint and stuff like this right so if anybody want to take it continue training they can do it right um yeah as well as uh like us uh obviously we are bit more focused on the inter 2 um Direction right now but uh yeah theoretically you could just uh continue to training basically uh indefinitely Or how big a data set is yeah and so with intellect to what are the big changes like is it different is it the same architecture is it the data set composition or what are the what are you guys looking into um what we're looking into is uh yeah fixing some of the limitations we have right now with the local run right so one of the things obviously want to get to bigger model sizes right so uh yeah the the current like network bandr is it's actually not the liming factor in a sense um in some regions it's more also about the memory requirements you have right like if you want to train the 10 billion parameter model on an h100 node uh yeah that's almost already limit right of what you can train on an h100 node if you Shard like your parameters gradients and optimizers right um so we need a way to to use less memory right there are different optimizers for doing this uh as well as like more the galor type research approaches right so we've been exploring over the last couple months all of those research directions to to make that possible um as well as the other component is to to scale to more more providers more compute contributors right so uh right now uh delu has uh diminishing returns after like 8 to 16 or in our case we we've been running on up to 14 NES right um and we've been working on researchers and have a a paper coming up to actually make it possible to um yeah scale to a lot more providers right or compute providers so uh that's one of the things things uh we we are trying to to to to tackle right um the the memory requirements as well as the number of Compu contributors we can scale to and then uh yeah we want to train a Frontier Model for the next one right so it's uh not about only uh the model right and the local scales but we also need to make progress on um yeah the data set size and a lot of other components so um we've been uh exploring a bit in the large scale St data generation domain and uh yeah um have some some upcoming uh releases on that front basically um but yeah that's that's uh all the research Direction we've been basically exploring for for an inter 2 launch yeah so I guess one interesting question here is that um you know we discussed earlier that in your original paper that you published in October I guess or earlier uh the Loco was not doing as well as uh data parallel right but you're saying that you know down down the you know with a large number of tokens it converged so there's a good chance that you know like if you if you run deu with with a 30 replicas that it's doing poly at the beginning but maybe you know long term it does as well as the data parallel with f replica so it's it's a it's something that is very hard to experiment with without actually running a long run so it's a but but you know it could be that Delco actually just you know it could be that it just scales to larger number of replicas we just don't know it yet without running an expensive experiment but but also something I wanted to ask is uh what in the current run a single node is it a single machine with eight h100s or is it like a small cluster I see yeah right now it's a h100 node basically is a is a single worker so um we made it open for anybody to basically contribute jut resources through the platform basically they could just launch an instance there right uh as well as everybody else uh that provided compute like a h face obviously uh had their own compute resources to to put into the training one right um yeah so in a short term right like let's say we want to train a 70 billion parameter model right and you mentioned that 10 billion uh is sort of the limit of what fits on a single machine right so in the short term something that presumably will work is that you say hey a single node now is not is not a machine it's a it's a small claster of eight machines because majority of providers give you like like you know like if you go to Lambda or yep like any of them they give you like they give you small clusters with interconnect right it wouldn't be impossible to imagine especially you could do it right now if you uh want to do a launch basically over a 7 billion model decentralized in the same setting that we got in a sense right like we had also like single nodes and data centers where there wasn't other nodes available because we've running on spot instance Z or something like this right but for uh the general use case there's a lot of data centers with more 81 than 8 h100 um which could interconnects and that's also the one of the reasons why this approach is especially interesting for the big labs in a sense right it's not only for like the open source Community right trending across their different uh smaller clusters right it's one of the um things you could use it for but also for for the big labs to train across like uh yeah four to maybe 16 different data centers that they got right yeah exactly especially Google right that has a you know historically has a lot of locations with the tpus right and and I think so so I see 112 I will not be able to divide it by eight on my head I suspect it is 14 right 14 yeah yeah right so so right now you use 14 replicas right so so like finding 14 clusters of eight machines is extremely doable right so it's a so so like short term there is a simple solution because I think the Paco results were very underwhelming right so it's it's presented as a super cool solution but once you start like you know running math in your head you realize that deak is very data inefficient which I guess a sort of expected right given that it's actually training a much smaller model than it's more like a mixture of experts yeah like I'm still like uh yeah interested obviously in the more mixure of expert type of ways right on how we can train decentralized right I'm not sure if it's a approach they've been going for there with like a sequence level mix of expert right so um what you usually have is you uh when you train your mix of expert right you route per token and it's uh actually extremely like communication intensive right um and they route per per sequence right um to like different experts which made it possible to uh yeah train this decentralized um but yeah also not 100% convinced yet that's the way if this is actually producing the best model that is is possible and uh it's not worse than a normal mixture of expert yeah that that that yeah so so so I noticed when you were talking about ideas for Prime intellect 2 for intellect 2 you didn't notti you didn't mention that direction I think it's more about the novel like architectures obviously we've been launching something like a non GPT challenge basically or like uh somebody did and we Rec created a leaderboard for it right um which is uh an interesting challenge to uh yeah basically get uh better model performance um yeah with with L like gets the same model performance on like a gbt2 benchmark to the same loss on F to uh uh yeah just less tokens right um so those are more approaches we've been interested in to to innovate for an interl to yeah what is the timeline when do you guys I guess not even just what is the timeline the first question is are you training something right now right you have you have those resources are the ized or are they waiting for intellect to we've always been we've uh we are always running ablations in a sense right to to make sure what we want to train next right um but we don't have intellect to running right now I can say that much yeah so I I guess couple interesting so so at some point I want I want to discuss the step four of the master plan uh I think it sounds very exciting but before we even go there right so um so so like training uh train training a large model is very expensive right so like let's say if you want to train a 400b uh that will be something that is extremely expensive right it's like uh I mean it used to cost $100 million maybe it's cheaper now h100s were uh continuously getting cheaper but it's still at least an eight digigit number right so it's a what is your thinking in terms of uh uh like I was thinking of mon ing the models eventually uh or or are you thinking of it more as a you know uh Humanity pulling resources to create public good without an expectation of the returns like how do you think of the business model that's basic two Dimensions we we see on this too right like uh either you're on the front of like monetizable open source models which is an interesting research Direction I'm happy that people exploring it in a sense right to to make that possible uh and uh yeah um seems interesting to my my current guess or like how we see it basically is that uh yeah everything will be commoditized in a sense right what we are doing here we we commoditize compute with for our computer exchange right um to always give you access to to the cheapest computer resources make completely commoditized as well as on the intelligence side right um like the the models uh that are out there seem to be uh yeah becoming commoditized in a sense too right and we we see it more as like a public good of people coming together want to pull resources to actually create of the art models and yeah will be the greatest public good in History to have like open source AGI yeah so so so you're thinking more of like people just like voluntarily contributing compute into a very large cluster exactly like I I rather think that it's going to be more like a public good than uh being monetizable but happy to people Pro them wrong and people figure out how to uh monetize open source models uh in a way um maybe to to give a c there uh like on different directions I'm I'm more bullish on monetization right um so there are a lot of foundation model use cases where it seems reasonable to to monetize the the model layer so um for example for like a large scale scientific Foundation models uh which we've been really interested in with uh yeah my background but also my my co-founder's background in in the D siiz bace um to to innovate a lot there I think there's a lot of possibility for monetization uh but on the on the Frontier Model lab side uh it's seems to be a lot more commoditized in the sense right um but for for scientific Foundation models or like any kinds of other models in this direction robotic Foundation model would be interesting um we've been training a a meta metag genomic Foundation model for pandemic prevention that's going to come out soon um which is also very easily to train in a decentralized fashion in a sense right so for for those kind of use cases I think we love to explore that to on the monetization side yeah what are the so you mentioned some research from monetization side I I don't know if you have some particular favorite approaches that you heard of I think uh some of the ones uh are to to actually make it possible that not everybody gets the weights of the of the model right that's the one of the approaches I've been uh caring over the last couple months right um um so uh yeah I I really wonder how that works out in like still having the best perform model and also the the fastest INF people want to have fast inference obviously right and it can't be expensive there so we've been not uh following or like exploring that closely um yeah as well as all the the fingerprinting type Solutions right um that uh yeah have people have been proposing um yeah um not not not my area of expertise yeah I was thinking a lot about it because I think um so I as I mentioned I'm primarily interested in the situation where the models continue being being bigger and bigger and better and better even though I'm not fully convinced that that's that's what will happen it it's definitely there's definitely not a trivial probability of that happening right and so how do we make sure that the open source Community continuously has you know resources to train those models right especially if they require you know hundreds of thousands of h100s then you know that does the does the community have even collectively access to those resources right and and monetization I was very interested in it uh and I I think one of the most interesting approaches that we're looking into is um h100s have trusted execution environments meaning that you can run code without the cost seeing what's happening there or like anyone seeing right and h100s have has an overhead of like 30% I think people say between 10 and 30 the but b100s will not have any overhead meaning that you can run in trusted execution environment as fast as if it was not right and so there's this idea of training a model in such a way that nobody ever sees the weights they always encrypted right and then the model would only do inference for you if you open a payment Channel and pay for it right so anyone can copy it to their machine right you don't have to trust anyone you can you know build your own rig bring the model to locally got the internet you know you use the QR code to send the payment channel so it's like fully isolated but uh but you still have to pay to use it so I think that's a very exciting Direction yeah definitely sounds interesting um and uh excited to see some research there um and uh yeah hopefully the T like supply of the of the next generation of Nidia chips is going to be even yeah greater in a sense right that you could do it on on any B100 or b200 um so uh yeah that would definitely solve a lot of our issues too so excited for it yeah and it was very fascinating right h100s came out last year right uh and like by the end of 2024 it's a commodity as you mentioned right like I can go to GPU le. uh which is like it's like an aggregator of all those providers right and and it's like you know $2 per GPU is a like there's a lot of them especially on the large cluster size there still so much like uh yeah um yeah different and um yeah reliability the providers right so the the rck I Forks had this great paper of like showing like 100x difference in terms of reliability on the Clusters they've been running on right so I think we have to solve the the for toin part on the software St if n doesn't solve it on the hardware side right um uh where they're obviously still the best obviously in their game but uh they still have issues right and the providers have issues and uh yeah to to fix it on the on the for tolerant side is probably the way to go for the next couple years yeah yeah and I don't think Nvidia is investing much into that because you know people already used to the fact that Nvidia gpus fail in a variety of ways and you know everybody who trains at Large Scale already have mitigations for it yeah in one way or another okay uh I think the most interesting thing uh which you know I left for the end is the the the step four of the master plan which is uh actually one thing I wanted to mention is that I think of prime intellect as a I don't know um I mean I mean you guys want to build a decentralized protocol so I think of it as a you know some somewhat in this crypto Anarchy you know set of companies and and it's quite surprising to see a company saying hey guys we had the first step of master plan and we have more than one right and then check two months later and you noted the first step you you you further along that doesn't happen very often in the right so so so I think that's that's a very uh that's a pretty cool speed at which you move through your master plan right so so so that's a pretty good indication that step four will happen you know at some point so so so I would love you to talk more about your vision for Prim in the future um I mean we we uh structured the master plan in a sense like this way because we obviously have to solve a lot of the research challenges right now to to even get to like more like a prodal based uh system where we can incentivize people to contribute to resources to train like Frontier models and uh yeah create the greatest public good in History to build open source AGI as well as like making progress on science and stuff like this right with uh scientific Foundation models and so on they've been uh yeah being trained on uh um yeah people contributing GP resources so that's why we structured the way like uh the master plan like this obviously we want to solve aot of the fundamental research challenges right now right um scaling uh your intellect to to like the the largest model as well as like making more the the trest compute component work right so um we've been exploring different components on um with the least amount of overhead in a sense to to enable uh yeah people to provide compute resources where you can verify they are not sending malicious gradients um as well as uh uh yeah it's actually an h100 node that they are providing right um so we we are still on the way I have to solve some of those like trustless compute challenges in a sense as well as like uh proving with intellect to that we can actually train a Frontier Model right and then uh yeah we'll uh move ever closer to um yeah the last stage of our master plan and um yeah so a lot of that if you trust Nvidia which we sort of have to a lot of that can be achieved with the trust trusted execution environments in the sense that like Nvidia will send you an station that this is 8 h100s and if you run the code inside of the te you know that the gradients they send you are correct so so I think short shortterm this is the most viable solution even though right now there's a lot of like Supply shortage in that sense right we've been talking to a lot of different providers that we have on our platform obviously and those uh uh unfortunately have like not the expertise to actually set up like an h100 node with Ed fter execution environment or don't have the specific uh like CPU chip you need wor uh actually enabling that right the CPU extension umf TDX you need a fifth generation Intel processor and we we talked to a lot of providers and in fact it is not something people have today exibits will have some of them in the future like in a very short future but uh that's uh I mean having one provider is precisely opposite of having yeah Supply issue now so uh we've been thinking more about more uh yeah more approaches in terms of like staking a slashing and stuff like this right where you have a lot to lose basically game theoretically if you uh yeah send malicious gradients or um yeah provide computer resources that are not as uh um yeah not the ones that we need for the 21 right um so yeah yeah yeah yeah that's that's obviously a more stand approach but my thinking with TDX is that first of all um it will probably change right like same as what happened to h100s which you know like two years ago they didn't exist and now they quite abundant uh but but also for people who will be joining especially people who will be building rigs in the future you know if they know that fifth generation Intel processor gives them more opportunities to use it it uh definitely bullish on on the future there if if we want to launch something uh early next year there's probably not as much Supply to actually do an inter to launch on on uh on such a system as well as like obviously right now we we've been still limited to homogeneous Hardway right we've just been running on h100 noes um but we we also are working on approaches to make uh yeah our Del work and more to uh be able to use henus compute right so uh we will still mainly be focused on more the data center GPU uh just because of the rerum requirements we got for actually training state-of-the-art models right um but at least uh like it's still good to to train on A1 100s right um so there's nothing wrong with that a lot of researchers don't want to but um yeah there there's no reason to also include A1 100s and the training one multiple noes and stuff like this and a 100s also has 80 GB of memory right so it's like the same size of the model will fit it's just the number of steps you can do it smaller exactly we didn't mention in The Deco Paper what happens if in the same amount of time you know so so you cannot have the number of steps constant anymore right now it has to be variable presumably right because a 100 I cannot do 100 steps in the same time exactly or you run on two a 100s and then you are roughly the same time but that's not what we went with um it it needs to be a more like asynchronous approach right right now it's very synchronous right they Al synchronize after 500 steps and yeah we we've been very interested in how to make the local work asynchronous um there's also a paper actually from from Google deep mind uh that did D asynchronous right um but also just scaled it to the I think only 150 million parameters and and not more and had to do a lot of tricks actually to make it work uh asynchronous um but yeah we we we thinking about mik an easy approach there um yeah yeah so so actually couple couple more interesting questions one is um like the number of steps you take between uh gradient synchronization what is it like how does it map into the realtime clock like how frequently the guardian synchronization happens in terms of like seconds oh yeah um so for inct one run where we were sying every 100 steps right um we've been sying every 40 minutes so they were running completely independently on those different islands for 40 minutes and then yeah depending on our configuration we are different synchronization times so we've realized for example in the US um you have pretty fast interconnect between all those nodes right so we had a 4 gigabit interconnect between all the US nodes uh that we got like all over the US um so uh we've been then the all reduce basically of all the different notes and the took one minute then um and then we uh in the training one uh we added Europe right uh so we had some some notes in in Sweden and Finland I believe and then our all reduced time went down to five minutes right which was still pretty efficient right we we trained for 40 minutes and then we sink for five minutes which is uh still acceptable right and then I think when we added Asia some Singapore notes and indiaan notes uh we were at six minutes or reduced time on average um so uh yeah that's about the computer efficiency you can expect there with 100 steps yeah is already speaking uh uh geography into consideration like is it more efficient to all reduce all in us than all in Asia and then like you know the four-way or or that's not happening yet that's the plan it didn't happen yet so right now what we do is like a literally ring all R so we we've been literally uh when we like uh yeah yeah um all reduce between all those different locations we we've been doing a ring the around the whole world um but obviously there are even more efficient ways to do it like one of the things we could think more often in like a continent for example or generally where you have fast interconnect sometimes it's actually pretty weird like we had some connections to Europe that were extremely fast because uh uh yeah some kind of internet routing uh um uh yeah internet routing um reason right and uh yes um but uh yeah fundamentally usually it's like in a continent you're you're fast between the nodes right um so there's definitely ways to make use of that for for our next run yeah and do you know already how big intellect 2 will be let's see um let's see how how much progress we make obviously on the um yeah on the on the ream side size how much we can actually fit into a Noe right um that's one of the things uh as well as we want to make progress uh not only do data portal training obviously which we do right now right uh which is fundamentally limited do you have to do the whole training on a note um but actually split up the model with like pism for example right um so there we could actually scale to a lot bigger sizes um but it's also another engineering challenge that we have to solve under in too yeah Al so so you think that will already be you're already expecting to have progress on that for front by the intellect too it's a good question if we need it like we have uh uh yeah two streams basically going on there right now right like you could either like work on a PR just to reduce a VR requirements on a note right to actually train your model on a note um like larer models on one note or you can split up the model right um and uh do do pipeline portal training um like uh yeah one uh um yeah great researchers from from together AI Maxin who's been uh yeah uh like fundamentally inventing the whole field of decentralized training right he's been working with us on the intellect t or advising us for for research there and here has this great paper on Swarm uh which uh yeah is like a pipeline parall decentralized training approach right which uh nobody has scaled yet for some reason so I'm excited I would love to yeah if some people want to do open source contribution work to to our Prime Reaper or uh whatever else right in terms of like making pipeline p training work across the world right uh that would be amazing and yes there's a lot of potential I guess into uh yes sing the Swarm algorithm whose paper is that Max vinan um is uh yeah a researcher who has been doing his PhD actually in decentralized training right so a lot of the fundamental work in in terms of uh yeah this research direction is coming from him okay cool we still have uh five more minutes give or Tex so I wanted to also uh briefly talk about uh uh the the post training so so I see it's done by uh by AR I don't know if AR is completely separate entity if you just guys work with them or what is the relationship there who's comput it is who's incentive Etc uh yeah asdi is is a yeah a really good friends of us basically that are um yeah one of the leading Orcs I would say on uh you're doing post training of those uh uh Foundation models right so uh they have those RC Supernova models which are pretty popular and uh yeah have a whole team like just specialized on post post training of uh of of Bas mods or uh even continuous post training of uh um yeah instruction T models right um so uh yeah really grateful for their support basically on the interleg one uh release uh they actually made it possible that this space model was able you could actually chat with the model right um which uh uh yeah we didn't in house have the expertise to to do it as quickly as they did so we we sent them the weights uh on the day when uh the intellect one run was done right and immediately uh yeah started cooking on it and had a pretty good mod um at the end of the week then um which yeah we deployed in a chat form they everybody could chat with it right as well as like sour it right um and yeah they they did a bunch of different approaches um so one of the things they um always do on those models that um uh yes they innovated in a sense or um uh is the merge kit work so basically uh merging different open source fine tuning uh models right uh to get one bigger better model out of it right so um that's also what they've done on our in one run so doing a bunch of different different fine tunes basically and then uh yeah merging all of those the best funds at the end to to get the best mod out of it yeah so so if people want to chat with the intellect one where do they go uh they go on chat. Prime intellect. um I I didn't see it in the Tweet storm so it's a Well hidden yeah was a long tweet storm yeah okay that's uh uh and uh where is is Prim itself distributed or or where you guys located um most of the people are in in San Francisco um so uh yeah most of the people are here but obviously we're happy to uh have a lot of like disabil uh community members in a sense right uh that uh can work on different research challenges right uh so um yeah if anybody interested we have a lot of challenges to solve and uh hope isn't interesting research uh problem to work on um yeah yeah okay uh that was a uh that was that was a very fun discussion I I don't know if we discussed everything if there's if there's something uh I didn't ask that you think is important uh in the I think you asked yeah great question I wasn't even expecting such a technical discussion so it's amazing that you went so deep on like all the the local work previously with Y to so really enjoyed it and uh that I could share some of the research stuff here too yeah yeah we can put like any links below the video to the to the relevant stuff so it's a uh yeah but uh yeah I think I think this is one of the most exciting directions the decentralized community can and should take today so it's it's exciting that there's actually someone who's who's working on it actively yeah and so yeah let's see how two you know when it comes out and uh yeah everybody who wants to play today it's chat. prime. right exactly yeah lot for joining us today thanks Alex","transcript_synced_at":"2026-11-03T05:13:14.947Z","segments":[{"start_seconds":5,"text":"blockchain","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=5s"},{"start_seconds":11,"text":"flows near the one we're close to won't stop us","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=11s"},{"start_seconds":18,"text":"now together we'll redefine the game when near we'll find the flame","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=18s"},{"start_seconds":26,"text":"empowered Minds rise and S use your own AI we'll","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=26s"},{"start_seconds":40,"text":"explore hold the future hand in hand in this tax driven","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=40s"},{"start_seconds":47,"text":"Wonder okay hello everyone uh it's a a new episode of near office hours uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=47s"},{"start_seconds":54,"text":"today with us is johanes from Prime intellect um I uh I learned about prime prime intellect last October when you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=54s"},{"start_seconds":62,"text":"guys started training intellect one uh and mostly what I knew was that you're training a large model using Theo","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=62s"},{"start_seconds":70,"text":"but I've been reading up in the last couple days leading to these office hours and I don't know I don't know how we going to squeeze all of that into one","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=70s"},{"start_seconds":77,"text":"hour that's a a lot of exciting things uh that U there are to discuss do you want to give a quick intro on on","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=77s"},{"start_seconds":85,"text":"yourself and the prime intellect yeah for sure happy to do it and yeah nice to meet you Alex um so yeah I'm johanes one","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=85s"},{"start_seconds":92,"text":"of the co-founders of Prim and uh yeah my background is more on like uh J side so I've been um yeah turning Foundation models that a yeah Foundation model that","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=92s"},{"start_seconds":100,"text":"appear in Europe since end of 2021 and um yeah basically what we do at Prime intellect is we aggregate Global","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=100s"},{"start_seconds":108,"text":"Computer Resources from all providers all data centers and have them yeah all in a single platform as well as um yeah","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=108s"},{"start_seconds":114,"text":"working on um yeah training distributed across all those different clusters so we are making like um yeah working on fundamental research to make disted","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=114s"},{"start_seconds":123,"text":"tring work in scale and um yeah obviously had the intellect one launch where we actually scaled up our like research that's been we've been doing","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=123s"},{"start_seconds":130,"text":"since uh end of last year basically uh to the 10 billion parameter size and uh yeah from here on um on a scale it up","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=130s"},{"start_seconds":138,"text":"further obviously and uh yeah generally are interested in all types of uh research directions which are important","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=138s"},{"start_seconds":145,"text":"to make open source AGI a reality so not only on The Descent training side which will be one um yeah important part to make like last compute resources","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=145s"},{"start_seconds":154,"text":"available obviously but also um all other research directions from like large scare synthetic data generation to um yeah obviously not just training a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=154s"},{"start_seconds":163,"text":"pre-t trend model but also maybe going more into this test time computer action yeah okay so I I suggest we begin with","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=163s"},{"start_seconds":171,"text":"the lowest level and go up uh uh you know climing up the master plan right so so the first step was to to pull the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=171s"},{"start_seconds":178,"text":"compute and and that's something that many companies do right like there's a cash Network exhibits hyperbolic so my understanding is that Prime intellect is","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=178s"},{"start_seconds":186,"text":"sort of orthogonal to all of them right you're not trying to compete with them uh you're trying to take advantage of all of them I saw hyperbolic was participating uh in training of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=186s"},{"start_seconds":195,"text":"intellect one right so so how do you see uh the role of prime intellect here and the how does it work with all of those","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=195s"},{"start_seconds":204,"text":"yeah on the on the aggregation side I would say um we are a bit on a on a layer on top in the sense right just by being more like a computer exchange","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=204s"},{"start_seconds":212,"text":"right like having all those providers on the platform basically and letting people like tap into the cheapest compute resources through it right um so","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=212s"},{"start_seconds":220,"text":"we have a bunch of like of the centralized like new Cloud providers on the platform which have like varying uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=220s"},{"start_seconds":228,"text":"like availability right uh like often it's been a joke over the last couple years right that lamb Labs didn't have like the h100s available to train U AI","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=228s"},{"start_seconds":236,"text":"models or something like this right but somebody else might have it right and uh yeah that's the approach we've been uh going for there like having more of like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=236s"},{"start_seconds":244,"text":"a computer exchange basically you can always tap into the cheapest computer resources and yeah obviously uh um yeah","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=244s"},{"start_seconds":251,"text":"we we aggregate all of the providers right from like the repb two players to also more like an aash for example is on our platform we have an easy like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=251s"},{"start_seconds":258,"text":"interface to to make use of it and uh yeah just give more access and cheaper compute to um Source","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=258s"},{"start_seconds":266,"text":"developers right and so my understanding is that uh the the interface with the provider right is that they give you an","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=266s"},{"start_seconds":274,"text":"SSH access right your SSH to the machine do whatever you want right and there's a but","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=274s"},{"start_seconds":282,"text":"uh like naturally if you think about it on a very high scale right it's uh there's certain trust that needs to be between you and the provider right that","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=282s"},{"start_seconds":290,"text":"they will not you know give you inferior Hardware pretending it to be Superior Hardware and whatnot so so right now there's no attempt to fix that problem","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=290s"},{"start_seconds":298,"text":"right like we just asse that if they give us a machine with 8 h100s then it's a machine with 8 h100s right yeah in a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=298s"},{"start_seconds":307,"text":"sense right now we rely a bit on the provider in a sense right obviously we make it easier for people to to know which one is a trust Prov trustful","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=307s"},{"start_seconds":315,"text":"provider in a sense having Community Cloud offerings and a secure Cloud offering um the other thing is uh yeah","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=315s"},{"start_seconds":322,"text":"um it's not only about like trust uh or like trust's computer in that sense right it's also like reliability of a provider right um especially if you're","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=322s"},{"start_seconds":329,"text":"an AI researcher and want to scale up things um yeah to more than an h100 node for example there is uh yeah a lot of considerations basically that some of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=329s"},{"start_seconds":338,"text":"the providers fail way more often than the other ones right so uh there's been some interesting reports of that in the in the Lama 3 paper and so on right um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=338s"},{"start_seconds":347,"text":"that uh yeah how reliable different providers are right um so uh yeah I think that's another component right","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=347s"},{"start_seconds":354,"text":"those two reliability of the provider as well as uh yeah can you trust the compute resources it actually giving you an h100 right so um yeah yeah yeah okay","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=354s"},{"start_seconds":365,"text":"and so like the next step of the master plan was to build Prime I guess it's called right u","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=365s"},{"start_seconds":372,"text":"i don't it's a Transformer implementation right an Optimus Prim is a Transformer I don't know if that's intentional or not um but you know where","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=372s"},{"start_seconds":381,"text":"the name Prime intellect is coming from no I don't know where is it from um there's a nice sci-fi book about it basically um about like an ASI","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=381s"},{"start_seconds":390,"text":"um that's called Prim um this where the originally name is coming from","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=390s"},{"start_seconds":396,"text":"yeah yeah I I'm reading ungodly amount of Science Fiction but I haven't heard of that one yet so that's a that's a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=396s"},{"start_seconds":405,"text":"I'll be sure to read it is it a good book um ah I'm GNA get cancel for um it's uh it's definitely","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=405s"},{"start_seconds":414,"text":"interesting in a sense I don't agree with the whole conclusion obviously there but uh um it also Paints the interesting ASI future where everybody","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=414s"},{"start_seconds":423,"text":"has everything everybody is Immortal in a sense but they also don't have anything to do in a post I would yeah it's not a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=423s"},{"start_seconds":433,"text":"yes it's yeah but uh so I saw so so there was a paper right you first published the open decal paper and then","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=433s"},{"start_seconds":442,"text":"there was a technical report later I I guess very recently right uh and technical report has more things right there's fsdp","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=442s"},{"start_seconds":450,"text":"and whatnot so I guess let's dive into all of those so let's begin with dilco right uh can you give us like a quick","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=450s"},{"start_seconds":457,"text":"overview of what Theo is and why it is needed for sure to to give like a timeline there basically so um end of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=457s"},{"start_seconds":465,"text":"last year it's a it's a work by Google deep mine called distributed low communication training um which is basically a technique uh they've","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=465s"},{"start_seconds":472,"text":"published for um yeah doing disability training with relatively low communication requirements so um they","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=472s"},{"start_seconds":480,"text":"have yeah they train their models on different islands of devices basically and then usually how you have to do it in your distributed training setup you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=480s"},{"start_seconds":488,"text":"have to Sy after every step in your training basically so every step you have to send quite a bit of data basically of the gradients uh in like a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=488s"},{"start_seconds":496,"text":"an all reduced fashion and um in this uh paper they basically show they could get the same performance out of it by only","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=496s"},{"start_seconds":503,"text":"syncing every 500 steps and yeah are training independently for about 500 steps on each of those machines and then","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=503s"},{"start_seconds":511,"text":"they sink between those and uh yeah do an auto optimization Loop basically to uh yeah still achieve the same performance as you would get in your","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=511s"},{"start_seconds":519,"text":"centralized training setup and um yeah maybe to to to go to open the Loco basically for for the timeline there b","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=519s"},{"start_seconds":527,"text":"uh so we started reproducing that uh deep mine work at the beginning of the year so um yeah at uh yeah I think in","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=527s"},{"start_seconds":535,"text":"April or May or something like this we we launched open de loo which was the first replication basically of that work and we scaled their work to the 1","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=535s"},{"start_seconds":544,"text":"billion parameter size because in their paper they only showed it it was working until 400 million parameter model sizes","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=544s"},{"start_seconds":550,"text":"so we scaled it up to the 1B size and um yeah then from there on obviously did more research on the on the algorithm","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=550s"},{"start_seconds":557,"text":"side of things obviously so um yeah we we kind of realized you can very much also um yeah quantize those pseudo","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=557s"},{"start_seconds":565,"text":"gradients so usually your gradients are in uh uh 32-bit Precision um when you tried to send them and we got them down","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=565s"},{"start_seconds":573,"text":"to 8bit basically by comp quantizing those and was still working as well as you would send the 32 bit uh um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=573s"},{"start_seconds":581,"text":"gradients and um yeah that was one of the things on the algorithmic side as well as also on the on the framework side so we we switch basically to our","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=581s"},{"start_seconds":590,"text":"own um yeah decentralized training framework uh called Prime um where we basically build a stack on top of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=590s"},{"start_seconds":597,"text":"pytorch uh for for tolerant trade so um in that sense uh yeah note can drop out","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=597s"},{"start_seconds":604,"text":"on the training and the training is still continuing basically um it yeah has a lot of great features in a sense have to to go into detail on some of the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=604s"},{"start_seconds":612,"text":"things um like for example you can join with a note obviously during in the middle of the training one right um then it peer-to-peer downloads the weights","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=612s"},{"start_seconds":621,"text":"from the the other note and the network basically right and uh yes able to to contribute then to the next outer step basically with zero gradients at the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=621s"},{"start_seconds":628,"text":"beginning and then it actually starts contributing right um so I think the the for tolerant part as well as the onboarding offboarding dynamically um is","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=628s"},{"start_seconds":638,"text":"like yeah one of the the main features of it as well as uh pretty good efficiency but yeah happy to to go into more detail with","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=638s"},{"start_seconds":645,"text":"questions so so we did U we did here in San Francisco reading group on the Loco so I had to dive very deep into all","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=645s"},{"start_seconds":653,"text":"their results we did the Loco and there's a second one um gaco right so so I I guess we can","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=653s"},{"start_seconds":660,"text":"we can talk about that a little later but for the Loco generally I was a little disappointed with the uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=660s"},{"start_seconds":667,"text":"baselines they chose to report against I I think in many cases the Baseline I would want to see was missing right so in this regard you only had two","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=667s"},{"start_seconds":675,"text":"baselines of which one I think was the the most meaningful one right which is a training in data parallel fashion with","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=675s"},{"start_seconds":683,"text":"the same number of replicas right but but I think Deep Mind paper had like the basic report against that one but everything else was against the single","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=683s"},{"start_seconds":691,"text":"replica which is a very weird Bas like obviously it performs worse it's a right right but I I think a very","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=691s"},{"start_seconds":699,"text":"interesting question is so so in your report you show that uh yeah the uh the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=699s"},{"start_seconds":705,"text":"G is slightly worse right than uh than data parallel but the question is what would be the uh what would be the size","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=705s"},{"start_seconds":714,"text":"of the uh like what should be the badge size for the Loco to precisely mat match","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=714s"},{"start_seconds":720,"text":"for for replicas of data parallel it's a good question right and uh how how I think about it is like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=720s"},{"start_seconds":728,"text":"obviously uh if you want to match then the the data parallel Baseline like in our report maybe to to go back to that uh we we match the data parallel","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=728s"},{"start_seconds":736,"text":"Baseline um uh at a later stage in training basically right so right now it seems like the convergence that's the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=736s"},{"start_seconds":742,"text":"initial phas uh with a lot of steps basically is a bit worse right so um you're saying eventually they converge","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=742s"},{"start_seconds":750,"text":"with the same number of exactly oh that's great so that's one of the things um as well as uh yeah it is all like a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=750s"},{"start_seconds":760,"text":"bit of um yeah depending on the uh number of in steps you already choose right like 500 inner steps are quite a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=760s"},{"start_seconds":768,"text":"huge number especially if we do in eight quantization on it right we can get down the um communication requirements by 2000 right um and for for a lot of the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=768s"},{"start_seconds":777,"text":"things we we didn't need that right for our inter one run for example right we uh we're fine with 100 steps basically doing in eight quantization so we have a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=777s"},{"start_seconds":785,"text":"reduction of 400 basically and we're running uh pretty efficiently across the whole world right um between all those nodes so pretty comparable to to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=785s"},{"start_seconds":794,"text":"centralized training right um so um yeah obviously unfortunately don't have a baseline for that one if anybody want to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=794s"},{"start_seconds":802,"text":"sponsor it happy to to run a baseline experiment but it specific care it it gets too expensive in a sense right","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=802s"},{"start_seconds":809,"text":"um yeah so I have couple technical questions maybe they answer it in the in the report I didn't read it very carefully but one is so so the actual","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=809s"},{"start_seconds":818,"text":"number you use for intellect one is 500 right the number of steps no 100 in our steps so 100 in steps with our quation","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=818s"},{"start_seconds":826,"text":"technique so we have 400 uh X reduction basically in 20 and the and the optimizer use the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=826s"},{"start_seconds":835,"text":"same as in paper like Adam Adam internally n of momentum exactly we didn't want to like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=835s"},{"start_seconds":843,"text":"take risk basically on that front but yeah but but you don't have like in the original paper they train uh in","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=843s"},{"start_seconds":850,"text":"a like a single replica for the first several steps you don't do that right that's a good question like we didn't do","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=850s"},{"start_seconds":858,"text":"that no we we started uh uh yeah randomly initialized from uh uh from scratch basically right um we've been uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=858s"},{"start_seconds":867,"text":"obviously in in contact with different folks have been doing stuff on their Loco of a himself right and uh he was also showing it it works also if you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=867s"},{"start_seconds":875,"text":"just start from a scratch uh not sure why he did it in the paper to uh yeah start from the 24K steps uh checkpoint","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=875s"},{"start_seconds":882,"text":"do Las I I think in the paper they show that it doesn't matter so their point was more like hey if you only do one replica you save some compute but","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=882s"},{"start_seconds":891,"text":"uh I don't know if the complexity of this extra step you know justifies uh no not using Compu for","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=891s"},{"start_seconds":899,"text":"24,000 steps which is negligible anyway okay so uh the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=899s"},{"start_seconds":907,"text":"U let's dive a little deeper into into intellect one so so I guess generally the um something so so so we discussed","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=907s"},{"start_seconds":917,"text":"uh with Ilia a a lot the idea of training a model in a decentralized","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=917s"},{"start_seconds":924,"text":"fashion uh and uh there's a couple interesting things one is that uh training model is easier than training a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=924s"},{"start_seconds":931,"text":"good model right and uh I I I I wonder how you approached uh choosing like I I presume you didn't do","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=931s"},{"start_seconds":939,"text":"a lot of you know like grid search and what not for hyperparameters right so how did you approach using the the particular model architecture hyper parameters and data composition right so","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=939s"},{"start_seconds":948,"text":"like for example I see that you use 55% of where did I see it yeah uh so fine web edu right so so I guess one","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=948s"},{"start_seconds":957,"text":"interesting question is what is fine web EDU and you know how did you approach you know choosing that data composition yeah uh happy to share uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=957s"},{"start_seconds":965,"text":"about the data composition we had help from from huging face they were interested in obviously what we are doing here right and uh yeah uh propos","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=965s"},{"start_seconds":973,"text":"this like kind of data set we are going for right now um and yeah find U is one of the latest data sets uh yeah at least uh in terms of like benchmarks","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=973s"},{"start_seconds":982,"text":"performance and stuff like this one of the best data sets out there um for um yeah doing pre-training basically right as as well as we we put a lot of other","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=982s"},{"start_seconds":990,"text":"data sets into the mix just uh to have more a bit more variety right we had the stack in there which like a coding data set right we had a open rep math in day","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=990s"},{"start_seconds":999,"text":"so more like a a um yeah math data set right um as well as let me guess dclm or something like this I think it's from","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=999s"},{"start_seconds":1007,"text":"Apple um yeah some of the other uh yeah uh open source data sets that came out so there's been a lot of progress","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1007s"},{"start_seconds":1015,"text":"actually in the open source world to yeah just create better pre-training data sets um that wasn't really around like a couple years ago when we all were","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1015s"},{"start_seconds":1022,"text":"training on on the pile by Uther yeah so I I don't know St V2 but I was I was playing with st V1 I remember","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1022s"},{"start_seconds":1030,"text":"St V1 was just individual files so you wouldn't get much in terms of uh uh you know code in context I don't know if V2","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1030s"},{"start_seconds":1038,"text":"is solving it or if it's uh similar and fin has no code right um probably not as much no well","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1038s"},{"start_seconds":1046,"text":"very little um yeah I I was going through fine web with the filter that would say hey only show me the things","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1046s"},{"start_seconds":1053,"text":"that have line break space space which I think is a you know relatively simple filter for code and uh like I went","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1053s"},{"start_seconds":1061,"text":"through the first 2% and I didn't find any I think that's sufficiently good indication probably doesn't I mean they probably","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1061s"},{"start_seconds":1069,"text":"intentionally filtered it out as a out of not a Target use case I guess makes sense yeah um I may to share","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1069s"},{"start_seconds":1078,"text":"a bit more there right on the on the data set side as well as obviously the intellect one model it was the the first release to basically show it's possible Right to to train efficiently uh a 10","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1078s"},{"start_seconds":1087,"text":"billion parameter model around the world right obviously we want to uh make it a decent model right we we Tred on a a decent data set obviously we trained on","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1087s"},{"start_seconds":1096,"text":"a full like one trillion tokens right obviously the current llama series and uh a lot of the other models basically","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1096s"},{"start_seconds":1103,"text":"out there in the open source right now are trained on a lot more tokens right um they get better performance we just want to show hey you train on one trillion tokens in a decentralized","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1103s"},{"start_seconds":1112,"text":"fashion such a model and you get a model that you would expect to to be um you of this quality in a sense right um so that","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1112s"},{"start_seconds":1120,"text":"was the plan for intellect one but obviously for for intellect two we we have a lot more planned there decentralized training needs to show that can actually train a Frontier Model","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1120s"},{"start_seconds":1128,"text":"obviously right and uh yeah that's uh yeah the the moment like all the like more like engineering challenges are currently solved right we we had a lot","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1128s"},{"start_seconds":1137,"text":"of like issues with for Toler and the training obviously right um where uh we we fix stuff in production in a sense","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1137s"},{"start_seconds":1144,"text":"right to uh fix all the failer modes for for tolerant see where where was the failing right and then at the end of The","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1144s"},{"start_seconds":1152,"text":"Turning one I think we got a pretty decent stack now of uh yeah people being able to to onboard to the training one as well as like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1152s"},{"start_seconds":1160,"text":"offboard yeah and and it's like architecture Wise It's just uh it's like Lama right like Gago RMS Norm","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1160s"},{"start_seconds":1169,"text":"exactly we we took the Llama three architecture and uh yeah just added a couple more layers basically to to the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1169s"},{"start_seconds":1176,"text":"Llama 38b one same tokenizer um so it didn't really innovate there but also very excited obviously on all the uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1176s"},{"start_seconds":1185,"text":"yeah architecture improvements but yeah for for the initial one didn't want to take the risk to uh uh yeah that it could be failing because of those kind","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1185s"},{"start_seconds":1192,"text":"of reasons to to change the different architecture yeah but and and it's not sure of experts it's a it's","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1192s"},{"start_seconds":1202,"text":"not yeah yeah so I'm looking at the table with the comparison right so it's a and uh specifically intellect one","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1202s"},{"start_seconds":1211,"text":"versus llamas I I think llamas is the most interesting Benchmark at least because you know actually don't know some of the other models I don't know maybe some of them are more exciting but","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1211s"},{"start_seconds":1220,"text":"llama is a I mean it's an industry standard more or less uh and so on certain benchmark","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1220s"},{"start_seconds":1228,"text":"I would say my my vision of what I see is that on most benchmarks it's a little worse than 7B right which I guess is","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1228s"},{"start_seconds":1235,"text":"still pretty good for the first model uh but uh interestingly on grade school","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1235s"},{"start_seconds":1242,"text":"math it is outperforming even the 13 B Lama which is uh it could be I guess because of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1242s"},{"start_seconds":1249,"text":"5% data mix but it's interesting right so it's already outperforming on some benchmarks yeah for sure um there could","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1249s"},{"start_seconds":1257,"text":"be a reason uh find web U I think has a lot of like great uh um yeah more educational data in there so would expect it to be good at uh great school","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1257s"},{"start_seconds":1266,"text":"Marth in a sense right so uh that could be one of the reasons uh too um but in general we were pretty happy with the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1266s"},{"start_seconds":1273,"text":"with the benchmarks results that um yeah was scaling uh at least to that size in a sense right all the other models we","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1273s"},{"start_seconds":1280,"text":"compared that to on the Lama 2 Series at least right we're trained on like two trillion tokens right so um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1280s"},{"start_seconds":1289,"text":"that's that's a good point right I guess if Prime intellect was train for twice as long that would be right but but even","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1289s"},{"start_seconds":1296,"text":"like I guess you know La llama has a you know like 60 people centralized cluster it's uh like I think for the first model","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1296s"},{"start_seconds":1305,"text":"even even if it underperforming not because of the smaller number of tokens it's still an outstanding result and and a pretty good prove that that the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1305s"},{"start_seconds":1313,"text":"approach is viable which is uh which I think is quite important I think generally it's it's it's my it's my very big uh I don't know if I'm fully","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1313s"},{"start_seconds":1322,"text":"convinced uh that scale will continue right but if it does continue uh you know the the fact that","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1322s"},{"start_seconds":1330,"text":"uh only you know three entities in the world can train can train a one trillion parameter model is not is not where we","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1330s"},{"start_seconds":1336,"text":"want to be as a society right if uh so so I think it's a you know the effort to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1336s"},{"start_seconds":1344,"text":"to make it work right now is uh I mean the best time was last year uh but the second best time is right now exactly","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1344s"},{"start_seconds":1353,"text":"yeah so something else uh which is interesting that I wanted to ask is uh I spent quite a bit of time last year","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1353s"},{"start_seconds":1362,"text":"training large language models uh and one problem I was fighting with which uh uh there's very little information","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1362s"},{"start_seconds":1369,"text":"online about fighting it it's mentioned as an existing problem but not much in terms of a solution spikes so I wonder if you observed spikes and how you dealt","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1369s"},{"start_seconds":1378,"text":"with them if you did yeah honestly we we didn't have to uh uh yeah let me think about it but I don't think we had any like um yellow","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1378s"},{"start_seconds":1387,"text":"Spike basically in the current tr1 right um which was interesting like that's one of the things you need to test in like a large scale train1 right it could have","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1387s"},{"start_seconds":1395,"text":"very much happened that Delo is uh not as good in terms of uh yeah training stability in a sense right that it could","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1395s"},{"start_seconds":1403,"text":"have like especially in the end of training basically have l l spikes right um where yeah you just have to restart","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1403s"},{"start_seconds":1411,"text":"from a checkpoint and hope that it is not happening again in a sense right um we we obviously we integrated some research on top of actually the Lama 3","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1411s"},{"start_seconds":1420,"text":"architecture with the set loss uh that one of our researchers a huge fan on for fof for like observing how um yeah","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1420s"},{"start_seconds":1429,"text":"stable the training is in a sense right um where you can basically identify lost spikes earlier than they would happen basically um so we implemented that one","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1429s"},{"start_seconds":1437,"text":"looked all good uh didn't uh yeah look like it's going to have a lost Spike right but yeah yeah I guess the reason I I thought","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1437s"},{"start_seconds":1446,"text":"of a loss spike is that there is like a little spike in the training but now that I look at the picture that's because the data composition has","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1446s"},{"start_seconds":1453,"text":"changed oh yeah that's a good point um at the the end of training so basically maybe to to share there a bit um what we","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1453s"},{"start_seconds":1461,"text":"went with with the learning rate scheduler so um usually when you train your model you have to Define in the beginning how many tokens are you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1461s"},{"start_seconds":1468,"text":"training on um so you have a a specific learning rate schedu that is just uh um un leing the learning rate to zero at","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1468s"},{"start_seconds":1475,"text":"the end of the training R and like a um yeah Co and curve and uh what we did in the training one from like research from","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1475s"},{"start_seconds":1482,"text":"also I guess from hugging phase but also other uh um yeah organizations uh yeah released this over the last couple","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1482s"},{"start_seconds":1490,"text":"months that they've been training models like this is basically to have a constant learning rate after warm-up phase where you can basically then decide at which point do you actually","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1490s"},{"start_seconds":1498,"text":"want to do um yeah in an needing phase to actually get the model yeah a need the learning f rate to zero but you you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1498s"},{"start_seconds":1506,"text":"basically have the option to train on as many tokens as you want right and um yeah we obviously publish the checkpoint and stuff like this right so if anybody","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1506s"},{"start_seconds":1514,"text":"want to take it continue training they can do it right um yeah as well as uh like us uh obviously we are bit more","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1514s"},{"start_seconds":1522,"text":"focused on the inter 2 um Direction right now but uh yeah theoretically you could just uh continue to training basically uh indefinitely Or how big a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1522s"},{"start_seconds":1531,"text":"data set is yeah and so with intellect to what are the big changes like is it different is it the same architecture is it the data","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1531s"},{"start_seconds":1539,"text":"set composition or what are the what are you guys looking into um what we're looking into is uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1539s"},{"start_seconds":1547,"text":"yeah fixing some of the limitations we have right now with the local run right so one of the things obviously want to get to bigger model sizes right so uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1547s"},{"start_seconds":1555,"text":"yeah the the current like network bandr is it's actually not the liming factor in a sense um in some regions it's more also about the memory requirements you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1555s"},{"start_seconds":1563,"text":"have right like if you want to train the 10 billion parameter model on an h100 node uh yeah that's almost already limit right of what you can train on an h100","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1563s"},{"start_seconds":1571,"text":"node if you Shard like your parameters gradients and optimizers right um so we need a way to to use less memory right there are different optimizers for doing","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1571s"},{"start_seconds":1580,"text":"this uh as well as like more the galor type research approaches right so we've been exploring over the last couple months all of those research directions","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1580s"},{"start_seconds":1588,"text":"to to make that possible um as well as the other component is to to scale to more more providers more compute","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1588s"},{"start_seconds":1596,"text":"contributors right so uh right now uh delu has uh diminishing returns after like 8 to 16 or in our case we we've","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1596s"},{"start_seconds":1604,"text":"been running on up to 14 NES right um and we've been working on researchers and have a a paper coming up to actually make it possible to um yeah scale to a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1604s"},{"start_seconds":1613,"text":"lot more providers right or compute providers so uh that's one of the things things uh we we are trying to to to to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1613s"},{"start_seconds":1621,"text":"tackle right um the the memory requirements as well as the number of Compu contributors we can scale to and","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1621s"},{"start_seconds":1628,"text":"then uh yeah we want to train a Frontier Model for the next one right so it's uh not about only uh the model right and","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1628s"},{"start_seconds":1635,"text":"the local scales but we also need to make progress on um yeah the data set size and a lot of other components so um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1635s"},{"start_seconds":1643,"text":"we've been uh exploring a bit in the large scale St data generation domain and uh yeah um have some some upcoming","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1643s"},{"start_seconds":1651,"text":"uh releases on that front basically um but yeah that's that's uh all the research Direction we've been basically exploring for for an inter 2 launch yeah","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1651s"},{"start_seconds":1660,"text":"so I guess one interesting question here is that um you know we discussed earlier that in your original paper that you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1660s"},{"start_seconds":1667,"text":"published in October I guess or earlier uh the Loco was not doing as well as uh data parallel right but you're saying","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1667s"},{"start_seconds":1675,"text":"that you know down down the you know with a large number of tokens it converged so there's a good chance that","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1675s"},{"start_seconds":1682,"text":"you know like if you if you run deu with with a 30 replicas that it's doing poly at the beginning but maybe you know long","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1682s"},{"start_seconds":1690,"text":"term it does as well as the data parallel with f replica so it's it's a it's something that is very hard to experiment with without actually running","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1690s"},{"start_seconds":1698,"text":"a long run so it's a but but you know it could be that Delco actually just you know it could be","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1698s"},{"start_seconds":1706,"text":"that it just scales to larger number of replicas we just don't know it yet without running an expensive experiment but but also something I wanted to ask","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1706s"},{"start_seconds":1713,"text":"is uh what in the current run a single node is it a single machine with eight h100s or is it like a small","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1713s"},{"start_seconds":1721,"text":"cluster I see yeah right now it's a h100 node basically is a is a single worker so um we made it open for anybody to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1721s"},{"start_seconds":1730,"text":"basically contribute jut resources through the platform basically they could just launch an instance there right uh as well as everybody else uh that provided compute like a h face","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1730s"},{"start_seconds":1738,"text":"obviously uh had their own compute resources to to put into the training one right um yeah so in a short term right like let's say we want to train a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1738s"},{"start_seconds":1746,"text":"70 billion parameter model right and you mentioned that 10 billion uh is sort of the limit of what fits on a single machine right so in the short term","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1746s"},{"start_seconds":1755,"text":"something that presumably will work is that you say hey a single node now is not is not a machine it's a it's a small claster of eight machines because","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1755s"},{"start_seconds":1762,"text":"majority of providers give you like like you know like if you go to Lambda or yep like any of them they give you like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1762s"},{"start_seconds":1771,"text":"they give you small clusters with interconnect right it wouldn't be impossible to imagine especially you could do it right now if you uh want to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1771s"},{"start_seconds":1779,"text":"do a launch basically over a 7 billion model decentralized in the same setting that we got in a sense right like we had also like single nodes and data centers","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1779s"},{"start_seconds":1788,"text":"where there wasn't other nodes available because we've running on spot instance Z or something like this right but for uh the general use case there's a lot of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1788s"},{"start_seconds":1795,"text":"data centers with more 81 than 8 h100 um which could interconnects and that's also the one of the reasons why this","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1795s"},{"start_seconds":1803,"text":"approach is especially interesting for the big labs in a sense right it's not only for like the open source Community right trending across their different uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1803s"},{"start_seconds":1811,"text":"smaller clusters right it's one of the um things you could use it for but also for for the big labs to train across like uh yeah four to maybe 16 different","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1811s"},{"start_seconds":1820,"text":"data centers that they got right yeah exactly especially Google right that has a you know historically has a lot of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1820s"},{"start_seconds":1828,"text":"locations with the tpus right and and I think so so I see 112 I will not be able to divide it","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1828s"},{"start_seconds":1837,"text":"by eight on my head I suspect it is 14 right 14 yeah yeah right so so right now you use 14 replicas right so so like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1837s"},{"start_seconds":1845,"text":"finding 14 clusters of eight machines is extremely doable right so it's a so so like short term there is a simple solution because","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1845s"},{"start_seconds":1853,"text":"I think the Paco results were very underwhelming right so it's it's presented as a super cool solution but once you start like you know running","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1853s"},{"start_seconds":1861,"text":"math in your head you realize that deak is very data inefficient which I guess a sort of expected right given that it's actually training a much smaller model than it's","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1861s"},{"start_seconds":1869,"text":"more like a mixture of experts yeah like I'm still like uh yeah interested obviously in the more mixure","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1869s"},{"start_seconds":1877,"text":"of expert type of ways right on how we can train decentralized right I'm not sure if it's a approach they've been going for there with like a sequence","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1877s"},{"start_seconds":1885,"text":"level mix of expert right so um what you usually have is you uh when you train your mix of expert right you route per","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1885s"},{"start_seconds":1893,"text":"token and it's uh actually extremely like communication intensive right um and they route per per sequence right um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1893s"},{"start_seconds":1900,"text":"to like different experts which made it possible to uh yeah train this decentralized um but yeah also not 100%","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1900s"},{"start_seconds":1907,"text":"convinced yet that's the way if this is actually producing the best model that is is possible and uh it's not worse than a normal mixture of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1907s"},{"start_seconds":1916,"text":"expert yeah that that that yeah so so so I noticed when you were talking about ideas for Prime intellect 2 for intellect 2 you didn't notti you didn't","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1916s"},{"start_seconds":1924,"text":"mention that direction I think it's more about the novel like architectures obviously we've","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1924s"},{"start_seconds":1933,"text":"been launching something like a non GPT challenge basically or like uh somebody did and we Rec created a leaderboard for","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1933s"},{"start_seconds":1939,"text":"it right um which is uh an interesting challenge to uh yeah basically get uh better model performance um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1939s"},{"start_seconds":1948,"text":"yeah with with L like gets the same model performance on like a gbt2 benchmark to the same loss on F to uh uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1948s"},{"start_seconds":1957,"text":"yeah just less tokens right um so those are more approaches we've been interested in to to innovate for an interl","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1957s"},{"start_seconds":1964,"text":"to yeah what is the timeline when do you guys I guess not even just what is the timeline the first question is are you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1964s"},{"start_seconds":1973,"text":"training something right now right you have you have those resources are the ized or are they waiting for intellect","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1973s"},{"start_seconds":1981,"text":"to we've always been we've uh we are always running ablations in a sense right to to make sure what we want to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1981s"},{"start_seconds":1988,"text":"train next right um but we don't have intellect to running right now I can say that","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1988s"},{"start_seconds":1994,"text":"much yeah so I I guess couple interesting so so at some point I want I want to discuss the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=1994s"},{"start_seconds":2003,"text":"step four of the master plan uh I think it sounds very exciting but before we even go there right so um so so like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2003s"},{"start_seconds":2011,"text":"training uh train training a large model is very expensive right so like let's say if you want to train a 400b uh that will be something that is extremely","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2011s"},{"start_seconds":2019,"text":"expensive right it's like uh I mean it used to cost $100 million maybe it's cheaper now h100s were uh continuously","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2019s"},{"start_seconds":2026,"text":"getting cheaper but it's still at least an eight digigit number right so it's a what is your thinking in terms of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2026s"},{"start_seconds":2035,"text":"uh uh like I was thinking of mon ing the models eventually uh or or are you thinking of it more as a you know uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2035s"},{"start_seconds":2042,"text":"Humanity pulling resources to create public good without an expectation of the returns like how do you think of the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2042s"},{"start_seconds":2050,"text":"business model that's basic two Dimensions we we see on this too right like uh either you're on the front of like monetizable open source models","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2050s"},{"start_seconds":2058,"text":"which is an interesting research Direction I'm happy that people exploring it in a sense right to to make that possible uh and uh yeah um seems","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2058s"},{"start_seconds":2067,"text":"interesting to my my current guess or like how we see it basically is that uh yeah everything will be commoditized in a sense right what we are doing here we we commoditize compute with for our","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2067s"},{"start_seconds":2075,"text":"computer exchange right um to always give you access to to the cheapest computer resources make completely commoditized as well as on the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2075s"},{"start_seconds":2083,"text":"intelligence side right um like the the models uh that are out there seem to be uh yeah becoming commoditized in a sense","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2083s"},{"start_seconds":2090,"text":"too right and we we see it more as like a public good of people coming together want to pull resources to actually create of the art models and yeah will","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2090s"},{"start_seconds":2099,"text":"be the greatest public good in History to have like open source AGI yeah so so so you're thinking more of like people just like voluntarily","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2099s"},{"start_seconds":2108,"text":"contributing compute into a very large cluster exactly like I I rather think that it's going to be more like a public","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2108s"},{"start_seconds":2116,"text":"good than uh being monetizable but happy to people Pro them wrong and people figure out how to uh monetize open","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2116s"},{"start_seconds":2123,"text":"source models uh in a way um maybe to to give a c there uh like on different directions I'm I'm","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2123s"},{"start_seconds":2130,"text":"more bullish on monetization right um so there are a lot of foundation model use cases where it seems reasonable to to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2130s"},{"start_seconds":2136,"text":"monetize the the model layer so um for example for like a large scale scientific Foundation models uh which we've been really interested in with uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2136s"},{"start_seconds":2145,"text":"yeah my background but also my my co-founder's background in in the D siiz bace um to to innovate a lot there I think there's a lot of possibility for","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2145s"},{"start_seconds":2153,"text":"monetization uh but on the on the Frontier Model lab side uh it's seems to be a lot more commoditized in the sense right um but for for scientific","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2153s"},{"start_seconds":2161,"text":"Foundation models or like any kinds of other models in this direction robotic Foundation model would be interesting um we've been training a a meta metag","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2161s"},{"start_seconds":2169,"text":"genomic Foundation model for pandemic prevention that's going to come out soon um which is also very easily to train in a decentralized fashion in a sense right","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2169s"},{"start_seconds":2178,"text":"so for for those kind of use cases I think we love to explore that to on the monetization side yeah what are the so","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2178s"},{"start_seconds":2187,"text":"you mentioned some research from monetization side I I don't know if you have some particular favorite approaches that you heard","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2187s"},{"start_seconds":2196,"text":"of I think uh some of the ones uh are to to actually make it possible that not everybody gets the weights of the of the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2196s"},{"start_seconds":2203,"text":"model right that's the one of the approaches I've been uh caring over the last couple months right um um so uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2203s"},{"start_seconds":2212,"text":"yeah I I really wonder how that works out in like still having the best perform model and also the the fastest INF people want to have fast inference","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2212s"},{"start_seconds":2219,"text":"obviously right and it can't be expensive there so we've been not uh following or like exploring that closely","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2219s"},{"start_seconds":2227,"text":"um yeah as well as all the the fingerprinting type Solutions right um that uh yeah have people have been proposing um yeah um not not not my area","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2227s"},{"start_seconds":2237,"text":"of expertise yeah I was thinking a lot about it because I think um so I as I mentioned I'm primarily interested in","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2237s"},{"start_seconds":2245,"text":"the situation where the models continue being being bigger and bigger and better and better even though I'm not fully convinced that that's that's what will","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2245s"},{"start_seconds":2254,"text":"happen it it's definitely there's definitely not a trivial probability of that happening right and so how do we make sure that the open source Community","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2254s"},{"start_seconds":2262,"text":"continuously has you know resources to train those models right especially if they require you know hundreds of thousands of h100s then you know that","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2262s"},{"start_seconds":2271,"text":"does the does the community have even collectively access to those resources right and and monetization I was very","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2271s"},{"start_seconds":2279,"text":"interested in it uh and I I think one of the most interesting approaches that we're looking into is um h100s have","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2279s"},{"start_seconds":2286,"text":"trusted execution environments meaning that you can run code without the cost seeing what's happening there or like anyone seeing right and h100s have has","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2286s"},{"start_seconds":2295,"text":"an overhead of like 30% I think people say between 10 and 30 the but b100s will not have any overhead meaning that you","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2295s"},{"start_seconds":2303,"text":"can run in trusted execution environment as fast as if it was not right and so there's this idea of training a model in such a way that","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2303s"},{"start_seconds":2312,"text":"nobody ever sees the weights they always encrypted right and then the model would only do inference for you if you open a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2312s"},{"start_seconds":2318,"text":"payment Channel and pay for it right so anyone can copy it to their machine right you don't have to trust anyone you can you know build your own rig bring","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2318s"},{"start_seconds":2327,"text":"the model to locally got the internet you know you use the QR code to send the payment channel so it's like fully","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2327s"},{"start_seconds":2335,"text":"isolated but uh but you still have to pay to use it so I think that's a very exciting Direction yeah definitely sounds","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2335s"},{"start_seconds":2343,"text":"interesting um and uh excited to see some research there um and uh yeah hopefully the T like supply of the of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2343s"},{"start_seconds":2352,"text":"the next generation of Nidia chips is going to be even yeah greater in a sense right that you could do it on on any","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2352s"},{"start_seconds":2357,"text":"B100 or b200 um so uh yeah that would definitely solve a lot of our issues too","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2357s"},{"start_seconds":2364,"text":"so excited for it yeah and it was very fascinating right h100s came out last year right uh and like by the end of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2364s"},{"start_seconds":2373,"text":"2024 it's a commodity as you mentioned right like I can go to GPU le. uh which is like it's like an aggregator of all","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2373s"},{"start_seconds":2381,"text":"those providers right and and it's like you know $2 per GPU is a like there's a lot of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2381s"},{"start_seconds":2389,"text":"them especially on the large cluster size there still so much like uh yeah um yeah different and um yeah reliability","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2389s"},{"start_seconds":2397,"text":"the providers right so the the rck I Forks had this great paper of like showing like 100x difference in terms of reliability on the Clusters they've been","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2397s"},{"start_seconds":2405,"text":"running on right so I think we have to solve the the for toin part on the software St if n doesn't solve it on the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2405s"},{"start_seconds":2413,"text":"hardware side right um uh where they're obviously still the best obviously in their game but uh they still have issues","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2413s"},{"start_seconds":2420,"text":"right and the providers have issues and uh yeah to to fix it on the on the for tolerant side is probably the way to go","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2420s"},{"start_seconds":2427,"text":"for the next couple years yeah yeah and I don't think Nvidia is investing much into that because you know people already used to the fact that Nvidia","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2427s"},{"start_seconds":2435,"text":"gpus fail in a variety of ways and you know everybody who trains at Large Scale","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2435s"},{"start_seconds":2443,"text":"already have mitigations for it yeah in one way or another okay uh I think the most","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2443s"},{"start_seconds":2450,"text":"interesting thing uh which you know I left for the end is the the the step four","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2450s"},{"start_seconds":2457,"text":"of the master plan which is uh actually one thing I wanted to mention is that I think of prime intellect as a I don't","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2457s"},{"start_seconds":2465,"text":"know um I mean I mean you guys want to build a decentralized protocol so I think of it as a you know some somewhat in this","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2465s"},{"start_seconds":2473,"text":"crypto Anarchy you know set of companies and and it's quite surprising to see a company saying hey guys we had the first","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2473s"},{"start_seconds":2480,"text":"step of master plan and we have more than one right and then check two months later and you noted the first step you you you further along that doesn't","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2480s"},{"start_seconds":2488,"text":"happen very often in the right so so so I think that's that's a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2488s"},{"start_seconds":2495,"text":"very uh that's a pretty cool speed at which you move through your master plan right so so so that's a pretty good indication that step four will happen","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2495s"},{"start_seconds":2503,"text":"you know at some point so so so I would love you to talk more about your vision for Prim in the future um I mean we we","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2503s"},{"start_seconds":2513,"text":"uh structured the master plan in a sense like this way because we obviously have to solve a lot of the research challenges right now to to even get to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2513s"},{"start_seconds":2520,"text":"like more like a prodal based uh system where we can incentivize people to contribute to resources to train like Frontier models and uh yeah create the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2520s"},{"start_seconds":2529,"text":"greatest public good in History to build open source AGI as well as like making progress on science and stuff like this right with uh scientific Foundation","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2529s"},{"start_seconds":2537,"text":"models and so on they've been uh yeah being trained on uh um yeah people contributing GP resources so that's why we structured the way like uh the master","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2537s"},{"start_seconds":2545,"text":"plan like this obviously we want to solve aot of the fundamental research challenges right now right um scaling uh your intellect to to like the the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2545s"},{"start_seconds":2553,"text":"largest model as well as like making more the the trest compute component work right so um we've been exploring different components on um with the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2553s"},{"start_seconds":2561,"text":"least amount of overhead in a sense to to enable uh yeah people to provide compute resources where you can verify they are not sending malicious gradients","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2561s"},{"start_seconds":2570,"text":"um as well as uh uh yeah it's actually an h100 node that they are providing right um so we we are still on the way I","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2570s"},{"start_seconds":2579,"text":"have to solve some of those like trustless compute challenges in a sense as well as like uh proving with intellect to that we can actually train a Frontier Model right and then uh yeah","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2579s"},{"start_seconds":2588,"text":"we'll uh move ever closer to um yeah the last stage of our master plan and um yeah so a lot of that if you trust","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2588s"},{"start_seconds":2597,"text":"Nvidia which we sort of have to a lot of that can be achieved with the trust trusted execution environments in the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2597s"},{"start_seconds":2604,"text":"sense that like Nvidia will send you an station that this is 8 h100s and if you run the code inside of the te you know","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2604s"},{"start_seconds":2612,"text":"that the gradients they send you are correct so so I think short shortterm this is the most viable solution even though right now there's a","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2612s"},{"start_seconds":2621,"text":"lot of like Supply shortage in that sense right we've been talking to a lot of different providers that we have on our platform obviously and those uh uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2621s"},{"start_seconds":2629,"text":"unfortunately have like not the expertise to actually set up like an h100 node with Ed fter execution","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2629s"},{"start_seconds":2637,"text":"environment or don't have the specific uh like CPU chip you need wor uh actually enabling that right the CPU extension","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2637s"},{"start_seconds":2646,"text":"umf TDX you need a fifth generation Intel processor and we we talked to a lot of providers and in fact it is not","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2646s"},{"start_seconds":2653,"text":"something people have today exibits will have some of them in the future like in a very short future but uh that's uh I","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2653s"},{"start_seconds":2662,"text":"mean having one provider is precisely opposite of having yeah Supply issue now so uh we've","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2662s"},{"start_seconds":2669,"text":"been thinking more about more uh yeah more approaches in terms of like staking a slashing and stuff like this right where you have a lot to lose basically","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2669s"},{"start_seconds":2677,"text":"game theoretically if you uh yeah send malicious gradients or um yeah provide computer resources that are not as uh um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2677s"},{"start_seconds":2686,"text":"yeah not the ones that we need for the 21 right um so yeah yeah yeah yeah","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2686s"},{"start_seconds":2693,"text":"that's that's obviously a more stand approach but my thinking with TDX is that first of all um it will","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2693s"},{"start_seconds":2703,"text":"probably change right like same as what happened to h100s which you know like two years ago they didn't exist and now","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2703s"},{"start_seconds":2710,"text":"they quite abundant uh but but also for people who will be joining especially people who will be building rigs in the future you know if","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2710s"},{"start_seconds":2719,"text":"they know that fifth generation Intel processor gives them more opportunities to use it it uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2719s"},{"start_seconds":2727,"text":"definitely bullish on on the future there if if we want to launch something uh early next year there's probably not as much Supply to actually do an inter","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2727s"},{"start_seconds":2734,"text":"to launch on on uh on such a system as well as like obviously right now we we've been still limited to homogeneous","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2734s"},{"start_seconds":2742,"text":"Hardway right we've just been running on h100 noes um but we we also are working on approaches to make uh yeah our Del","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2742s"},{"start_seconds":2750,"text":"work and more to uh be able to use henus compute right so uh we will still mainly","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2750s"},{"start_seconds":2757,"text":"be focused on more the data center GPU uh just because of the rerum requirements we got for actually training state-of-the-art models right","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2757s"},{"start_seconds":2766,"text":"um but at least uh like it's still good to to train on A1 100s right um so there's nothing wrong with that a lot of","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2766s"},{"start_seconds":2774,"text":"researchers don't want to but um yeah there there's no reason to also include A1 100s and the training one","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2774s"},{"start_seconds":2782,"text":"multiple noes and stuff like this and a 100s also has 80 GB of memory right so it's like the same size of the model","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2782s"},{"start_seconds":2790,"text":"will fit it's just the number of steps you can do it smaller exactly we didn't mention in The Deco Paper what happens","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2790s"},{"start_seconds":2798,"text":"if in the same amount of time you know so so you cannot have the number of steps constant anymore right now it has","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2798s"},{"start_seconds":2805,"text":"to be variable presumably right because a 100 I cannot do 100 steps in the same time exactly or","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2805s"},{"start_seconds":2814,"text":"you run on two a 100s and then you are roughly the same time but that's not what we went with um it it needs to be a more like asynchronous approach right","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2814s"},{"start_seconds":2822,"text":"right now it's very synchronous right they Al synchronize after 500 steps and yeah we we've been very interested in how to make the local work asynchronous","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2822s"},{"start_seconds":2831,"text":"um there's also a paper actually from from Google deep mind uh that did D asynchronous right um but also just","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2831s"},{"start_seconds":2839,"text":"scaled it to the I think only 150 million parameters and and not more and had to do a lot of tricks actually to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2839s"},{"start_seconds":2845,"text":"make it work uh asynchronous um but yeah we we we thinking about mik an easy","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2845s"},{"start_seconds":2851,"text":"approach there um yeah yeah so so actually couple couple more interesting questions one is","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2851s"},{"start_seconds":2860,"text":"um like the number of steps you take between uh gradient synchronization what is it like how does it map into the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2860s"},{"start_seconds":2868,"text":"realtime clock like how frequently the guardian synchronization happens in terms of like seconds oh yeah um so for inct one run","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2868s"},{"start_seconds":2877,"text":"where we were sying every 100 steps right um we've been sying every 40 minutes so they were running completely independently on those different islands","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2877s"},{"start_seconds":2885,"text":"for 40 minutes and then yeah depending on our configuration we are different synchronization times so we've realized for example in the US um you have pretty","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2885s"},{"start_seconds":2893,"text":"fast interconnect between all those nodes right so we had a 4 gigabit interconnect between all the US nodes uh that we got like all over the US um so","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2893s"},{"start_seconds":2902,"text":"uh we've been then the all reduce basically of all the different notes and the took one minute then um and then we uh in the training one uh we added","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2902s"},{"start_seconds":2911,"text":"Europe right uh so we had some some notes in in Sweden and Finland I believe and then our all reduced time went down to five minutes right which was still","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2911s"},{"start_seconds":2920,"text":"pretty efficient right we we trained for 40 minutes and then we sink for five minutes which is uh still acceptable right and then I think when we added","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2920s"},{"start_seconds":2928,"text":"Asia some Singapore notes and indiaan notes uh we were at six minutes or reduced time on average um so uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2928s"},{"start_seconds":2937,"text":"yeah that's about the computer efficiency you can expect there with 100 steps yeah is already speaking uh uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2937s"},{"start_seconds":2947,"text":"geography into consideration like is it more efficient to all reduce all in us than all in Asia and then like you know the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2947s"},{"start_seconds":2954,"text":"four-way or or that's not happening yet that's the plan it didn't happen yet so right now what we do is like a literally","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2954s"},{"start_seconds":2961,"text":"ring all R so we we've been literally uh when we like uh yeah yeah um all reduce","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2961s"},{"start_seconds":2968,"text":"between all those different locations we we've been doing a ring the around the whole world um but obviously there are even more efficient ways to do it like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2968s"},{"start_seconds":2976,"text":"one of the things we could think more often in like a continent for example or generally where you have fast interconnect sometimes it's actually pretty weird like we had some","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2976s"},{"start_seconds":2985,"text":"connections to Europe that were extremely fast because uh uh yeah some kind of internet routing uh um uh yeah","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2985s"},{"start_seconds":2993,"text":"internet routing um reason right and uh yes um but uh yeah fundamentally usually","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=2993s"},{"start_seconds":3002,"text":"it's like in a continent you're you're fast between the nodes right um so there's definitely ways to make use of that for for our next run yeah and do","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3002s"},{"start_seconds":3011,"text":"you know already how big intellect 2 will be let's see um let's see how how much progress we make obviously on the um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3011s"},{"start_seconds":3020,"text":"yeah on the on the ream side size how much we can actually fit into a Noe right um that's one of the things uh as well as we want to make progress uh not","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3020s"},{"start_seconds":3029,"text":"only do data portal training obviously which we do right now right uh which is fundamentally limited do you have to do the whole training on a note um but","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3029s"},{"start_seconds":3036,"text":"actually split up the model with like pism for example right um so there we could actually scale to a lot bigger","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3036s"},{"start_seconds":3043,"text":"sizes um but it's also another engineering challenge that we have to solve under in too yeah Al so so you think that will","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3043s"},{"start_seconds":3051,"text":"already be you're already expecting to have progress on that for front by the intellect","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3051s"},{"start_seconds":3059,"text":"too it's a good question if we need it like we have uh uh yeah two streams basically going on there right now right like you could either like work on a PR","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3059s"},{"start_seconds":3067,"text":"just to reduce a VR requirements on a note right to actually train your model on a note um like larer models on one note or you can split up the model right","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3067s"},{"start_seconds":3075,"text":"um and uh do do pipeline portal training um like uh yeah one uh um yeah great","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3075s"},{"start_seconds":3083,"text":"researchers from from together AI Maxin who's been uh yeah uh like fundamentally inventing the whole field of decentralized training right he's been","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3083s"},{"start_seconds":3092,"text":"working with us on the intellect t or advising us for for research there and here has this great paper on Swarm uh which uh yeah is like a pipeline parall","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3092s"},{"start_seconds":3101,"text":"decentralized training approach right which uh nobody has scaled yet for some reason so I'm excited I would love to","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3101s"},{"start_seconds":3108,"text":"yeah if some people want to do open source contribution work to to our Prime Reaper or uh whatever else right in terms of like making pipeline p training","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3108s"},{"start_seconds":3117,"text":"work across the world right uh that would be amazing and yes there's a lot of potential I guess into uh yes sing the Swarm","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3117s"},{"start_seconds":3124,"text":"algorithm whose paper is that Max vinan um is uh yeah a researcher who has been","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3124s"},{"start_seconds":3132,"text":"doing his PhD actually in decentralized training right so a lot of the fundamental work in in terms of uh yeah this research direction is coming from","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3132s"},{"start_seconds":3142,"text":"him okay cool we still have uh five more minutes give or Tex so I wanted to also uh briefly talk about uh uh the the post","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3142s"},{"start_seconds":3152,"text":"training so so I see it's done by uh by AR I don't know if AR is completely separate entity if you just guys work with them or what is the relationship","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3152s"},{"start_seconds":3161,"text":"there who's comput it is who's incentive Etc uh yeah asdi is is a yeah a really","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3161s"},{"start_seconds":3169,"text":"good friends of us basically that are um yeah one of the leading Orcs I would say on uh you're doing post training of those uh uh Foundation models right so","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3169s"},{"start_seconds":3178,"text":"uh they have those RC Supernova models which are pretty popular and uh yeah have a whole team like just specialized","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3178s"},{"start_seconds":3186,"text":"on post post training of uh of of Bas mods or uh even continuous post training of uh um yeah instruction T models right","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3186s"},{"start_seconds":3195,"text":"um so uh yeah really grateful for their support basically on the interleg one uh release uh they actually made it possible that this space model was able","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3195s"},{"start_seconds":3204,"text":"you could actually chat with the model right um which uh uh yeah we didn't in house have the expertise to to do it as","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3204s"},{"start_seconds":3211,"text":"quickly as they did so we we sent them the weights uh on the day when uh the intellect one run was done right and","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3211s"},{"start_seconds":3218,"text":"immediately uh yeah started cooking on it and had a pretty good mod um at the end of the week then um which yeah we","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3218s"},{"start_seconds":3226,"text":"deployed in a chat form they everybody could chat with it right as well as like sour it right um and yeah they they did","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3226s"},{"start_seconds":3232,"text":"a bunch of different approaches um so one of the things they um always do on those models that um uh yes they","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3232s"},{"start_seconds":3241,"text":"innovated in a sense or um uh is the merge kit work so basically uh merging different open source fine tuning uh","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3241s"},{"start_seconds":3250,"text":"models right uh to get one bigger better model out of it right so um that's also what they've done on our in one run so","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3250s"},{"start_seconds":3257,"text":"doing a bunch of different different fine tunes basically and then uh yeah merging all of those the best funds at the end to to get the best mod out of it","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3257s"},{"start_seconds":3267,"text":"yeah so so if people want to chat with the intellect one where do they go uh they go on chat. Prime intellect. um","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3267s"},{"start_seconds":3281,"text":"I I didn't see it in the Tweet storm so it's a Well hidden yeah was a long tweet","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3281s"},{"start_seconds":3288,"text":"storm yeah okay that's uh uh and uh where is is Prim itself","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3288s"},{"start_seconds":3296,"text":"distributed or or where you guys located um most of the people are in in San Francisco um so uh yeah most of the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3296s"},{"start_seconds":3305,"text":"people are here but obviously we're happy to uh have a lot of like disabil uh community members in a sense right uh that uh can work on different research","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3305s"},{"start_seconds":3313,"text":"challenges right uh so um yeah if anybody interested we have a lot of challenges to solve and uh hope isn't interesting research uh problem to work","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3313s"},{"start_seconds":3322,"text":"on um yeah yeah okay uh that was a uh that was that was a very fun","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3322s"},{"start_seconds":3331,"text":"discussion I I don't know if we discussed everything if there's if there's something uh I didn't ask that you think is important uh in the I think you asked","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3331s"},{"start_seconds":3339,"text":"yeah great question I wasn't even expecting such a technical discussion so it's amazing that you went so deep on like all the the local work previously","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3339s"},{"start_seconds":3347,"text":"with Y to so really enjoyed it and uh that I could share some of the research stuff here too yeah yeah we can put like","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3347s"},{"start_seconds":3355,"text":"any links below the video to the to the relevant stuff so it's a uh yeah but uh yeah I think I think this is one of the","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3355s"},{"start_seconds":3362,"text":"most exciting directions the decentralized community can and should take today so it's it's exciting that there's actually","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3362s"},{"start_seconds":3370,"text":"someone who's who's working on it actively yeah and so yeah let's see how","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3370s"},{"start_seconds":3377,"text":"two you know when it comes out and uh yeah everybody who wants to play today it's","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3377s"},{"start_seconds":3384,"text":"chat. prime. right exactly yeah lot for joining us","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3384s"},{"start_seconds":3392,"text":"today thanks Alex","url":"https://www.youtube.com/watch?v=r0pweLNP21w&t=3392s"}],"youtube_url":"https://youtube.com/watch?v=r0pweLNP21w","url":"https://near-pedia.xyz/videos/r0pweLNP21w"}