DiPaCo paper: Decentralized Model Training
Nothing is loaded from YouTube until you press play, and then without tracking cookies.
About this video
The DiPaCo paper from DeepMind, highlighting its challenges and potential in decentralized model training. This video sheds light on the future of machine training and how it may reshape the landscape of AI research. #DiPaCo #DeepMind #DecentralizedAI #MachineLearning #AIResearch #ModelTraining #ArtificialIntelligence #TechInnovation #ComputationalModels #FutureOfAI #NEARai #NEARprotocol
Transcript
143 wordsCaptions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.
- 0:00
yeah so so there's a follow-up paper from Deep Mind called deaco which tries to solve the problem with models not fitting on a single machine and that one is way more involved it's not just you know a small change in the optimization step and the experiment section is uh um if you try if you like try to extract Apples to Apples comparisons you will notice that daku is not doing as great which is probably sort of something you would expect when you um because it's only using a subset
- 0:29
of parameters on pass so but it is it is a great forward first step right and like from here again in in a form of a competition we can push it forward and hopefully figure out a way to train a massive model in a in a decentralized manner