Training 7B LLMs on Edge Devices: Bridging Decentralized AI and Resource Constraints

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:10

   Uh hello everybody. I'm Jerry. I'm an uh independent IML researcher. Uh previously I was a senior AI engineer at digital currency group and a deep learning engineer at Intel. Today I want to talk about something that every builder in this room cares about and that's how to train your 7 billion parameter model on uh the same um hardware that your decentralized nodes are running. Um a little bit um yeah a little bit background before we go on at

2. 0:42

   digital currency group um I was part of developing and designing the decentralized AI systems. I have experience uh fine-tuning open source large language models like uh llama quen Microsoft across GPU backed infrastructure and at Intel I was part of the TensorFlow core optimizations team where some of the code that I shipped um improved convolutional operations by 400% on uh CPUs using

3. 1:14

   Intel's 1DNN core library. My research focus is uh resource efficient fine-tuning and uh so this isn't just research for me. I've been building and deploying as um AI systems for a while and that's the lens that I want to uh share this with you.

4. 1:32

   Okay, so here's the problem. Um so a 7 billion parameter model right now would require around 100 to 120 GB of VRAM just to train. That's a cluster of H100 GPUs. It would cost around $50,000 to buy or $500 for a single cloud run on GCP.

5. 1:55

   But um our edge node, our edge nodes um consumer GPUs like 4090s, 3090s, they have around 80 to 24 GB of VRAM. So we see there's like a huge gap here. And this gap matters more in 2026 than ever because models are getting bigger with architectures. Uh public data is running out and it's estimated to be completely gone by 2032 and soon privacy will be non-negotiable which means that um

6. 2:26

   ondevice training would be not just preferred but compulsory. Why does decentralized AI need this? So in a decentralized network not everybody first this heterogeneous hardware. So on a decentralized network not everybody would be running the same hardware.

7. 2:44

   Everybody would be running completely different like some some person would be running like a 4090 someone laptop GPUs. So you need a uniform stack for all of this. Second is like I said before privacy may not be you know preferred it may be compulsory. So data needs to stay on device. And thirdly, uh decentralized AI gives something that central AI doesn't, which is verifiable

8. 3:08

   contributions. This means uh onchain voting and uh incentive mechanisms which could enforce quality. So how do we close the gap between the 100GB of requirements to 8 to 24GB of reality? Okay, so here's the full landscape from the most expensive to least expensive.

9. 3:31

   So full fine-tuning in this method um around all the parameters of the model are updated. So that could take a 100 plus GB as I previously talked about and the next less expensive way is adapter layers which is actually a much better option here. What happens is trainable modules are um placed between frozen layers and because of that that reduces the model size from 100 to 60GB because you're freezing the model and after that is the real breakthrough called Lora which

10. 4:10

    stands for low rank adaptation again here the model is frozen but what happens is along with the attention layers uh a few amount of tiny matrices are trained along with that and this completely reduces the VRAM to 20 to 30 GB and after that is Qura which is basically Laura plus quantized. So I have to tell you Laura stands for lower rank adaptation. Qura stands for quantized lower rank adaptation. So what

11. 4:41

    happens here is uh the model would be all the um operations would be in 32-bit. Here what happens is it's reduced to four bits called NF4 and this makes sure that the memory requirements come down to 6 to 10GB. So with this along with a couple of uh stability uh techniques like gradient checkpointing, early stopping, gradient clipping, this could definitely make it run on our edge devices. And if you add it with this technique called flash

12. 5:16

    attention 2, basically what that happens is it recomputes uh attention and kernel fusion to run on the fast SRAM and it's much closer to the CPU. So what that h what that gives you is it gives you a longer context length. So uh Laura has been actually there for a while now. I think it came in 2021 or 2022 but nowadays there have been more advancements since then and first is uh DORA which stands for weight decomposed lower rank adaptation. So here what

13. 5:50

    happens is the weight updates are split into magnitude and directions. They're updated and then projected back. Second is galore which stands for gradient low rank projections. In this method, it's actually um completely different because it doesn't touch the weights at all. It takes the gradients, projects them to a low rank space, updates them, and projects them back. Third is Vera, which

14. 6:16

    stands for uh vector-based random matrix adaptation. So basically here what happens is a lot of the frozen matrices are similar. So instead of storing these matrices what happens is it stores only tiny scalar vectors which contains only the scaling factor. So that reduces the memory a lot and this is very popular in distributed uh environments. And lastly is mobile fine-tuning which I also came across only a few days back in which

15. 6:47

    models can be fine-tuned on our phones which is like ultra edge which is even more than edge and uh they use C++ back propagation for this okay so what is that that you need to use which framework that you need to use I would suggest if you have a data center of A100s or H100s you should definitely go ahead with Laura or full fine-tuning but anything after that from RDX 4090 or 3090 3080 which has less

16. 7:19

    than 48 gigs of RAM I think Hora should be your baseline and that coupled with flash attention and the stabilizing techniques that I previously talked that would definitely um help to uh be able to train the 7 billion model on your edge nodes.

17. 7:39

    Okay. So what's worked and what's really hard. So this is the exact stack that I used at DCG. Um this helped me to be the top performer across seven decentralized u AI tasks on the flock network and this surprisingly this techniques also helped me to reduce the accuracy gap while uh full finding. If we like for example when we go a few slides earlier I was talking about how full fine-tuning is

18. 8:14

    100% quality and Qura comes to 90 to 95%. With those techniques it is actually possible to make it just less than 1% gap between the two if you know how to use it right. And here's what's hard and what most people don't talk about is that in a decentralized network as I previously mentioned there's a hardware variance. So when in situations like that, it's extremely vital that you uh make your training stack that is uh

19. 8:46

    good for the lowest tire and not just something that's uniformly work. It has to be uniformly working but the lower tire needs to absolutely work. And secondly is quantization has a floor. So earlier in Qura we are quantizing it from 32bit to NF4 which is 4bit but um if you go below that there is a significant change and you would start noticing it immediately. So quantization is not unlimited. It's not infinite. You can only quantize to a certain level.

20. 9:20

    Thirdly is there's a coordination tax. So since we're doing it on blockchain, there's additional overhead that you need to budget for like um incentive mechanisms, staking, validators, and this would additionally occupy some space as well. And this is something that you need to look forward to. And last is uh data distribution gaps.

21. 9:43

    So since everyone's data is private and everyone's uh training completely different you don't know the exact distribution of their data. So because of that the same model could converge faster on someone's data and converge slower on other people's data. So because of that uh you need to be able to find ways to coordinate between that as well.

22. 10:09

    And so if there's anything that I would like you to take away is that you don't need a data center to fine-tune a large language model a 7 billion parameter large language model and stack beats hardware with the right uh training techniques. You can definitely be able to train those on the edge nodes and decentralized AI needs this bas because of heterogeneous nodes and privacy constraints and I truly believe that the error of data center only LM fine-tuning

23. 10:41

    is over and now we can do it on edge devices as well and thank you. So I'm Jerry you can find my uh details on screen and you can contact me for implementation details. Thanks.