NEAR AI Paper Club DeepSeek V3 & R1 - Architecture & Training Insights

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   well I guess deeps was already big but it wasn't that big at an absolute Apex of uh of hype um okay so the paper for V3 is pretty big and it Go and it goes all over the place uh I will not cover everything everything from there I will skip some of the data preparation parts and whatnot but uh we're going to dig very deep into the things I understand well and we will not very deeply into the things I don't understand well uh so architecture wise so generally what's cool about deeps a

2. 0:37

   it's a state-of-the-art model uh which outperforms um you know many many other open source models and closed Source models uh and allegedly it's extremely cheap to train right and so there are multiple reasons for both of those achievements right so presumably uh the performance is probably has a lot to do with data preparation which paper doesn't go very deep uh into uh but the

3. 1:03

   speed of training and the cost of training um especially given that they have Hardware which is inferior to the hardware we use here right they use H 800s which is uh the same as h100 but the speed of communication uh is twice as slow it's an intentionally um you know intentionally crippled GPU uh for like whatever regulations on whatnot Okay so because because they had worse Hardware I I think that's an the irony is that because they had worse hardware and the

4. 1:35

   worst networking they had to figure out uh how to make it work and so they made a lot of advancements that wouldn't happen otherwise right um okay let's let's begin with the first one in the paper uh multi-ad latent attention so on the left here I have um the basic attention right yeah you you have the input Vector in in this case it's called HT so HT is whatever coming from the previous layer or from from the embedding and then you compute you have three matrices w k w vwq often

5. 2:09

   times it would be single Matrix that we will later splice into three uh and you multiply um the so so here I use the same notation that they use when you multiply matrix by Vector but in reality what happens is that you're not using a single Vector you have a whole batch of uh of inputs and so what you actually is you multiply the batch of inputs by The Matrix right so it's going to be the other way around but right so you multiply input by those two matrices you get keys values and queries uh and then

6. 2:40

   you um you compute inner product of keys and uh queries and keys and that for every token um like for a particular token for which you're Computing uh the attention it will give you the similarity of the query to each of the keys uh preceding it and then you multiply that similarity uh by the values right so here couple other details is that for keys and queries you also compute positional embeddings which

7. 3:11

   today almost exclusively are um rope which stands for rotational position embedding uh okay so let's uh let's look at what they do differently so the problem they trying to solve is that uh after you computed those uh Keys uh values and queries uh you need to keep the keys and values in memory uh for the future for the future tokens and memory uh is at the premium so ideally you want to reduce the amount of memory you consume so what they do uh is they

8. 3:46

   instead of computing uh keys and values directly uh they effectively they they do something very similar to what Laura does namely they multiply the input Vector which is wide by a matrix which uh project it into a smaller Dimension much smaller Dimension right and then they multiply by another Matrix which up projects it back to the original Dimension uh and the key here is that

9. 4:12

   the uh keys and keys and values that they store in memory are the small down projected right and so they they occupy much less memory but also as a side effect uh multiplying by uh down projection projection Matrix and up projection ction Matrix is cheaper than multiplying by single Square Matrix right so they both save time uh and memory uh with a caveat that uh because they only store uh down projected version of keys

10. 4:44

    and values uh they have to recompute like they have to up project right so like when you when you use multi-ad tension for each consecutive token uh keys and values already stored you only do inner product uh and you know soft Max multiply by values but in their case they only store the blue ones in memory right so this um like here they have to recompute uh keys back right so so they have to multiply by a projection Matrix again they have to recompute values multiply by by a projection Matrix again

11. 5:17

    but you know presumably that is still cheaper than than uh full multi-ad tension another interesting thing maybe maybe that's how everybody does it uh but that's not how I saw it being done is that generally uh so so the way rope works is that it takes uh every pair of uh so like you have a vector of size 2048 for example it takes every pair of consecutive elements and rotates them uh by different angle uh based on a

12. 5:50

    particular rule right and so usually people just apply rope to the actual Vector that you get as a result of you know for cure for for query they don't do they don't do it this way they compute two different keys and to one of them they do apply Ro but for to another one they don't right so part of the key right yeah the the question is uh why is there no information loss when you when you go to the like like the beauty of

13. 6:21

    the of the Transformer right that initially Transformer is completely random right so whatever constraints you put on it it will train under those constraints right and uh like people have been studying it for a while but there's still a lot of places where you can like where it doesn't need all the uh resources given to it so presumably for keys and for like like a very good example is group query attention right so group query attention like back in the day you would

14. 6:46

    compute as many keys queries and as you have values uh but eventually uh people figured out that you can okay and there was some other reason why they apply rope not on the whole Vector but only on part of it I don't remember it now maybe as we go I will remember okay now the next thing is that the uh the actual model is the mixture of experts uh we've seen mixture of experts before I think uh it's pretty well known that all the GPT GPT is a

15. 7:20

    mixture of experts mixol is a mixture of experts um okay so so very quickly what is mixture of experts um um here I have an example where you feed a token corgis and it goes through the whole Transformer so the Transformer consist of embedding which embeds the token into the space um in which the model can can reason then it goes through multiple layers and it does un embeding which gives you potential tokens right and every layer uh does two things it does a tension

16. 7:51

    which we just discussed and after that tension it does what is called MLP uh where without mixture of experts it just multiplies it by a matrix which is learned applies nonlinearity swiglo in case of um llama and in case of U uh dsek then multiplies by another Matrix right so here the thinking is that instead of always multiplying by the same Matrix uh which is like one size fits all what you can do is you can have multiple matrices called experts and

17. 8:23

    then for each row for each token uh you first run like let's say you have 16 Matrix is 16 experts you first run it through a matrix of the size uh dimension of the model Time by 16 and it gives you the confidence of the model in terms of which of the experts is the best for the token right then you sample some number of them like let's say two and then you multip then you use those

18. 8:48

    two matrices uh to compute the MLP instead of using the same Matrix right and the original idea behind mixture of experts is that the total number of parameters now is much bigger uh than the number of parameters you actually activate on the forward pass right and so if the model learns to use those parameters efficiently you get a more capable model cheaper uh the second Advantage is that you can have those experts to be on different gpus well well I guess it's

19. 9:20

    not yeah you can you can have those experts on different gpus effectively spreading parameters into different gpus and then you can apply MLP efficiently taking advantage of all the parameters right so in this case um I have on the on top uh a simplified version of mixture of experts here right you comput so X is the input uh you an add at is the P torch way of saying ma mold you multiply it by G so G would be dimension of the model times 16 or like times number of experts it gives you the probabilities of each expert you

20. 9:54

    you soft use soft Max and then you sample one expert or you know several and then you compute input times the expert nonlinearity times the second expert okay so so a couple things couple things here a is so here say sample in reality in deeps they don't sample they they take top top one or like top K right so if they want k k experts they will not sample K of them they will just take the

21. 10:23

    top ones uh and they uh so let's look uh on the bottom I um um it's the same let me see okay right so so G would be so so capital G is the probabilities of the experts right then you sample multiple experts K of them not one and then the output Vector would be the sum of U okay yeah now I see the key here so the interesting question is how do you train the The Matrix that will be predicting

22. 10:56

    probabilities of experts it needs to be learnable right because if you only use it here to sample or or like to choose top K that's a nondifferentiable action right so the gradient cannot flow so what they do uh is when they choose several experts like K experts right so K has to be more than one when they choose K experts they multiply the output of every expert by the weight that was assigned to the expert right and then they divide by the sum of the

23. 11:20

    weights so the expected the the the the expected magnitude of the output is the same as the expected magnitude of each expert right but the this multiplication is differentiable now right so on the backward pass if one of the experts was giving a better more useful output than another uh then the gradient will flow and say hey it would have been great if that expert output was multiplied by bigger number right and so the expert

24. 11:45

    Matrix now can can learn the problem is that uh early like initially the model is completely random and one expert will start learning a little faster than the others naturally right and so the model will say oh that expert is better uh right let's give him um higher weight right and so some experts will be getting higher weight now what will you will notice here that in the in in the

25. 12:09

    on the backward path if you think uh what is the derivative of product right the derivative of product uh is like derivative of KX is K meaning that the gradient which flows into the expert is scaled uh by the probability right so if an expert is chosen with a smaller probability it also gets less gradient right so now the expert which is already smarter and gets chosen on itself is

26. 12:33

    also learning faster right and so what happens eventually is that there's a complete collapse where one expert is always Chosen and is being trained and all other experts have zero weight no gradient is Flowing they do not get better and so the way people deal with that is they introduce a separate loss auxiliary loss which penalizes uh if the experts are not chosen uniformly and the

27. 12:57

    problem with it is that if you add this auxilary loss now you have two losses one loss says Hey predict the best token possible and another loss has nothing to do with the quality of the token right so that loss is tampering with the learning uh and so that is something uh yeah so this is this is a paper a very old paper from Google that introduces auxiliary losses called the outrageously large neural

28. 13:21

    networks and this is like at the bottom you see the the loss uh they propos to use as a auxiliary loss anyway in the Deep Sig V3 they completely remove the auxiliary loss and the way they do load balancing is they introduce a bias right so there's another learned Vector called B though B is not learned through gradient descent instead so so G uh you will notice here that g is used is used for two things you use G to sample experts

29. 13:54

    and then you use G to multiply the output by the probabilities right so they compute now two G's G and G Prime where G uh is uh what you will multiply the output buy and G Prime is the probability right so they decoupled them and for the probability they add the bias and the way the bias works is that they they run a whole batch then they look which experts were chosen how many times and and if a particular expert was not chosen sufficiently

30. 14:24

    frequently they bump the bias but because it only affects it's not sampling again so top it's top K in their particular case but because it only affects choosing the experts but not the actual computation that bias does not influence the um the output um in the like like it doesn't it doesn't it doesn't it doesn't temper with the output in in in any way besides the the expert choice and I uh I I don't

31. 14:54

    remember uh in the paper whether they use this bias at the end when when the model is fully Trend like during inference time but presumably no like during during inference time you can probably just drop it and so this way you have pretty high certainty that the experts will be balanced because the bias is consistently balancing them uh without having any auxiliary loss without tempering with the with the quality of of

32. 15:18

    training uh that also so uh the the thing about mixture of experts also is that the U you want not only you want experts to be balanced for the sake of uh uh of them all learning you also want them to be balanced for the sake of computation because if one expert has way more uh tokens than another then assuming they on different machines which is the case in their case right that computation will be longer than like the computation is now just not balanced right one machine will be Computing longer another

33. 15:51

    one will be idle uh and in their case because they're pretty balanced uh they don't like if you if you read Google papers you will you will find many of them that they that they drop tokens right they will compute they will assign tokens to experts and if some expert has more than some threshold of tokens they will just drop some tokens like some tokens will have no computation made for them uh and in deeps they say hey we don't have to drop anything because the

34. 16:14

    balance is pretty big and the only thing I don't have it here on my on my slides but they do add a little bit of auxiliary loss to make sure that within a particular sequence uh the experts are sort somewhat chosen uniformly cool uh and another thing uh I think that's the last one in the architecture uh is multi toen prediction so so effectively they say well we we have the input token we did all the expensive computation of the Transformer and at

35. 16:42

    the end we only predict one token but in many cases presumably you have enough information to predict a couple like if you think how we think we usually have like we don't need to know the next word we about to say to know our thought right so it makes sense to reuse this expensive computation with the with a lot like on this picture something to pay attention to is that in the leftmost block there's L Transformer blocks there's L layers and

36. 17:08

    L is a pretty high number it's like between 60 and 90 right uh but in the second and the third block here there's only one Transformer block right so what happens is that you feed uh the sequence of tokens like the first four tokens into the Transformer it goes through all the layers goes to the output layer predicts one token and then this um this final Vector that goes into the

37. 17:33

    output layer it goes into one more Transformer block does that Transformer blocks goes into the output layer predicts the token after next right and so that gives you more signal that allows you to train this uh you know it's more more loss more signal with a very little very low cost and on this picture they show multiple tokens ahead of time in reality in practice they they predict one more right so in practice

38. 17:57

    they predict the very next token and the token after next cool uh that is also presumably could be helpful during inference I don't know if this is used anywhere but because you can sort of pre- predict next token um I I don't know if this is used in any way okay now um uh this is the next section which is uh the engineering behind behind um deep seek so training generally uh distributed training is

39. 18:29

    relatively complex because you're doing a lot of computation and you're doing a lot of communication and ideally uh and communication is expensive and in their case communication is twice as expensive because they have crippled Hardware right so you want ideally what you want to happen is you want whenever you do communication to do some computation at the same time right so if you if you're doing some computation when

40. 18:51

    communication happens that's uh like effectively your communication is free right and when you do um when you do model scaling there are multiple approaches to it right one is called Model parallel where you know when you do when you multiply one matrix by by another you can slice one of the matrices move them to eight different gpus right and then you compute 1/8 of the matte mole on each of them and do all use this is something uh that DP doesn't do at all because I guess with their overhead of networking it's too

41. 19:24

    expensive then the other one is expert parallel where every GPU is running a different expert right and so after you compute the expert assignment you dispatch the rows to the corresponding experts compute the experts and then you combine back the the data the the outputs then there's obviously data parallel which is effec saying hey we're going to have the copy of the model many many times uh and we're going to be

42. 19:48

    Computing on different batches at the same time different mini batches and then there's pip planning so the idea of the PIP planning is you slice the model vertically meaning that the first like one GPU will have the first several layers the next GPU will have the next several layers next GPU will have the next several layers and then the input comes into the first first GPU it computes its share sends data to the next and as the next GPU starts

43. 20:13

    Computing the second set of layers the first one is ready to compute the second uh batch of batch of inputs right and so um here on the picture you will notice like naturally when you just begin the um the batch right only the first GPU is active all other gpus are blocked right so here on the picture you will see the green is the first batch of data right and you feed it to the first GPU the second one is Idle um right and so in

44. 20:38

    the middle you will notice it's very busy right so once the pipeline is is busy um doing stuff it works great but at the beginning at the end you will have those holes and because generally you want to keep your batch relatively small the reality doesn't look like this picture the reality looks like this picture uh condensed horizontally meaning that the percentage of the wasted time is actually much higher than on this picture but also another problem

45. 21:03

    is so here on this picture the narrow the squares a forward pass and the wide rectangles is a backward pass because backward pass is generally slower right and so here you will notice that um like if you look at the yellow yellow batch right it does four forward passes on different uh gpus then after the last one it goes computes the backwards pass and then it goes backwards right and so here everything here is condensed

46. 21:28

    however uh look at this you can see my pointer right so here the the blue one has finished the forward path it sends data to uh to the next layer and the next layer just finish the yellow backward pass sends data back uh and here if you draw it on the paper you will notice that there is no way to Overlay Communication in computation uh you have like at some point they all have to stop and do and do communication

47. 21:54

    there's no way like to move them in a way that there's no Deadlocks which is highly undesired ideally you want to do pipelining in such a way that uh whenever there's any communication someone is doing computation and so they do it differently uh to understand what they do let's uh let's quickly revisit all the steps with export parallel right so first thing you do so so you have the input Vector uh sorry input batch uh

48. 22:18

    from the previous layer or from the embedding it's on every machine right all of them can do attention that's compute uh then after attention they assign experts that's compute once experts are assigned they dispatch dispatching is networking now every machine has their share of the every expert has the share of inputs they compute the the expert the MLP now they have to combine their outputs

49. 22:43

    that's networking send to the next pipe depth also networking if if if that's the last layer if it's not the last layer it goes back to attention right and on the backward you do all all all of those in Reverse right so combine combine Prime is so now you have the output gradient coming from there and you need to send it back to to The Experts right so it's the derivative of combine is dispatch right so combined

50. 23:08

    Prime is dispatch FFN dispatch Prime we just combine expert assignment backward attention Okay um okay so the key to their optimization is to remember what is the backward pass for the MLP layer for the MLP layer for the M Mo on the backward pass so on the forward pass pass you have X which is the input to the layer and W which is the weights and you compute the output as a m Mo on the backward pass you compute the derivative

51. 23:37

    of the input for the chain rule uh for that you get the derivative of the output that came to you from the chain Rule and multiply by the weight Matrix transposed and then you compute the derivative of the weights that's the actual gradient uh that you want from the backward pass and that you do by multiplying the transpose of the input uh with the output gradient now the key here is that the the X Prime you

52. 24:01

    need uh to send back right so so before you can send to the previous pipe depth you have to compute it right there is a they have to be serialized but W Prime is not blocking anything this is the this is your final computation it's a leaf of your graph of computation so you can you can compute W Prime whenever uh when so whenever you have three Cycles when you when you're doing some uh communication you can compute W Prime and so they have this picture uh which you there's a couple of things to pay

53. 24:32

    attention here the the the thing to pay attention number one is first of all let's see let's see what's depicted here right so the green one uh is a forward pass sorry the yellow one is a forward pass of one batch uh the green one is the backward pass of another batch and the blue one is the backward pass for the same batch uh that computes the weight gradient right so so this so this

54. 24:56

    is uh this is a yeah so this is yellow for one batch this is green for another and this is blue for another right so a couple things to to notice here one is that they not aligned uh in the sense that you will notice that for the yellow one sorry for the um for the green one it begins here right so it begins with MLP okay I should so FFN is MLP right so it begins with the FFN with MLP then dispatch then

55. 25:30

    um it doesn't show exper assignment here but then attention this is attention right and sending back to the previous pipe depth this is PP so PP is sending is the pipelining um the communication but for the forward pass it begins in the middle right it begins with dispatch which is right here right so they not align the backward pass and the forward pass are not aligned uh they they like slightly

56. 25:55

    slightly shifted relative to each other the second thing to pay attention here is that if you remove the Blue Block you would still entirely or almost entirely overlay communication and computation because uh presumably computation is slower is than communication right so here if you if you imagine removing completely MLP W on top you would still mlpb like dispatch F will just get smaller and dispatch B will get smaller because here you don't use like you don't need sorry like if you look at

57. 26:27

    this graph the green mlpb is a m Mo dispatch f is this is sending the experts presumably dispatch F takes smaller less time than MLP right so uh so part of this time is not actually spent on networking so if you remove MLP W here you would still have computation here and communication fully overlapped uh and right so dispatch F will finish while mlpb is being computed right so by the time you you're ready to start LPF you have the data and you're ready to

58. 27:00

    dispatch B on the right side I believe it is also the case right so if you remove attention W here attention B so after after we're doing attention B we need to send to the next pipe depth this is blocking uh before we can do combine right so attention B some pip planing will have to happen here before we combine to combine some pip planing before attention but presumably you can fully overate in a way that uh you do

59. 27:25

    combine F this is this one right then while you're sending to the next pipe depth attention B is still being computed right and so by the time you you sent the combined data to the next layer on the forward pass right so attention f is ready to be computed attention B just finished so it's ready to do its pip planing sending to the previous pipe depth and then you're ready to combine right so the key here

60. 27:47

    is that you can remove the blue ones and still have computation and communication fully overlaid right this is not emphasized in the paper but it is uh it is clear from this following picture that they show where they show an example of pipelining like it's a whole example of multiple like the S so there's one other thing here which is U they don't have a single uh every every device has uh not one but two different pipe depths uh one

61. 28:18

    um so like for example here the device zero has the first pipe depth and the last pipe depth and the second one has the second and the penultimate the next one has the third and the third from the end right and so this way you can start sending data from two ends which further reduces the bubble right and so here uh and they only annotated the top one I annotated a couple of bottom ones here right so so a b are also two batches right so that you can

62. 28:43

    see where they going right and you will notice right so at the beginning they start multiple forward passes so at this point no backward can be computed because you know still no no forward pass has been fully completed right and then at some point uh the forward passes have completed right and you see a here um or rather let's look at the one they annotated right zero here it has completed the forward pass it completed

63. 29:05

    the backward pass and now it's uh um it could have already computed one uh like I guess one and zero here can be swapped the blue one and the yellow but the key here is that no matter how you arrange them if you remove the blue ones there would be there would be some bubble right so they have some available computation and they fill it in with blue they fill it in with Computing the um the uh the the weight gradients right

64. 29:28

    but here in the Middle where the pipeline is fully busy you will notice there is no blue right go ahead yeah so so the backward pass the backward pass consists of two things Computing the input gradient Computing the weight gradient Computing the input gradient is Green Computing the weight gradient is blue and and the weight gradient is not sent anywhere over the network so you can compute it at any point the gradients are

65. 30:02

    always you can do the the together you can the the point is that you don't have to right like the point is that in order for the chain rule you have to compute the input gradient before you send it but you have an option to compute the weight gradient later you don't need to compute the weight gradient to send the input gradient backward right so part of the computation you can delay until later it's not blocking you right and so that

66. 30:28

    that gives you flexibility in arranging your pipeline and here you will notice that um but importantly you will notice that in the middle uh when they when the forward and backward passes are fully overlapping from from multiple um from multiple mini batches they do not compute at all the weight gradients right because because they they can fully utilize uh they can fully utilize the computation and at the end where you

67. 30:52

    would have a bubble otherwise uh they they compute the tail of the blue ones actually in retrospect I might be wrong here it could oh yeah sorry uh I think I think the wide I think the wide green means Green Plus blue yeah but potentially take back take back everything I just said uh yeah yeah the the white green is green plus blue it's uh it's both um it's both backward for input n backward for weights okay so that also means that

68. 31:26

    that potentially my my statement before that you can remove the blue here and still fully overlay communication computation might be incorrect right there's a good chance then that dispatch f is actually taking longer than mlpb right and then this blue in the middle is what enables okay so that's pipelining um so so in general I I think again like the the irony of the fact that because they had SL slower Network they had to be creative here and come up with all those School optimizations is a is

69. 31:59

    pretty interesting okay quantization initially I didn't plan to talk about quantization today so I don't have all the slides uh but but I think quantization is very interesting so quantization during tra so during inference I guess everybody's using quantization right you get the model which is trained in fp16 or bf16 and then you quantize it to like eight bits or four bits or even smaller right so that you can run it locally in

70. 32:23

    your machine but training uh with smaller Precision has been challenging historically right because the you do need for certain operation you do need the Precision right and so here they they go into quite a bit of detail how they manage to train in eight bits and so so so the the key the key interesting ideas here one is that when you do 16bit training you usually just use bf16 right bf16 is a floating Point

71. 32:52

    format where I guess exponent is wider uh and myti is narrower and then fp6 is the other way around right so but with bf16 uh you just use uh like like you you just use whatever the the actual range of the format and don't do any uh rescaling but when you use fp8 uh you cannot you only have you only can represent 256 different values right so the actual range of the of the type is not adequate so they do rescaling and

72. 33:26

    there there are two things here which are interesting one is that how many bits they allocate for mantisa and for exponent right and the other one is what is the range sort of the range of the um uh fp8 what does it map into and usually what people do is that they they get a tensor they compute the minimum the maximum value and say Hey fp8 will just map to the range from the minimum to the maximum value right but that that

73. 33:51

    obviously degrades the bigger your tensor gets right so it's it's one thing if your tensor is 256 by 256 right then you only have 65,000 um values right but if your if your T is 4K by 4K suddenly like the outliers the me and max value can can significantly um reduce the the Precision of fp8 so what they do instead is that they they quantize per some number like they don't quantize per per tensor they quantize per like slice of a TENS right so for input for example they take a single token uh they take 128

74. 34:26

    consecutive um values in the tenser and they they only use those 128 to find the minimum the maximum and scale those 128 to that range and they get the next 12 128 compute the minimum the maximum and scale there right so they quantize per slice and because of that no matter how big the tensor is the quantization quality Remains the Same and then because of that I guess there was some previous state of the art uh in terms of training fp8 and in that uh in that approach they used three bits

75. 34:59

    of of uh three bits of mantisa four bits of exponent for uh forward pass but for the backward they would use only two bits of mantisa five bits of exponent because for backward they needed more range but they say hey with our quantization strategy you don't need that you can use three bits of mantisa four bits of exponent both forward and backward which increases the Precision okay cool then uh certain things need to stay in uh in high Precision I guess one

76. 35:30

    thing is the the actual weights are always in fp32 right and the gradients are always in fp32 right I think those are the only two things maybe there's a couple more and and I think here one arrow is missing or maybe I'm not I'm misreading the picture right but the uh in Adam so so the biggest memory waste besides the activation during training is adom right because you need to like naturally you have to store the weights and then you need to store two tensors per so for each tensor for

77. 36:01

    each weight tensor for each parameter of the model you have to store you need to have one more tenser for the gradients and then you need two more tensors for adom one with momentum one with a second order momentum right and so they say well those two can be bf16 that's chill uh the gradient has to be fp32 and and the weight has to be stored twice in memory one is in fp32 one is in

78. 36:25

    fp8 for the actual met right so this is a little bit of overhead but uh despite this little overhead for the weight uh then you can have significantly smaller like the input activations can be fp8 they also Sho them over the network in fp8 reducing the network overhead right all the M moles except for attention attention they do they don't do an fp8 but all the math mes for experts are fp8

79. 36:50

    and then they aggregate an fp32 and then there's a there's a part of the paper which I did not I think my knowledge of Cuda is not sufficient to understand it uh but uh evidently for some reason aggregating in FP is not trivial and required some engineering cool yeah but anyway uh training in FP is possible uh and that's uh that's pretty cool okay pre-training they talk about a couple

80. 37:21

    things one is so there is this concept called fill in the middle the idea is that it would be great uh if we could not only predict the next tokens knowing the prefix but if we were also able to see the prefix and a seix uh and predict what is what is in between like if you think of co-pilot right the co-pilot doesn't just see the prefix of the code it sees everything before the cursor and everything after and it needs to fill in the what happens in between so the way

81. 37:46

    people do that uh is they convert um a subset of the in of the training set right so you have a training set like 15 trillion tokens or whatnot you get a a subset of it percentage and for that percentage you rearrange the tokens right so you say you you you insert a special token called f begin uh then you put the prefix then you say fim ho put a suffix another special token middle and the final token right uh and then just part of your data set is in this format and

82. 38:16

    what the model learns to do is it says hey if I see this special token this special token and this special token oops then what I'm starting to predict here has to be consistent with this beginning and this ending right so the model is just naturally learning to predict the middle uh and the only concern that you might have is that if you con like let's say you're converting 10% this is what they do they convert

83. 38:40

    10% of the input data set to this format that means that for the 10% of the data set you don't see it in the proper order right and so the the question is does it affect the performance and the answer is no right so 10% if you convert 10% to fim are the performance of the model on the regular non-im uh inference is not affected at all right so converting 10% is acceptable and then you get the F

84. 39:05

    model right uh one interesting thing that is not mentioned in the paper that uh quite a few people when they train when they train fim they swap the suffix and the prefix right and the beauty of that is that if you swap the suffix and the prefix here uh then you don't need you don't need a separate token between prefix and the middle because prefix and the middle like the middle is a natural continuation of the suffix of the prefix story right so you would have a suffix special token and then you would have

85. 39:35

    just everything before the safix right you don't you don't like here you have to choose two places where you split the sequence but if you do safix prefix middle you only choose one place to split right and then you like effectively have in infinite number of midles for free they they don't talk in the paper why they chose this approach yeah yeah and another beauty of a safix prefix middle is that

86. 39:57

    uh presumably the prefix is more important for the middle than the saix so it's great if it's close right because obviously Transformer learns to to utilize the recent data better than the old one right so if you do prefix saix middle by the time you do middle your prefix is very long very far behind but if you do suffix prefix middle then by the time you do middle the prefix is right here available to you cool yeah so they do fill in the middle

87. 40:24

    uh then they have a very large context window and naturally nobody trains with a large context window from the beginning people train with a smaller context window and then they upscale it and here I have the formula uh so this is like if you were reading hiker news like I don't know a year ago uh uh gorgi griov was showing that hey guys I managed to increase the context window

88. 40:46

    of llama right and the way he did it was using this formula where um like on the left you see so the only thing you need to change is rope right everything else uh is unaffected but the positional embedding has to be changed and here on the left before the left side before the equal sign is the how rope is usually computed right you have the input you have the input token you have the um position in the sequence and the frequency and they say well if our

89. 41:17

    old maximum was L um Prime sorry our old Max yeah our old maximum was L Prime our new maximum is L then instead of feeding the current position we will fit the position times Old Times new divided by old right so effectively you stretch uh the positions in a way that the the range of inputs to this parameter Remains the Same right and if you just do that the model will break so anyway this approach uh works right so so like you know we we know that if you get llama for example with a short context

90. 41:48

    apply this to rope uh and and just keep on training it for I don't remember the number like 100 thousand examples million examples it uh it learns to use the longer context and and gets pretty good at it uh but there's this paper called yarn which digs deeper into it uh and they analyze so so the way again rope uh we didn't we didn't get very deep into how it works but the way it works is that

91. 42:13

    every two consecutive tokens uh the the vector by which they rotated uh the the uh the frequency at which rotation happens is different for every position right so for the first position the frequency is very quick right so every couple tokens it will do the full full circle for the next one it's slower for the very last two tokens it goes very slowly like maybe not even doing the full Revolution uh throughout the sequence right and so the people in the

92. 42:37

    paper in the yarn um it's generally a very good paper to read uh it's very insightful but couple of simple insights is that a for the first for the small frequencies uh the actual values like you don't want to stretch it because the actual relative value between two consecutive tokens is very important right because for for recent memory you would be looking like you know if if you need to look five tokens back the model

93. 42:59

    sort of learned uh the exact rotation it needs to apply to get there if we if we stretched it it will now overshoot and look in the wrong place right so in yarn they say hey for the small frequencies let's not stretch them let's keep them like as is for higher frequencies it makes sense to stretch them because you know it's like for looking for older stuff and and there's more insights there right they uh they they have three

94. 43:23

    different um three different approaches to look at it and every approach uh gives them different insights for different frequencies and then they have like a final um final modification which works better than this simple formula and deeps is using deeps is using um yarn okay now V3 and R1 are heavily inter interdependent so if you if you download the V3 paper the one they published originally before R1

95. 44:00

    became a big thing you will notice that the tail end of the V3 paper actually has most of the R1 information already there because R1 was used in the post- trining step of the three so let's talk a little bit about R1 and then how they um how they are um used to train each other okay so first of all a very quick uh insight into to PO so they use their own reinforcement learning

96. 44:29

    approach called grpo group relative policy optimization which is a flavor of po po is a uh is a paper by John Schulman from very long ago I think it's 2016 with the basic idea is uh here I don't know if you can like can highlight it in a way but um the the core idea is that you have a a policy a policy is your uh generative model that produces your output right and then you want your model to get better at a particular task for

97. 45:03

    which you have a way to estimate estimate the quality right and so you uh you have the so so here you will see uh P policy right so policy old is the policy that you have right now on the previous step and the policy without all policy my my my Greek is uh not up to date I don't I don't know that how you say this letter uh but the uh the new policy is the policy you're choosing uh you know to replace the old policy with and so

98. 45:36

    there's a very old algorithm called policy gradient which effectively says hey let's compute let's optimize this right this part like effectively uh the behavior of the so so so you have some number set of queries so queries is something you sample from a set of queries you can use uh then output is some output from the policy um right and then you say hey I have some number of those queries I will sample old policy

99. 45:59

    and the new policy um and then I will compute what is called Advantage Advantage says how much better the new policy is better than the old one and and then I will optimize um I will optimize the U this uh uh this function the expected value of the advantage multiplied by how confident the new policy in a particular trajectory divided by how confident the old policy is right so if there some trajectory for which advantage is high and the new policy is much more confident you want uh you know uh to uh to give her a

100. 46:35

     reward if the new policy you know there's a lot of Advantage but the new policy is just you know slightly more confident than the old one then you know you give it a little bit of reward anyway so this is very unstable and the way poo is fighting it is by doing two things one is the clipping uh this value right so this ratio can can is clipped between Yus Epsilon and 1 plus Epsilon right so you cannot like if your policy is five times more certain than the old

101. 47:00

     one you're only getting reward for one plus Epsilon and the second one uh so so here I use the formula from the paper from the geps paper but if you go to the old poo papers you will see that there were many other ways to do that but the idea is that there's also another policy called reference policy reference policy is your model before you even started reinforcement learning right so you have three models one is your original model

102. 47:25

     before reinforcement learning and then at every step you have old and new then you go to the next step the new becomes old you have old and new and you're effectively saying that the K Divergence of the final model from the reference model has to be as small as possible effectively the reinforcement learning says get as good as possible in this skill but not deviate too much from what you already know right

103. 47:49

     and uh so this is okay so there are two problems here one is that Advantage is generally computed using yet another model called value function right so effectively we say hey this is the reward we computed and this is what we expected to it to be right and the value function is hard to train and and that's another model to maintain in memory right so it's a it's an unfortunate um thing to have here and and I guess

104. 48:21

     the second problem they mention is uh uh the way the k dience is applied they they move it so they move it here right so so they they they do they they make two changes right the like if you look at this formula it is exactly the same as poo uh with the difference that they uh subtract the kale Divergence in the actual expected value not per so here it's per uh per sequence here it's computed once uh and that is something like if you open the original poo paper

105. 48:53

     I think it also does it this way right so so I guess there are many ways to um to to apply K versions but the key the actual uh key here is that they don't have a value network instead what they do is that they sample um they create um multiple um give me one second yeah yeah they they compute multiple they unroll policy multiple times they unroll it like's say I don't know 100 times right so they have 100

106. 49:26

     different sequences uh and they compute the reward as a reward minus like effectively the advantage is how much better the reward of this output relative to all the all the rewards we collected across all 100 right so so instead of having a value Network they estimate the value function by averaging a lot of outputs uh together right so that gives you um that allows you not to have a value network uh and

107. 49:57

     uh and you know we like by now we know that this works very well right because if you go to Twitter uh a lot of people tried this exact approach on smaller networks on a lot of like you know every several hours someone is publishing new results and you know grpo works so this is a this is a pretty cool pretty cool reinforc learning algorithm go ahead the is the ratio right between the and the r there no our I is the reward

108. 50:28

     okay and so you're not it's not linked to that ratio yeah okay let's quickly I guess go deeper right so what happens right so we begin so we have again let's say we already did multiple iterations of reinforcement learning right so we have the current so policy policy is the fancy name for the neural network we have for the for the Transformer so we have a reference policy that's the one we started with we have the old one

109. 50:52

     that's the one we just computed and then we're about to have a new one right so we sample uh we sample multiple queries so queries will be the tasks on which we evaluated uh right so we have multiple cues then for every Q we use the old policy the the current state of the model to sample uh G outputs right so you have multiple trajectories you have some math problem and you sample from the current policy like 100 different thinking trajectories and each of them results in a particular output and then you can put reward for

110. 51:23

     each of them so Ri so here I is from one to G and r i is the reward right which in case of a math problem would be zero or one it's either correct or not but in in case of some other applications it would be you know potentially more insightful more granular and then you compute uh the the value function sorry sorry the the loss function uh as uh you you take each of those outputs you compute the

111. 51:52

     probability of this output under the new policy and under the old policy um and multi and and then it's clipped right so so like it's Min of these two right so it says hey which one is and multiply by the advantage and Advantage is computed as reward for this output relative to rewards for all other outputs right normalized right and then you subtract diversions like saying hey do not do not go too far from the reference and and

112. 52:26

     they DK is slightly different uh they um in this particular paper they don't say why they use this one uh maybe maybe it is explain somewhere cool so so okay so couple so what happens with this reinforcement learning is that they they begin um they begin by having so they have one network called V3 base so V3 base is they finished pre-training of the uh of the V3 that's I presume this is V3 based or maybe that's after uh supervised fight tuning but that's before any reinforcement learning right they use

113. 53:05

     that to run AEL uh and so when they do that AEL uh has AEL is working pretty well but has issues uh such as you know sometimes it's unreadable sometimes it's not formatted very well because you know for a real formatting doesn't help right all it wants is the reward right and and it can read text which doesn't have formatting just fine and it mixes languages right so what they do next is they they train that model now they use that model to sample a bunch of

114. 53:35

     trajectories of high quality right and uh there my my reading of the paper could be incorrect but I think what they do is that first of all they guide it with the prompting and second of all they use humans uh to choose the high high quality trajectories and then the next thing they do is they get the base model again fine-tune it on those high quality trajectories and then do reinforcement

115. 53:59

     learning again right and so that one is good uh that one presumably doesn't mix languages very much though somebody told me today that it still mixes languages every now and then right and then uh and then V3 in particular is also trained on those reasoning trajectories question is where do the rewards come from so for the original uh for the first set of reinforcement learning right the one they do to get the reasoning they use math problems programming problem problems where the your word is exact

116. 54:31

     right so there's a math problem that says you know like compute something and there's the integer answer right and so the model either produces that exact integer answer or not it's one or zero programming problem is the same right presumably like there's some output it needs to get um then later after they do that reinforcement learning uh and they find you V3 on that uh they do more

117. 54:54

     reinforcement learning this time not for thinking trajectories but like for general you know like for uh like how well it answers the questions from people how well you know like how safe it is ETC for those they use uh um they use neural network to to give the reward right so that yeah so so then it's not just zero and one I guess so the question is how would that work in the areas where there's no explicit reward so even for the last stage of um

118. 55:22

     training V3 they do reinforcement learning uh for like alignment right and uh there like I guess a pretty standard approach is you just use another good neural network to tell you how good the output is right because they um they they are pretty smart and in many cases they can give you very good um value there's a there's several breakthroughs we talk about today right so like you

119. 55:45

     know the the model is cheaper I think the model the model being cheaper is just good engineering right and uh there could be multiple hypothesis why uh that didn't happen before I guess one of them is that the intersection the set of people who understand machine learning very well and the set of good system system Engineers is not is not as big as any of those two sets right so like like like meta for example probably has very

120. 56:08

     good system as Engineers open does but but they also not well I guess opena doesn't publish too much of their infrastructure uh specifically for grpo uh that's an interesting question I guess I guess part of the thing here is that until 01 people didn't know that it's like a four four minute mile right like like with o1 we all knew now that it's possible right and I guess the amount of time that passed is relatively small so maybe there were other people

121. 56:35

     who were very close to publishing it oh yeah or or potentially yeah potentially just the base model finally became good enough right because also uh deep SE V3 on itself is a state-of-the-art model right so they they working on top of a very on top of very capable model right so yeah like a year ago you would have you know Lama one maybe it just wasn't capable enough yeah I guess I guess we can conclude this part here and then there's Pizza uh yeah

122. 57:04

     and then more discussion in the smaller groups can continue with pizza okay thank you everyone for coming that's