NEAR AI Office hours #15 - Reading Group discusses the Future of Efficient AI: The Era of 1-Bit LLMs

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   the one we're close to won't stop us now together we'll rine the game with near we'll found the flame empowered Minds rise and soore use your own AI we'll explore how The Future Has in in this Tech driven Wonder okay hello everyone we have a special episode today uh we have a reading group uh about a paper Alex will introduce in a second and we have a neari team joining as well

2. 0:53

   uh to discuss it so I'll let Alex to take it away okay you can hear me right y okay awesome so the paper we will be discussing today is called U this one the uh era of onebit llms uh it's about training large language models with the without multiplication using only uh ones zeros and minus ones for weight matrices before we get there let let's do like a very quick uh recap uh let's spend like couple minutes to remember how Transformers work and what

3. 1:33

   components they have uh so we have a stream of tokens coming in um each token is an integer uh it's an integer from some somewhere from you know zero to let's say 60k the number of tokens in the vocabulary um where those would be like the the most common combination of bytes right so every individual bite is a token the is a token hat is a token and then the first thing it goes into is called an embedding and embedding Maps those integers into the

4. 2:11

   vectors of size what is called G model um and the g model is the size of the vector that represents an internal I I like to call it the state of State of Mind of the model uh and uh as the uh as as the sort of information flows through the Transformer at each point the state of with is represented by a vector of D model floating Point numbers right and then many many times um this Vector will go through

5. 2:53

   a uh like a single Transformer layer um many times and then at the end it goes through what is called an un edding where uh in the UN bedding it goes back from a vector of size D model into the space of tokens right if you have 60 tokens they unembedded will be effectively a matrix of size d mod times the size of the vocabulary and it will return you a uh a vector of size and vocabulary where

6. 3:44

   every element is proportional to the um logarithm of the probability that that token should be sampled right and then you exponentiate them you get the actual probabilities well and every Transformer layer uh consist of two parts the first part is called uh attention right and then attention um you um that's where you you mix the information between tokens together right so like if the later token enters

7. 4:23

   here it goes into the attension so internally here cck into attention and MLP wring the attention uh you um you you start by multiplying this Vector of size de model the input let's call it x uh you multiply by by three matrices one is let's pull the first one q w sorry yeah like very good thanks let's see will this work yes then you have x q um w x right so so you get three matrices um

8. 5:18

   and effectively K and uh K and V are kids and values that you computed for the future tokens uh and Q are queries that you use to query keys from the previous tokens uh I'm not going to go too deep into attention right now what is important is that there are three Matrix multiplications here involved and then after you compute qk andv the actual out of the attention if I don't mess it up will be

9. 5:52

   uh something along the lines of Q transposed time k um time V except that this position probably goes here yeah so you compute sorry and there's a sorry okay uh so again what is important here is that there is a m Mo here and there's a metal here and then MLP is effectively just matrix multiplication um some nonlinear function in the matx multiplication so in each Transformer layer we have 1 2 3 four 5 plus two from MLP six seven Matrix

10. 6:42

    multiplications uh then unembedded is a matrix multiplication right so as you send as the token as a single token goes through the Transformer the majority of time it spends uh the majority of computation spent is performing m multiplications right and so when you run like if you if you download all Lama and you run llama on your Mac um the reason it is feasible the reason it is somewhat fast is that the Matrix multiplications are not being

11. 7:14

    performed in uh in the full Precision not in fp32 it is usually performed in something much smaller and even when you train the model right when you train a model uh the gpus a100 h100 they have a very fast uh kernels uh with not not even cron they have a very fast uh processing units on the chip that can multiply uh 16bit floating Point numbers uh approximately four times faster than they can multiply 32bit numbers right and so uh as you TR the model uh the uh

12. 7:49

    the forward pass backward pass you do in forat mod you do in bf16 not an fp32 that makes training four times faster uh and after the training done once you have the model people quantize it to 16 to to eight bits and I think some people even quantize it to four bits uh and the question would be in the ultimate can we quantize it to a single bit where every Matrix every element in the Matrix is represented by by one bit and let's see let's see why this

13. 8:22

    challenging and what can be done about it so let's zoom in into single map mod so let's say we look into this one right so we have Vector X here this is the model and usually we don't look at a single Vector we look at a bunch of them uh so there's your whole context window of data right so if you doing inference you you fit the prompt let's say the prompt is already 100 tokens so this is your 100 tokens your prompt uh we call it

14. 9:00

    uh let's uh and this is multiplied by The Matrix of size D model by D model and after the model has been trained there's this a matrix of Si D model by D model of either FP 32s or bf16 so it's either two bytes or four bytes per element right like something like 0.6 0 uh and effectively what we saying is we say hey if we slightly lose Precision here right if we compress every element to the fewer number of bits the overall

15. 9:59

    output of the model will not be affected that much but the performance of the computation will be improved significantly right and and and we would uh during the cotization we would compress every element let's say to one bite so to eight bits right and then each of them will lose some amount of precision uh but uh you know like overall the performance of the model will not drop drop that significantly

16. 10:23

    because you know especially if you like you know round them stochastically into different uh uh to different uh like like some of them you run to the to the element that a smaller some of them to to the element that is higher and if the G model is large enough like on average it is not that far from the from the output you were supposed to receive in full precision and people measured it and it was good enough for eight bits maybe not that good for four bits but once you go

17. 10:55

    to the lower number of bits it gets worse and worse right and you see why um I think it's somewhat helpful to to to look at like uh what what is it even that we're losing when we quantize right so if you look at the bf16 right so let's say let's say the model was trained in bf16 and or rather let's look at fp32 first so fp32 you have 32 bits and out of 32 bits you will have um

18. 11:30

    let's think about it I think I closed it already but I think you have something along the lines of8 bits of exponent one bit for the sign and then you would have how many left 32 - 9 23 bits of uh mon I could be a little L here it could be like 10 and 21 or 11 and 20 I don't remember exactly but importantly so mantisa is what gives you the Precision and the exponent is what gives you the range right so you can trade one for another right in the extreme you can

19. 12:09

    imagine having you know 31 bit of mantisa that effectively be an integer uh type or you can imagine having 31 bits of exponent which will give you a massive range uh but uh practically no precision and in a p32 uh it has quite a bit of range and quite a bit of precision so that's uh there there is no real trade of there but once you go to lower uh Precision types uh the tradeoff

20. 12:40

    starts being more and more pronounced right so if you go to 16 bit types uh the default 16bit types called fp32 fp16 would have five bits of exponent uh and 10 bits of mon uh and that gives you uh decent range uh but sorry decent Precision so 10 bits of precision but very very low range uh and the problem is that quite a few values that you would actually encounter during training

21. 13:17

    uh would not fit into the range of of with p16 so what people do instead is they use a different data type called bf16 and that has longer so it would be like here right it would be eight bits of exponent seven bits of antisa so that gives you uh assuming my fp32 was correct that gives you as much range as p32 but much fewer Precision right so effectively all the bits that you trade as you go from fp32 to fp16 are taken away from Precision n them are taken

22. 13:59

    away from range cool H so bf16 is known to work pretty well for training right majority of training happens in in bf16 right so bf16 is uh the model is strength in bf16 so clearly during inference bf16 will work just fine right okay so as we go to BF to to 8 bit uh we get one still one one sign bit and then you have two uh big two major options effectively you either do four plus three or you do five plus two five bits

23. 14:40

    of range or bits of precision so notice that as you go to eight bits right so in bf16 there were eight bits allocated for the range right so that is already more than you have total right so you will have to significantly reduce the range uh and uh barely anything is left than for precision and I think um I'm actually not quite up to speed in terms of what people use for BFA but but either of them is is very limiting so so this data type the whole range is like from minus 400 to plus 400 you will take

24. 15:15

    and this one I think is from it's something like uh I have it's here practic fromus 57 to plus 570 right so in both cases the range is pretty low already uh and the Precision is the like you know the Precision you have is two or three bits so what is important is that like it's quite natural to assume that for quantized model to work the weights need to be at least somewhat close to what they were right so as the amount of

25. 15:47

    precision you have goes lower and lower the weights deviate more and more from what they should be right so like as you go from two bytes to one bite you already will be quite far from the actual value that you have here but if you imagine going to four bits right in four bits you still you would still have one bite per sign right so you have a total one bit per sign you have total of three bits for uh for your exponent and for your mtic right so so

26. 16:19

    like clearly you will deviate very far from the from the values right and in the extreme case if you have one bit then you literally just have the sign right one bit you have the sign you have no exponent and no Mantia right so uh and in practice I guess people tried it right and if you quantize the model to a single bit or even to two bits the model doesn't perform well right it loses it loses its

27. 16:49

    capabilities so what can we do uh so what the the paper suggest to do and there was a there was two papers in uh uh in exception the first one was called the first one was literally using one bit per uh per weight and this one is pre Lama right it's October 2023 I think this is even pre- GPT or like I don't remember when GPT came out was it November or October but it's like wait no I'm saying it wrong 2022 yeah

28. 17:24

    yeah okay I'm saying it's wrong so this is post Lama right llama was 20 22 as well L was early 2023 okay anyway um and in this paper uh it boils down effectively to the following and and this is the paper that introduces the whole concept it BS down to the following they say instead of uh quantizing the model after the fact what if we train it already with the idea of every weight being a single bit right and it's uh there was a reason why I I drew the

29. 18:04

    whole Transformer at the beginning because it's worth mentioning what what what they do quantize and what they do not quantize right so a couple things they don't quantize they don't quantize embedding to one bit uh and the idea here I think that it is important for the weights like like for this Vector initially to be high Precision right so if you quantize the embedding then your

30. 18:28

    uh you know mental representation of every token from the GetGo will be effectively binary it will be D model bit number which is not quite what you want right so they do not quantise embedding and embedding is fast anyway at least they don't quantize it to a single bit they do not quantize unembedded to a single bit or a thing at all uh with the reason being that after you do un embedding you

31. 18:53

    want to sample and ideally you want sampling to be done in high precision they do quantize uh each of the parameter matrices right so you uh uh query projection key projection value projection and both matrices in MLP are all quantized to a single bit uh and uh they do not uh for this product and for this product uh they do them in uh full Precision or by full Precision I mean

32. 19:28

    presumably bf16 uh they don't do them in U in low Precision okay uh and when I say quantize they do not quantize matrices at rest meaning that the actual Matrix they train the actual qw KW and VW and the two matrices in MLP are actually High Precision matrices right they they during training they are stored in at p32 or bf16 uh and and they trained in high Precision uh however uh the what changes

33. 20:08

    is how those matrices are applied on the forward forward path on the forward path instead of multiplying uh the uh the input vector by The Matrix itself The quantize Matrix on the fly right so the Matrix is quantized before being applied to one bit and then they multiply uh The Matrix the input M uh the input tensor by The quantized Matrix meaning that during inference you don't

34. 20:40

    need to know the high Precision Matrix anymore right because during inference you only have the forward path right so during inference you will only be multiplying by a one bit Matrix uh but during training uh The Matrix is full Precision right so so you can apply gradients all right so let's go into a little bit more of a detail right so so let's zoom in into single matrix multiplication right so we have this let me move it

35. 21:22

    oops okay so we have an input tensor and a matrix that we multiply by uh so so this Stace I canot do this this okay so we have the same input uh and we multiply by matrix uh so the Matrix changes specifically for every weight in the original one bit paper uh they would apply uh the following transformation they would subtract the mean uh so they would compute uh it's the mean of the whole Matrix right so they compute

36. 22:03

    the the average value of the whole Matrix subtracted from every element right so now it is effectively C centered around the mean and then they would just take the sign of every element right so if the element after subtracting the mean is positive it becomes one if it's negative becomes minus one so let's say the mean was like let's say we mean with something like2 and then this element will become one because it's bigger than me this element will become minus one right and so on average you will get

37. 22:34

    half ones half minus ones and then you multiply the uh the input tensor by this Matrix of once and minus on you get the output um now ideally you would prefer the output to have more or less the same distribution uh as it would have been if you didn't quantise uh so in the original paper where is it uh I I do not see them anywhere compensating for the subtracted mean I guess the expectations that mean is approximately zero which is not a

38. 23:09

    potentially completely unreasonable expectation but they do multiply by what they call a scaling Factor beta which is not described anywhere but it's presumably uh the same mean uh so let's say we multiply Again by the mean uh you you can imagine if you subtracted if you replaced every element in the Matrix or it's not going to be mean it's going to be the mean of the absolute value right if you imagine if you if you replace every element in the Matrix with one and minus one uh and then you multiply by by original Matrix

39. 23:49

    and you multiply by The Matrix that only has the signs and and and let's say for for the for the for the sake of Simplicity that the mean was Zero right so uh then the output will be uh if the uh if the average absolute value was one then the distribution of the output will be approximately the same right uh if the average uh you know absolute value was two then the output will be approximately twice as as little right so effectively you want to multiply later by the average absolute

40. 24:33

    value to get to the same more or less distribution and presumably beta is in this particular case and and for the second paper that that we're going to switch to in a second they do have code so we will see this is more or less what happens right so so effectively you replace uh so so before output was equal to x * W right then the new output would be uh would Beal to x times um let's call

41. 25:12

    it binarized w right uh and then you would uh yeah times uh uh mean something like this cool now the problem with this is that this operation uh this one uh is not differentiable right so as you multiply uh so as you compute the forward path you want to compute the backward path the backward path should be uh the derivative of the of the function that you applying the pass but this bation step is not differentiable you canot really compute the backward

42. 26:05

    however what you can do is you can say well because the output of this step is approximately you know has the same distribution more or less as the multiplying by by The Matrix itself uh we can pretend that minorization never happened on the backward pass right so you compute on the forward pass you will uh you will quantize The Matrix uh multiply by The quantise Matrix multiply by the scaling factor and on the backward path you will pretend that that never happen right you will just

43. 26:37

    literally pretend that so on the backward pass you will compute the gradient of the you know just say hey like the the gradient of w is equals to you know gradient of the output uh times UT trans I think this is how you computed plus minus uh order and transpositions right we just completely ignore that that is called a straight straight through estimator and the fact says hey we're going to estimate the gradient of

44. 27:15

    the of the weights and of the inputs as if um you know by by by estimating you know this no differental function with this fully differentiable function which is just a matrix multiplication s would say like gradi of the input is equal to I think it would be something SAR of the something like this probably anyway cool so that's for weights now as I mentioned before uh so during training

45. 28:09

    you don't save that much you do save a little bit right because now your forward pass is faster right during forward pass you do not do this matrix multiplication uh you you quantize the uh The Matrix and do a much faster operation uh but during backward pass you don't save anything because you still do two m mve to compute the gradient of the weights and the gradient of the input but during inference the back doesn't exist right so so this doesn't exist at all right uh you only compute

46. 28:43

    the forward pass which is which uses the Quant Matrix and as a matter of fact you don't even need to store the original Matrix it does not it is not necessary during inference you only store The quantized Matrix meaning that you use significant fber memory you use one bit per weight uh and for the forward pass you can use a custom kernel or even custom Hardware which doesn't need to do

47. 29:05

    any multiplications right it only needs like like if you think what happens if you if you think through what happens as you multiply by this Matrix right you have some input Vector right it has like some ABCD right at every step you start with accumulator equals zero and at every step if it's a one you add the value if it's minus one you subtract right so there's no multiplications involved whatsoever

48. 29:28

    right then there was the second paper uh where they uh it simplifies quite a few things and it introduces one other value which is zero they effectively say instead of only having once and minus ones we're going to have ones minus ones and zeros the benefit there are two benefits of doing that the first benefit is that you have more Precision right so now you now you have three values which increases the Precision of your model uh but the second benefit is that if an average one

49. 30:01

    out of three values is zero which is what they observing eventually that the distribution of ones minus ones and zeros is uniform more or less uh then one out of three times you perform no subtraction or addition right so you you reduce the amount of computation by 30% uh and they simplify the um the computation they do not subtract mean anymore you will notice right so they do round clip and round clip just just starts from uh

50. 30:39

    oops so so like X here would be a single weight and they sub they they do not subtract mean they just um yeah so so so effective they start they start with the weight Matrix they divided they divide it by the mean of the absolute value right clip it to from minus one to one and round it right Sur rounding effec means um you know it becomes either minus one zero or one whichever was the closest uh so there's no more mean

51. 31:20

    subtraction uh and oh one thing we didn't I didn't mention is they also quantize the the activations uh and uh they effectively say we want it to be between they want activations to be one one bite right so eight bits uh and they uh I don't think they do anything particularly fancy there it's just uh uh they just literally squish all the values into the range from minus 128 to 128 and round it okay so let's see how it looks on the

52. 32:08

    code they they did publish a PDF that has a pie torch uh implementation of it uh so let's take a look and see how all all all of what we just discussed Maps into code so so weight quantisation right so you have a matrix W Matrix W is a full Precision Matrix that you have in memory um okay so we compute the apps mean right so that corresponds to effectively Computing this um I I I don't know how to pronounce Greek

53. 32:51

    letters uh all right uh so this is effectively to prevent division B zero if it happens that your whole Matrix is all zeros uh you don't want to to end up dividing by zero so W Times scale means w divided by the absolute mean value then you round it right so what happens here is that after you divide every value by the apps mean you will get something uh the absolute value of every element will be

54. 33:30

    will be on average one right so after rounding it you will get zeros ones minus ones you still can get some other values right so the fact that the mean value is on average one doesn't mean you don't have out layers right and then it is clamped between minus one and one and Let's ignore this divided by scale yet so this part effectively computes the quantized model right and so what you would do then uh is that you would separately have have the scale factor right and on

55. 33:59

    the forward pass you would apply the quantize model right and then later you will divide by scale but during training I guess they short circuit it and they divide by SC The Matrix itself is divided by scale right so they do not use the custom kernel they just use this regular math Mo so might as well multiply by scale right away scale is a scaler right it's a single value um right so they divide so fact they they end up having not so the Matrix the output the U here will not be

56. 34:28

    the Matrix containing minus On's zeros and ones it will be a matrix containing minus one over mean absolute value zeros and plus one over mean absolute values right but ideally when you train the model in ship it to people you will publish this Matrix and this scaler separately right so then you can use the custom K or custom Hardware to apply it and then you divide separately and activation quantization is similar uh

57. 35:01

    it's effectively the same function with a couple of changes one is they divide by by the max maximum value not mean all right so this is more Reliant I guess on the activations being more or less uniformly distributed and you will notice it's per that makes sense it's per token so so so here scale is a it's a single scaler right so the for for the whole Matrix you compute a single value but here it's per per

58. 35:34

    token right so scale is a uh one value per token there right but the uh transformation is the same the difference here would be that uh the output of this is not uh the mean is not one the mean the mean absolute value now is 127 so when you round it you will round it to something between you know minus 127 and 127 and then you clamp it and again divide by scale this is something that

59. 36:16

    um potentially will be done as a separate step cool and in terms of implementation the implementation after that is very very simple right so so you build weight quantization you build the activation quantization the whole function the whole layer now boils down to having it has one parameter the weight you Canute Ms Norm this is this is the same thing that would be done in llama right so llama begins every layer by Computing in Rems Norm right

60. 36:50

    and then uh let's see what happens right so they say x quantized is equal to um x normalized uh plus okay so so so so let's go through this this is uh this is an implementation detail right so you will notice it has X Norm minus X Norm right so they will cancel out and then it has activation Quant of X Norm so this is the qu quantized X Norm uh you know fromus one 28 to 128 so the reason to let let's get back to it in a second but for now X Quant is just quantized X

61. 37:36

    Norm right these two things cancel out then we multiply then the same hack you see there's W minus W that cancel out and there's a weight quantization right so X quantized is a uh quantized input uh W quantized is a quantized weight and then they compute the out put as a product of quti X and quse W and this hackery with adding and subtracting X Norm uh is effectively a way to implement the straight through estimator uh that we

62. 38:14

    discussed where we want to compute the gradient as the quantization never happened so detach effectly means hey do not do not do not use this for backward pass right so if you look at what the backward pass will see for the backward pass X Quant is you know equals to X norm and and W Quant is equal to W and then Y is the product of X norm and of yeah n w right so on the on the backward

63. 38:39

    path uh the gradient uh will only see W and X norm and effec you know the gradient of X Norm will be a product of the output gradient and W quantized and the product and the gradient of w of of w will be product of the X one and output gradient right so so this this this uh little trick with adding and subtracting X Norm but then detaching one of them allows you to implement straight through estimator in a very uh concise way right and then once

64. 39:21

    you have this layer which is what is it like 30 lines of code maybe less 20 lines of code uh all you do is you replace the linear layer uh for query projection key projection value projection and the two um mlts and the and the two Matrix Matrix multiplications in MLP with bit linear it's called bit linear and then you train the model right and the idea here now that because the model is as you train it because the module is already quantized right it

65. 40:00

    learns the weights uh that will work with the quantized model right so so by the end of the training you get a model that performs well being quantied to a single bit and there's a couple of other interesting things here one is that at the beginning if your learning rate is if you if you start with the same learning rate you would start when you train a you know LV model uh the training will not uh pick up because applying a small

66. 40:34

    perturbation to the weight does not like effectively you need to apply quite a bit of change to the weight for the for the sign to uh you know to flip right and if you start with a low learning rate you apply small changes to the Matrix very few bits flip right so the training does not does not do much so they have to start with a much higher learning rate uh to the extent that the

67. 40:58

    learning rate they start with if you started training Lama with the same learning rate like a full Precision llama uh it will diverge right so so so effectively empirically this one bit model is more stable at the beginning of training uh which is uh which is nice right but then they I I believe like closer to the end they they actually go to prow learning rate and at that point the the

68. 41:24

    training goes pretty well okay I think that is all there is to the paper the um the cool thing is not only the fact that there is the is we interested the reason we interested in this uh and there's a good chance we're going to try to train something like this uh is that that allows you to have a much bigger model that fits into memory of you know of a machine and the inference would be much much faster right so if you have if you have a machine without a

69. 42:10

    GPU accelerator not having to multipay things is a is a highly desirable aspect right and so that can enable more people to benefit from more sophisticated models there's energy analysis on uh operations on bits versus uh eight bits and it's like 10 like way lower energy usage and so for mobile devices for example if you run such a model with uh this trender weights you potentially have way less energy uh exhaustion which is probably

70. 42:57

    one of the bigger issues actually of for running on edge llm right now and I I guess it's also it also makes it more viable to run models uh like I don't know if it's a good idea right but it it does it does make it viable to run models on uh on sgx in this on the CPU right so if you trust uh you know like if if you have a cloud with a bunch of sjx devices you can you can run a one bit model could become a viable thing if that's something that is

71. 43:32

    important to you cool are there any questions comments and yeah it can be custom fpga or as SEC as well yeah yeah so so like if the model is shown everything I showed right now is uh uh um actual hour five month ago I actually did not find if anyone managed to train a large model it's been it's been it's been several months so there there H someone already trained if nobody did we can we can definitely try it right and if it is a you know if

72. 44:19

    if a larger model is uh is functioning well than like investing into building in a and Asic that does it very fast is is a worthy Endeavor yeah I thought go ahead interesting to look at this from kind of the top down perspective um you know it makes a lot of sense to push towards binary we've been doing that you know for many years as Engineers I mean the initial computers you know used a lot more

73. 44:52

    different voltages to represent different numbers go 0 to 10 volts great of you know base 10 um but it was you know it was a lot simpler to to push that down towards binary partly you know because of just noisy Electronics but we also got some things that kind of dropped out of that out of binary that you know now you're working with Boolean logic and you know powers of two you get fast 4A transforms instead of for

74. 45:16

    transforms so so I feel like you know logically it it makes a lot of sense um and then you know it's also kind of similar to the the brain right you've got brain activations or kind of a binary type of thing um so yeah makes makes sense to go this direction and then looking at this paper it's like okay turn aray you know why why would you go turn AR from like just a conceptual level

75. 45:41

    level um and you you had mentioned you know the kind of quantitative advantage that you know a third of the weights are zero that makes a lot of sense I was trying to think about kind of the the qualitative differences between like ones and zeros and negative ones those those give you some different sort of things you know off is very different um in ter uh you know from from on if you

76. 46:07

    asking a neuron's opinion you know do you have an opinion on this feature or or like no no opinion um the negative seems interesting in terms of like being able to have inh inhibition you can you can have a neur neuron that's inhibiting other neurons um in your system here so so it's interesting this so first of all there's there's a disadvantage right to having zero because now you need to have two bits per weight right so the memory

77. 46:36

    footprint is H you either need some hackery to compress it to not to not actually spend two bits uh or you wasting like the the memory footprint time is 2X and something interesting something IIA mentioned two minutes before we we went life is that the paper does not actually compare one one 58 bits to to one bit right so so nowwhere here in the results uh is uh bit net one bit so it's interesting right it it it

78. 47:10

    it would be it would be a good experiment um to try uh to um to compare one bit to 158 bits and see how they compare two things I think are like worth pointing out is like one uh in your diagram Alex you're showing like minus one and one but there's also zeros uh effectively you don't need the need to load those weights into memory because they shouldn't affect like they won't affect the output um and also in the P torch code

79. 47:48

    they show using linear but yeah as you mentioned you can use like a better kernel to compute the forward pass like it probably just end up being a select and a sum effectively and that would be faster than doing a m mole in general so there's actually another paper that came after uh in in June I just sent it here and it's pretty much matm free um Transformer and it is uh pretty much toward what you're just talking about which is then going in you know okay

80. 48:32

    well if we're already if we're already in uh you know bit space like let's just remove Matrix multiplications completely and replace them with trary accumulators and few other tricks and so so I think it's um pretty much in that direction of like and they actually they have fpga implementation at least the reference in the paper uh that presumably allows to have this yeah okay so so this so the this is

81. 49:13

    the same bit linear right it's not a different yes yeah so they that's what I'm saying they took that and they can expanded it without multiplication yeah yeah it's worth mentioning I I didn't think too deeply about it but but I think on GPU the fact that zeros are not resulting in a addition does not give you that much right because on GPU you know like all the processors are doing the same work right so if if you have like like like you know if some of them are one

82. 49:46

    some of them are zero some of them are minus one it will be something like you know if it's one add if it's minus one subtract and all of them will have will have to go through those branches so zero doesn't save you much but when custom cardware um presumably you can where where do I have it yeah presumably you can have it um saving you quite a bit of time do they build it for like a is is it for pgs

83. 50:16

    or they have I think both the P torch code and fbga also depending on the distribution of like the number of zeros then you can also save on memory it's not hard to do on GPU yeah it's hard to be hard on GPU but for fbga like save on Space yeah I guess in the short term if we're simulating this instead of having an FBG a is it still worthwhile does it have enough gains I mean I think on the mobile device is just the the amount of energy

84. 51:05

    exertion is so much less um like I'm looking at the there's a comparison in the bit in the one bit what paper and they show that the on the 7 nanometer processor the addition on 32 bit bit is uh compared to on one bit let me do a division is like 40 times cheaper to do one bit and uh multiplication is like 300 times cheaper so you potentially just getting yeah such a massive energy and I I've heard the same thing from like folks who run on uh on

85. 52:01

    like llms on phones that they they're very interested in this types of models because uh because it's just like so much more energy efficient because right now if you try like even if you're running 3B uh model I mean especially if you're running 7B like your battery was just going to die and you know whoever ran gpus like when you're actually like spinning up all the training process or inference

86. 52:41

    process right you can see the energy usage just going through the roof right I I want to open up the discussion on limitations that come with like this model dis approach sorry and um like for example you said Alex that like for the for the weights to flip in the in the model itself it takes like a Higher Learning rate at the beginning because it's like very it's like insensitive and so what would happen in case of fine tuning for example what do you believe

87. 53:16

    interesting question let's see if they discuss fine tuning at all not much so uh well well let's begin with a simple fact if you only open source this then of course fine tuning is out of question because you do need the high Precision weights right but naturally I think it makes sense that you know if this model gets open open open source or or even if it gets open weights uh it makes sense to open weight the the high

88. 53:50

    Precision weight Matrix right then presum fine tuning will work right likely you will have to start with the higher learning rate than usual uh but the same you know the same training Loop you apply during training should potentially work with the fine tuning that's something to experiment with to see if uh if F tuning picks up because I I assume that uh it's an interesting question right because if you were like as you apply even with a

89. 54:21

    Higher Learning rate as you apply the the gradients um a lot of weight will not flip the bit right so so so like when you train on high Precision you apply the gradients every weight is changed right you get a lot of uh like on the next step you have a model which which sort of assimilated quite a bit of information from the previous gradient update right but if only a subset of bits flip right that means that a bunch of gradient updates did not affect the model at all so I would expect that during fine tuning with the

90. 54:53

    same number of samples it's uh uh it might require larger number of samples to get to the same level of quality but but that's something you know that's something to play with so it makes sense to try to train like a I don't know a largish model and try to find see what happens could you think of any like other kind of limitations that could come with these kind of models it's a h to answer question yeah again it's an easier

91. 55:36

    question to answer with experimentation right so we can um I mean I I it's actually interesting because indeed you need to like get over a bigger hurdle it might be interesting to see which bits off flipped when you are updating on a single batch because that that may be more informative than yeah like looking at gradient updates in in the regular models from just interpretation

92. 56:21

    perspective but I think I I I I I looked around internet there's there's few different people who've done fine tuning like instruct fine tuning as well as Laura seem seems you know it works I mean there's a little bit of handling of Laura required to quantize the Laura itself um oh so so so Laura gives you so so what is Laura in this case you will have two quanti matri on top of

93. 56:54

    the yes see seems like they are what's the meeting there's a link I sure there's a bit lower layer is it the same room it's not it's not the the guy who who built per right I mean PIV 2 AI right yeah um but yeah definitely need some experimentation get get this uh set up but yeah I think it would be really interesting to see the difference after like one batch and kind of to understand a little bit more like because indeed yeah you from a

94. 58:00

    every single update you have a little bit of difference and all in all weights every update which is really hard to interpret and so you kind of need to do like if you if you want to see how much information is extracted from single batch you really need to get um like a lot of kind of AB testing inferencing thing here you you can actually you can then isolate some of the some of the things some of the capabilities potentially

95. 58:38

    because if you are as you're training the model you can see when it gets like when it starts getting some capability right like you literally run a e Val of that capability at every step and then if you go and and see kind of what was the batch that led to it what what was the updates that can be interesting like analysis around this and if you have like set of capabilities and kind of testing for them so yeah just it can be really

96. 59:18

    interesting for just like understanding more the how information is stored and how how the more reasoning capabilities are being emerged yep okay we're getting close to the one hour mark any any final questions or comments people have I guess no okay cool thanks everyone for coming and watching and I guess there's going to be more of those coming uh in the future weeks so stay tuned yeah if you have any ideas for papers drop them in comments SL

97. 1:00:18

    repes and thanks Alex for deep dive