NEAR AI Office hours #30 w RWKV

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   hel the future in in this driven Wonderland hi everyone this is episode 30 of near AI office hours and today we have Eugene from a project called I'm afraid to mispronounce itak yeah you got it right uh though for most people in America I actually prefer to say uh r w KV the key value um which is a player on the famous Transformer qkv because every time I say rou people keep thinking are you Roku

2. 0:39

   TV yeah yeah okay so um RW KV is a uh it's an alternative architecture I I don't know if you guys still call it a Transformer architecture right that primarily so initially when when I looked into it I thought it only replaces the attention but it but it replaces all the components of the Transformer right including the forward so for the full RW KV model we actually replace all the

3. 1:08

   components uh for it um uh for the converted variant uh like the qrw KV we do keep the feet forward from the Transformers uh but I think those are like more towards the technical details because like like like some the core blueprints are still the same like we are once you look underneath the layers it's still matrix multiplication SE activation functions those things are still the same y cool so there's a couple exciting

4. 1:41

   things uh to discuss here so one is that uh uh so my understanding is that RW KV so so there are generations of it right and we at generation seven right and each of them one um six is the main main line right now uh where seven is probably going to be dro yeah and yeah very much like Ubuntu versions they all called after a animal or a bir uh that is that is starting with the right so we we are getting into

5. 2:12

   the goose uh and uh so so so there's a there's a PDF that explains versions five and version six and I presume there's another PDF that explains the previous ones but I didn't see that one I only read the one yeah there there's no PDF of version three two and one because uh we didn't we didn't write papers during that age yep yeah and so uh the PDF I don't know if it's available from the website right there's a website which is spe which is RW k.com I don't know if there's a link to

6. 2:47

   PDF from here um but yeah if anything we will we will we will post it together with on Twitter on YouTube so so that people can can see it um so so there's couple I guess there a couple questions to start with one is can you I think one interesting thing to to explain here is what is the structure of the project uh right who's who's driving it how many people are involved how how they communicate to each other I think this

7. 3:19

   is a very interesting uh this is a very interesting effort so I think one of the things that makes RW KV very unique and actually confuses the heck a lot of people in AI right is that we are driven more like an open source software than anything else U so typically in AI space right people will have a hypothesis and then they'll do they they'll draft a paper and then they will they will do the experiments

8. 3:45

   and then they they'll do the results so it's always like things will be released more in the sequence of paper model artifacts weight weights and code in that sequence uh ours is as with any open source project right is the code is is actually in the GitHub report it is being iterated live um the reason why some people already know about V7 is because the V7 candidate code is already

9. 4:09

   in the GitHub report um and and these things these things um so there's one channel called RK XDA Where we actually allow experim people to propose experiments and experiments get run uh is is that when new ideas can be introduced and and and in a collaborative way and if a call member picks it up do do a run and thinks that hey this actually improves the model congratulations it's a candidate

10. 4:34

    for the next version It's a u one of the candidate mergers for the next version and that's how we've been actually building up on the RW KB project since essentially version one or two like uh where where we showed that the core idea is that hey if you take something like an RNN structure but if you remove the bottex from it and you scale it up using using the same matrix multiplication and

11. 4:58

    activation functions techniques that we see being used in Transformers we could re-engineer uh this architecture to be extremely scalable at Transformer level will still be extremely uh compute efficient like an RNN and and when and after we solidify like certain key ideas right like one of the things I say is that no idea no component or idea is sacred uh we have ran experiments where

12. 5:23

    we where we where we remove or replace entire core components um just just to to see whether it works better and that's that's led to the iteration from V2 to V3 V4 V5 V6 and V7 so in terms of like Community contributions right uh in terms of the foundation model uh side right the call contributions is only around like I guess we say like eight to 10 people give and take uh some more

13. 5:49

    active some hey as with any other open source project have not seen you in half a year and then suddenly see you again kind of situation uh and and and we contribute the ideas and and and and the call members will try to to train it and and iterate on it and and so on um but beyond the call model um so so so the so the call the call the call team is being led by uh coet by me and um bling the the original Creator from version zero

14. 6:20

    and Harrison V who who's uh who's another call committee committee member um Beyond this call group that works on the foundation there is an extended group as well that works on the inference code um the the data sets and and basically things within the ecosystem so so that's how how how like like this project like balloons from like 10 call Members to like 20 30 additional auxiliary contributors and and and over 800

15. 6:52

    members into our Discord committee itself so uh so so uh sorry 8,000 members so so that's that's the the the structure for theb organization yeah and so um by the way you mentioned that you you removed the whole components which is which which is I guess exactly what happened when the Transformers were created right because I think before Transformers were published it would be blasphemy to think about removing recurrent connections from from our nns right but they removed

16. 7:22

    the hold uh the curring there and so so with rnms right so with r&ms we have uh each token depends on the previous one right and so you you go from left to right and so historically before Transformers are nms were very hard to train in parallel because you you had to go one token at a time at training time right so Transformers changed that so that now you can train on the whole sequence but at the same time Transformers are at the inference time

17. 7:52

    you pay panel to because you have quadratic attention right and so my understanding is that r wkv has a construction which which you can compute both in parallel during training but then you can sort of reformulate it uh in the way that it is recurrent right is it uh correct understanding or yes it's cor understanding uh but I I think it's also part of like one the one of the key things that we broke the bottom neckline uh so if let let me see

18. 8:24

    whether I can I can I can share my screen am I allowed to share screen yep okay I'm going to quickly share screen so so I'm not sure how many people it we will see but it's like like the one way I view it is that uh like every model be it Transformer orn at the end of the day right is just lots of layers with embeddings flowing in whatever directions and we just stack them um so

19. 8:49

    that's that's the core argument we making the the big leap for for AI right what's not attention is all you need it's actually more of like we have figured out how to scale uh matrix multiplication to billions of parameters and and and so so the if you remember the call the O RNN structures right is that the biggest bottleneck right is that your embedding from the last layer uh will need to be fed to the first

20. 9:15

    layer in the second token so the first token as as it outputs as the outputs come out the last output will need to be fed to the second second token and in F again this made every single token need to be wa exclusively one after another and that caused the the very slow botton so what end up for people is that you have your h100s okay it wasn't h100 at that time times have changed a lot but you

21. 9:41

    can have a H1 but if you need to do things step by step right it was entirely it was extremely slow why we fundamentally change is that in if you want to oversimplify is we remove that line and once you remove that line right you the the computational process right can happen in a cascading pattern so so once the first very first inference block is computed uh so so for example

22. 10:05

    like so so once you remove that line here and the first INF block is computed at the start it it Cascades very rapidly and this is like within the first three four tokens uh you can start saturating the GPU because of of the cascading pattern um this may not be exactly how it runs because you can think of blocks as chunks or gpus or layers um because it's apparently more efficient to do that when you write right the exra can

23. 10:30

    but from a visualization standpoint right um once you have pass the first few tokens you can Cascade and you don't you no longer have that sequential waiting bottom neck and that essentially allow us to scale uh a lot of our training or even inference because even on inference um we can apply these techniques for preing and and and to generate the state rapidly yeah so here x x is context and Y is layers right

24. 11:00

    say again so x x axis is that's the context right and Y AIS is the layers then y one layers yes yeah y okay stop screen yeah so um so like the general Principle as we scale or do iteration is that it it must not compromise on the compute it uh Boton and it just needs to improve better the uh the model and we iterate on that so we we we someh have kept this cascading structure uh where information flows up and to the right because it's the main way to ensure that

25. 11:41

    we have a fixed State between tokens but in a way that structured that uh can saturate a GPU yeah and so so there is a um there's a 34 billion parameter model trained with this right that's that's version five right um so for version five and version six we only train from scratch um though we are stretching the definition from scratch here um because some of it is converted from older versions of RW KB

26. 12:14

    um or up to three trillion tokens uh up to 7 billion parameters then we upcycle it to 14 billion parameters so those are like our Mainline RW KV models um that is trained on over 100 languages uh and and this so one that's interesting for AR because we are very Community Driven approach and we we actually work with a lot of people from different nations um we actually Focus very heavily on

27. 12:39

    multilingual from from the early get go our tokenizer is a completely different tokenizer U yeah it's a what we call the world organizer and it was train specifically on the top 100 languages uh according to Wikipedia and so we intentionally train it on multilingual from from the from from getg go and the data that we use is also reflective of that and this also ends makes us ends up

28. 13:03

    like uh like being one of the top rank model uh in a lot of uh foreign evals and test uh test set specifically for for for for different languages only because we actually work with native speakers for for the for their languages um though though sometimes this is like more like a case of like hey we might be the only three models that support the language kind scenario so so but that's

29. 13:27

    the scen that's the thing why we do that is because we want to make AI accessible to everyone so and we this intentional direction right actually hurts our English eval but it's not something that we are obsessed over we are actually more like focusing on like multilingual um yeah and that that's for the call outm K models but for the qm K model which we recently launched uh based on

30. 13:52

    the V6 architecture um that is not part of our main uh Main Line uh that is uh uh an experimental preview version where we took Quin 32b and Quin 32b unfortunately is only trained on from what I remember either 15 or 20 languages um and and the and it was an experiment where we froze the feet forward layers replace the attention layers uh with RW KV and then we train the attention layers to replace the original quadr attention so this brings

31. 14:24

    about the and somehow it worked that was a crazy thing is it's like I I'm I sound like I'm explaining a Frankenstein experiment where I deleted half the brain and insert a new brain on it but but during the training um we just a few million a few 500 million tokens we were able to like fix the attention layers and we realized that it were and and the big benefits of this is that it it end

32. 14:46

    up being one of the best uh linear model and that is exciting because um Transformers are expensive to run uh because of the quadratic nature uh especially over long context or if you want to do ow reasoning uh and this is because like with every token you output um you need to expand the amount of memory space and compute space so if I to use a human energy that's a little bit absur because like basically for every second you need to leave your brain needs to expand so if if that was true like if you're 10 years old the brain would have exploded and and but uh

33. 15:19

    and but the thing is like um somehow we manag to get it to world and Transformers and through Ro force of the the computation we have and to me to that's something that uh doesn't make sense and if we are able to like scale it down to a more fixed State size it'll be much more efficient and that is essentially what our architecture does so even though the converted model is still we only replace the attention

34. 15:45

    layer RW KV and we don't use RW KV Feit forward layer which we call chomic so we um uh it is still able to retain most of the benefits that we have from rkv um without The Upfront training cost essentially and this is a paradigm that we are still exploring and scaling uh as of now we are converting the Llama 7B and we are actually looking forward to that because if that converts successfully and works with 32k context

35. 16:16

    L as far as we know I don't think there's a most Enterprises uh 90% workload is can be serviced with this model and if that's the case right if most AI workload can be done without quadratic attention um is attention all you need anymore well you can still I would say it is still attention right it's just no long longer quadratic or or you wouldn't you wouldn't qualify an attention at all yeah these are B inol debate it's like is self attention considered

36. 16:47

    attention as per the paper and I'm like okay I can leave that to someone else to debate it yeah yeah um and so so I think you mentioned it but just to uh reiterate so um so so so first of all the resulting model it is competitive with original Quin right so it's a uh but but the inference time now is uh noticeably faster right because you don't need I guess it's not just faster but you also you don't need to store the KV cach

37. 17:17

    right so also much more memory efficient um relative to uh relative to regular quadratic attention and uh the actual V6 right right right we mentioned at the very beginning to V6 both replaces attention and the fit forward but in this case the fit forward is still just a mat move right so you don't use the fit forward changes that you introduced in this six only the attention right yes exactly so so we we keep the

38. 17:50

    original Transformer feet forward in this case yeah yeah and it's so so I think it's interesting and the the Feit forward is Frozen for the for the entire duration of training right you don't unfreeze it at the end um and so so it's a two step process the first step uh is you need to freeze the fe4 network to initially train the new attention layer uh even that uh is not sufficient because um it's like if

39. 18:15

    VI as a protocol like like like the attention layer has a certain communication protocol with the feed for and and then after that you need to unfreeze the fif layer we have a custom learning schedule which we are still kind of like tuning for best results um to to ease the integration between the two sets y yeah um okay so I have couple more questions about the the uh the tech

40. 18:39

    itself so I think one thing I found interesting is that uh each version introduces new interesting Concepts right so so it's interesting that five and six are described in the same paper right so it's every advancement uh right and there's already so on the website there's already explanation of seven and seven go seven like even just comparing the visual representation of 7 to six seven looks

41. 19:03

    definitely more sophisticated right so it's getting um I'm curious so so one thing about seven I'm curious if it uses uh it mentions that the trains something at test time um do I I don't know if you know the the uh deep details of what exactly is happening there oh okay so so um um there's a there's too many things that happens so one things uh benefits and curse of Open Source Community benefit is uh nothing stopping members various members of the

42. 19:39

    team going into various Direction uh and and some sometimes the experiments can go faster than than the results so like um I so one one of the thing is that I'm sorry to our own internal RW KB Moe team uh because because um we were actually also doing Ane model and we we announced it on we in announced it on the same day as the qrw KV model but no I'll be honest no one's going to pay attention to thee model because the qrw KV model

43. 20:09

    performance jum is so huge compared to the model it's like the M model is not relevant yes and and that's kind of like uh how also part of how we run the reason why V5 and V6 paper came out together is because by the time we were done done with the V4 paper we were already at V6 kind of situation so so it was like Hey okay let's do both paper at the same time and and and yeah so so it's like there is that that step jump so like I

44. 20:40

    would say like in terms of like historically like the big jumps that we had right was from V3 to V4 then V5 is an incremental of V4 then V6 uh V5 and V6 is uh kind of like increments and then V7 is a huge jump from V6 itself and and yeah and hence the diagram got got changed dramatically um it's very in but if you see everything right it's still kind of like hold a lot of lineage

45. 21:08

    down to even like V2 or V3 because because it's still incremental ideas being built on top of each other um uh through experiments um as for like test time uh um test uh test time um inference compute right um so so so there are two two things that is that makes our architect very favorable for this kind of new paradigm where during INF time one one is that because it doesn't scale quadratically uh and skills L to us spending additional tokens is

46. 21:41

    significantly cheaper uh in terms of inference so this means it can it can it can it can do do do that process respectively another thing that's rather interesting because it's a state based model is that it actually changed the way you um and this is not Deploy on our API because um because um it's not something that people are used to yet but it's something that we experimenting

47. 22:04

    with it can change the way you you you you handle prompts for example I could let's say have do a a single set of forward path on all my documents several times so that the model learn the documents residue memories of it uh and and and with certain set prompts and I save that state so this is so you can view so then I do the next prompt from that state that means every subsequent inference right my state right the state that I

48. 22:37

    have trained to quite training might be might be might not be right like that I have pre- prompted uh it's now part of the model memories somewhat then I can just prompt from there and so this is like this is kind of like uh something that uh that that will that that will benefit uh like kind of like this kind of uh inference paradigms there's also another technique which we we are still working on right uh that is you once again unique to R because we have all these State fixed states that that we start

49. 23:10

    with is that another thing that we can do is we can take data from a data sets and we do fine tuning but instead of fine-tuning the model weights we fine tune the state and this state is 40 400 megabyte depending on model size and and how we design it and and when we do inference we start with that state so this is a very it's not even it's not the same thing as Laura but it's another form of lightweight fine tuning that can be done at near inference time essentially and and those those are

50. 23:41

    techniques that we are exploring further to to actually push this paradigms uh respectively so so the state in this case is what moves from one token to the next one right yes and yeah fact you're saying that the initial state is also now a model parameter exactly y and it's a very small parameter in the large scheme of the billions of parameter but it's very important on the model and it's very effective to train it yeah yeah and I like how

51. 24:13

    uh it also potentially opens the whole area of uh investigating maybe again that's what that's what you meant uh right but you can you can run multiple proms get multiple States and potentially find a way of mixing them yes in some way yeah yeah okay we are still string the surface of understanding how our states work uh every time we we barely understand how one of our version State Works um that was two versions

52. 24:43

    ago okay so so I guess one other interesting question uh for uh is so so this this is U uh this is effectively a nonprofit effort under Linux Foundation right or uh or that's not quite the case say again so so this is so the the uh the project is done under Linux Foundation right correct so the rwk project is under the Linux Foundation uh and and and that means all all the call

53. 25:13

    RW KB code IP and all that is uh is under the Linux foundation so so so this provide assurance that our commitment to the open source side of things like even if let's say let's just say hypothetically me or other key members go completely Rog and close source and stuff like that L Foundation holds the IP for for what has been released today and they will continue maintaining they'll be able to continue maintaining the project and we did that because um rather we we actually had lengthy

54. 25:45

    discussion in like do we come up with complicated uh bo structures open source structure and it's like like like what happened to other ai ai groups and we're like why don't we just use this very oldfashioned try and tested uh Linux or aach Foundation kind of structures and it's like yeah why don't we do that and then we went ahead with it because is is reliable in insurance so so I guess an interesting question here is you also besides uh RW KV you run a for-profit company right so

55. 26:18

    how do you balance your time commitments uh between the two so at the end of day right uh so RW KB's key mission is to make AI accessible to everyone regardless of language Nation or compute and and and these these open source models are the path forward um so when when when we decide to create a corporate entity to support uh the open source movement um we look into the history of Open Source uh what worked well essentially and one

56. 26:49

    the one of the key key shining example of uh so there there different structures like there there's like the the support structure so you provide support services but I think one of the key big shining example for us was the database and database had had multiple Decades of trying to sort this out and and and like for example like mongod DB Atlas elastic um and a few others they

57. 27:13

    have an open call model where essentially any of us can run the database um but running databases is not so easy as well like this setup work maintenance on it so a lot of us end end up just running the the platform instead and so the commercial arm benefits from hosting the databases be in the public Cloud private cloud or on premise but the database itself is open and and as long as we are able to like balance that respectively uh we will be able to help

58. 27:48

    essentially monetize these models and so so we decide when we when we build the commercial arm which is TR Fus AI um we we decide to to move in that direction where where we provide inference uh and manage inference uh support for any open source model not just RW KB uh even though started with RW KB we actually expanded to even non RW KV models and and that essentially becomes our Revenue line uh to uh for that help us sustain the open source business and it also

59. 28:20

    keeps incentive in line so for us at the end of day right the real customer question is are they going to use open AI or one the open models be Lama or us and and as long as we keep supporting the open source Community um to make better open models then they are more likely to actually do their inference on fedus instead of let's say open a or entropic cloud or Google and and that in

60. 28:45

    sense gives us additional Revenue which then feeds back into US supporting the open source community so it g it makes it makes the company align to to to that common goal as well and and that's how we balance it in between uh if you want to say like the more specific how's my time balance oh gosh I wish I got I I I know how I even juggle my time on this but yeah I would say like um 30 to 40%

61. 29:09

    is like me just juggling all the research initiatives or getting the gpus for the for the open source research and the rest of my time is just juggling managing the the company to to just scale the inference and get more customers yeah yeah sounds exciting uh um so yeah for for everybody who wants to find out more the website is RW k.com U and uh uh yeah it already has all the diagrams for the future version and a lot of details on the previous ones uh so yeah check it out and thanks

62. 29:46

    lot Dre for uh for joining us today yeah and yeah have a good day ahead