NEAR AI Office Hours #10 with Schelling AI

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   hey everyone great to see you again at our uh AI office hours today we have a special guest IM mud I'm super excited I'll let dead introduce himself uh and kind of uh we can also talk about what he's working on now but also there's some uh reasonably recent breaking news we can discuss and then we'll go from there take it away thank you having on ear team um s was founder of stability

2. 0:31

   AI um best known for stable diffusion um but we also gave away 20 million supercomput hours and grants for the whole ecosystem uh with organizations like the Luther lion and others and end up with 300 million open downloads so hopefully you know contributed to all of this going on uh now at shelling AI uh doing the decentralized version of that stability was meant to be a dow Dows

3. 0:54

   originally but the technology wasn't quite there but I think it is now which is super exciting awesome yeah I'm excited to dive in into kind of what uh like how shelling approach is it uh how people will be able to participate but uh I just saw that llama 3400b was released and so uh given you're a big proponent of Open Source and open weights I'm curious to hear kind of your reaction and uh as well as just generally how you see kind of uh

4. 1:27

   this evolution of Open Source space um given given this you know comparatively large model now is available uh what what can we expect yeah I think um you know the numbers have just come out and it looks like we have the uh first model that basically competes against GPT 40 and uh Claude Sonic which actually has been one of the best models I've used recently it's definitely better than 95% of

5. 1:57

   graduate programs I to say that um I think you know there have been these charts of closed models going up like that and then open models not quite catching up but if you look at llama 405b the 405 billion parameter model so it's like 800 gigabyt of Weights fp16 16b Precision uh lower if you've got sper Precision it was trained on 16,000 h100s for three months you know that's like probably double triple the comput of 4 originally and So Meta has 350,000

6. 2:33

   h100 they got 35,000 now just for this that you've just had Elon announc his 100,000 h100s for kind of grow 3 and um it's good that we have access to this technology because these models are like you know graduates in a way like they go through High School University and then you can take them and tune them if they're open or if you have planing capabilities and then make them your own and I think you know mistal llama these

7. 3:00

   other models have created a really nice base from which we can go but there is still for now the need to have these monstrous supercomputers again some of the fastest in the world um to give an example of how fast it moves we stability in summer of 2022 um had a 4, a100 cluster so that's the generation before this one it was basically like 10th on the world's Global list of supercomputers this new meta one with all the optim izations that have come through now is the of 64,000 not

8. 3:34

   4,000 and so again it's been so quite a leap up that we've seen in the computer requirements and uh more and then the final thing I'd like to say as well is if you look at the Llama 3.1 like documentation just had a quick skin before we came in theyve basically Incorporated many of the Innovations from the open source ecosystem on model tuning or merging together weights and more like looking at what the ecosystem

9. 3:57

   is doing that's something like what we did at stability everyone built these fine tunes on top of the base stable diffusion level and then we combined them together and it learned from all these varians to create something closer to human preferences which again I think something very exciting when we start talking about decentralized figure to AI how something can learn from our collective experience but then be personal yeah that's a very good point

10. 4:22

    that I think as they had an opportunity to see what people are doing with the L 3 they they could actually bring it back and pick um whatever whatever was kind of interesting into this I'm actually curious yeah because indeed the this is a size and and you know potentially quality although obviously we yet to see this at like at practical use cases right which is which is the actual the actual test but um yeah I'm curious how

11. 4:58

    do you see now competition of you know like open AI is charging pretty pretty hefty fee over how much it cost to do run entrance on just kind of B compute uh and obviously you know now everybody will start optimizing like they already released you know8 bit which is potentially fits on a single machine with a GPU so you can have like a more like a cheaper inference prage uh in many cases so I'm curious from economical perspectives because on my

12. 5:32

    side I mean that was always an interesting like the point I made in beginning of 23 is like I think open source will catch up uh but it's even more is like open source is the way because you get all this improvements and insurance and then you're not limited by kind of your setup uh and now you cannot incorporate those improvements yourself into uh if you have a close Source solution yeah I think that you know um

13. 6:03

    when you were doing the groundbreaking research that led to all of this you were doing research artifacts right and so you were scaling the compute on the attention or you need paper you were now looking at these and then we had gpt3 and other things come out but they were really research artifacts they were not optimized you know they weren't in the Real Environment with people trying out different ways to optimize it with stable diffusion we saw it go from 30 seconds an image to 300 images a second

14. 6:27

    once the community got a hold of it like stable video which is still probably the best open source video model out there despite being released last year was 40 gabes of vram the day we released it within a week the community got it down to eight you know which massively increased kind of the availability here um I think we saw something really fascinating last week when open AI released GPT 40 mini which was a 99% decrease in cost over

15. 6:57

    the original chat GPT model per token 99% we haven't really seen that anywhere and if you kind of look at what's inferred they've also taken some of the stuff from open because again we haven't seen that much actually amazing originality from the open Ai and others of the world everyone in AI actually builds upon the shoulders of others because when you look at the actual code basis for running these models it's like

16. 7:20

    a few thousand lines of code at most and you replicate Google's pal model I think Lucid Reigns one the most productive AI corders in Source he did it in 200 lines of pie torch you know um so it's not about that it's more about understanding data it's about understanding architecture and bringing all this together but it does seem to be a bit of a race towards zero on cost and what is the cost of doing a specific

17. 7:43

    unit of intelligence from writing a piece of code to doing these other things the optimizations that are occurring now means that we've seen this competition occur really really fast especially because they're so well funded and especially you have this open source equival so deep seek actually a Chinese lab is the one that maybe kicked this all off so deep seek was a hedge fund they got

18. 8:07

    10, A1 100s and said let's go and build models and deep seek actually has some of the best AI models in the world that were the first to compete against GT4 they went um from the average price was $15 for million tokens to 30 cents because they created this new mlae architecture for DBC V2 and then the whole Chinese market did that now the whole Western Market is doing that but with the open source stuff you don't know how fast you will get with things like grock taking it and adapting it and

19. 8:38

    you don't know how cheap you'll get with people as you said taking the quantization down so when the weights come out and they're trained at 16bit Precision you'll have 800 gigabytes for llama for 100 bat but you know even with um 8 times 80 gigabytes um 640 gbes on a single h00 node of 8 chips that's still too much but when you quantize it down to eight bits then you get 400 gigabytes then you quantize itally down to four or two you have Precision loss it gets even cheaper

20. 9:09

    where's that key benefit we'll know because everyone in the world's going to now try it we'll try it how cheap how can we engine exactly and people will now okay well can we quantize and then fine tune for our problem so it's actually better yeah can we take one of the really interesting things now is that they change their license on lava so you can use it for synthetic data creation and distillation as much as you want so you can use the big giant model to teach a

21. 9:39

    smaller model you can distill down you know and you can create synthetic data so they use the big model to train and improve llama 8B and 70b as well and so now we can do that as well with the first commercially license model apart from Deep seek that you can do that with which again is super exciting so I think we will just continue going down I think we've got probably two more orders of magnitude through data optimization model optimization and Chip speed up as we get

22. 10:10

    the Next Generation chips coming over the next two years so I think the cost is going to go from $15 per million tokens to 15 cents to 0.01 cents and a million tokens is like seven what is it probably 700,000 words or something like yeah it's like a two three books like size sizable books yeah for 0.01 cents without even thinking about the edge stuff that's insane right yeah intelligence too cheap to mag up yeah I mean that that would be

23. 10:50

    exciting to see I mean even now on on like 70b right you can get 7080 cents per million tokens which which is already pretty cheap and uh um I think I think the the for me the interesting always question was right now we're unrolling obviously Chain of Thought and so we kind of we're spending what output tokens are like we're measuring kind of compute but with tokens that that's really what it is yeah um which you know it makes sense to actually measure compute like how much

24. 11:23

    compute it took uh not how many tokens it spit out and then you know being able to let model think without spitting out tokens constantly and using that more as a in general representation but that's a probably an interesting topic um there's actually very no no those are a very interesting paper where they were doing kind of Chain of Thought unroll uh kind of in Laden space um on

25. 11:53

    every token and so and then teach it to see where when should it do it or not right so like if it isy if it's easy for itself to predict the next token it wouldn't do it and if it's something hard it would go drop into kind of in space do a Chain of Thought uh which wouldn't come out as output token and then use that to predict output token and like I think some form of that uh

26. 12:19

    machinery would be really interesting yeah I think that you know it was interesting you said tokens versus compute and then moving away from just these models guessing the next word having more internal planning basically mapping more parts of the brain it's almost like type two like it's a type one thinking instinctive thinking is what we have right now these models are instinctive animals you know they're like not really reasoning but if we can

27. 12:42

    start unifying those together with agents and I think this is where distributed Ledger technology becomes so interesting with kind of output formatting and other things then you can actually have intelligence because one of the really fascinating things I think here is that llama 400b is too big consumer right but it's not as big as you can make it you can make it 10 times as big right there's less people that can run

28. 13:07

    it a lot of these models have be optimized for the size of one node like Google when they released their Gemma 27b model said well it works on a single chip but you think about intelligence if I've got a piece of code that's very important that I want to write then my Compu energy budget for that might be $10,000 yeah not .1 cents and so if the cost goes down another order of manit even where it is now at like 30 cents per million tokens or lower then I could

29. 13:38

    throw a lot of tokens a lot of energy a lot of comput at that but how am I going to chain these models together and this is where things where the mixture of expert stuff where you have different experts for different things lamb and I had a very interesting thing here they had a million experts and then they took the accuracy of rag from 60% to 95% the retriev which is great you had to get AI kind of releasing their mixture of

30. 14:00

    Agents thing where it routed to the appropriate best models and out performed all the models there you'll have this type of intelligent routing intelligent agents with memory solving complicated problems and we've seen that with open Devon Devon itself and others where once you take away the compute restriction from an energy output and the need for it to be like zero shots just instinctive you can do things that

31. 14:23

    just amazing andless is just like when I started my career as a software developer I was writing assembler 23 years ago right didn't have GitHub and all these things these kids today have now it's like blocks we still got to build those building blocks and that's I think amazing exciting so maybe that's a good segue into shelling kind of how how you see the world I mean you mentioned stability

32. 14:46

    right was originally kind of thought of as a dow of dows um I don't know if if you're comfortable talking about any anything there but um would be would be interesting in into kind of how shelling sees the world and how uh how kind of you see engaging with research and kind of doing some of the things that didn't work in stability yeah so stability we have these communities like a Luther that we incubated and spun out and others we had hundreds of thousands of developers and

33. 15:18

    you know we' had things from like language models to image video everything because I was building a multimodal AI team and I was like from these communities hiring giving grants we can get amazing people that are passionate to build great models and that's what we did the business side wasn't so good but we still you know broke a lot of Records there and we're heading towards profitability I was looking earlier this year at stability and I was like I built a multimodal AI team I'm now going to have to go and

34. 15:41

    raise $500 million to compete against these other guys like Co here today just raised 500 million 30 million of Revenue it was available but then there was no sustainable model and the reason we moved away from these like Dow Dows concept so I was something like if we air drop to the community it's going to mess up the community you have a mixture of these researchers conventional non- conventional and raccoons you know it's going to be messed up the way comp the

35. 16:07

    this is all going I saw that there was basically two gaps number one is every regulated industry I believe will require open data and open models because of the poisoning super agent type stuff you get so let's coordinate and build those for every nation for every sector all the cancer knowledge in the world available to everyone with a model that count from Human doctors is want diagnosis and empathy that should be available open education model for all the kids so we're starting with education Healthcare we making some

36. 16:38

    announcements shortly um and also building models for each Nation one after the other after the other so we b a Japanese version of stable diffusion that reflects the local culture you know we built Spanish models and Argentinian ones and kind of other things every nation needs self- Sovereign AI every person needs help one every community so the idea of shelling is basically to be the highest marginal dollar for

37. 17:02

    compute solving the demand side of deepend and then use that directly and by giving it to research Partners to build open infrastructure in research that the world needs and so you know Bitcoin uses 160 ter hours of energy all of the data centers in the world use 350 which is insane a lot of Bitcoin miners are moving towards AI compute now but they don't know how to use it we will use that all to build stuff that makes people's lives better and then you can take these data sets like we've got

38. 17:34

    Lon for images and pile for language and we released the pile code the code on 3dxl and models and layers and you can just combine them fully open source to whatever you want to build the building blocks of these systems so you know multimodal AI team that's just building building building and then having this uh store of value digital currency backed by people work system most likely we're still ironing that out that can then be used to fund the building of

39. 18:03

    vital infrastructure for everyone not trying to do you know the complicated stuff keeping it simple um long way to go but you know started out well and I think at the very least the worst is that cancer autism dementia Alzheimer's all these conditions we will have a fully open source high quality data set and model that app human doctors on empathy and diagnosis for all of them in

40. 18:27

    the next 6 to 12 months that will make a difference to the world you know how how do you valuate empathy what what's a bench there are empathy kind of diagnosis ones whereby it's um almost like an ELO score that they do on the cohorts so Google had that with output of Med pom and then others are Dom like Stanford as well these tests the reality is human doctors have a low empathetic level um because they're overworked

41. 18:57

    right because current medical education system has to take everyone down to just numbers that's why everyone gets the same parac or coding prescription despite being individualized whereas the future that we want you need to said user own self- Sovereign AI you can bring intelligence to the edge how much knowledge do you really need for 90% of medical queries or educational queries enough to put on the

42. 19:21

    MacBook honestly within a few years and then you should be able to call to bigger models when you need to have a double check right I think that's the way that things will evolve but as I noted I believe that every regulated industry will need to have fully open data sets this where L is fantastic but it doesn't do enough we don't know what the data inside is we don't know if it's in program said that Boer was awesome El kind of

43. 19:45

    sucks you know yeah that that was a joke conspiracy that I was uh joke like to explain to people why why open data matters is because yeah like we have no idea if it's now sadly manipulating everyone but this will be important because like just like not your keys not your crypto get to the point of not your models not your mind we will rely more and more on this technology and so again

44. 20:10

    from my perspective all the regulated Industries education healthare finance will all require fully open models so let's go and build those right and then you can call the open weight models for certain things and the close Source models for other things because close Source models will always outperform open ones so you can always bring private data to them you know and optimize it but it doesn't mean that the

45. 20:33

    base level has to be that different like you know we have GPT 4 than everything else now it's caught up and there will always be a bit differentiation but they're good enough now for just f everything yeah and and I think I mean the benefit is they cannot bring your private data to their closed Source model which you can do with open source model because you can run it yourself and I think that that's where the big quality differentiation when we're

46. 20:56

    talking about assistance where we're talking about medical Stuff Etc because it just has all your information available uh to understand and that's you know that's why I call it shelling I like shelling point you know the focal point in game theoretic terms like Bitcoin is by deliberately creating amazing high quality National data sets vectoral data sets and models it makes a

47. 21:21

    lot of the work though you and others are doing a lot easier because you can have pre-install models with really understandable standardized lat spaces like why stable diffusion 1.5 is still so widely used because the base model is so understandable that the fine tune things that we put on top are compatible right and so we want to have these graduates that have got to high school level where we all know what that is but then they've gone to local

48. 21:45

    University and they've gone to vocational school and those are predictable that makes a lower requirement for fully homomorphic encryption or tees or other things like that to achieve the types of goals that we want which is that you should be able to have control of your data but then also opt to in and get rewarded for it you know we want to be able to contribute to a whole and then benefit from the whole as again I think that this is the big unlock this Edge

49. 22:08

    intelligence that we've got yeah exactly maybe uh diving in into the data because I think that that is a very important question and I mean stability indeed open sourced a bunch of data sets uh which then became like which I mean for example for lion right you could just download image from the URLs um how do you see uh kind of this conflict right now around hey this is public data this is like copyrighted data but it's public this is

50. 22:43

    copyrighted data but on the platform that you write off all your permissions to right like Facebook Instagram um yeah is did they train on public data which is public to them and and Etc like kind of I'm curious how you see that space because and I I'll yeah I guess like after that we can talk about synthetic data in how to evaluate synthetic data for the same things like slipper agents Etc so you know it's very fascinating

51. 23:17

    like I say this is someone who's been sued for $1.6 trillion dollar by gy and that's company because they were like use Getty Images we didn't use gey image data you know but this is a big topic because the previous Norms of you know you have robots. text and then you can't scrape are now being challenged and one of the really interesting thought experiments I like to say is this we're going to have robots right five 10 years

52. 23:40

    there'll be like a hundred million a billion robots they'll be walking around are they going to cover their eyes every time they see a Coca-Cola sign or their ears or they got Katy Barry probably not right that' be stupid like again there's this public consumption of stuff and then there's the whole questions around train training but we're already seeing regulatory Arbitrage so for example Japanese and Singaporean law is very permissive there whereas other countries

53. 24:06

    are much less permissive most of them don't even know what to do because this is a complicated topic from my point of view I was like there's a Commons and it's okay to fund research and build open data on anything that's publicly available we opt out all the robots.

54. 24:21

    text then we opted out anyone that wanted opt out so we had a billion images opted out of stable diffusion 3 from um the spawn SL have I be knowed pwned trained kind of team gave us that data and I think the ultimate what we need to get through though is no scrapes at all because scrapes are stupid like what are we really trying to build like it' be like we take our kids and we let

55. 24:44

    them watch YouTube all day and then we put them into the workforce no wonder our models come out slightly insane right well and that was again a reason for shelling it was like I want to create high quality data sets for every nation called topic subject area so you don't need to use sprakes anymore you know and then the actual amount of data you need I think will be lower because right now it's like we take crap quality

56. 25:08

    stakes and we use giant compute to tenderize it yeah to like get out the information that you need yeah get out the information but then like you know I was having this discussion with like yosio and a bunch of others are like you know want to to train on biological weapon data don't teach it biological weapon data you know let's have on the input versus the output side right the

57. 25:31

    downstream versus Upstream things I think you know eventually we're going to move to a point where any AI That's used for decision making that's important will need to have ingredients just like the food that you eat you are want you me and again I think that we need to as a society move away from these big data scrapes to that but in the meantime the law can't keep up in time so we see things like meta made an announcement

58. 25:56

    llama 3.1 is the last llama model that will be available in Europe llama 4 will not be downloadable in Europe it will not be usable in Europe like you can download it VI VPN or whatever right because according to European Union law meta is not allowed to train on Instagram posts yeah yeah you had the same thing at Google where Google couldn't use YouTube data and other data the longest time because of certain other weird kind of things and so that's just think just

59. 26:26

    absolutely fascinating kind of view because Mom just look like okay fine don't have intelligence you know your yeah I mean it's an interesting question because yeah like gdpr says you should be able to remove all the data from anywhere it's used and so if the model saw the data that was then requested to jar you need to go and modify the model to unse see it how's it going to work right but the example of

60. 26:50

    like a robot you know needs to close its eyes or like have a bunch of filters pretty much on on the things it's SE is is a good example I mean from my perspective I think yeah we'll just see the copyright needs to completely be revised because indeed we we're we're in the world of generative generative information versus published information and so that in itself changes how things

61. 27:16

    are attributed well I mean this this transformation thing like if you've got a lot of the copyright law is actually at the point of download but then if I tokenize a data set and I download the to I tokenize data set so if I tokenize it somewhere that's regulatory compliant and then I download and use it for somewhere that you can't download the the tokenized data set is transformation literally in code right of the original

62. 27:41

    so you can have people doing these weird regulatory Arbitrage things unless you kind of come to some solution but the reality is that unless you're someone like we were at stability we were trying to do things right people just don't talk about their data sets anymore yeah I mean nobody nobody nobody even just tries to like open source them yeah they're just like here's parameters you know deal with it so like you know when we were doing our

63. 28:07

    image generation people were like why aren't you good as Xyz and I was like because they downloaded 10,000 movie torret and trained on that you know like ultimately people will just do it because there's no way to trace it back necessarily particularly for proprietary models and this becomes very interesting from a geopolitical perspective because China doesn't give a damn about IP and copyright and so they actually care a lot about

64. 28:31

    what goes into the model they do so for the for the media models it was interesting because the media models coming out of China Now train on everything Hollywood everything because you think they're going to win Disney going to go into a Chinese company now you know yeah but they care about it for language models which is very different but again if you're a Chinese company or others you'll be able to use all of Siad for example right and

65. 28:55

    these other high quality data sets whereas again it's getting very very very strange very very very quickly yeah so so maybe that's a good segue into and and we talked about that 400b can be a good source of synthetic data um do you see actually I think and job py tweeted about it which is like hey we see consistent Improvement of like smaller models all the time being more and more intelligent and that's because we started yeah just by throwing random data at them and now

66. 29:28

    like we slowly reorganizing data with this bigger models that are more like more intelligent right now we can reorganize information in such a way that they actually generate the synthetic data to train smaller models and so I think that the interesting question is W will there be a step change there where as you create those data sets right and and what you said as well the like I mean right now it's used for

67. 29:58

    mostly for distillation right but and some fine tuning but given you can have this larger models kind of running for long for a while and creating synthetic data sets that are not anymore predict the next token problem but actually maybe a little bit more sophisticated uh training setup and then training on that uh will actually just be better and it removes a lot of you know I mean it it leaves a lot of

68. 30:28

    questions because like those models were trained on all of this data but at least now you have all of the training data it's like it's synthetically generated it potentially can be open source because it is fully transformed data and now everybody can go investigate what are the challenges with the data and potentially fix them yeah I mean like I think this is a big like shift now with this model coming out so now you have deep seek and llama as fully synthetic

69. 30:58

    dates are reliable at gp4 plus levels 4 levels right um so Microsoft had their paper textbooks are all you need that led to the Fire series of models that are entirely synthetic but actually the interesting thing is that the five models are not entirely synthetic they're majority synthetic and they inject a bit of web data at the end for the generalization so it's mostly synthetic and the mixture of those two are performance because right now when

70. 31:23

    we talk about high quality data we're not sure what we actually mean like is New York Times high quality when we did the original stable LM model like stable LM eventually got to MRA level before we left but the first one was the one of the worst language models ever released we broke the scaling law because of Reddit data and we didn't think it'd be that bad but it was bad you know what is the type of data that goes into these models that optimizes it there's a huge

71. 31:48

    amount of work still to be done there where you can do these Dynamic ablation and this is what where dpin could be very fascinating right like just putting it on ,000 gpus and saying optimize for this so we had a paper from our RL lab carp AI called Q quality and diversity through AI feedback actually it's about the quality of the data it's about the diversity of the data taking a Wikipedia article looking it from a hundred

72. 32:13

    different ways taking a piece of code and having various abstractions on top of that code and then rewriting the code to see how it kind of runs and having again feedback functions this is where I think you're going to see an explosion in synthetic data understanding research more to figure out you know just like a mama bird feeding a baby bird big models feeding the little models which typ of because mentally I look at these and like I said it's like we use this

73. 32:37

    pressure cooker soid to take large amounts of crap and we squeeze out a bit of gold right but the internals of these models are a mess and we see this with things like models have multiple layers we can literally remove entire layers of the model and they still perform the same it's like makes no sense right it teaches you that the models actually even though they're dense have a lot of craft in

74. 33:01

    them and so when you look at things like the rectified flow work on the stable diffusion 3 paper it's actually very elegant the way that the internal structure of stable diffusion 3 was kind of set up I think just one one of the best papers I see very proud of that team what's the internal structure of these models and what's the optimized end form of that versus where they are now my take is that we probably don't need more unique

75. 33:26

    data than you or I have seen through to University to have an amazing high quality model the trillions of tokens we use right now are the intermediate stat but again moving synthetic and understanding synthetic the ingredients and how to pull them together will be the next big unlock towards these smaller models that are hypers specialized because if you have a medical model does it need to know about civil

76. 33:49

    engineering probably not right if You' got a coding model does it need to know about CFA with some of the code bases yes but you know well yeah I mean modern education says you need to you must know about Kafka right uh to be a human being well like cap toone basis we've all seen a lot so I think that again this is where the Deep ins side becomes very interesting the user own data site becomes very interesting

77. 34:19

    because the big super computers took crap data and compressed it to good now we have models that can take data and improve it and look at it from various different perspectives with long context that work on the edge so we can access the exop flops of compute that we've got there for dynamic model optimization through LS you know you look at the work of people like answer Ai and others

78. 34:44

    getting it working on just individual chips or EXO Labs or just TI of in here with training together multiple MacBooks there's a lot of Compu we can use to build high quality data sets for generalized everyone and of course once you've got a high quality data set once like the pile was very high quality for language now you've got um data comp being much better you know we funded

79. 35:08

    that one you don't need to repeat it over and over again just like you don't need to redo a syllabus you know and so that's what becomes very very very interesting I think because that will lead to this next order of magnitude or two more optimization in these models yeah yeah I think I mean both having as you mentioned in the beginning some mixture of Agents which underneath use a more specialized models um and those models are

80. 35:39

    fine-tuned or or even trained from distilled uh SL synthetically generated specialized data sets specifically for problems that they will be solving like that kind of pipeline potentially can be a lot more optimal per token per per per unit of compute kind of getting out intelligence like and then being able to route to the right places I think there's probably two more things here one of which is continuous pre-training of these models you know or fine tuning as it learns about your preferences and you've

81. 36:13

    got like the Lura on top of the base model or the series of those and the other thing is the long context window work that's happening now like you know with Google using likely ring attention or something like that now scale to millions of tokens like llama 3 out the box is 28,000 tokens so 100,000 words of input proft yeah and now that we have the state space models and things like Samba that are combining these we have almost linear scaling which is insane

82. 36:39

    you know and so when we kind of look at that you know some of the KV cach work um on persisting prompts Etc you're like you can combine these in really interesting ways because then you'll have a constantly updating base model and the ability to expand out context length in a predictable way so you can take the Bas Laura move it to the cloud maybe and then use cloud compute for

83. 37:03

    giant context window stuff to look at a whole bunch of different data but with your personalized knowledge design patterns like that become super interesting so you see some kind of combination of like local like generation but uh offloading long context on onto a cloud yes that's an example of uh modality right because if you know again I'd be happy to spend $100 on an inference for a long context 400 billion parameter model but you know

84. 37:37

    having the ability to import my local Laura to that to look at my entire code base in one give me a report and send it back you know and so this is why you know I think you will have I have a task to do and then your agent will figure out the budgeting of compute of that task can I do it with my existing things or do I need to go and provision something else and access that and that's why I thought you know with

85. 38:01

    shelling if we can standardize that for all the open stuff fully open that make life easier you know and then you'll still have lava and a few these other things around that because you want to have things that are relatively deterministic utilizing large context window and function shaping and other things like that you can have a high level of determinism and outputs versus short prompts with high

86. 38:26

    temperature and language models like it'll just say different things all the time yeah that's just like a graduate though right instru give a graduate that matter you don't explain them what what is the tools available to them right then they yeah gonna run around yeah so do you do you think we're going to get to a postgraduate level oh I mean we're already there right it's just that they're not as depressed as postgrads yeah

87. 38:58

    that's a key concern yeah I mean like again it feels that you're only just the start of this like at stability we built a system called comfy UI for images whereby rather than having a prompt it takes every hyper parameter and model and then embeds that entire chain of node logic in the end image so if I shared that with you it reconstructed the entire Flo because I was like base models and control of models and

88. 39:22

    composition of models we're still really at a compositional stage in llms and language and knowledge whereas knowledge should be dynamic structured trees almost they get adjusted based on your prior your axium and others again those design patterns haven't emerged yet because we're only two years into this literally we're two years into this explosion again you figured this out way before obviously but that pick up yeah

89. 39:48

    it it took a while to scale it to to a level where yeah yeah it started to be intelligent yeah um yeah yeah I'm actually I mean my question been kind of as I was looking at a lot of like some recent research is that I I haven't seen anyone thinking uh more how to build better representations and um how to train them more effectively right because I mean you mentioned like right now the representations are mess the you know you can throw away like some of the

90. 40:21

    weights and it's going to be pretty much the same uh and at the same time like even when you find tune right it's actually forgetting some of the things um and you kind of need to like balance it by upsampling you know other data to like make sure it doesn't forget it and all the stuff right so there's clearly a lack of um kind of understanding is one but also just in general like how do we more effectively optimize the uh this models in general yeah I mean to be

91. 40:56

    honest model training is D until recently just been more of an art than a science right like even because you always compute constrained when you were doing your oblation studies and even at the largest places because it was always like well a bit more scale is good so how much do I allocate towards moral optimization and we had love and we had things like that but there's still so much more to do which is insane but even

92. 41:19

    now like the whole of Wikipedia is 26 GB compressed and the new llama 8B is let's say gabes or 4 gabes Q4 but it can write most of Wikipedia like we have figured out how to represent knowledge and context incredibly small the fact it knows anything is amazing and again as you said maybe we move Beyond tokens to just direct input maybe it needs multimodal tokens or Jack like joint representations to kind of move the next step but the thing now is that there's

93. 41:53

    all the smartest people are now looking at generative AI so the talent quality is increasing the capital is crazy um and there's a real business incentive to get this done now you know so I think that will hopefully continue to see open advances here new systems and ways of bringing these models together and again a move away from zero shot to really intelligent thinking of what is knowledge how does that come together what are the representations and internal structures of these and how do

94. 42:21

    we chain them together yeah totally agree I think that that would be big Next Step although some some people believe it's just you know thr more compute uh maybe put some RL on it and it's going to work well then Google will be the winner right like with the v6s and v7s and the interconnect fabric that they've got between the data centers you'll be seeing multi-million TPU run soon like it's insane again but I don't think that'll be the case I think probably what's going to happen is the

95. 42:54

    compute will this is the last big take up in for performance above the level of the data underly and then it will be about optimization combination swarming collec cols but who knows I don't know you don't know yeah we don't know that that's one thing that's like the science is not uh predictable right like you everybody will try something and then we'll see what works and then we're all going to swarm on that to continue expending it so it's not very scientific

96. 43:27

    actually one of the things that really astonishes me is that you know again like sometimes you just got to do it like who is building open- Source education and Healthcare models you know we built the best Radiology model with um Stanford check agent other like nobody but obviously we should build that now because the outperforms and all the benchmarks and we should make it open source it's available to everyone

97. 43:49

    like we don't need the AI to get better now to have individualized education for every child at $10 per year for kid you don't need it so so that no person without simal or dementia doesn't have access to all the knowledge there through retrieval augmentation in their language how do you deal with your family to what the latest therapies are and so I think you'll see the class of research artifacts continuing and again we'll try to optimize but the really

98. 44:14

    amazing thing like I said now is the technology is there that we can proliferate real world impact to people which something that's super exciting to yeah I think that's people kind of in in this race forget that Yeah we actually have something that's like if you if you show any anyone two years ago that like hey we have this people would be like holy this is amazing right and like now we're like hey you know it's not really good at this things like you know when

99. 44:43

    is gb5 coming right everybody's like I'm just waiting for gb5 it's like why are you waiting for gb5 like you have the most amazing technology of of it's like I don't want to say the fusion like two or something came out and people like I can't do fingers and I was like can you draw a finger like fingers are hard I think part of this as well is you know the flow um thing like if I use super Maven or Claude Sonet coding is really fun that's what I kind

100. 45:15

     of found but Bo it wasn't fun with the previous models and so I think again you get to this level of performance where okay we take it for granted but we can do much more now like I can try different things otherwise they're way down my list you create a to-do list for the house that's doesn't suck that's one of the projects I'm doing now you know it's like um and I think that just unlocks a lot of mental capacity and the

101. 45:39

     ability to flow and then our organizations can improve as well like the one thing I advise every organization to have is just an AI tinkerer that just tries everything and thinks about how can we implement this to make life better you know yeah I think there there will be a lot of interesting how organization will transform I actually I I personally more excited about new or newer organizations

102. 46:02

     at least which are adopting this and becoming more productive than existing like larger Enterprises introducing and making thems even though they have more data they presumably should be in a better position but like the processes and kind of the I would say like overlap personal and corporate structure I would I would expect that to be challenging well I think that you know organizations make you sad because of

103. 46:34

     the lack of information flow right and context so organ you know when we start our organization again this is one to Lear stability I kept very close to the research team so nobody left last year you know and they offered ridiculous amounts of money because it was fun and they were enabled and they could do their job very well and the information flow was pretty good but then when we scaled the corporate side of things this something that happens again we started okrs and jira and other things

104. 46:59

     responsibility agency and accountability was split if you kind of look at how organization go now these model of context machines and their F shot Learners right so again if we design properly then every meeting we have now is automatically transcribed and added to our Global Collective knowledge base and we've got a chap PL that helps out with everything makes life so much easier right like we scal the research on Discord and I'm just glad that

105. 47:23

     everyone's had their ad all through that because it makes you so much more a D where Discord with memory right like well Discord in general is for DH for ADHD so I don't think it makes it better I think it makes it worse like dis memory but the the thing is you said organizations that AI first should out compete organizations that are AI enabled or bringing AI in because the data structures and the organizational structures and bureaucracy will stop them from out competing so I think

106. 47:55

     humans plus a i CS will always before but are you AI first as an organization are you embracing it or are you just where can we use AI to improve because you look at it a very different way you know it's like are we trying to build agents that are autonomous or we trying to make people more agentic most organizations don't want to increase the agency of their employees yeah you know and so that's why organizations that are

107. 48:20

     all about the agency of their employees I think will outperform that's true that's a good point and that's I mean in general that's what in web 3 we've been trying to do is how do we make individually and and Community wise more agency you know through decentralization through empowering to own your own assets and kind of make your own decisions in Dallas like it's all toward that world and it is I would say like there's a pretty big

108. 48:52

     differentiation when you try to even build an organization in three and if you still use the principles of like traditional business building like you always have this challenge where like the space requires agency and so if you're trying to build okrs and hierarchy Etc you're kind of failing in there I mean Al this this self- Sovereign kind of aspect it's the contributor kind of thing and what we're trying to do is build protocols for various things right and there was this

109. 49:23

     lack of intelligence within web Tre because when you had a big shelling Point like Bitcoin people who get around that they understand it you know even though that's changed from peer-to-peer money to store value or whatever there is again I know what I need to do I need to mine it I need to buy it I need to profitize it Etc when you got to down to the Dows and the governance side we didn't have anything to check the

110. 49:46

     outputs so you had like let's put 200 million into gaming or something like that like 4.7 not referring to anything specific nothing specific 4.7 billion dollars has gone into crypto X gaming and I can tell you as a former game investor they're terrible it's like again this m allocation because we don't have the checks and balances but now with these context machines we do like

111. 50:10

     if you've got a constitution everyone agrees to you still have differing objective functions and local Maxima and so let's face it governance been premature for all these things in the decentralization of governments solid teams that are building with the Define road map will outperform but then we can now build the tools that enable governance and constitutional checking I mean indeed anthropics big Advance was

112. 50:35

     constitutional AI you know AI that matches these things and now with a large context windows we can say given these decisions and these are the good ones and others does this Dow decision make sense for the context of what we want to build give independent view I think we'll see this from more and more things whereby before you have medical decisions by AI every medical decision will be checked by AI

113. 50:57

     before you have Dow decisions made by Ai and maybe you know in Wyoming you'll have the first fly autonomous AI Dow you know yeah yeah um before that you will have every down decision should be checked by an AI you know every single piece of code and smart contract should be checked right now definitely exactly yeah especially for yeah I was I was actually testing this out and yeah

114. 51:23

     even GPD 4 was pretty good at finding water ability is in code so if if you prompt it right and like point it at you know uh to the reviewer function thing we should have a collective knowledge on all these vulnerabilities and that can literally fit in one context window right if you offered that audit as a service that would be fantastic of course we should be checking all our code because how

115. 51:47

     much money has been lost with these vulnerabilities and CS you know but then of course we should be checking all our decisions because the context of each individual member come from so many different places right and so many ds we all know that in suboptimal ways yeah I think I mean one of the ideas that people discussing on on our governance is creating an AI delegate before indeed AI decision maker so you you you can actually have people delegating to it uh and then it it's

116. 52:19

     voting on behalf of its de kind of delegates and so the interesting thing is it kind of creates a economic Force where if it's making good decisions you know more people delegating to it so over time it can become you know have more and more voice um which is I think is an interesting way to experiment and kind of ramp it up yeah I'd say that's that's really interesting because I said like there's a optimization equation there and like I said the other thing is you have a set of agreed principles at here around the

117. 52:52

     government stuff everything all that should be uploaded to a large context window model along with what the community have used has been great decisions that have been made along with local context that's relevant and you should be asked again and again like please look at this from all the different perspectives and scenarios that the community wants how does this work I think that Technologies at the point now will actually have really interesting feedback this comes down to you know

118. 53:16

     citizen assembly type things whereby if you take groups of citizens and you actually inform them about a topic over a few days and cut them off from the internet they actually make really interesting informed decisions you right and now we have the ability to almost scale that mhm yeah I mean that's kind of what the government like US government has all this agencies it's supposed to like evaluate the budget the EPA all this they just don't very work

119. 53:40

     efficiently right so here yeah it's because the lack of context shifting an inadequate local Maxima right like the whole thing is the sociopaths Rec TP the D principal you know the office and then because everyone's writing reports all the time and ring reports is hard like it all becomes about just perpetuating yourself and Gathering more more political power and that's why you have like a railway line in California of 100 meters or something that cost 10 billion yeah five five

120. 54:13

     kilometers yeah in the middle in the middle of California yeah it was a test um but again like I think there's this really fascinating thing right now where I was like we can really build the open infrastructure of the future because you said a organization that Embraces this technology for healthc care for Education will outperform those that don't like literally one toone personalized education for kids using a choronic level model implemented

121. 54:43

     intelligently for a kid in Nigeria will probably lead to that kid outperforming a kid in Detroit or Washington or something like that you know which amazing exciting I think important as well personal tutor that knows exactly how you learn it knows how to present information so are you visual are you auditory you know have you got dyslexia and that will be there like we've got the cost on tablets down to $7 per year per kid in Africa now with solar power and starlink and so I think

122. 55:15

     that'll be a huge unlock and I think for web 3 as well 76% of web 3 developers are outside of the US so this is a blessing and a curse the curse is that you have again these local Maxim and people don't understand each other's context and you get these tribes and everything like that there's also a lot of potential if we can use a technology to break down the barriers and augment and we get people to agree why are we building near why are we building

123. 55:38

     shelling why are we doing these things right I just need a little bit of support yeah well I mean that's why we're doing this is to educate people and kind of and get get people to understand why we doing this and uh how it all fits together and it's not just you know yet another web 3 yet another hype there's a lot going on to actually enable and keep keep it user focused keep it Community focused uh efforts yeah with this intelligence

124. 56:13

     toing yeah super exciting time give my The Tipping Point now again like it was just really fun using sonit you know and now llama 3.14 5p at the same level it was really fun using stable diffusion you know I think a lot of people come into we3 because they want that high degree of agency and control and self- sovereignty and now if we can get this technology to that group and increase

125. 56:40

     that group people want agency they want control you know they want to be able to believe they can create and so let's unlock that for as many people as possible I agree well um we coming to the end so I will wrap this up thanks much for coming I think it was amazing conversation I totally agree that you know as as more of those tools are empowering agency I think we'll see amazing results across the world uh so thanks and tune in next

126. 57:15

     week for more Che everyone for