NEAR AI Office Hours #9 Fireworks AI

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:07

   okay hello hello we are live uh really excited to kick off this episode today we have a dimma uh CTO of fireworks uh I'll allow him to introduce himself and his background uh but um the background that I have is that we actually were on the same team in ICPC back in the day where I was sadly way worse than dimma uh but uh yeah I I'll let him introduce himself yeah thania uh yeah super super

2. 0:39

   excited to be here and yeah as AIA mentioned we kind of actually all three of us go way back through programming competitions so I I mean I've been on the same team with IIA in the same school and with Alex I think we were competing against each other on many many different programming competitions so that's been fun um but yeah I mean uh I I am now ounder of NC fireworks and we're going to I guess talk a lot about Ai and fireworks and what we do uh but yeah before that I was uh many years

3. 1:14

   working kind of AI infrastructure platform space over at meta uh particular for the past five years for the last five years there I was uh one of the four core maintainers of figh so kind of we went from early humble beginning and research to adoption everywhere I know were in early days of Tor flow a little bit so sorry but yeah won sorry but no sorry sorry I'm sorry ja is great but uh yeah so that's been a fun

4. 1:48

   journey of kind of bringing AI to to to the masses and productionizing of deep learning kind of across meta products and across the industry and kind of with fireworks it's kind of Contin of that uh but now with the generative AI L language models wave awesome yeah so I guess I mean we talk a lot about here about open source and user owned Ai and so the kind of exciting part obviously been uh using

5. 2:20

   like we've been using fireworks for open source models and it's been uh really really good experience so do you want to introduce kind of what fireworks offers uh how you guys view the space overall and what you're trying to achieve yeah of course so I mean uh so generally fireworks is like a platform for building this Genera I with foundational models like large language models or generation Etc

6. 2:49

   uh targeted at developers or businesses uh uh starting with just inference of of Open Source models but also all the differences including customization of deployments customization of models funion and various building blocks which you might need to actually build an application and end and as you touched it we like my personal background is a lot with kind of Open Source and open

7. 3:14

   science through like P related things and like what we really believe that like open weights models like open source models will I mean they already have pretty big footprint uh in the industry in kind of the usage but that that footprint will even grow over time and I can expand why why why we believe so um so our bet overall is basically to be the best platform for building this generative AI with open weights models to allow de developers and businesses to

8. 3:44

   customize and tailor for their needs and build endtoend applications uh which they can control and they can take from small s small developer phas all the way to like production scaling and uh kind of Enterprise scale but specifically on specifically an open source part I think there is this kind of big big kind of two two ways of philosophy right like kind of the way how the space is

9. 4:09

   developing on one side there are Frontier Frontier models build by Frontier Labs you know G you know GPT from openi clo from entropic some other models from various some some Chinese Labs Etc like all of those models uh I mean they're great right but they are fundamentally aiming on you know building the the biggest and greatest and most awesome model for absolutely all use cases kind of trying to aim at AI which is great for a lot of use cases especially like some of the consumer applications kind

10. 4:42

    of an arbitrary like CH GPT conversation but then if you look at kind of a lot of applications and businesses and concrete stuff several problems emerge uh I mean one you actually don't necessarily need the biggest and mightiest model which is good at all use cases because majority these cases are more narrow or more specialized uh you know if you're trying to build some customer support assistant

11. 5:06

    you actually explicitly don't want to that that system to chat with you about all the possible Pokemon types or write you poems Etc you actually want to wanted to stay on message uh so you don't want all those capabilities you actually want to have control over quality in relatively narrow domain uh and as a consequence you actually often want to have lower cost and maybe like higher performance characteristics which are achievable with tailoring for this

12. 5:35

    particular case maybe using smaller models or serving it in a different way which is specialized for this use case so yeah all all these factors kind of P push you know use cases in business more towards being controlling customization for their needs which fundamentally means yeah like uh starting starting with open weights so you have control of what you're running and have ability as you as you as youc graduates to be able

13. 6:01

    to go and poke into it and customize it and uh and build an it further so that's that's one motivation another motivation which we hear a lot from our customers is just being being in control of your own destiny so you all seen you know uh thre on Twitter like oh my CH GPT got Dumber or got smarter or whatever uh people are really kind of worried about that so they actually want to know which

14. 6:25

    you know version of the model they run and they they want to be in control not kind of some external API provider even if it's that even if that's great so all this reasons like control and ability to customize and optimizing cost and performance characteristics and improving qual quality in narrow domain kind of factors for which push a lot of these cases towards open weight models so the pattern which we often see is that people start with CLA or gp4 to

15. 6:51

    kind of prototype and pro that application is working and as it's getting more scale or needs to specialize it needs to like uh focus on costs they kind of reach out for for open models uh which is great and now you can ask like hey why do you need actually fireworks because you can also go and like run PL cppp on your 4090 Etc which is great for uh kind of Enthusiast use case but as you as you probably guys know from building infrastructure whether it's for a for crypto like infrastructure is is hard specifically

16. 7:24

    for uh for AI like gpus and accelerators generally tend to break can be be sitting even just bitting them is not fun uh models change all the time so you want to optimize those models for performance uh you want to make sure that you they tailored to your use case if something breaks somebody's on cold even just this basic layer you know people run into the problems with that so that's something which we can provide

17. 7:48

    and as you kind of go further there is most customizations needed on performance side of sense maybe have real time voice application where latency is like super important for you maybe but bu process PDFs uh and ser is really important or if you are actually trying to build like assistant application C you should don't just need a model inference you also I mean you also want to probably have some ground

18. 8:12

    in like GRE you want to have some maybe interface with external stuff so we can talk about those kind of application components further on so our goal is to basically take care of the all this like hard but fundamentally boring stuff for application developer like stuff which is which should just work and uh like allow uh allow customers to actually build this Ai and like focus on quality

19. 8:36

    and enabling their application while we take care of all this you know underlying platform okay that's a short short pitch yeah short short yeah yeah as a customer of fireworks uh I can I can assure that it's extremely fast uh yeah maybe maybe one of the definitely one of the two fastest we know but as we all know one of them might not be future proof without naming names starts with G ends with Q starts with G ends with Q yes

20. 9:16

    but rhymes with Elon musk's model right yes yeah um I mean yeah we kind of pride ourselves to be being the fastest or one of the fastest uh as you say uh in those applications um I mean again not gettings like I would just as high level the space for like L like for Gen models inference and specifically for like llms is pretty kind of multi-dimensional and complicated um I mean fundamental reason for that is that those those models are AO regressive so they actually don't map

21. 9:54

    that that well to GPS accelerators and there are a lot of tricks need to do to make them run make them run fast enough so short of like some fundamental algorithmic advances there is a lot of work to do on performance optimization side which is I guess good and bad news uh but um the factors which kind of impact your performance right there is obviously all the various Hardware choices and we can go and Bash in

22. 10:18

    different about different like as6 and gpus and Etc that's pretty dynamic in fun space uh but also just even from applic like application use case perspective there's like a lot of factors which um which kind of impact performance whether you have long prompt or short prompt what you know which load level you care about etc etc so you you were you were mentioning some other of our competitors and yeah they're doing great job in some use like some worklow use cases for example if you have like super short

23. 10:51

    prompt and short generation you can like respond super fast what we see for example is that majority of production in these cases actually do like grounding as through regular like big system prompts so actually have like super long prompts uh which sometimes might be repeatable Etc and yeah in those cases we are often like multiple times faster which what you would get from from competitors again like optimizing all the levels of the stack from K kernels all the way to how

24. 11:17

    Services deployed uh just just a tailor for like workload characteristics and generally be faster and yeah in the b c i mean we kind of we've been doing a lot of uh like our founding team kind of comes from Google and meta and we've been doing a lot of AI platform optimizations for many years myself included so we kind of bring this experience to uh custom build and customize our own uh stack so we basically wrote Our Own inference engine from the ground up uh so we are fully kind of in control and we are able to T

25. 11:49

    customize it po models use cases so so I have a couple of uh uh interesting questions going into the the weeds of uh of some of the offerings uh so uh we we happened to uh to spend a lot of time couple weeks ago uh playing with the different instantiations of llama and llama is notorious for having a relatively short context uh compared to say mistel uh but mistal is not as good right so so like

26. 12:27

    you're choosing between quality and the context L and quality uh like in some cases the quality of my is just not sufficient right so you sort of forced into shorter context right and however uh you you guys do have a long context l so so I guess one interesting question is uh generally like if you look at the models that fire fireworks has it has a long context Lama it has a function fire functions right how how do you generally

27. 12:57

    approach creating those custom modelss where do you get data is it like synthetic data are you annotating that data is it is it some open source data sets that that are available for such use cases yeah uh so yeah I mean okay so let me answer the uh the long context part and then maybe go to uh to kind of how we train some of the models which we ourselves fun so non context part we actually post published a blog post maybe like last months kind of just

28. 13:27

    evaluating a lot of long context models uh I mean the the set part is that there actually not that I mean like there are not that many like open weights models right now which are like really good at long context and let me clarify what I mean by that is I think like long context is generally Super O overloaded and it's pretty hard to measure uh you've seen probably a lot of these charts of needle and hastech floting R and Twitter uh which I mean they do like very basic stuff right they

29. 13:58

    take like I don't know they take a book uh they take like a l long book stick it in the context and put like some absolutely random sentence in the middle like I know how how many lollipops does the unicorn have or something like this and ask a question in then it's like oh how many lollipops like the sentence was like I know unicorns have seven lollipops or something and they ask this

30. 14:21

    question at the very end and kind of if the model can retrieve it then they say it's great which is actually like pretty simple test right because you know if you have attention Asia can tell you right from early days right like this unicorn probably never appears in that book anyway so it's super easy to like match find that sentence and just pull out the right in for the model but this is not usually what you want right like

31. 14:46

    you want instruction instruction Following over the long context window you want kind of some reasoning abilities Etc and the more kind of sophisticated you make the test the faster the models kind of fall off from there so even for you know like GPT and clouds like when once you get start pushing kind of past like 64k context window or like 3K window window it gets a little a little bit icky I think one

32. 15:10

    good I mean there was a good paper since there were like ruler Benchmark they had like a bunch of tasks which are more complicated but basically come to the come to the point of if there are multiple you know if instead of asking model to just like retrieve a single sentence uh I try to come up as a benchmark which requires understanding broader text or at least have like a bits and pieces of the same story splattered around a big text right so

33. 15:36

    instead of asking just about unicorns you know they it inserts sentences somewhere like oh Mary went to the kitchen and then like somewhere later picking up from that story or like she took something from the stove or whatever and try to Splatter that that story in and ask about kind of multi-step reasoning and then like a lot of models drop off from quality uh yeah so we like evaluated a lot of Open Source models and yeah sadly

34. 15:59

    like yeah the the quality actually drops off even for many models which do claim to be kind of super good and long context they in on more complicated Tas they often fall off uh we try to do our our own kind of uh little bit of funion for that but yeah frankly it's really hard to get super good results something which you actually want to want to like for a bus production usage precisely

35. 16:24

    because yeah it's it it requires a lot of there are two two problems like one you actually need like high quality training data for long Contex like which incentivizes model to Long context because a lot of kind of documents and paragraphs are pretty short by themselves second problem if you're trying to fune something like llama uh then they like llama instruct actually had very good like supervis f data set

36. 16:49

    for instruction following which from the paper is like 10 million tokens in like super high quality curated so it's actually very easy to screw that up and basically make model for like by doing something else undo instructions yeah undo instruction following kind of on that part so that's uh so that's pretty pretty tricky so yeah while we have no we have variety of offerings and you like we should try various models I mean

37. 17:12

    some of the E models actually do pretty like I think and like now D actually doing I say like from from available open source model actually doing pretty well in like long context I hoping that I mean the rumor has it like yeah llama eventually will also release officially tuned Lan context window I think for actual usage like that that is kind of required in the meantime yeah you're welcome to approximate this but your

38. 17:38

    mileage will very dependent on the try various different models and platform but your mileage very dependent on the task if it's simpler it might work if it's harder task might dis great but your second question it would be nice to have that high quality F instruction fine tuning so you can fine tune longer yeah uh I mean we trying to build that but it's yeah partially it's it's really

39. 18:01

    hard to get to exactly that quality what you are getting from you know from front models I would say if you guys want to collaborate on that that that would be pretty awesome uh uh second part which I can I mean where we had I guess more success so far was like function colon F so you mentioned fire function so we actually spent M kind of multiple months it's like our second iteration of of the model which we just released like in June fire function V2 but the

40. 18:32

    idea is basically to kind of produce the best model as a kind of Base building blocks for kind of compound applications centered around function calling for example for like chat Bots which can do different actions Etc so for folks who are like I mean are not familiar with like function calling I mean originally was popularized by same way open AI but the idea is like you train the model to

41. 18:57

    kind of like translate user requests into uh some API calls or like some tool invocations which it's it's aware of kind of execute them in the background so ex to the background get the result of the function call back to the user back to the model and model can decide to do further function calls or actually summarize the result and return it back to the user this is very useful for kind

42. 19:22

    of just like routing request or kind of any kind of support agents uh type of stuff because you effectively like translate in between structured world of like apis DBS and uh kind of interactive with Software System and natural language which users want to chat uh so for example if you if you have like some guess hello world example for that is hey can you can you please plot me the stock prices of I know top um uh top

43. 19:53

    Cloud providers right the model needs to figure out first what's what are the top Cloud providers maybe it know it it needs to go call some API to uh as the latest stock price and maybe go call some API to plot the resulting graph so we like as an example example case of May bit it's booking some flights or you know retrieving some information so whatever uh but yeah that's that's generally function Callin and situation is kind of similar with uh loan Contex

44. 20:19

    as you mentioned there are a lot of uh if you go to hugen phase there's ton of models F in for function Colin they all claim to do great but there but devil is in details a lot of what people try to find tune for is like single function call which is basically just structur data extraction I have a query I want to convert to Jason that's often not like we actually started with that but what we heard a lot from customers like that's not actually what they want for

45. 20:43

    many cases but many case what they want is I said like kind of like customer support type of use case where you want a little bit of general chattiness and help right he still wanted to say like hey if somebody says hey hey how are you like still respond to it normally not spit out a bunch of Json like user request hello status okay or whatever so you still want to preserve a bunch of

46. 21:07

    chattiness but you want to have this like robust intent detection and conversion to like structured uh backend requests when needed and finding this balance is actually pretty like actually pretty tricky so that's a lot of what our focus with f hunon was how to balance like General chattiness and kind of intent detection when actually there needs to be some request made on the

47. 21:30

    back end satisfy the user user question uh second is like doing multi-stage like kind of parallel like nested function calls so if you need to do a chain of actions how to how to do that well and yeah like from our rations I think we're doing better than lot majority of like kind of open open weights models and kind of on again all benchmarks are conditional but like on kind of mix of public benchmarks people use it's doing on par with like GPT 40 uh for like simple simple conversations so that's

48. 22:04

    been like a super fun journey to kind of how to do this fun Etc and by the way the model is obviously available on the platform uh running really fast but they actually like it it's open weight so the weights are available on uh on higen face as well so you are welcome to run it elsewhere even though you shouldn't uh but to to your question Alex about like how to how to how to train like

49. 22:28

    what was the cooking the cooking process it's it's based on llama on llama instruct so we kind of have this like Baseline chattiness already built in so yeah as always the secret is in you know cating good good F tuning data sets so it's a combination of uh kind of synthetically generated data with other models so you kind come you basically write a bunch of code to generate synthetic scenarios with multiple steps and multiple uh kind of multiple of actions

50. 22:59

    multiple types of user requests function call in It's relatively simpler because it's basically if you can Source representative set of like possible functions to call just like open API specs Etc you can ask like big models to kind of generate you plausible scenarios of how to call this functions and then generate you kind of possible requests which could have generated this and kind of come off

51. 23:22

    with this very structured uh kind of multi multi- turn user interactions which do Translate into some into some API calls which the model needs to needs to do uh and uh then there is like a tricky tricky balance to kind of see where it starts over fitting to just call in the functions and generate enough data which you can actually generate from the model itself uh generate enough data on just conversational side so it doesn't kind of forget just broad instruction following and if there is like retrieval

52. 23:53

    of based knowledge of the world the model is still capable of like answering for that so that's kind of broadly speaking kind of yeah so it's like synthetic mostly synthetic pipeline from this kind of two sources but actually like getting it right requires a lot of iteration so as you as you guys probably doing yourself because I know you f some of the models too yeah you you kind of like created this set and you look at

53. 24:18

    like what's there uh what the distribution s you try to train you you play with the model you do else you see what which scenarios Works worse you try to augment the your your data F plan with with those scenarios you make sure that like it didn't break other Generations along the way so even just like looking at your data set all the time is very helpful and yeah you kind of like rinse and repeat until you get to the good quality I mean the benefit

54. 24:48

    of actually doing fine tuning is that yeah you can do it on a pretty high frequency and because this is not a a training run where you have to like Yol and and Hope uh you can actually kind of do it iteratively and improve quality along the way without worrying like oh fire up all my gpus for three months and ended up with some garbage is fire function sorry is it the Laura fire functions uh so we tried both like Laura and Laura actually for this case like

55. 25:21

    quality is about the same so yeah it's like Laura but we serve it like was merged and unmerged but it's like pretty pretty high rank you had a question I interrupted you right no no I just said yolo and prey yeah YOLO and prey well that's that's a game for pre-training yeah po training f is a little bit more controllable yeah I guess maybe uh changing topic a little bit given I think our audience is interested in uh

56. 26:01

    more combination of where this approach is you know open source in general and uh as well crypto decentralization blockchains intersect um maybe Switching gears a little bit on how do you think like as you know this is evolving right let's say we're g to you know we potentially have a llama 400b finishing training right any day soon and then let's say they decide not to open source it um right I'm curious kind of given you know

57. 26:39

    you you're coming from a different perspective like what what do you think is kind of future for open weight and ideally actual open source models um and and do you see the opportunity where kind of the approach we're taking right like how do how can we coordinate a more a decentralized community of people to come together and no this there's a lot of technical challenges there's a lot of kind of uh Financial like questions as

58. 27:11

    well but um do you think there is an opportunity for uh having like reasonably competitive solution get assuming you know a a benefactor and your previous employer decides to not open source the Next Generation I mean hopefully they will and like this is proba for next year but I mean I I I mean I I don't know for sure of course because I'm not affiliate anymore but from all the rumors I heard that yeah it's it's imminent and happening next week uh I'm

59. 27:43

    I'm just saying like let's imagine it like not to right I first of all I really hope that yeah that incentives in the long run such that like open rights Community will continue to strive and kind of meta will continue to be uh the kind one of the lead one of the leaders there and I mean frankly so far it seems that all all the pointers are to the fact that it will not change which is great uh I think generally there's I think there is like

60. 28:12

    enough incentives for people to like to open source broadly right especially for like I mean companies for meta like meta who like don't have direct Pro like Direct business benefit from you know keeping the model closed and building products that I they going to they have user base they're going to build products with it anyway right but there there's like kind of incendia from from open source like just from General Advan in science to kind of commoditizing the space broadly uh

61. 28:45

    to kind of future proofing uh and kind of like like red teaming if you will from the community which I think worked pretty well so far uh you know people build like or just people building bunch of research and kind of building bunch of experimentation on top kind of fostering Community then benefiting from this community you know I've seen this firsthand from like working on P like why the heck to open source p in put

62. 29:09

    resources into it uh and especially just like purely on open source side but I think that benefited D that benefited meta a lot by bringing back like all the Innovation or just like making it like standard which everybody uses so you can use libraries buil by other people you can hire people who already you know kind of P Etc so it's just like future proof the technology because people take

63. 29:32

    it in so many different directions before maybe you explore those directions uh I think the same is true for like foundational models and that's the reason why uh you know meta has been open sourcing uh I mean it's it's actually pretty good be like we saw very capable models coming from some of the Chinese Labs as well for the past year so uh but I guess like I mean so I think first of all I don't think the space will will disappear uh on yeah to your to your crypto part I

64. 30:03

    mean I would I would I would be very curious like I me you guys are expert this like is there a way to like tie it up this financial incentive somehow in like is there a way to you know open source something but bring back some royalties so like some kind of financial incentives to the model provider frankly I haven't been I haven't seen like very viable kind of use case for that so far

65. 30:28

    because yeah this models are big you know you you will not put you will not put the whole model on chain and there are still inen so just like copy the weight and run it on your on your own so that's that's kind of tricky uh for really big models uh there might be some maybe some way to structure this you know we I mean we also work with some like model providers in like partnership uh form I for example like Ser

66. 30:53

    stabilities stabil Fusion three so if you call their API actually it actually goes goes to all servers uh kind of behind the hood and yeah maybe there is like a setup of like doing it licensing wise you know if it's really big models people don't really going to run it uh locally on their computers because it will not it will not fit for example if for 100 B for like synthetic data generation or whatever maybe there is like some way where you can craft the

67. 31:24

    license and incentive such a way that you like work with a bunch of kind of posted providers which offer it as an API and there is some like R share deal or whatever uh and thaten devizes kind of model Builders uh while still kind of like having open weights for like uh fcking around and debugging but not like industrial use whatever and if it's big enough then probably people not going to

68. 31:48

    do it industrial use locally anyway maybe there is like some way to craft it license wise I don't know yeah frankly I don't know how to like put it in code or like put it on chain so it's like programmatically enforceable but maybe some combination of like technology and careful license crafting might work out I see I definitely see like people some people experimenting with this with like

69. 32:11

    interesting licenses where you need to buy some membership well stability was the first one right yeah stability is wor one yeah I I don't know how it will work out in the in the long run that's specifically on the on the model side uh the part where I think it's much more plausible is like either like curating specialized data sets or kind of cating fine tunes and like model customizations right if you

70. 32:35

    you know if you believe this world of uh like many small to media media models like tuned for different use cases yeah maybe there is some like Marketplace Dynamics to build here right I and if I have a use case for I mean basically if I have a if I a business and I have like use case for particular narrow domain yeah maybe I can have like some compet competition you know keggle style like some kind of way to like put this bid out and you have various people

71. 33:08

    try to try to come up with like a data set of f tune for this use case and depending depending on how much that model gets used uh get back some you know fin some compensation Financial incentives to those those people so you can kind of extrapolate it almost to the way of hey do I everything time I'm using particular subset of the data set somehow through some model like whoever builds the data set gets compensated U I mean I know there are many folks kind of on intersection of a crypto Community

72. 33:39

    try to try to explore the space so maybe something something there Ian we we haven't invested that much in this in this particular space more focusing on like foundational infraspace but that's I mean that I find kind of more plausible on yeah on training purely like pre-training uh yeah given how Capital intensive it is it's like unclear how like how you can Crow Crow fund it kind of explicitly effectively because yeah you basically need to concentrate a lot of

73. 34:11

    capital and buy those accelerators or gpus or and put them in you know put them in one Data Center and run for a while to do pre-training so that that becomes that become potentially harder I know what what are your thoughts on on this because I'm sure you guys are think thinking about this a lot I think I think I think in terms of training for any foreseeable future it will all be happening in the big centralized clusters right so uh I think Deep Mind has some interesting

74. 34:41

    papers uh I don't know how to pronounce them second one came out yeah Loco there's a second one came out as well pretty much bringing back yeah it's like uh you can you can paralyze over experts right so each expert can be on a different set of machines and like they don't need to be as synced and then you can have a pretty much parameter server approach where you have shared parameter

75. 35:06

    servers that kind of everybody sends updates and then you fetch uh new parameters which is kind of how pre- tensive flow uh training was happening and so if if you cook it right with all the right parameters right you can actually get something uh it still requires you know reasonable connectivity and and liveness of servers but indeed has way lower requirements than the current setup where everything like wired up in the one script infin band yeah on

76. 35:38

    infinite band yeah uh so yeah so there's I mean I think there's an interesting and then the other thing is if you assume that a lot of the a lot more work is actually going to start happening on distilling models and like creating synthetic data and you know doing Chain of Thought on rolls uh and like a lot more training going into that I think there can be an opportunity using a more distributed compute to do that even if you're like exact training updates are

77. 36:08

    happening on the smaller cluster yeah um Again by smaller cluster it's still like few thousand you know h100s not but at least you don't need like a 20,000 uh cluster yeah what we call a small cluster was something that five years ago was the biggest cluster yeah I mean it was like GP gpk2 trained on like basically one like gp2 was train on one machine with eight A1 100s yeah yeah like the gx1 or original so it probably was B 100s right

78. 36:41

    given how long ago I mean I trained first Transformer on CPU so I yeah doing doing translation so it's all it's all relative yeah but yeah I definitely agree that like kind of want to I mean it's more plausible for like data curation or specialize in models maybe like synthetic data generation because it's more paralyzable the actual training is harder but yeah if a lot of focus actually goes you know a lot a lot of cooking happens in post training

79. 37:12

    these days right which is not as computer intensive so and as you said like curating data for post training is like more expensive even compute wise than than actual training run so that that becomes more plausible and paralyzable and how to set up incentives for that uh is interesting but at least it's more plausible because like yeah uh I mean like straight straightforward trying to paralyze just the train like SGD updates itself I feel like a you know Federated learning space kind of

80. 37:42

    tried this a lot in like Edge uh Edge setting and frankly it had very very like mild success so far just just just because it's hard to like yeah it's hard to paralyze core algorithm I I don't think actually the problem was I don't think was because it was hard it's because nobody needed it that that's honestly like you can do it all on your data on your in your data center and experiment a ton or you like deploys this multi-tenant setup right which you have no logging and no ability to debug

81. 38:19

    and like hope it works right that's kind of like why would you do that to yourself right yeah yeah and then you try yeah I motivation was like Regulators but then explain well exactly I mean exp sending pii is bad but sending gradients of pii is fine I think that was challenging too but yeah yeah yeah so I mean we'll see this is this is actually the interesting question that I was thinking about because I was asked

82. 38:48

    uh by few people around in indeed Pi let's say like let's say you are you know one of those big companies with a big chat app right right or or whatever right and you already actually capturing a lot of the data from people using the app can you train on that because if you train on that and all that pii data actually you know goes into the weights and then you use like and then people

83. 39:12

    use it and be able to query it out that's a very bad situation right so and like you know Google traditionally clean does a lot of cleaning for pii from Google search queries because people keep putting like addresses and phone numbers and all kinds of stuff into search bar um even before it hits like kind of uh uh storage but the I mean obviously like with more sophisticated queries like it

84. 39:36

    it it becomes more complicated but but this is a good question can can we actually detect pii based on like the like like L likelihood pretty much uh of the of the model and being able to filter it out or even better predicted inside the gradient update and uh kind of uh in some way uh avoid it and this is like in general the topic I've been I don't know kind of thinking about is how do we

85. 40:07

    improve kind of the update function because obviously the the current gradient update is it's either you know like go in that direction but a little bit right uh or like well we over overstep let's go back right that's kind of a like it it doesn't have the Nuance of hey like there's some facts here that we should remember and uh you know the rest is kind of just you know General like let let's let's know the syntax type thing or hey you know you're

86. 40:38

    reading an article and there at the end there's an author like you will not be able to prct to AU from the article right that's highly unlikely so stuff like this like there's a lot of nuance around that so how do we how do we make it more nuanced how we uh making updates to parameters yeah uh I mean yeah that I mean that's exciting I think yeah it's still probably a little bit far out from figuring out some something people barely understand how SGD works so far

87. 41:10

    right there's exciting if only it was like 40 years or 50 years of it yeah exactly well not at the scale I think we kind of understand how to train ptron pretty well yeah um yeah so that that's fun uh but um I mean your question about like pi and like prediction I think again like I think more and more what people are doing is more like prepressing like synthetic I mean you can call it generational like basically cleaning with llms themselves right because you can ask I mean yeah you can train you can train model to remove Pi

88. 41:45

    you can train model to like rewrite stuff and like rewriting and cleaning works really well so yeah your example about I mean about auor maybe you can like restructure it so it's more useful uh I mean the the rumor has it that like big Labs basically you know for for Next Generation models like even that is like to lot of synthetic data even for pretraining but certainly for like kind of post training like fine tuning that's I mean that the RO like spend like spend

89. 42:11

    spending time and like cleaning that those data sets uh and making them like kind of robust and easier to predict and making the models kind of remember facts and Style by by just like providing very high high quality data definitely is like the trend of the last uh you know last year or two or so and I mean that's that's kind of more plausible and theable than trying trying to cook something with GD updates now get me

90. 42:37

    wrong I would I would love to understand how how SGD so far SGD every time wins right every time you try to add more structure SGD comes in like on that level but you can give the right right input yeah yeah I mean I think uh well the other interesting topic I think to touched a little bit is like Hardware um I guess right now I'm assuming most of the stuff you guys running are on Nvidia there there is indeed alternative Hardware A6 and uh that either exist or

91. 43:23

    or people working on there's you know all of this like transform former kind of hardwired kind of how how are you looking at that market how are you thinking about you know I'm sure you just like whenever it works we'll just use it but uh but I guess like obviously whoever produces them can offer like a inference on top of it thems uh as a cloud so H how how are you kind of thinking about that yeah uh I mean so

92. 43:53

    first of all it's like very very fun and dynamic space right the whole computer architecture was like semi dead like I think like for 2000s or whatever where people were trying to optimize last 2% of Branch prediction and I mean here first with gpus and now with A6 and all the different chips there's explosion of various architectures which which is I just technically find super cool and interesting that's because I haven't

93. 44:17

    been tracking crypto Krypto been playing with hardware for a while now yeah I know I mean I say it's about about parallel there like also wave of like original uh I mean there was a way of like original a64 computer vision circle like 2015 and there are some lessons to learn from this I guess uh yeah so I mean first of all I think you know in the short term gpus gpus are great and versatile right and yeah

94. 44:44

    like Nvidia is great and it works and that's why their stock pric is so crazy uh that's why people keep keep buying this uh I think for inference specifically it's more plausible to do something different right because there's like less flexibility all this as6 usually fail and like oh we're going to go change my model a little bit in in research and nothing works uh in happens a little bit less often so I'm hopeful

95. 45:08

    that it will P out in like few years time frame uh I mean there is also story of MD which we're also looking at and you know MD is one of was also like one of their our investors for like series B which is which is great together with Nvidia and um yeah like I think I I mean AMD got like it's also GPU I mean they got good Hardware to be honest uh their software stack historically has been lagging a lot they come a long way so it's now working but I mean there are

96. 45:42

    gaps and misses but it's it's in the working stage when we started working with like you know MD in PCH land Circ like 2017 2018 that was like very raw to to say the least uh now it's getting better so even on GPU Market it's great to have like plausible competition and like for example for inference for inference domain yeah they have for example more hbm than current generation

97. 46:05

    300 has more hbm than h100 which is great and it's actually good for inference stuff so that in short term that why I'm that's why I'm hopeful hopeful about um I think generally for for startups kind of trying to do as it's like a it's like a crazy trade-off right like you basically you kind of you need it's like a gamble with the life it's a gamble with life yeah and you need to like find this like gold theog zone right because on one end you can overfit a particular architecture I'm sure crypto guys did

98. 46:38

    this right because they were like Bitcoin Bitcoin mining chips which couldn't do anything else um in you know AI space you probably remember all this like 40 startups which were doing CNN inference C like 2015 2016 out of which I guess only two or three survived because by the time they they they taped out a chip which can run vgg really really fast the world moved out to resnet and stuff like that so it's really it's really tricky to like not over fit the particular architecture but also not go very very Broad and very

99. 47:11

    general because if you go very general then you're effectively trying to beat Nvidia in it in its game you're basically trying to build a GPU and yeah Nvidia video MD spend like a lot of years building good gpus it's really unlikely you're going to you're going to do something better so that's that's an interesting kind of trade-off and yeah some startups kind of try to get this get this right I like last one was just making the news from the space is like

100. 47:37

     was like HED I think which is still kind of like you know same tradeoffs as G it has compute it has hbm so it's kind of like generally similar architecture to GPU broadly speaking but way way way less programmable so you have like less flexibility there easier to build more power efficient so yeah uh I mean I would probably give like smaller chance than 50% for for them sued but it's still much greater than zero which is which is awesome and yeah there are like

101. 48:07

     other startups along the same line and then you have this like generation of uh people who like bet on SRAM like crazy I think Gro is one of them um I me like cerebras and I cerebras is kind of in the same category more or less uh but basically they they say like okay we're going to dit hbm this moderately fast but really really big memory which is going to be bet onless Ram which is basically on chip registers more or less uh which is very very fast right so you have like insane uh in

102. 48:38

     insane memory B but it's very limited in space right so I think Gro was saying that like one of their chips has like 200 like 130 megabytes of SRM like memory at all like which is allram which means which is really like a gamble in terms of model um ter like model architectures and model popularity because even if you even if you try to you know serve llama 7v you can do the mess how many chips you need you need to do that just to load the model and serve

103. 49:11

     one request right you need like hundreds of chips to just load it in this R yeah your requests will be like super fast uh but like economics will work out only if you have enormous amount of traffic coming to the single model so in effect you're kind of betting on the future of uh single model serving lots of lots of use cases and and having this like basically half a data center dedicated

104. 49:34

     to serving this model in super paralyzed way with very good interconnect uh which might might very well be like a lot of use cases but the world which we talked about about a lot of specialized models and smaller models Etc like that world actually looks very different Hardware wi than uh a thousand chip holding single model in aam and running it very fast so it's actually like this open weight world world looks looks different with a lot of customization lot of specialized models and that's and that's why you see

105. 50:04

     you know rock having like to three models off right because that's that's the fundamental bottleneck just like the fixed cost is really high so that's that's an interesting design Point uh and I think you know it it rocks in some use cases it's doesn't do so well in other use cases but it's good to have like diversity and kind of balance balance in balance in the community and then there are of course you know some I think startups try to do completely kind

106. 50:29

     of experimental architectures I don't know analog Computing or whatnot like like Optical analog Etc like I think that's that's even maybe we'll have models that only need you know minus one zero and plus one and yeah exactly maybe one 58 bits is all you need and then you exactly and and then you can do something like that I think that like I personally find that extremely exciting

107. 50:54

     but like chances of that succeeding are like because of H this huge gravitational pull of like existing you know Transformer architectures which had some hand in help him to create right and like kind of it works well enough so there's so much hardware and software and optimization kind of knowledge sunk into making it work it's like really hard to pull out of this like a lot of people tried know few years back capsule networks right down there is like mambas and state space models which are like relatively small

108. 51:26

     twist on that uh yeah even those like you know even if you run on gpus you run into like software optimization problems if you get try something more extreme and sparely connected to whatever then you run into Hardware problems and yeah just like effectively train at like foundational model scale with new architecture just to see whether it will them work if you feed trillions and tokens in yeah it just becomes like exponentially harder it's hard enough to train with existing architecture and

109. 51:55

     gpus it's much harder if you're trying to innovate across the multiplayers which is again sad I think from exploration perspective but yeah it's kind of a like if we Stu in local Minima and there's actually like another architecture that you know over a hump but we cannot get over that hump to it's like evolutionary evolutionary Branch right it's like maybe we like all dinosaurs and like

110. 52:19

     being like all all our models are like dinosaurs right and being like warm-blooded and we just need to much better right like it needs to be something you know some big cataclysm happen to to explore the different branch well I hope it I know I'm more of I mean obviously I'm biased here but my belief actually like any any big enough model has like with with some inductive bias has a

111. 52:50

     capability of learning anything right and it's more of a question how we're learning it versus like that that's what I was saying like I think the fact that we don't understand how it's actually learning and kind of uh you know trans transitioning from like capability stages from one stage to another uh like to me that's that's a problem not like hey you know we can go and like you know change the architecture you know and like we pass now the information through this like vector instead of this Vector like I I

112. 53:23

     feel like that that's like not going to change fundamentally our understanding of how this model work which I feel like requires for the next stage of innovation here yeah but I I just wanted to make one comment for like crypto Community because notice how dimma outlined different trade-offs of different solutions and said it's great to have a competition and different people taking different approaches in crypto it would be like no my Approach is the best and takes over all the use cases and everybody else's

113. 54:01

     mind just polite yeah I'm try to be diplomatic here no I I think it's like uh I think generally Community wise it's pretty I mean I mean there there's a first competition but I think there is enough friendliness and cross peration yeah I think there's just the market is growing so quickly that everybody's kind of yeah figures out and and so fast that everybody's figuring out how to all work together um yeah I'm just like I I just

114. 54:26

     came from a crypto conference so I'm you know my brain is still uh hardwired into those uh uh discussions yeah but of course you should use only fireworks for open waste mod because I have a quick technical question uh from from the early discussion uh so uh you mentioned again that uh I guess it's sort of implicit right that fireworks is betting on on a lot of small uh fine tuned

115. 54:58

     llamas uh as a way to compete with the you know like the world in which GPT 5 exists uh I guess a question is so so let's say yeah let's use the example of uh like a support support chat right uh in your experience how many how many samples does it take uh for a customer to make a uh model sort of good at their use case uh and are they like how good is synthetic data for this use case right so like if I'm if I'm like a

116. 55:34

     hypothetical company that wants to have intercom uh B yeah like if I just use Lama 7B to generate you know like 20,000 examples will will I get a decent model for my use case or do I need more examples and do I need to involve humans um I mean it really depends on the use case I would say right um and how much effort you I mean how good are you generating data just number wise I mean

117. 56:04

     yeah like tens of thousands is is fine right I mean the more mer if it's high quality of course but again depends on this uh I mean the way I think about it is like if you can prompt engineer the foundational models into in the particular pattern is like multistep prom Etc like there is you can basically compress this find tun no question right uh something more complicated uh where you can't just prompt engineering s in uh requires more like more effort

118. 56:35

     comparatively because it's like harder for like if model can't you know can't grasp when you tell it like in very boss form to follow some instructions it's harder to translate that knowledge into it's harder to make it work through through the weight updates because it's not just T in the style a little bit uh yeah so then then it requires more focus on maybe like more data and in incorporating incorporating new knowledge or or something like this but

119. 57:00

     yeah it's definitely know plausible and you I mean what you also see is like you usually people start building this know big Mighty model and then yeah they realize what's their product Market fit is and what their bucket of use cases is and They al already have some like real data from users they can train on or like or they can like use as a reference for for generating some data and that's

120. 57:25

     I mean that's where you have like much better idea of like what the typical interactions are and you can kind of compress it into into a smaller into a smaller case and yeah like the more you know restrictive domain this the better right like for example you know if customer maybe customer support it harder but like you have for some like simpler like code completion cases which where like maybe latency is really important but you don't want like

121. 57:49

     perfect accuracy like latency is all you want like yeah that that's like a clear-cut trade over like smaller Model F tune and can like generate a lot of training data because it's just code which you're completing uh from interactions right that's that's like easier to find for example right and we have some of the customers kind of from coding assistance type type of cases which like do this kind of in their products whether it's the cursor Source

122. 58:13

     graph Cod or Etc so there like I guess like really really depends on use case and uh I mean like going forward I think yeah like having more permissively licensed big models so hopefully 400 vs this week like next week uh yeah that you know that that makes Game Inc like incrementally easier which is which is awesome and you guys ready to have 400 b on fireworks right away right yeah we hope we are yeah we see I mean

123. 58:48

     you can just initialize a model with 400 can serve it right yeah later yeah I I guess I'm actually I mean we talked about long long Lama but uh I mean this is just like probably can ask it in our slack Channel later but uh you guys have the like sharded attention that can span multiple machines when you go for like a longer context uh yeah so I mean basic yeah basically if you like I mean if you have

124. 59:19

     uh multiple so I mean yeah you you there are like two ways of like shard in attention basically right you I mean you can do like regular tensor parm like Megatron style which for attention basically means that you split by head right uh across multiple gpus and that actually works fine for inference also in in many cases but there is like more communication between uh I mean more

125. 59:47

     blocking communication both like weights weight SES and and on kind if if your cont is long like on on activations Ki there there another option is to yeah another option is to kind of split by sequence lens I think in training this paper was called like ring attention but it's like training equivalent of that but it's basically like chunking the sequence lens when you do when you do prefill like Chun in Chun in the sequence lens and passing on the like

126. 1:00:18

     kind of pre like KV chunks in parall with like how you compute them uh yeah so we we have all the possible versions of pism implemented yeah it really depends on like you know settings and like what's how big the prills are and how how big your prompts are basically how like what's the model you know is it ET like how proportionally heavy MLP versus attention is compute wise what's like lad level Etc and yeah there's like lots of knobs you can tune you know which where which is what one of s which

127. 1:00:51

     well can like configuring like finding the right configuration for particular case so if you come to us and say like hey I have this like real time coding assistant or I have this like voice support chatbot where I need that as much stup as possible under this super tight latency constraint on this type of work Cloud those are kind of type of knobs which you can do under which we can do under the hood to kind of support

128. 1:01:16

     support you best or you know tun speculative decoding stuff or like some deployment configuration so least goes on and on on and that's kind of goes back to those are like some of the things why I was saying that you know even for open weights models there's a lot of just pure even before you touch into quality uh there's a lot of pure infra kind of setup needed and like that that can make huge difference both user experience- wise because your stuff runs

129. 1:01:46

     faster and cost wise because your stuff runs you know the the same traffic can run on fewer accelerators so you pay less money fundamentally makes sense all right well uh if folks want to learn more about fireworks it's uh fireworks.

130. 1:02:04

     AI uh and uh yeah we will uh I mean thanks dimma for coming and kind of having this great discussion I think I I hope it was interesting for everyone to learn more a little bit from a different side right on folks who are optimizing and kind of serving a bunch of clients across uh various are um so yeah thanks for joining everyone and uh we'll see you next time with more conversations yeah thank you and happy to have near as one of the as one of the

131. 1:02:39

     customers and looking forward to kind of work together on ways to connect maybe crypto and AI ecosystems both technically and cure ways cool awesome