NEAR AI Office Hours #12 with Arize AI and NetMind

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   hand in hand in this Tex driven Wonderland okay hello hello everyone uh great to have you on the new episode of our uh AI office hours uh we have today dad and Henry I'll let them introduce uh in a second um hope you're gonna enjoy the conversation Dad do you want to kind of uh introduce and then Henry yeah sounds good thanks Ilia uh really nice to see you my name is D no uh I'm an AI

2. 0:32

   architect here at arise uh maybe a little bit about myself um I guess uh I was a data scientist for a long time uh was a software engineer before that um I built a lot of you can think classic ml uh nowadays you'll catch me uh really building architecting uh with a lot of our customers here at arise so if you don't know what arise does we're uh kind of AI observability platform so I work

3. 0:56

   very closely with some pretty amazing teams like uh you know Uber Uber's customer Obsession team Spotify DJ booking a lot of the airlines but a lot of folks building in the the Gen space so really nice to to be on the show cool yeah I can uh continue so I'm Henry I'm the C of net mine. so a little bit more about netm so netm is a decentralized uh machine learning platform offering um a variety of AI services and we build on top of a decent centralized and distributed um

4. 1:32

   GPU infrastructure um so I need engineering team at nm. focus on focusing on building infrastructure and AI Services uh prior to netm um I spent um over like uh 10 years in soft engineer industry so I work at Microsoft and coinbase prior to nine uh I'm also excited to be here with near AI for AI discussion awesome well uh great to have you and hopefully we have interesting discussion

5. 2:05

   for us and everyone watching uh maybe to kick it off um I don't know that you you've sent a few topics beforehand so I'm curious if you have like specific questions uh we we can uh zoom in yeah I had some fun topics that I think we might chat about um maybe I'll just bring up one and we can just talk about it but uh I think one like I don't know if you'all been watching the markets

6. 2:30

   recently um but yeah I was curious you know your opinions on and I'm happy to share mine as well um you know AI Roi um and then the current market conditions that we're in when we think about spend versus uh return where are we at in that kind of life cycle I think people want to have a conversation or are we in a bubble or are we not but I'd be curious to hear y'all's opinions first I I have plenty of opinions and I'm not so um but

7. 2:56

   really excited to hear kind of what you guys are thinking in the space a Roi yeah I mean it's definitely a complicated question right because um I mean I I kind of similar to you guys worked uh on machine learning stuff for for a long time where this was not a question because you usually applied machine learning already when you have a business right and you have clear Roi and

8. 3:26

   uh now we have this kind of race to the top of you know who's going to have the biggest cluster who's going to have the biggest model um which is akin to you know building out uh kind of Internet super highways back in the day and some other like infrastructure build outs like railroads is a good example in beginning of 1900s um and so I think on one side AI I do think is a very transformational technology right and kind of what

9. 4:05

   we envision is something that you know Gonna Change how we you know work and live fundamentally but it's unclear what's the timeline of that right similarly how it was unclear you know five years and 10 years ago uh it's it's been very much kind of faster than Skeptics think and and slower than optimists like me would like to and so I think we we're in similar State stage right now where everybody is kind of really building on infrastructure but um you know the Skeptics are like when is the revenue

10. 4:47

    going to come from right it's uh you know like the generative AI is going to always be just mimicking and paring the statistical data and this is going to be the how it how it works and this is not going to you know give us much and then the optimists are like hey you know like we're using it every day this is like super awesome but obviously there's still like some fundamental limitations and so I think that the Cru is somewhere in the middle and so sadly the big companies right now are the ones who can stomach

11. 5:19

    that spent because they actually don't have any anything else to spend productively money on and smaller startups are not and they're being bought out right now right or more like licensed and aqu hire people uh so yeah I think that that's kind of the state we are right now um and and then one more item like I actually I'm more optimistic about a new newer companies that are buildt AI first than adopting kind of AI in existing like

12. 5:58

    bigger Enterprises and that's mostly because like the adoption in Enterprise is extremely hard and and some of the things that AI is really good at is kind of requires re-engineering how things are done and kind of sometimes putting person as quality control versus the doer and so like you kind of need to change the processes and models and it's fine for a new startup who is like thinking that way and it's really hard when you kind of need to like undo everything while you have business

13. 6:31

    running forward right and you have kind of a lot of uh political and kind of personel questions to answer so that's where my other like people expect Enterprise to pay a bunch of money for this stuff and I'm just like it will be some but it will not be as much and and it will become from this new startup just growing faster because they're like so much more productive you know that that's kind of a few points

14. 6:56

    there Henry yeah so that's a really interesting topic and question so uh I do have some ideas for Roi in AI so uh because like I work at UH infrastructure and cloud computing industry for Nots of years right so um it's similar with the cloud computing about like 20 years or 15 years ago like for example AWS right so when the cloud computing at their early stage it's also a really sort of a big investment for the company to spend

15. 7:36

    lots of money building the computing power machine pool data centers and infrastructure Cloud infrastructure and at that time uh people has have no idea how cloud computing can involve and uh how there uh investment can eventually T into uh the event Vue um and uh but they still like spend lots of time uh including uh Hardware software and also engineer and all those kind of stuff but eventually like AWS Azure gcp

16. 8:15

    the win and uh it sort of like um has similar similarities with like AI nowadays so you don't know how AI can involve and can have uh like impact or the potential Revenue in the future or in like two three four five years but like big companies like meta and Microsoft they spend tons of money investigating in AI um so like we know like AI is a really really big thing uh but we still unclear what's the next like um big

17. 8:58

    Point like of like sort of like um revolutional point that could happen and could change the industry can change the world um so uh and also like you know I think the biggest um investment in AI now has like several like can can uh includes like for example GPU power like for example if you want to train a really big model like Nas one uh 3 point one like they require massive GPU clusters and all

18. 9:36

    those GPU are really high brand like H 100 right so this is actually a really really big investment I don't think like a small or mediumsized company or startup can afford this and and also like um you know investment on the scientist and research can also be very expensive like sometimes like you put lot of effort on research it like cannot guarantee you get the um like the outcome that you um are like happy with

19. 10:12

    right so but eventually like if you find something like chat gbt right it will just like explode and you you just like get into next stage um so I would say like because like at like my right we um uh work with lots of small startups like the because they also need GPU computing power right there this this the span of or the cost of GPU is pretty expensive especially for Star small Stars right so

20. 10:45

    they are not um they are more willing to use our GPU instead of using the uh Cloud platform because we offer cheaper much cheaper and easy access uh at much cheaper price and easy access for the GPU um so in their perspective uh they have different stages like initially they want to build their model train their model find T their model and also work on POC MVP so at this stage they don't need to spend lots of money so

21. 11:15

    they just use the uh a small portion of GPU resources in our platform and when they want to scale they can sort of like um decide or to use a large amount of GPU or GPU classes class clusters um so for this one like I think uh the conclusion is like they need to uh Play Safe and eventually sort of like uh to know their product and know their customers and then decide which kind of

22. 11:48

    investment they want to spend in Ai and then eventually to sort of like uh grow their business grow their customer space and get the return so yeah so overall like I think it's uh complicated but uh uh I don't know if there is a model or algorism for II in AI for now but uh um I think like for them like some uh key factors are sort of like um cost savings Revenue growth and also uh decision making depends on the market depends on

23. 12:27

    the um product feedback yeah that's that's my thoughts awesome love the takes here um yeah I think I played Devil's Advocate to Ila's Point um I do think you'll have some super high growth startups Ilia that like they're AI first and they're like the recent YC batches or wherever they're coming from but I do think uh like so I work very heavily in the Enterprise space right observability all that stuff pretty I do think like the startups will move FAS but I do think there's like a

24. 13:00

    distinct Advantage because it's like it's not just AI that gives you a competitive Advantage it's like AI Plus data right um because like at the end of the day I think we all have access to the same AI right it's like GPD 40 if you consider that the best or sonit 3.5 so I do think it is like uh I it will be interesting to see um I I do think there's a lot of parallels between like

25. 13:25

    the the.com boom right it's like some companies took advantage of the internet but you had a lot of winners that came out of kind of nowhere the people who I guess didn't adapt um and I do think bubbles are good I think like with the recent like call the the investor called like I think the with Google and Sundar just like talking about hey we're you're spending a lot of capex into Ai and then

26. 13:50

    their stock price took a nice eight% drop um but I think I agree with his take and I think Zuck said the same thing but I think it is more risky to under invest rather than o like overinvestment you've all the knowledge work is being kind of you know delegated out that's one part of the ROI I think the second part is something a little bit deeper that we haven't really been thinking about was like new knowledge um I've yet to see an AI produce like you know

27. 14:46

    like yeah exactly but could you imagine a world where that's true and it's like we haven't gotten there yet but that's like well so there is actually some of that it's just not in pure generative AI space right like a friend of mine worked on material design machine learning right where they pretty much can score uh material molecules ahead of actually trying to compose them to understand their properties and so like they could filter down to similarly one

28. 15:19

    of the co-authors of attenti you need is doing this for uh kind of drag design where you could actually you know you know predict what the possible kind of impact of specific mRNA and how it will affect different proteins and then you use that like Alpha fold as well so so I think like we have that in specialized Fields right now it's not a generic like you know come up with new stuff and it goes and like does that right you kind of have a specialized models right now

29. 15:51

    but you know if anything we've learned from the past few years of is that like well if you train on a bunch of these things at the same time and you know maybe mix in some of the search uh Machinery uh it can actually start learn like you know cross-learning and finding interesting um examples and so yeah I mean I'm excited about combination of kind of G generative like model and some

30. 16:17

    form of search and then some of form of environment be that real world or simulated where it can actually test its hypothesis um and so the I the the Imo R thing right from Deep Mind is kind of a good example of that yeah totally I was just to say I work with a lot of biotechs and they're definitely in the deep deep ml space on on that stuff but I I think I completely agree with both of your takes is like I think we're

31. 16:42

    super early stage I think bubbles are good honestly I like that capital investment there's a lag time right from capital investment to like actual value it's like the internet um the internet companies didn't like think about like Cloud came from the internet and that was like in the like the 2010 era and then the Doom bubble was 10 years before that uh it's just these effects that you

32. 17:04

    have no idea about um but I've got some maybe other topics if he wanted to to bring those up and as well but yeah I don't know if you had any other ones though yeah no I think I think uh we can we can discuss those cool um I think another fun one was um I was just curious about this one for maybe uh you Ilia and Henry and then uh myself uh what are you making bets on in the future like where are you making investment not capital investment just like maybe energy or what else but like

33. 17:37

    um but what do you think is going to really be a really cool area that is like uniquely your opinion or your take um so what's near betting on what's net mine betting on and I can maybe give a take on what arises betting on do you want to go first hentry yeah that's a great question but uh so like this is also related to AI right so like in AI domain right yeah in the AI domain okay okay cool uh yeah that's a great question so for me

34. 18:09

    um one of the um because like um we are building a decentralized um infrastructure and also we are focusing on offering AI service in a uh in decentralized domain right so one of the um I was say opportunity or uh the thing that I think it could take off in the future is the uh decentralized AI so um so uh there are several aspects in decentralized AI right so one thing is like uh is a decentralized

35. 18:51

    infrastructure uh so basically like um um so unlike the traditional cloud computing where all those kind of GPU machine and GPU power are sort of controlled and democ uh like sort of like controlled by the uh big tech company so um de decentralized like GPU network uh all those kind of GPU are contribute by the uh GPU owners and then this like creates a large machine pool and on top of those like large machine pools like we build reliable and scalable decentralized and distributed computing infrastructure so this is

36. 19:41

    actually a really uh good point that we can offer much cheaper price and easy access for uh GPU users um and also like uh we buil a variety of AI servic is on top of the decentralized uh infrastructure uh like GPU like for example training fine tune and inference right so uh with this kind of Rob robust infrastructure and AI services like people are can use the eii service in a much uh cheaper price and affordable and

37. 20:21

    flexible uh way um so yeah so um you know like uh we we are like spend lots of time and resource in investigating or researching some cutting Age Technology For example asynchronized training and also surance uh inference um so like those are sort of like um uh combined with our decentralized infrastructure which are very interesting and also uh can sort of like um lower the GPU uh cost uh a

38. 21:03

    lot and uh like I think uh I think like um for uh like for the um like for example asynchron training right so this can sort of like improve the training um efficiency by split the model and also split the data and also eventually make the training in a synchronized way to reduce the um high speed and high bandwidth connection required by the traditional steps uh and also like for serverless training right so you know like for

39. 21:42

    decentralized Network inference is one of the biggest scenario for making use of all those kind of decentralized machine and GP power so uh we build a sort of like a serverless influence platform that can maximize the use of GPU resources and also uh sort of like we we we sort of like investigating the like model loading model inference acceleration technology and those kind of thing combined with our decentralized uh infrastructure can uh can sort of like also meet the grow demands without

40. 22:24

    being limited by the centralized GPU supply issues and uh and we also partnership with data centers and individual contributors to ensure a steady and scalable supply of computing power um yeah so basically I think uh there are lots of interesting topic for uh decentralized AI um ranging from infrastructure uh model and research and those all those kind of thing I think like has a big potential in the future

41. 22:57

    yeah yeah I actually like that take I think whenever you guys were talking about like it you have to be one of these big players in order to come in here I think decentralizing some of this would be super super nice um so I really do like that take it's like I think democratization is like a very common theme over the last 25 years yeah be very honest like democratization of everything um like you know I think Hollywood's going to be changed in the next five years pretty significantly um and the rest but I really like that take

42. 23:28

    and I like the way kind of you're going about it I'll go second just to keep it different we'll make Ilia go last but uh what's what's a recent bet that we've made um G play a fun little game with y'all how many softwares do you think you interact with in a day like just give me a number both of y'all 25 probably hundreds if you include all the different website servers well web side okay then like probably a lot right yeah over 100 and so like here it Ares we do observability

43. 23:58

    right we we let you know when you know we started off in the classic ml World obviously this geni space we we were doing NLP before even 3.5 came out so we were like really well set for this wave um and so when I think about some of the bets we've made and it's like when you think about how do we observe software today right the Myriad of softwares you work with well we use software for that right we use things like data dog um we

44. 24:22

    use things like um you know open Telemetry so we use software to observe software today and then now my next question to you two is in five or maybe 10 years how many AIS do you think you'll interact with in a day give me a number hundreds probably same amount right so what was our bet here uh our bet was really like do you think do you think it's a and this is like a fun question this is really like do you think we're going to use software to observe these AIS right

45. 24:55

    and our take was no our take was like no we need you need something of equal intelligence right um and so we started working on our co-pilot um so in the same way we have observability tools um to go and let you know like hey these are where like you're hallucinating like everybody here is probably running LM as a judge eval on like their their AI we should buil for example uh an AI whose one job is to troubleshoot and observe other AI systems so like a little bit of

46. 25:26

    AI setion um and so it's been really fun building that out we had a big launch maybe two weeks ago um but the idea was like um yeah I think it was like a stepping stone into something a little bit more scalable because if you imagine these things they're non-deterministic systems if you want to think of them like humans it's like well how do we police and observe other humans and it's uh probably another human I think in

47. 25:50

    like like police versus citizens kind of thing and so in terms of an area I've heard people talk about this I think the glean of CEO Arvin and then um I think Richard no he shares my last name but he's like uh he was on the AI alignment team at open AI still is actually um talking about like if we scale this out into the future uh what does like how do we how do we make sure these things that we've built uh kind of performing in the way that we expect so that was a big bet that we made recently uh and it was been

48. 26:23

    pretty exciting building it um just we had a lot of learnings a lot of failures a lot of lot sucesses but uh that's one area that we we've made a b in yeah no I mean totally makes sense as uh the complexity and intelligence of the software enhances I mean in a way we had this on a on last week's episode as like agents are just you know apis which are instead of syntactic are semantic now

49. 26:53

    and so so you kind of need all of the software that was syntactic before and was monitoring syntactic things or interacting in syntactic ways you now need them to be moving up into semantic space yeah I guess I'll I'll go on my side so last year I was really excited about Edge models and kind of this was before I would say like 7B were gotten like way smarter and so so we kind of been started this Edge intelligence Community right uh that has uh you know

50. 27:30

    an office in San Francisco now and uh really kind of working kind of across few projects that are uh doing work in the space both on more hey how do we bring kind of from a privacy perspective like how do we keep data on a device and then bring the model to the data and make it actually better like results for the user and for for Advertiser for you know recommendation system for whatever you're doing on the other side right for

51. 28:03

    product um so how do I make it more private you know cheaper to run because you don't have inference costs and and data storage costs and uh better results better clicks or ratio right so uh custos is doing for example this in very interesting way um and they could to be coming out with like a like separate product for that uh and there's a few others who are kind of in this more Edge uh models and obviously you know the space kind of

52. 28:35

    caught up fine-tuning those models and I'm interested in this even more fing on the next stage um because I think I mean I'm kind of I'm I'm in a way disappointed that like we're using exactly the same stuff we've done right for like past you know 10 years of deep learning which is like we just Hammer with gradiant descent and you know and like babysit learning rate I mean there's a little bit of like

53. 29:07

    advancements but like Adam has been around for a while right that's the last advancement we've had in learning algorithms and so so from my perspective that's kind of the missing piece right now is I think we have like the model architecture is okay I mean you can you know there's there's obviously some really good ideas coming up but I think the problem now is how we're training

54. 29:30

    these models and so there's an interesting combination of learning algorithm and kind of synthetic data preparation that I think will actually allow us to have smaller models to be even more intelligent so we're already seen that right like you know you take 400b model you synthetically generate data to train 7B model and get better right and so I think we'll see that uh kind of uh progression continue happen happening especially if we like go away

55. 30:01

    from Pure SGD right but actually uh leverage a bit better um kind of update algorithm uh for parameters and so so there I'm interested like I mean I've been reading a lot both on like what is the meta learning algorithms right now or what are the also there's people like researching on how uh kind of you know universality of parameters and how like if you train different models and different modalities there is actually uh you know the features learned are uh

56. 30:31

    you know some trans kind of potentially a fine transposition of each other so like it's there is some learning that's happening that like is you know I kind of the way to think about I mean in in in the geometric space right it's trying to find this Global Minima right and so like that Global like how do we navigate to that Global Minima faster than you know hammering with thousands of examples of the same thing right uh the

57. 31:02

    model and finally uh because you mentioned the observability and one of the things that we really critically need is uh making sure that models uh agents and in general code that runs is correct and I think uh we will have S this more semantic but we also need form modification and so that's something that my co-founder Alex's looks a lot at how do we actually introduce formal verification and like realistically

58. 31:36

    people will not write formal proofs uh for the code written and the code written itself will be AI generated so like we need AI to generate formal proofs and the benefit of this formal proofs is actually they're self-consistent right right if you have like the right you know preconditions that you want to validate like it's either works or not and so so that has like it kind of similar to like z z you know recursive proofs right like if if

59. 31:59

    you can prove the some condition then everything underneath is correct as well and so I think that's going to be fundamental for us to be able to compose this especially more complicated kind of software behaviors in intelligent software behaviors that like hey it can maintain this set of you know preconditions or post conditions even after whatever AI magic is happening in in the middle but it's not going to spend more this this much of your money for example or go and like try to start

60. 32:31

    the war or something right it's G to it's going to stay in this you know specific set of conditions so I think that's and again this is important because most of human written code has bugs and so like we kind of need to rewrite all of that and the only way to do that now is with AI and so we will need to have valid code there so that that's kind of how I'm thinking about like you know like Edge models making

61. 32:55

    them smaller making them smarter through better updates and then on the other side using all of that to actually build that software build better and that is formally verified yeah do you imagine that the I guess the purpose what's for the everyday you know Layman what where do you think these Edge models will really help out I mean like on device on phone like that seems like the natural First

62. 33:19

    Step um but where do you think like do you think it can go into other parts of like industry like I don't know you have like you have computers everywhere right so is it just like um even in like I don't know chemical refining or construction or um is that where you see really a lot of the value for these smaller Edge Edge models going I think it's definitely the consumer part gonna come first just because it's like the

63. 33:45

    cost of run like imagine you're running like a you know 100 million user app like if you need to serve AI inference on gpus like I mean it's just going to be like expensive and and um either way right so you want you want to push the cost on the on the user side and so and again I mean this is why like you know Apple intelligence is good example of kind of moving that direction but I think like we need that to be open source we need that to be like accessible to every app not just like

64. 34:15

    operating system kind of controlled and gated um so I think that would be a step one the step two for sure will be like just a new type of software right as as you mentioned right it's going to be like llm inside right and so like when you deploy stuff you're going to deploy like combination of you know kind of a GIC framework with llm and that's a software that's going to run on something that then you you want some observability and some set of things that it's not going to like blow up your

65. 34:49

    plant that right but but that this way it actually like is able to handle all of the like you know errors and situations which like software cannot and then you like need to escalate to human and again there you don't want to go to hit some API that may go down you don't want to leak all your data that somebody can then use I mean as a maliciously or for trading or for anything else like inside of training type thing so like you want to have as much local control and uh privacy as

66. 35:21

    possible and so similarly with companies right like we we running you know internal search AI powered right like I don't wanted this to run on you know hitting a open AI I want to run it in my cloud on top of my data uh and so like I want an open source model I want this but ideally uh you know it's like robust enough that I can run it on kind of um small small enough gpus so I think like it is going to kind of propagate everywhere yeah it's like you need the variability like I think there's

67. 35:55

    advantages to being like um just just running everything economies of scale right the same reason why Cloud works but there's also advantages of like being distributed right uh and I think you serve different markets that way um your other take too just wanted to add a comment on like um uh what we see in the space too is like uh about like uh making sure software or these this case agents behave as intended yeah I think

68. 36:20

    when you think about the solution space of like a fully autonomous agent right when we talk about like the planning stage execution and then uh reflection and all that stuff uh I think we all know that like poly aous agents kind of suck right now um and then you're exactly right what we see happen in the Enterprise is people bound the problem down because like if you want deterministic behavior from something is

69. 36:43

    like not deterministic you kind of have to bound it a little bit um so I think a lot of people here's my hot take is like a lot of people like use LMS too much for things like you know what else is like really good at orchestrating and executing is like code like just just normal code um whereas like I think LMS are very good at like like interfacing with a human in like unstructured text

70. 37:07

    so I definitely agree with you on that one I think we need to mature out some of the Frameworks too I think there's like it's like either fully autonomous like agent can plan its own thing and then what works in the Enterprise today which is these like router where you've bounded the problem like you can do x y and z um I think you're right Ilia there's like some place in between like hey here's your box and like you can live every anywhere inside that box but it has to be inside this box but that

71. 37:31

    was my take um yeah so yeah I can I think I do have some idea about Edge so you know like Edge Computing is pretty um interesting and also like I agree that you know like for most of the uh so there's a trend for the model to be smaller and to be capable of doing things like agent right but it's sort of like much smaller then like the like like trillion or like like Lama 45b so it's really not that like it's too much

72. 38:08

    for a like um for that to solve a specific task or specific thing right so um in the future like the smaller model can sort of become more capable more capable like to doing things and also be much like cheaper and also much smaller to to run you know small devices so uh you know like um we like netm also are investigating how to fully leverage all those kind of edge device because like

73. 38:46

    for um Edge device like they are sort of like consumer or even consumer um brand GPU or even like CPU they can also sort of like contribute their uh computing power to like our platform and like uh we we can leverage their um computing power to do for example model inference um so um but as I agree that like uh security and data privacy is really really important like we don't want to risk like uh customers like data or any

74. 39:25

    of these kind of data to leave in the um uh in the users uh devices so um but like we are investigating how to connect more and more different kinds of edge devices to our GPU Network and also run inference on top of them but also ensuring the privacy and security um imagine like in the future like there are uh sort of like uh tons of like decentralized uh like device clusters and and with a sort of like security and

75. 40:08

    all this kind of compliance buil on top and then like everybody can just like use all those kind of services in a multi- region and decentralized way and then like uh and we have H hybrid model like for smaller model to solve specific task and for task that they cannot solve maybe they can just like um use like larger model or use different kind of models but eventually we want to build all those kind of clusters with cluster

76. 40:40

    Federation and then to so serve all those kind of inference um with a secure way so that's actually one of is in our road map but I think it's uh challenging but I think it has uh very big potential yeah aome I'm curious actually on on this side in general um how are you thinking about new hardware uh new types of Hardware as well as kind of evolution as well as um kind of cost structure on

77. 41:13

    existing Hardware because I was actually trying to do some math and from my understanding if if it's not the smallest model and like if we're talking about kind of 70b for example on uh prev like not a 100s but like previous generation like 4090 3080 like you're getting a pretty uh like really bad performance right you're getting like two tokens per second kind of thing uh out of 70b Model

78. 41:44

    right which given you know the cost the cost uh kind of is defined by h100s at this point uh it's not that much revenue being made so there's like pretty massive opportunity cost SL like just like electricity cost that comes into this so I'm curious kind of how you see that how you see new hardware coming in which you know maybe have different properties or uh is is uh kind of

79. 42:13

    provides different tradeoffs uh yeah so like I think um I agree that like um so if you want to have like the best performance you need to use the hybrid um GPU like h00 a 100 but for inference like uh because like we with the latest like model acceleration technology uh it's actually can get like relatively good performance on like 4090 uh if you just like uh for example inference a uh 40b uh 70b um

80. 42:57

    model um also like you know what do when you say good performance what's uh what's a rough numbers uh so there is like I think like um I cannot get the real like concrete number for that it depends on the model right but I think like it's not really bad as you mentioned like you get like two second latency for the first first to token so I I meant like two tokens per second that's kind of the with the uh speculative decoding using 7B and

81. 43:37

    uh 70b B like the model it's like two two toket per second on uh yeah so like um if you search the um like latest M acceleration Benchmark I think it's much better performance uh than like two seconds per token so yeah and uh I I I can say that because like we actually did a bunch of uh testing and Benchmark for model acceleration and also like we uh because there are like lots of Open Source uh framework you can use for accelerating the

82. 44:19

    inference um and uh I also want to talk about the Cs so because like um you know like uh you can actually serve the model in a seress way that um that like you can sort of like make full use of all those kind of GPU um and also like um so like I I'm not sure if you know like the Lambda in ad right so like people are really um interest in like using the seress by saving the money because like uh if you use the if you

83. 45:04

    are Enterprise customer right and there are a bunch of way to use a inference API you can sort of like use a dedicated server like you pay for a GPU server like Edge 100 or a 100 daily or hourly and then this is like your specific server to um to serve your uh API inference API but you can also just like pay for the use and use the servus so in this way like it's actually um give you

84. 45:35

    a space to loading the model dynamically and then to sort of like make full use of the GPU and to serve different kind of uh customers so this actually can sort of like save money in terms of the uh infrastructure side um and also like I think for new GPU right I believe like um I I believe like I'm not sure like how uh because like the the hardware and the AI model are involving together like with the uh high performance GPU come

85. 46:18

    out we can have like much bigger or better model which can sort of like depends on those kind of Hardware so yeah it's pretty tricky to um to to to see the future but uh I think like the model can be much smaller and then eventually I hope that like in the future we can have like the smaller device or add device can run fairly good model and with all without GPU yeah that's my

86. 46:55

    take yeah I think the a yeah I the whole your comment about like how the I guess model inference model training is is really adapted to the hardware that you're on um since you're on that side of the space Henry where do you feel like how much more Innovation are we going to get from the hardware side which your like hot take yeah I don't like are we are we maxing out like is like is invid like

87. 47:19

    are we still early stages like I definitely know from the application side from where I sit it's super early stages right but on I heard some news that like the architecture can be changed for specific or type of models like for example like the GPU for RM I think like this can be the um opportunity or potential for to sort of like accelerate the GPU performance so yeah but I know like we are sort of reach a

88. 47:53

    limitation because I'm not an expert in Hardware I I don't know how far we can reach to the limitation but I I I'm thinking like maybe Invidia or other big tech company can come up with some like new architecture specific for AI models right and uh there's a couple startups that are doing that yeah like specifically for Transformers the the challenge there is like and and this is true also with some of the other competitors who've been

89. 48:25

    trying to build kind of AI specific accelerators is that it's really hard to do research on them because like it kind of anything beyond just like a standard you know whatever Lama does anything you try to do it like blows up one way or another and so uh and what it means is like all the code that researchers write that then goes into becoming an next model is not written for that hardware and so so there's like a very weird kind of negative look there and so and because this things have I

90. 49:00

    mean come on this has been two years since people started like seriously looking at this stuff uh the reality is like whoever's targeting stuff like you know the time to design to tape out to get prototypes to get production ready like that usually takes like year and a half two years so by the time they do that the research already moved on right you know somebody comes up is like hey actually instead of you know attention

91. 49:24

    we should use whatever this thing right and boom it like now you literally design and build a company so it's it's kind of a like in a way it's a high like high bat that like high risk bat that you're like hey I'm gonna like optimize for this architecture and then hope that it's not going to change for two years um fundamentally and so that's like as you said right the software is just evolving

92. 49:56

    so quickly that is just like it's a very risky betat and like I mean there's some people taking it and you know obviously VCS have funded that uh just because like Nvidia has such a massive you know earn earning uh but if you talk about like generic GPU like generic Mass processor right that you can do anything with the real challenge is software Cuda has been like chiseled out for 10 20 years right into like what it is now yeah and uh a

93. 50:28

    lot of people kind of like AMD and others are trying to catch up and the hardware is probably like you know maybe not one to one but like comparable and it's just like again software problems is like kind of killing productivity and so uh but I think like for AMD it's and for other for a few others it's like if you go production and you need to do it at high scale you can do it because like it's worse investment into like fixing all of the issues down the stack and like you know fixing drivers and fixing

94. 50:59

    like whatever uh but if you're doing like research and you're kind of pushing the envelope right it's not and so that's kind of the trade-off I agree if you look at the pace of innovation from like model Builders you know like it's very clear like anthropic has caught up right in a very short amount of time but no one's like AMD is nowhere near close like yeah AMD is nowhere near close to

95. 51:23

    Nvidia don't even get me started on Intel so yeah uh that those are my two fun questions but I'm happy to open up the floor to this is your show so yeah no I mean like it the idea to have you know discussion on some of the topics that's top of mine so I don't know Henry if you have any top of Mind questions on your side uh yeah I think uh uh like I do have one topic um I want to discuss

96. 51:59

    so um so I want to like bring up a topic about AGI so yeah I know it's a very broad a broad like topic but what do you think about AGI like what's the uh challenges for eventually achieving AGI and what would be the opportunities and uh uh also like the barriers to achieve the AGI yeah I guess my first question is how do you define AGI yeah but like AGI is just AGI right like uh or like I want to say like AGI

97. 52:40

    is something like the AI has like really high intelligence and uh it can just like do what or it can sort of achieve what human can do for like taxs for like helping people helping human to do things yeah yeah I mean my my formal definition is uh the it's a system that is able digital system that is able to do anything that human can do in digital realm better than the best human interesting take I feel like I I

98. 53:21

    can I have like many lenses that I want to look through it's like if you're saying like can can this like I don't know can an llm outperform a human on an IQ test right well technically we're at AGI because it outperforms some humans I don't think we have AGI until this thing has intention um that's where I think it's like it's got It's conscious and and can decide like yeah I mean I would argue that like the intention Consciousness is not the requirement for having something that is

99. 53:52

    generally intelligent for sure I fair enough I I see that take but I don't know it's like I artificial general intelligence I don't know it just depends on where you draw the line I guess um saying like it it should be able to do any task any skill in digital realm better than a best human at that skill and task oh okay better than all humans at that time yeah yeah so like it like if if if for that skill there's the best human in the

100. 54:19

     world it should be better than that okay we're definitely not there yet but okay fair enough not no uh what if the skill is like intending to do stuff you know what I mean Ilia so it's like you know I'm just I'm just saying I think intention to me it's it's in there it's like uh that's my definition but I I think I think why why I don't agree with that is because the intention may come from the

101. 54:44

     user and and then the you know in this case AGI is still a tool right like what you're saying is like hey it's a sentient being right yeah yeah like you can imagine a sentient being that actually is not that intelligent like a dog like dog is not very that's a good intelligent but it has a very much you know intention okay that's a good point I I kind of retract back what I'm saying um yeah you know I I think my take on AGI is like I don't know I see this stuff in the wild um

102. 55:18

     we're kind of nowhere near it to be very hon like getting getting actual in production llm applic ations to work even something as simple as like rag people still struggle with it it's like the latest flavor of like oh how do I get it's like graph rag is all the rage the last two months um so my take is like we're still still early stage I think like is Transformers part of the equation it's like super possible I think it was a big unlock but there's so

103. 55:47

     much and here's another hot take I feel like so much attention is on like llms that we've lost like investment into other areas um and it's kind of like homo like we're more homogeneous now we're like all focused on this where I think as before it was like kind of more widespread um so I think we're very far away from it um I the Direction's okay I mean but I think we just need more investment into other areas like I think there's a lot of promising areas outside of like Transformers but no offense you

104. 56:20

     but no I mean again for for me like as you said like it is you know Transformers or some some deep learning architecture is a component of a system uh but it's not going to be the whole system um like for for me the the things that like if if we would design it right now right just like kind of high level um you want something that is able to obviously like you know use tools and

105. 56:54

     process data Etc and and the reality is like the current models are not trained to do that right like yes we fine tune it a little bit you know we like put a little bit over it but the reality is like if if you think of like the way this model is trained is you know it's like if you were getting dropped into a new room every like 30 seconds and then you need to like find you know the ne like and touch you know three pieces of furniture and then you drop to the next

106. 57:24

     room and then you're like okay build me you know a great Furniture like furnished apartment it's like um like you don't actually know why this article was written when you're trying to generate it and so I think there there's like definitely uh better training procedure that's kind of I was saying right is like how to update parameters but also like how the procedure is done versus techto in protection um there is some form of search and kind of you know self

107. 57:56

     evaluation that needs to exist uh because this is how you're going to find new things it's not going to be because you know you're just like rolling the tokens um and this is what we see with you know all of this IMO stuff and other things um the some type of world model right again Transformers maintain a form of a world model in parameters but it's very um it it's limiting in a sense that like you don't update it from the inference right so like how do we do Active

108. 58:30

     Learning now to and is that then should live in parameters should be in activations like should we have like all of the activations ever be available well that doesn't make sense like okay well how do compress that into either parameters or something else so all all of those like things is where as you said it feels like the research kind of a little bit like narrow down versus like investigating I mean like some

109. 58:55

     people but like it's it's uh not enough and actually this is something we are looking at is like how do we accelerate open source research because from my perspective the challenge been is that in universities and in General open source World outside of three big Labs you actually don't have much capacity to even experiment with these things and it's felt Less open source over the last 18 months oh yeah exactly it's like no one wants to share anything and it's

110. 59:24

     like I don't I don't really like the direction we're going it's like the opposite and what like these companies have like oh we're going to you it's like the opposite direction which they plan to go it's funny like the pack to hell is paid with good intentions but you're still on your way to Hell well yeah I mean that's why like the the the business incentives of big companies right is and I mean meta thankfully has like business incentives to to that are

111. 59:49

     opposite but uh yeah I think I think again I mean this is something we working on is both coordinat a bunch of people but also like how do we create an environment where uh researchers can actually work together and then benefit in some way from the research they've created uh and also have resources to be able to do experiments at scale right uh and they and have new data so yeah I think that that is something we really need uh going

112. 1:00:20

     forward that decentralization piece is super important there like you like let's let's democratize a lot this stuff via like sharing oh yeah we're we're out of time right yeah we're out of time so thanks guys for joining uh was amazing discussion some hot takes uh I'm sure we'll see some of that on YouTube and in short videos now um but uh yeah exciting exciting to be building this with you guys awesome likewise thanks for having

113. 1:00:49

     us on you cheers what