DevHub Live 22 - getting started with AI agents and data with with Masa AI (@getmasafi)!

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:27

   m oh what you want what you want want what you want want want

2. 1:58

   what you want want what you want want oh what you want you want you want want what you want what

3. 3:54

   want want me what you want what you want what you want want

4. 4:59

   you want hey everybody we are live again here at devhub I am back with my co-host Owen thanks for being able to join us today I know you it's pretty late for you great to be back well it is late but I had to make it just to see you J I appreciate that awesome we got a really uh exciting uh Workshop today and I'm really excited because if you guys saw the yesterday uh with M of that I think this is something that everyone should get involved in uh it's great for the hackathon it's easy to add into your

5. 5:54

   project um but today we're going to dive into the technical pieces of of mass and actually had to start building right away so I don't want to waste any time I want to make sure everyone can maximize so let's bring Brendan up hey Brendan how are you I'm doing great yeah I can see oh good morning super excited for for this uh you know so we got awesome chance to talk yesterday and a lot of great things kind of came from that conversation about building agents and how to kind of dive into that so I'd love for you to kind of

6. 6:26

   just drive a little bit of this of uh where should we start and what exactly let's let's go one step higher case you missed yesterday what exactly is mastera and what can we start building with today yeah it's a great question so mastera is a decentralized um AI Data Network AI machine learning so what we enable we enable anyone in the world to both contribute to the network that is run our software run our code we're

7. 6:56

   going to go through how you do that today to start contributing to the network um typically that's scraping data it could be scraping data from a source like Twitter that's be something we're focused on today we're going to build a a rag agent that allows you to ingest tweets from KS we're going to do this in real time show you exactly how you can do it um and other sources like web so if you kind of want to for example figure out what is the weather

8. 7:21

   in San Francisco today and have that available to your agent uh you can use our our web scraper to do that we also have Discord scrapers Telegram scrapers and speech to Tech so you have all of this kind of Suite of data aggregation software that has contributed either in an indexed format so these are workers that are contributing data or developers can make real time requests I'll show

9. 7:46

   you how you do that through our um node API today as well and what this does it gives anyone anywhere in the world access to data which historically has been very difficult to get hold of um Andor um it's been challenging for you know developers who want to really focus on Building Product to do both things building data pipelines plus product is hard yeah no I I you know one of the

10. 8:13

    things that we highlighted yesterday in our conversation was just the ability to actually get this data in a normalized fashion uh to be able to get it at a way that is uh avoiding having to go through all these different separate apis and everything else like that and so you guys are really providing an opportunity for again we tried to highlight for the hackathon especially like an easy way to add on something to that you're already building uh and agents and things like that are an easy way to kind of get into

11. 8:43

    it uh oh and do you have any questions before kind of uh we jump into some of the stuff because you didn't weren in party yesterday well yeah I I obviously wasn't here yesterday so I'm going to have a few questions I think I'm just interested to see what it looks like and I'm glad I'm here today rather than yesterday because I always love to like dive into the more practical aspects of

12. 9:09

    things so I think maybe we could just get in get started and I can learn and ask some questions along the way perfect great awesome um and I might even start like today with a with a whiteboard just like come out like a very high level um walk through all of the components go through what the goal is going to be today in terms of what we're going to be building and then we can actually break it down and go through the code to sort of actually see how you go through step by step and get this running perfect areas where you might you know

13. 9:43

    might trip up so B on second I'm gonna put everything up onto one screen and then move things into this screen as we go through this process so bear with me while I'll just get this up I think one of the uh one of the most one of the things I'm most interested about hearing is like how you're actually getting like the data on chain and sort of bridging that that gap between like Twitter data and the chain

14. 10:19

    so that's be interesting to see yeah absolutely and I'm going to make this actually available to comment to anyone I'm I drop this in the chat with Jared so as Jared can share that with everyone right so people should be able to get into this whiteboard either during the session or after it and see exactly what kind of going through which would be really helpful um so I'm going

15. 10:42

    to start at a very very high level um what are the components of Massa first going to keep this really short feel free to kind of stop me and pause me at any point during the process we can kind of get into uh specific components once we've gone through these components we're going to look at what is the key um building blocks for a rag chain um based agent uh and how we can kind of put that together in literally we'll do it in like minutes on this actual

16. 11:13

    session session see how easy it is show you the key components kind of break it down and uh give you all an opportunity to get some inspiration for what you're building um also so um on the left here we have data sources um the this is a range and flavor of data sources that typically Encompass things like uh X I'm start using XX Twitter I'm still very Millennial with my use of Twitter we have web

17. 11:47

    scraper uh and then Discord Telegram and we have speech to text we won't really be touching on speech tot text today um but it is available if anyone would like to use speech to text please just drop us a note this is things like podcasts so if you want to take an entire Corpus of podcasts break them up diarize them for example this particular session uh you can use uh one of our pipelines to do that so we have these data

18. 12:19

    sources um as a worker on the mass protocol you configure uh a node to to run one or more of these scrapers um you also need to stake the node um as part of your uh process of getting up and running and the node itself has an API that allows you to interact with a range of different endpoints and we'll review those review those shortly um this node can run in isolation mode so today we're run it as a single node a

19. 13:00

    little bit like if you think of a cck cluster in early ethereum you can kind of have your own private Network or you can connect to the bigger Massa test net and I'll show you how you can do that today um but for sake of Simplicity we're going to have a a single node running an isolation um we still need to stake that node to get it to function uh and that will unlock US using a range of these different scrapers so as a developer you can choose to have your own private Network that's got dedicated

20. 13:32

    resources for yourself or you can join the main protocol and you can access uh an entire host of um scrapers and workers out there without having to configure um one of the scrapers yourself so it operates in two modes you're either a worker or you're a developer node which is like the equivalent of an ethereum like archive Noe where you're reading from the network but you're not providing an

21. 13:57

    impact the network so just pause there for a second s any questions so what I'd be interested in hearing so you say you can like operate a node by itself um or join this network so is it always just one node that is like polling the API or if you join the network will it be like multiple polling the same API to sort of make it verifiable I suppose so a node doesn't lie great question so on the uh test net we have let's call this just uh 500 worker

22. 14:38

    nodes and let's just assume that all of these worker nodes are Twitter worker nodes um I'm going to bring in my like equivalent of what is an RPC node so this is the a node that has zero configuration um it's just staked so this node can't do work it can't scrape and it's going to be sending requests directly to the network out to these 500 nodes and it will select um in a selection mechanism which

23. 15:12

    node is going to return data these workers are run using a consensus model that ensures that data that is returned to your zero configuration node is uh valid and true and each of these for example let's assume all of these are Twitter noes like we have here um can return a response uh to your client uh node here so we have validators on the network that ensure that all workers are

24. 15:49

    honest um and that the data you're getting returned is is in fact uh good quality data and represents your request and we'll show you exactly how that works in just a moment nice I'm uh very glad to get that cleared up because my main concern coming into this there's you could just like lie about data but um well this is obviously uh obviously solving that so that's great I mentioned to Joe yesterday we have something really cool coming to the network in the next month or so um I'm just going to change this

25. 16:23

    color here we have an Intel sgx uh containerized wrapper for um every worker node that's running on this network and what that does it enforces that the code that's running in this container and joining the network is a verifiable code configuration that includes certificates for for example validating that you're connecting to the Twitter API and you're not connecting to a mock API that's being routed through like a proxy for example so you could mock an API route through a proxy that

26. 16:57

    has uh the Twitter URL as like um a spoofed URL and return data from a mock that could be incorrect um this actively prevents this we're rolling this out both onto our kind of bit sensor subnet the protocol and then open sourcing it so as other projects and protocols can do the same validation for example if you've got a compute uh protocol and you want to verify and ensure that all of the uh capacity that nodes on that Network are delivering is

27. 17:28

    true you can use this containerized application to verify that it's a GPU that has a certain clock speed certain amount of memory and is at a certain spec and you can ensure that's enforced through this like hash verification and remote attestation that a validator does on your codas think of it like hashy C Vault but for runtime interesting I know I am curious as to why you're not so you're not call

28. 17:58

    calling this like a an oracle as such we used to um we used to call it that um we've sort of slipped away from that it's still naming convention in a lot of our codebase you'll you'll see that um but yeah it was what we uh what we started with from a nameing convention perspective but we've moved to just like node at this point or worker or no that's kind of the F that we've adopted fair enough yeah I was wondering if it was just more like naming convention or or whether there was like

29. 18:30

    distinct differences yeah no I I think that when we started we we thought that we were kind of positioning a bit like Shain link for real world data hence Oracle kind of made sense and and we've just slowly shifted away from that from like it it in the web3 space there's a very definite perception of what Oracle is for pricing data and it gets people in a certain mindset of what you're doing and and we've tried to kind of break ourselves out of that a little bit yeah nice cool okay that's that's all my

30. 19:01

    questions at this stage great question I I appreciate it and then on top of the API you can use our SDK we'll be using the SDK today um and that gives you a lot of flexibility on the way in which you can actually scrape data makes your life easy in addition to that we have in the near AI Hub um uh we're going to have and this will be probably live tomorrow we have a huge um I think it's several million tweet data set that

31. 19:31

    encompasses um a whole range of keywords uh around cryptocurrency cryptocurrency prices token prices that anyone can use so you have two options to get up and running it you can run your own node you can customize what you're scraping and you can create your own data set if you don't want to do that as an initial step you just want to go straight to building you can take one of our example uh data sets there's a

32. 19:57

    bunch on hugging face and we're now migrating all those to the near aiub so there's data sets on the near aiub that you can use and expect to see those posted in the next 24 hours to 48 hours for you guys um okay so this is basically the architecture and the layout of um the the protocol at a high level now if you want to be building an AI agent you're going to start off by uh getting your dat like I said there's two ways

33. 20:30

    that you can get your data there is from the near Hub or hugging face we have a whole most of um data sets up here under the Massa Foundation hugging face you can see we've got YouTube collection Twitter um and then there's uh 517 other data sets under here but likewise you also be able to access these from the U near AI hub um and in addition to this you can start scraping your own data set scrape your pH um we have example

34. 21:13

    repos in GitHub that will allow you to plug in these data sets and get up and running really quickly to create a very simple uh rag chain agent and uh in this example today we're going to be using as C toing we'll be using Lang chain um we're going to be using an open AI embedding zpoint but again uh we're going to migrate that over to a near AI Hub embedding zenpoint using the LM

35. 21:43

    infrastructure over there so very soon you'll actually see this example codebase transition um and we're working with the near team and Gabe at the moment to to get that transitioned over um and most of the llm work is going to be done locally we're gon to be using a llama 3.1 uh to run a local version you can see it running up here of llama 3.1 8 billion uh model and then we're just going to use a very simple

36. 22:15

    streamlet uh interface which will allow us to give ourselves a flexible UI you'll see a few like formatting issues on there we do like also to use um this is a front end there is this open source uh web UI um framework which gives you a chat GPT like experience I think this got say like 50,000 Stars 41,000 Stars extremely wellmaintained make your life easy use something like this this open source or something simple like stream it so as you can uh simply quickly and easily put a front end onto your agent like make your life

37. 22:56

    simple focus on the rack chain focus on the logic I will drop this into the chat as well these are some great tools for you all to be using um and uh from here we'll be able to prompt so we'll walk you through all of these steps we're going to get the data we're actually going to scrape some ourselves we're take a look at the example repo that outlines this simple rag chain we're going to run it using Al llama 3.1 locally you can also use any of the llms made of ailable through the

38. 23:29

    uh near Hub router um and then we're just going to Simply use a very very vanilla streamlet front end to be able to prompt the agent and get a response and you'll see in this what I would call lowest common denominator example like anyone can build this um how quickly you can get up and running with uh with data and start building and finessing uh an agent so I pause here see if there's any questions otherwise I'll kind of dive into it

39. 24:01

    I don't have any questions yeah I mean it right now the way you have this set up and again I feel like it's structured in a way that again it goes back to my whole product doesn't have to just be around this like there are spots where this can drop into my product and so or this can be Standalone so it feels like there there is a lot of flexibility for someone to come in after this flow absolutely absolutely and like your key

40. 24:33

    components outside of Massa is your is your rag chain um with your LM so these two components here this is what I would say is like when you're building an agent your your lowest common denominator set and then you've got your UI and you're kind of prompting here and we'll see how changing prompts kind of changes the interaction with the agent as well um so yeah absolutely if you've

41. 24:55

    got this kind of conon set of tooling uh you can bring in um you know any data for example if we have a step in our rag chain where we um want to get the weather today uh you could use a llama to construct a uh a query that can then be used against the web scraper endpoint for example to go out and get weather.com weather or for example if you want the the current price for Dogecoin today you can do that I'll

42. 25:25

    actually show you how we can how we can do that today as well awesome so our first step is in scraping our own data if we're not using the near Hub or hugging face data we want to scrape our own so the place we start here is with um that's a GitHub so you're going to come to our GitHub uh you're going to take a look at the Massa Oracle because we still have the old naming convention like I

43. 25:53

    mentioned earlier um from 18 months ago uh clone the repo and I'm going to show you how we really quickly get this up and running locally so let me just make some space on the screen here minimize this we're going to be using a I like using warp as my Okay cool so go into my master Oracle folder uh you should use the latest tagged release um when you're running um the actual uh protocol itself so always make sure that

44. 26:43

    you're running the uh latest tag release we actually have a new release coming out today we have a an 8082 release which gives the functionality I'm demoing today so in a few hours we'll release that and make sure you use that latest tag release it will give you latest fixes and all the all the all the um latest features so I'm going to clean up my um latest build and I am first of all going to configure my

45. 27:13

    environment file so I'm going to pull up uh this which is my Cent repo um and again if you go to the Massa developer documentation which is on our website uh most of these guides will be there and we will also uh do like a a how to video so you can easily kind of check this out make sure you configure your uh environment file here I'm just going to delete this information um and in the environment file you have some default configuration which is

46. 27:49

    your information here and then we're configuring uh the scrapers that we're using as part of the protocol and today we're going to use uh the Twitter scraper so I'm setting this to True um I've set custom user agent which helps our scraping you can actually create a comma separated list of user agents here if you want to cycle through multiple ones we would recommend doing that um and then you can go into like configuring Discord scraper

47. 28:21

    telegram scraper and I've also on here I've got my web scraper as true because I want to use my web scraper as well today in this demo so you'll find the environment file in um uh in the developer documentation there's a set of default environment files that you can use I'm just G to move this off the screen so as I can put my credentials back in here um for logging into Twitter you need a pro account with Twitter um and your uh username and

48. 28:52

    password you can add a string of usernames and passwords I I've added in two accounts here so it will cycle my requests between two accounts you could have up to 20 accounts um and I'll show you exactly how requests get routed um to those accounts each account can probably scrape around a million tweets per month which is a um which is a Twitter based limit so now I've got my EMV defined I am going to build the binary this will

49. 29:24

    take you know 30 seconds to do this and while I'm building um the binary I am going to go out and install um a couple of other packages that we have uh that I'm going to use for putting together this example so I would like to uh get up and running with the Massa SDK as I outlined on our previous example in our whiteboard just pull this up right here uh we use an SDK to intera the API and scrape data and you can find

50. 30:08

    that at our Massa repo here uh we suggest you use like a cond environment for running any of our python code um and you can easily install this so click install Master Ai and then we can create a request list of the data that we're scraping and I'll show you in just a moment how we're going to do that this saves all the data locally um and makes it available in a Json file so as we can um input this

51. 30:39

    into an embedding model uh and then into a local Vector store so as it can be retrieved by our agent um in addition to that we're going to use the uh Massa awesome Massa reper which has examples in it so you want to clone the awesome Massa repo here and within the awesome Massa repo we have um some scraper examples that don't use the SDK this may be like a one-off scrape you want to do maybe you just want to

52. 31:15

    kind of use one of our older um scraping scripts but we'd always recommend Now using the SDK to do scraping um and then it has uh an example agent we have have a k agent and we have a mcoin agent and within this agent file there's a very simple uh setup here using Lang chain and their local Vector store with a rag chain that is set up here with a with a prompt we'll change this in a moment using um a local Al llama model

53. 31:50

    you can change this to point to a a near model um and this is a very very clean uh boilerplate example of using this agent package with uh streamlit um to uh get up and running with an agent so in short get the master SDK get your node up and running and um you want to be using the awesome Master repo so we're now going to put this all together I'm going to make run the uh node when you run this the first time um this

54. 32:31

    because I have it run somewhere else one second let me just a couple of processes here I know exactly what's happening this is this is running on another I need to all this H okay a second it's always it's always uh chaotic when you're trying to do it live I I have my SDK docs running on the um on the same port as what I do on my on my node and I don't normally do that so I'll come back and kill that in just a moment and show you how we can get that running so bear with me one second let

55. 33:24

    me just stop sharing this screen real quick I let give me give me 30 seconds to server one sec's looking for that Owen what are some things that stick out to you so far as you see this I think I've never really heard I obviously heard about getting data onto chains with uh oracles and such um but I've not really heard of that being used like specifically for agents so I think this is where it really becomes um interesting and I'm I'm sort of curious

56. 34:08

    like how these agents are going to use the data and act certain ways based upon that y so I'm uh going to be going to be interesting and I wonder what sort of crazy products people are going to build based upon this maybe agents that are transacting and buying and selling meme coins based upon like like what people are tweeting right right right yeah it's the real time component Mak it is really interesting okay done this now

57. 34:57

    running in the background back I just share my screen and thanks for being patient there I appreciate it oh no problem okay all right so you should now see this again okay so we've we've started um our node um it is at this address you can see that I have staked it one of the steps you G have to do before you run the node is is stake it with test Massa there's a guide in our de docs to do this um and I'm just going to use Postman here to uh

58. 35:30

    take a look at our docs um you can see a local version of swagger that's running from the node so as you can see the end points that we make available through Swagger there are more endpoints on the actual mode itself but we don't always expose them through the Swagger docs we're be using Twitter specifically today so um this is a node that's configured uh locally in isolation um it has two sets of Twitter

59. 36:03

    credentials um and we are going to demo how I'm going to get the profile for go here of Twitter that's because I'm doing wrong should be Local Host and um you can see that it just sent a request out to my node and it's returned uh in real time the Twitter account for God I could also do my account here Twitter account um this is helpful if you want to do for example a um a digital T pretty quick yeah and you want to ingest

60. 36:45

    um my profile to kind of see my profile to your digital twin you could use my Twitter profile to do this really easily similarly we can um actually get tweets so with this query I'm getting uh any tweet with the hash mcoin zo a little bit here um from the 16th of September until the 17th of September so I'm going to send this query out and it's returning 25 of those tweets um in

61. 37:16

    real time directly from uh directly from Twitter so this is how you get up and running just showing you how to actually retrieve data we're now going to dive into the awesome was this um sorry were these tweets um the accounts owned tweets or tweets from like its feed or basically it is you can access any tweet that is available on Twitter so you can construct a range of queries and and these queries are actually in

62. 37:51

    my uh Dev docs that was actually running in the background could query by some by like a hashtag or something example yeah give give me an example like maybe we can do um the yeah this hashtag here let's do what's a good hashtags do Bitcoin right now and we're going to do that from the 10 three until 10 Z one so the last like three days and that's going to go the network and I have essentially these hashtags here

63. 38:33

    it's getting all of the tweets have the hashtag Bitcoin in it um we could also do from what's near near AI uh okay near protocol is the username I could also get all the tweets from near protocol so you can see here nice yeah last latest tweet and I can actually verify this by taking the permanent URL that I mean to do that get tweets sorry poting can sometimes be a little bit

64. 39:09

    pickle um and you can actually see that this is the tweet that is at the top of uh n's um Twitter feed this is the first one cool does that help yeah yeah really interesting amazing so I want to go one half step back here so people are just really clear on it so if I want to run an agent I have to run a node first right I'm not I can't piggyback on another note I can't have like if I'm is is it like a oneto one relationship kind

65. 39:44

    of thing yeah that's a good question um you have three options um to have your own single node with your own logins you could have your own private cluster of where you bring other people to your kind of private Network that contribute data so you could ask or invite 10 people with their own logins to come and join your private cluster and create a uh single Network for yourself and then the third option is

66. 40:16

    you can piggyback off of the aggregate users on the Massa protocol and the way that you would do that is in the environment configuration which is let me just pull it up one second in your environment configuration here let me just remove my password information here in this environment you would configure a boot node here this boot node is two addresses here of the main Network boot nodes and when you connect to this you could have

67. 40:57

    is Twitter scraper set to false and your request will go out to all other nodes on the main Twitter Network so if you want to use the main Twitter Network um main Master Network you can do it that way so there are the three ways that you can you can configure it up and running I got a couple Community questions while while we on this topic of nodes here real quick so first one up here is can I

68. 41:22

    write in my node on a Raspberry Pi I a good question if you've got a Unix uh or Linux environment on there yeah absolutely very very low power device can run this we typically run them on a 2 to four CPU and 8 gig um machine very very low power device about this great question and then uh do I need to run validator too or can I just run the node uh you can just run the node you don't there's no requirement run validator absolutely question that's some pretty

69. 41:54

    uh pretty concise responses there um awesome he knows his stuff um so let's just dive into the awesome Master repost so we're GNA Point um this particular script and you can I'm just using this because it's in the same repo as my example code that makes it easy for me to do today um I would strongly recommend that you use the Massa um SDK repo which I've got up here um there's uh healthy and sorry there's there's useful documentation that you can access

70. 42:41

    for the SDK right here and it includes all of the advanced Twitter search so it gives you instructions in how you can actually Define certain keywords in your search terms so as you can really get specific with the data that you're searching for and today we are going to um use a k World tweets from the last let's call it week um to build this agent so we're going to do from this account and we want to be calling up up to 150 tweets and we're going to get

71. 43:31

    tweets for the last so we're starting off today which is the 3D of October we're going to start on the first here and all of these other configurations stay the same and we're going to run this script yeah Spell python wrong and this will okay one second I need to remove my here my state file do that again okay great and you can see on the left right here we're scraping tweets it's going to start saving those tweets

72. 44:21

    uh up here into this Json file um and it will qu keep sling through these days so we can see like it's fetched 58 tweets from today 55 tweets from here 73 tweets and it will continue going until it's gone through the entire date range so you can see we've scraped 186 tweets from the last three days now what we're going to want to do is we're going to want to load um this into our rag chain

73. 44:52

    and we're going to do that by configuring let's just make a little bit of this a bit more clean we're going to use our K agent template um I am going to want to load data from to path the file that I just saved um everything else on here stays the same and let's say I want to configure my prompt so I'm going to go into my rout chain set up and here you're going to see the system prompt

74. 45:30

    that I am putting in here so you're an AI assistant specializing in analyzing to an analyzing and summarizing Twitter conversations about cryptocurrency based on K tweets let's just change that to tweets um from the author's Twitter replies let's just call this from the author's tweets it's a bit more concise uh your task is to provide size so this is where you can like really Define what your agent is going to do um it's

75. 46:03

    currently October the 28th okay that's October the 3 what am I doing 24 so make sure you're getting data that is relevant to that date and then Etc so on and so forth um it inserts the question that comes in from the prompt uh the Tweet data and the system prompt here as well now we're going to load up our data give me one second make this smaller to okay to streamlet it's too Rong my bad

76. 47:00

    second okay and this is going to spin up a local rag agent it's um creating a local Vector store it's creating embeddings so it's already creates our embeddings it's setting up our workflow graph and our workflow graph is defined in here which is retrieving generating and it also has web search functionality which we're not using today um but essentially it will allow me to ask can you please

77. 47:31

    summarize the tweets from my author um the author here is Satoshi flipper it's going to go out betch in beddings and immediately use the context from our uh data that we scraped uh to generate response and this response is going to be generated using our local version of llama 3.1 um and here we go so it understands who the author is um what he's doing um what he's talking about and gives you information about the tokens what does he say about Mass

78. 48:15

    token because he occasionally posts about a token um and you can continue to have a conversation here so this just shows you how fast you can get um up and running with Massa data scraping a custom data set and then bet getting into a rag chain agent in no time at all like 20 20 20 minutes there um now something like this you need to though them programmatically continually scrape data or how does how does that then fit into this yeah it's a really good

79. 48:49

    question um we currently uh there are two ways in which you can do this we as an organiz ation the foundation constantly scrapes a keyword based data set that encompasses some of the most requested keywords we get from users um you can find that list in our GitHub under our subnet our bit tensor subnet because it's our subnet that actually scrapes these keywords so the keywords are uh in a list here if you would like any of those keywords adding to the

80. 49:28

    index just please let us know or submit for request and we can update this that's what I recommend or you can use the um SDK to continually scrape a list of keywords so you'll see here that you can create an array of Search terms and you could have a Chron job that periodically let's say every day will will scrape and refresh the tweet you and put them into whatever store you're using okay that's I was next

81. 50:00

    thing so yeah so I I can just whatever my data set might be I can go in and I can just set it to check you know even every hour if I needed to or whatever might be yeah absolutely absolutely absolutely and and so where is this model actually running in like reality it's running on on the Node is it the model is not so the we are purely focused on the aggregation and structuring of data um and building those like data pipelines when it comes

82. 50:33

    to model delivery um we see that as very nicely commoditized within for example the near Hub where you can access any model that is specific for your use case I'm just using uh local llama 3.1 just because it's easy right now but you could similarly Point your uh inference to the near Hub and use one of the many U models that are available through the L okay cool yeah that that answers my question for sure great any other questions from the

83. 51:08

    community or you guys yeah I mean I didn't think say a question I think it's more of just a comment and so maybe it might help frame it better for people who are the hackathon just wanted get in this anyways so what you just showed here is not only how so you can easily start scraping data you get the note up and running when you actually then uh start everything up you actually are putting

84. 51:38

    together a full web interface that someone can then interact with directly correct is that available to uh open as like an API within your app so can I can just can I make calls to that directly or do I need to now program something else separately is that built into that code set where I can just call that as an API um the what do you mean specifically um so so right like there a chat if I

85. 52:05

    don't want to use the chat interface if I want to actually just make the call separate to have have that because I want to display it someplace different or I want to just do something different with that response and that data can I do like have a have a smart contract like do something based upon the uh the model I suppose yeah absolutely you've got two options you can build your own wrapper around the the node API um

86. 52:29

    similar to what I was showing you here and you know the web straper is also available so I could use this to scrape um all of the current data from the memes page on Co market cap and you can see it's got like the current price for bunk with so on and so forth so any of this data is available for direct access from uh the actual node API um so if you run that locally or in the cloud similar to accessing an ethereum rpcm Point all that functionality is available to you

87. 53:00

    we will have very soon a serviced API which kind of wraps our protocol so you don't have to run a node um we're looking to ship that uh in the next week or so and you would get an API key um and then you'd be able to just use the API direct like a centralized service similar to using infu or block D to interact with a typical kind of smart contract fa and one more question I do have

88. 53:33

    actually is so we talked about like web scraping and getting data from Twitter say I wanted data from like a specific API that's provided and it's not displayed on like any any web page or whatever are you able to Ping any sort of arbitrary API with this system or not great question um at the moment we're focused on uh data sources and M points that we found the highest amount of

89. 54:06

    product Market fit with um so to answer no not any um we are looking at over the next month or two which isn't be time to tack on opening up like a platform so as a developer you could contribute a package to the protocol which adds a plugin for another API for example like I had a request bizarrely last week for an only fans endpoint you know gambling and sing in web three is like really great product Market fit um and they're looking to prepopulate only fans profiles on a

90. 54:39

    particular app so that's something we're we're exploring integrating um but similarly you know we' like to be able to have people contribute those scrapers as well to the protocol very cool interesting uh one of the questions came up in regards to that though is in uh do you actually need the Twitter based plan for this you know yeah no you do not so um you do not need an API plan um you are using our infrastructure that

91. 55:09

    extends the API uh if you're running locally like I was you need just a normal user login with a pro account which is going to cost you about seven bucks a month um it will give you the same limits that um the I think it's Pro account on the API which is 5K a month so you're getting 5K a month value for $7 right very cool yeah right now and it's using your infrastructure is free

92. 55:40

    at the moment right right right right cool the amount of questions we're getting here from the community seems like uh they're quite interested in this so you might be getting a few submissions that's great that's great I'm excited excited see what everyone can build and you know this isn't just limited to building an agent the amount of data that you can get from these sources also lends itself very much to machine learning so if you're thinking about trading strategies or doing

93. 56:13

    regression analys analysis from like a data science perspective there's some really interesting modeling work that you can do to really understand like what price trends are doing what particular sentiment is happening in the market so from like a you know D gen product perspective you can imagine some of these data feeds going into you know alongside your uh troll boox you can have you know data streams that are kind of feeding in the best content from K so

94. 56:40

    you have a dashboard there's just a a NeverEnding way in which you can use and render this information that's outside of the kind of agent class as well um and is there anything you spoke about quite a few good like ideas there and what you can do is there anything you particularly want to see build like what would what would be like number one on the list of submissions that is a great question um I think the space is is really

95. 57:18

    missing real user facing product uh that has a very clear value proposition uh I think there's a ton of benefit from taking large amounts of noisy data large amounts of noisy information and making it very concise and easy to understand so what do I mean by that we are constantly bombarded in the web web fre space with noise I think there is an incredible way that you can build an agent that takes a range of information from the space

96. 57:49

    filters it down and makes it consumable and understandable in a way that anyone can get access to trust worthy information um so an agent that really delivers a lot of value to a specific segment of users is what I'd love to see and for those users to become really sticky with that product and engage with the product you know lots of projects out there building agents but would love to see an agent built that has a thousand users or a hundred users that

97. 58:20

    would be a massive win awesome awesome yeah I put on the screen your actual Bounty for this event but you know again I think it it's goes hopefully without saying that you you don't have to just build for the hackathon uh definitely check this out uh as you've shown here it's like 20 minutes to get up and running and to start exploring so you know if you're just hacking around on the weekend uh

98. 58:45

    it's worth checking out and it's free absolutely and use some of the open source Frameworks like uh open web UI like really cheat your way through to success um and I would strongly recommend using AI to build AI like inside our organization we use cursor which is an AI IDE um it will 10x your velocity and make this even easier to consume so use tools like cursor cursor.

99. 59:12

    a I think is now their domain um and and use the the AI to build AI it's like what we should all be doing yeah yeah for sure I I will second that I that's a great tip yeah yeah it's been fun awesome this was this was awesome Brandon I I really can't thank you enough for taking the time and walking through this thank you guys oh no no um answered all the questions I think one thing I did want to say is like we've we've come to the end of the hour now but for me it's felt like it's

100. 59:51

     been like 20 30 minutes cuz it's been so exciting like learning about all of this um so well this is my first teach um Joe and Owen I've actually made it this time so it's great to see what bounties are available and it's it's really fun to actually dive in and see what we can build in a such like short space of time that can be fit within an hour but well in this case it was more like 20 minutes

101. 1:00:23

     that we're actually building for yeah yeah and you can really see how you can get into testing your product with users so quickly you can get feedback really quickly from your users and then turn that into like tight iteration Cycles like almost daily so in this month you can go through so many iterations really quickly and really establish a product that like works for your users and you learn really fast and i' very much encourage people to not sit on their

102. 1:00:49

     hands ship something as fast as possible get people using it learn what those users want and like it'll be a lot of fun um there's been a lot of information today if there's any questions please join our our Discord Community um we have a very active developer community in there if you've got any questions having any pain points getting up and running please come in there and ask on our massive Discord um

103. 1:01:12

     and uh do you have a link for that by the way yeah give me one second um let me just get that and if you want to submit a poll request for any suggestions if you've got any questions please submit an issue to one of the GitHub um repos and that's a really easy way for us to like respond we we live in GitHub let me get a an invite link here sorry guys this is taking me a long a minut should to get a Discord invite link no problem second I feel like this should be right

104. 1:01:46

     on our right on our website there we go is there nice one sec and there is the link I just sh awesome right guys and thanks for the opportunity you know it's really great and really nice to be able to um dive in and talk about this in in detail yeah no I I mean this was this was exciting and uh as you guys continue to add more stuff and continue to add new features and I would love to have you back on some time and keep you guys are doing yeah we haven't even touched

105. 1:02:24

     like Discord scrapers if you've got a Discord Community you can add a bot bring your box open and then you can start getting conversational data like this is a great example I'll finish on this like we use um our own Discord uh data feed to understand product feedback from our users so we have a discol bot that pulls data through the protocol um into an llm and then we use the llm and

106. 1:02:50

     data feed to understand what features of the product what chain points are people seeing like what bugs they're having like what is the sentiment of our community and digest what is a huge amount of information coming out Discord and make it very very concise I think that's like again stress the point how do you take really vast amounts of information and make it very easy for people to consume that's just a massive value proposition it's nice to see when when

107. 1:03:15

     companies are actually using their own product for them for themselves you know that's what it's good I think what you touched on with like the Discord but I think maybe even the AI governance could come into play there where the uh like funds are distributed for and DOW for specific things based upon how the community is is like interacting thinking exactly exactly exactly yeah I love that Limitless

108. 1:03:45

     things wow making governance autonomous Bas that's fine there's probably some interesting civil effects that you got to think about within that kind of that's yeah an anti-il llm that would be an interesting yeah all right uh well great I mean again Brandon thank you so much for your time today uh this was great we encourage everybody to go check it out take 20 minutes and get up and running all right thanks everyone any questions let us know happy to support and um

109. 1:04:17

     thanks for joining us today perfect thanks again take care see you uh awesome awesome training I'm glad you got to join this one Owen uh great uh you know I want to encourage people to check out the hackathon again it's fully online you do not have to be in person you can check it out it's all the way till October 25th there's plenty of time to still jump in uh sign up for our newsletter here at devhub you'll have all the updates of what's Happening uh

110. 1:04:48

     when mastera drop something new in the ecosystem that people can start building on it will be in our newsletter uh so make sure you're keeping up with that as well well uh anything you want to sign off with here I mean I know you're out there at The Hacker House how's that looking yeah there's um there's a good vibe out here in changmai a lot of people just wanting to build and it's

111. 1:05:13

     quite nice like chill Vibe um I mean we're in the early days here but I'm interested to see like this sort of community get together and build in person over like a longer duration uh because I think that's when you really get the the ideas flowing and it's like we're living in an online world right it's it's more difficult for me to ideate with with someone uh but we well today this evening we've just been sitting down and and spit balling really and it's great to get feedback on like

112. 1:05:54

     what you've been thinking about since the uh since the last inperson event that's awesome so yeah good good vibe here very cool very cool well we defin look forward to more updates uh we will have more workshops next week more amas keeping all the way through till the end of October here uh if you have any questions or anything you can always join uh the near devhub linear developer

113. 1:06:18

     telegram channels uh we have dev's like Owen there to help support and anything else may come up please uh reach out to us but until next week this div Hub live I'm Joe this is Owen and we will see you then well