Data Platform Community Meeting (Jul '22)

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:10

   cool well welcome everyone welcome in um today is another uh lovely dev platform uh i'm sorry data platform let's start it again let's let's all right cut we'll edit this we'll edit this part out hey welcome everyone to another data platform community meeting uh i'm joined with vlad and tiffany from pagoda they're going to go over uh data um data platform things uh we're gonna we're gonna be talking about uh just

2. 0:46

   kind of recap and why indexes aren't important uh the relationship between between the index or framework and like framework um also gonna be chatting about lake framework for python uh and also go over more of the enhanced api features that we uh kind of discussed a little bit last time um so yeah yeah that's all we can stop it here thanks everyone tune in next time yeah if uh yeah again uh same format as always if you're new um just drop a

3. 1:22

   question in chat um drop uh um or or the q a we actually prefer the q a button but if you don't understand that's fine too uh but yeah drop a question at any point in time and we'll answer them either as they come or we'll find a moment to to answer it uh um during the presentation or afterwards um so yeah tiffany are you presenting uh a slide or a slides cool awesome yeah uh so yeah so like what josh mentioned we have a couple of items uh the only one that was added basically is the alerts uh demo uh

4. 1:59

   and we actually just released the alerts internally with a couple of the add-on feature from the last demo so you know looking forward to that um and without further ado i'll pass to mike to fro to discuss more on indexers yeah hey everybody i hope you haven't forgot about the indexers but for those who are just new to the ecosystem i decided to actually remind everybody uh where we are standing and what is the indexer and thanks for a question in the chat i saw what indexer

5. 2:32

   is so it's a right time to answer this question once again and you can find all the dogs i love the dogs they they're very friendly and also covers all things in details but i'll try to recap and just uh bring some essence uh to the table today uh so essentially when i will start with the problem uh that you can may face so the blockchain is quite a isolated environment and it doesn't want

6. 3:10

   to share their data it actually doesn't need to do it so the only purpose of the blockchain is that is to actually proceed with uh decentralized consensus being done and moving forward from like block to block including some changes from different parties and arriving to the next and like iteration of the data view so all the ecosystem is on the same page basically and indexers are the tools that and like approach or like ideas that basically

7. 3:50

   uh is it's very simple you just need to know what those changes are and the blockchain itself is not very friendly to do so so we actually started like when we were facing this problem as a explorer team initially we needed some way to record all the changes in the blockchain into our positive database in order to then uh efficiently serve them through explore ui uh so let's uh let's move to the next slide uh and i i'd like to uh present you with like how

8. 4:30

   indexers like a system look like um you could you can basically from the high level i would divide divide them into two pieces like first the tooling that we are talking like the the term of indexer is a bit overloaded by this time because sometimes people refer as an index indexer as something like a just a two like a a library or a helper to implement things uh but others at other times people refer to some specific indexer that was implemented with a tooling uh

9. 5:03

   that does something so on the tooling side we have we currently have basically two of them um it's the index of framework and like framework i will dive into details a little bit in the next slide but for now let's just remember those and go through the various implementations i just want to highlight a few that uh by by this time we have probably a couple of dozens of indexers that i i'm aware of

10. 5:33

    i decided to start with some like very very old ones so near index f4 explorer was pro like was the second um um indexer that we built the ver actually it was third but the the very first for explorer was not really like a proper indexer i would say it was very hacky solution there uh so the very first indexer was actually for wallet uh with the from there we actually iterated and implement the whole excerpt for explorer for all the needs to have all the data in the blockchain

11. 6:12

    dumped into the positive database and the from there once we tested it ourselves we were uh happy to actually share our tooling in the form of indexer framework and the very first community implementation was flux capacitor at least from what i know and they built the the indexer that allows you to selectively uh put some of the blockchain events into mongodb and then run from there so it was powering mint base and a few other

12. 6:50

    solutions over there uh it's it was also great uh and since the time we evolved and like we did a lot of experiments and we we always felt like it's we wanted to make the indexers much slimmer and smaller but the initial index of framework implementation didn't allow us to do so uh and uh because you you need to run the full uh near core node and it's it's uh it's not as heavy as like bitcoin node or theater node uh but it still

13. 7:28

    has quite high high bar requirements uh to the to the um storage specifically so we require a near coordinate requires ssd uh performance storage and a lot of that so if you want to have something like all blockchain data uh indexed you need archival node data on your server and it's terabytes of that in sitting on cd so without further ado we actually implemented this near lake index server which is very simple uh idea we put all the blockchain data that we collected

14. 8:06

    through indexer framework uh to a s3 aws s3 solution and from that point there was a whole family of new indexers based on display framework um we call them just near index near dash indexer dash something on our github so you can find at least six implementations by this time which do like index events um account balances um ft's and stuff like that basically so let's uh switch to the third slide at the next slide uh and uh i will i'd like to give you a

15. 8:47

    comparison of like how things actually fit together so both of the approaches of index framework and late framework allow allows you but like the basic baseline is to actually follow all the blocks and transactions in your protocol so that's strong yes on both sides though late framework is a centralized solution uh as you can see from the second line uh which affects the the fact that you

16. 9:17

    basically either need to run your own la lake indexer which is open source and you do do so just just fine but if you use our like pagoda offered data buckets you are limited to mainnet and test net networks because we only run lake indexer for those but the the benefit is enormous and we saw like cost infrastructure dropping like 99 uh from over a couple of thousand uh usd down to a couple of uh bucks like 20 50 bucks uh for the whole like infrastructure

17. 10:07

    uh which is a huge improvement on that side um also is amendments for the indexer framework you need to follow all the near core releases we had a new release just today and by monday you you gotta be updated on if you run indexer framework on mainnet uh on the lake side you don't need to do anything like if you have your indexer running it will just run just fine for you because we will update our indexer

18. 10:39

    like implementation which dumps all the data onto s3 and all the like upgrades will be uh just zero down time and like no done time at all um but and then when it comes to starring you note your indexer it um or is such a good strap in it it takes you days to sync uh with magnet and test net sometimes uh on the leg side you just spawn it and it's immediately there it reads the data

19. 11:16

    from s3 storage um local development is also cannot be easier than it is with lake because you can run the nose you don't need heavy heavy uh hardware requirements uh as the near core has however for localnet um indexer framework is totally fine it's it's not like the the empty near network is super lightweight and it's not that hard to do but if you depend on some network state uh that would be a bit challenging

20. 11:55

    and well the um the requirement that indexer framework imposes to you on you being the solution based on top of near core is that it's only available in rust which we love uh but we saw that as a limitation for some people um so if you are into um using python javascript or any other programming language you would be out of luck with inductive framework sort of by design and while on the lake framework side your by design can use any programming language

21. 12:33

    which can do http calls because you just need to communicate over http with this s3 uh buckets so and this is a teaser over here you or well joshua is followed but we never talked previously about python implementation of lake framework but actually that's what we want to mention today that we have the implementation sitting there it's even published on pi pi and you can run with it we are still um

22. 13:07

    testing it and like fixing some uh potential bugs we are not aware of any yet we we believe that it's quite stable but there are some um glitches in the network that could still cause some issues which we need to but i'll test further so far it is quite stable at least on par with javascript version i believe yeah and uh we encourage the community to build other uh link frameworks so i i can only say that

23. 13:41

    it's just a couple of uh well not a couple of hundreds but uh maybe a thousand of lines of code basically um that javascript and python implementation took to get us there um so you can contribute your own implementation to the community of lake frameworks and uh maintain it you don't need to be officially pagoda near or whatever it's at the community effort and that's yeah and the reason i'm not even talking about the specifics is that it is just as any other um

24. 14:25

    framework implementation this client as in rust or javascript we have tutorials on our indexer or near indexers.io website so you can find nft tutorial in javascript and python um we actually don't have one interest yet uh because we believe we have other tutorials in rust that just similar um so we can find all the tutorials on the website and yeah it's it's those will be just as

25. 15:05

    familiar as the previous uh like frameworks uh if you've seen them cool i think that's all from the lick from the indexers landscape and i i like to pass the word to tiffany yeah um thank you bro and i think there are a couple of questions or um or things in the chat do you want to address them first or just may ask you to yeah yeah i can go over them um one question is is there any chance of near lake using machina as a target for data

26. 15:45

    yes so the very like um first day makina is launched we will be there for sure like it's it's definitely the the direction we want to explore though um still wait wait it's not on lake or indexers blocker it's very straightforward to actually run that cool um could you provide some details on how you come about the 20 month fee for using airlink yeah sure so the the the major cost there is from a

27. 16:27

    double ss3 storage so we configured our uh buckets in the way that the requester pays uh for the usage and the in order to listen to all the blocks that are coming so that they are stored on s3 we estimated how much get requests get object requests and how much list objects requests we would need to serve two s3 from a three and our estimate is it's uh 20 bucks you can find all the details in in the readme

28. 17:07

    uh there is like a very nice breakdown uh and the from the terms of the uh node requirement like the actual instance that would run your in your indexer um you can even use some free instances out there which are like less than 500 megabytes of ram and share cpu power that would be just fine and just enough for rig like to follow the network if you need to so to do some complicated uh evaluation along the way definitely

29. 17:45

    would need a heavier machine um but yeah it could cost you a couple of bucks a month or a bit more but it is definitely not in the realm of the requirements uh near core node would force you to have so yeah cool yeah thanks for um posting those resources as well uh links in the bottom okay we'll make sure we'll add those to the youtube video when we upload it um a follow-up question to that is is there

30. 18:19

    um a time where in the future where we might be able to pay that fee with usn oh yeah so the the future direction graduates towards that we will actually uh repackage that into pagoda dev console and we will reduce the cost even further and the billing will go through the console and it's probably gonna be much cheaper because the ecosystem will grow and reuse the same resources so we

31. 18:54

    will need to hit aws as much and we will cut a lot of costs on on the shared resource basically um yeah one quick question i have on that actually um is we we are discussing you know enable payment with usn um with uh within the teams and one question i have is uh you know because enabling recurring payment with crypto is like a very unsolved question within industry and sometimes you know maybe people don't really like having

32. 19:26

    recurring payment with crypto with access key and all of that so just a quick like feedback getting uh do you guys mind of if it's not recurring and you get a notification and you pay for it with usn or with cryptal um so really welcome really welcome uh chats and just you know let me know if that's okay then you know that that's just like a really great cool cool awesome thanks um in addition to json are there any plans on supporting protobuf as output

33. 20:01

    is output of what sorry um malice can you clarify that question please yeah could you follow up that uh data so let me unpack it here so indexer uh mirror lake indexer reads the data from the indexer framework from the blockchain basically and packages it into json and stores those json files for about blocks and chunks on s3 and then any of those languages that implement those framework like client libraries can read it from s3 and 3.json and

34. 20:51

    repackage that into native programming language objects to work with and we so far we didn't plan to use anything else because json file just worked for us i would love to hear why you believe like protobox to be better cool uh is there a business model to monetize data with lake yeah i can speak to that so um we don't have uh the immediate plan to monetize at the moment but it is it is part of the plan to

35. 21:36

    um you know just like what pro mentioned right now you need to go on s3 and you know pay there and uh and all of that so we're we're actually hoping to making that process easier with a whole pagoda or like in your account uh on dev console and enable everything from there and you can manage your billing or all the services that you interact with the pagoda products there um and we're

36. 22:02

    planning on offering you know even like a better pricing for it um so it should be even cheaper and you know we hope to enable crypto payment like usn so that's even more convenient so yes there there is a plan to monetize but the model is not set but you know we definitely hope or like at least a goal is to make it just easier better and cheaper for for all of you cool awesome i think that's the end of the questions that we have so far oh i think uh one more are you going to disable the direct data

37. 22:36

    consuming from aws s3 after billing on dev console no we don't like it i think it's it's fair to keep the um sort of s3 access open and we have all the current tooling using the data but the the idea is basically that we can provide much better tooling and even cost efficient solutions uh so i think it i like it it will just eventually converge to the using pagoda instead of direct aws access would be just

38. 23:21

    better basically in all aspects we can provide you with some monitoring so if we if just for example if we would see that your indexer is not making queries to us we can alert you and say like hey probably something wrong with you indexer i i i'm as a developer i would love to have all those like nice features um built for me so i don't need to you know worry about monitoring of my indexer so features like that could be very

39. 23:52

    beneficial and we will definitely consider it to be one of those cool yes okay moving on to the next slide which is the next topic is the api landscape so um some conversations we've been having is you know there's too many places you can you know interact or consume primitive data from the blockchain um and there are a lot of there are a lot of things that we would hope our rpc uh calls will be able to achieve or you know just make everything easier or better or like you know easy

40. 24:30

    to consume or interact um so we create we introduced the enhanced api suite last time and we identified a lot of the primitive data needs sort of uh well required like for a regular pc archival rpc you have your customer pipeline to grab like a bunch of data from a bunch of places like prices and here and there and you need and now with our enhanced api it kind of you know maybe adds to this not skip

41. 24:56

    even to make it more confusing but so as a result we're actually going to um change this by offering this pagoda api suites all together which by by the user experience perspective it basically means we're going to give you a one-stop shop of all this kind of api most common or needed necessary needs on their data um and you know interact with the blockchain with the rpc with one endpoint so it's going to be something like

42. 25:25

    api pagoda testnet or maynet um we'll release that name once we really figure our finalize but it should be one endpoint um that enables access to all of this um and we're actually working on together with ro here also um to provide to to unify our regular pc and our cover our pc access so you don't no longer need to you know have like oh like this five app hogs i poke here and after that you know for prior and all you know history data i need to poke the the i cover note so we're going to all

43. 26:00

    offer them all in under one end point um so yeah that is the landscape change that we're very excited to bring you um and other than that for enhanced api were so the first batch last time uh olga introduced that um we're working on a couple of balances um and right now we're working on supporting legacy tokens and you know and a lot of a lot to make sure that offering is is complete and is as helpful to the ecosystem as much as possible without

44. 26:31

    you know the extra custom work uh that needs to be done and for the future batches we are going to offer like i mentioned um the price apis and combined price with you know guest spend with like transaction value and a lot of those to give you even more and better information um on the amount you spend and then we'll we'll also want to have more detail breakdown on account and contract states so you know right now uh there returns a lot of a huge strain

45. 26:59

    from the rpc call on accounts and contract state we hope to break them further down so that you can it's easier for you to consume and you know you don't need to manage or process that afterwards yourself um we are looking to provide more information account about account and transaction history given the long history of our blockchain amount of data there needs to be some sort of creative solution but that is definitely in discussion and we we definitely understand that this is a

46. 27:24

    very important um idea to need from the community and we also is planning on we're also planning on supporting logs or traces so it's helped you you know easier debug and we are thinking if it can be you know connected to other tools or calls to to make that you know the whole unified debugging process easier for for you all so yeah that's the plan for enhanced api we really welcome more suggestions or like you know or uh helping us prioritize what to support next it's a it's a lot

47. 27:55

    that we want to work on i want to provide so really welcome all the feedbacks that we can get from this and i see a question in the chat uh i think it's more of a more of a yeah it was pagoda super api uh yeah it could be for for for a premium uh price yeah so um so that is that is a that's like not really for super api as a price um but we do have a plan to start monetize um on on this front but uh we'll start with the regular rpc um

48. 28:35

    and you know the rpc yeah it's a spoiler alert so rpc is uh it's i think a couple of people probably already know um and you know we're we're half that we have to plan to um gradually you know supporting better our pc experiences but you know it but also from our perspective we're we're starting to create a monetization model for it um we'll have more information about it gradually and more but as as a as a result of that charging we definitely also want to promise a better service a better experience and more

49. 29:07

    powerful functionality so yeah that's the swole alerts but it will you will still have the free endpoint access until then so no worries about that um and we'll have more information done cool awesome thank you and the next topic is actually going to be a lyrics demo that we uh talked about a little bit i'm gonna go over to this site here so i'm already logged in uh for anyone who don't have a pagoda account

50. 29:39

    you know what yeah for anyone who don't have a pagoda account it should be very easy to i'm just going to log out and redo it so um uh am i sharing i'm still sharing the screen right yep okay cool um so yeah if you log into your dev console um and this is you if you don't have an account or you know you can sign up easily i already have my accounts like well i just connect with

51. 30:05

    github it's just easier um for me to do that and i have alerts test project that i created and there are a couple of tabs right here um well first of all very selfishly we actually embed it near lake into depth console too so you have the indexer tab on and within dev console um that you can just you know look at all the introduction that fro just talked about for the value propositional link

52. 30:31

    you know how powerful it is and how easy to use and you will have like the you know a couple of links um that you can just directly go into the most helpful uh examples and um and like try out near lake go ahead directly um and for analytics tab you uh probably talk about it before in other meetings but you can um also add your contracts within uh within this for example i can try you know for us the marketplace

53. 31:04

    oh it's not okay i actually don't remember what it is but you basically yeah you can you can add your contract here and then you will show up uh analytics tabs of your contract um how it's performing and everything but that's not the point of this demo the point of this demo is actually um demoing about alerts so i actually have alerts set up for successful transaction i'm just gonna delete this because i was

54. 31:27

    testing it um so let's pretend that doesn't happen uh so how you do is it's very easy it's just like you know hit create new alerts you go into alerts uh target address i'll put mine in case anyone wants to send me more token on testnet um so select condition we actually now and from the last demo we uh we actually have a lot more supporting this time uh we enabled events locked uh on on top of this alerts and we also enabled account

55. 31:57

    balance change and account balance change by percentage um so all of these are sorry uh all of these are the newly supported feature that we worked really hard to support on you know all the way from having more indexers to support this from the back end all the way to you know enable this from the front and the easier way um for the simplicity and you know i just don't have it's a it's a account uh account address

56. 32:22

    so i don't have a lot of things um to be able to demo here so i just use the simplest one on successful transaction and i'll enable this by adding a destination so this is also very easy you have a web hook or you have a telegram um select and you can easily enable this by just you know scan a qr code or just directly open the app uh you open your telegram and it will just pop up uh which i think can anyone see

57. 32:49

    this telegram bot yep cool okay um so yeah so you can just you know there's this def console def bot it's going to pop up and you know you are going to get your alerts from here um so i'll just enable this destination i'll create alerts what i'm going to do next is go on a test at knowledge this is like the tiffany testnet 3 account um and anyone want free test net tokens submit their uh addresses if not i'm just going to

58. 33:19

    send it to pro okay hey ben uh i'm giving you i only have six here i'm giving you three i'm very generous you know that uh and benjamin dodge has that okay send i'm giving you half my tokens ah and sending excuse me keep coughing okay awesome so it is processed here um let's look at the bots and see if it's actually um responding yay okay popping up you have successful transactions here yay

59. 34:04

    triggered and we have the reason it's popping up multiple times is because you know there are couple processing that um that's happening there so each of them refer to actually different receipts um so you can look into the details of your transaction but yeah so that's pretty much the alerts demo uh and definitely go in there try out we're going to have a public release um we're going to have a public release uh

60. 34:28

    you know starting next week on you know and you know it will be available for all the users on top of dev console um so this is a spoiler um so whoever is here you know you have the first see first view of the new alerts uh that we're gonna support so okay going forward um last slide i have here is pagoda user survey so our our developer uh designer team actually created this survey um and you know corvin our

61. 34:59

    designer actually created a forum post so that everyone can actually look at why we want to design why we want to do this survey and what we want to know about from you guys we really value the feedbacks and uh and suggestions and all of your experiences the survey is actually very simple and easy so just go in and they're only like you know a couple of questions um it's enabled with a google google form um so you know you can

62. 35:25

    just you can just answer this question selectively and really value that it's going to be super super helpful for us to really structure uh pagoda products forward and this is really you know going to be affecting across different products offering so we really welcome all the feedbacks and participation here so what we'll share the links um i'll actually post them all of them right now in the chat and we'll

63. 35:48

    also share the links after the call um in other areas and places and that's all for our presentation and i see one question about can we log in with your accounts or go to dev console um yeah so that is a great feedback we actually uh have been getting that feedback from a couple people and we're we're working on enabling anonymous login to dev console and we have actually created a github discussion about this so we'll love more discussion and

64. 36:21

    follow up on uh the status that we are going to have there i'm going to find that right now cool um yeah any other questions please feel free to drop them in the q and a section um yeah is that is that all the uh uh all the items that we're going to be presenting today you're still sharing your screen by the way tiffany sorry i was trying to find that repo we'll edit out your personal email stuff

65. 36:58

    yeah i'm really trying to find that repo no problem but yeah if you have any questions please drop them in the chat um we'll give it a give it a minute or so to see if anybody else has any follow-up questions before we conclude yeah no exciting stuff a lot a lot of cool things coming coming down the coming of the pipe uh any chance of having docs in spanish um yeah we do have like a crowd in i'm not sure about near indexer but i know

66. 37:37

    on uh docs.new.org we do have a crowd in uh translation feature that um community members have been helping translate um but yeah definitely yeah please please open up a pr would love to hear have some more community contributions uh uh for translating uh our documentation my spanish is on the level of uh uh um noble espanyol yeah which means i don't think i don't speak spanish and like

67. 38:09

    yeah dev the the folks on devereaux like we have five individuals that speak fluent spanish um so yeah hop into office hours if you if you speak spanish and wanna uh and want to talk to natively um yeah there's there's a bunch of us that um are definitely fluent in that language and um yeah could definitely help um with any pr's that you have um are there any problems for near lake framework assembler

68. 38:40

    yeah a lot of problems it's too low level all right any any any other questions cool i i i think we did it i think i think uh i think we answered all the questions that you have yeah and if you have any follow-up questions please uh yeah come hang out office hours um we'll upload this to youtube we'll also have links for feedback um that you can participate in when's the airdrop sir

69. 39:25

    yes uh yeah um other than that thank you so much tiffany and froel uh for another great data platform uh meeting and yeah i think next month what's that yeah i i i was just saying that it's it's a great opportunity to actually uh talk with fox and uh learn more so please reach out even if you are just uh watching it as a recording we are always present in the communities all over the places and if if we suddenly miss something the royal team will definitely help us

70. 40:05

    uh to like to be brought to the right discussion and helped you out so i am i'm excited to see more and more projects um building on top of all the tools that our our team is working on i am it's always a pleasure to see your success vlogs so go community absolutely go community and go to team all right have a good day everyone we'll see you next time