Data Platform Community Meeting (April '22)

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:12

   cool well um yeah let's get started hey i'm josh uh devrel at near protocol we are here doing our last community meeting of the month uh but not the least um we're really excited about near indexers and the data platform team that's going to be kind of doing a presentation for us and yeah answering any questions that anybody has on all things data when it comes to near protocol so yeah go ahead and do an introduction i'll start out with tiffany

2. 0:45

   yep uh hey guys i'm tiffany i am the product manager for data platform team at near protocol and i'm very excited to be here uh i'll pass it to fro cool yeah hey everybody um and today i'm as they i'm with with you guys in the role of data platform team leader uh you may have seen me a few weeks ago on explorer team as well so as you can see i'm jumping around trying to be everywhere and josh is definitely

3. 1:24

   even more than that i am so happy to see everybody here and hope you enjoyed this call i'll pass it to boyden uh hey everyone yeah i'm bahdan i'm working a data player from team mostly working on indexers so like and docs and everything is related so i'm passing to olga we will like introduce ourselves more during the presentations i have hey everyone my name is olga i'm dating engineer here at mir so welcome everyone

4. 2:03

   awesome go go team uh yeah just if you haven't uh attended one of these sessions before so at the very bottom of your screen you can see that there's q a section um there's also the chat so feel free to post anything in chat um if you have like a specific question fire off a question at any time during the presentation and yeah we wanted this to be more of an interactive um session with getting community feedback and

5. 2:26

   that's ultimately why we are having these is to get feedback from our community help us shape how we build things what we prioritize what's most important to you and help clarify any questions that you have about any of the tools or features on the platform so yeah at any time go ahead fill out a question in the bottom and we will answer it as we go and with that i'll pass it off to

6. 2:51

   tiffany to start the presentation that's good can you also see my green all good at least weekend yep sounds good yeah um so we'll kick off with a team objective for for the data platform team uh and we have top three objectives here uh first one is we want to make near the most accessible blockchain for webster developer we're very aligned with the tool team on this front because you know we interact with um developers our out

7. 3:24

   of our products interact with developers they use it on on you know a very frequent basis so we definitely want to make that experience as much as easy as possible for you know anyone to abort to use it um the second one is enable product based decision making for data platform products um and this really speaks to we really want to get as much of the feedbacks and engagement with our community as possible to really build the products that our community love and

8. 3:49

   like and needs and the last one is power on your project with stable accurate and um extensible data infrastructure so um really want to maintain our quality for our products and really deliver the you know the best accurate the ones that are going to be used for all kinds of cases and uh for our project and you know no matter where you are no matter what you do we want to empower that for all our developers out there and from there i'll pass to bogdan for an introduction of the history of

9. 4:22

   indexer yeah thank you can you open my slide the next one i guess sorry it's not really sharing yeah it's sharing but it's not like switching we can see it on leading objectives currently um let me do a ratio yeah that's right yeah meanwhile uh like a little introduction like uh i'm here to tell like a cool story about like the entire history of indexer what it is how it like was actually invented i would say uh yeah but

10. 5:04

    we can't see the screen unfortunately oh like i i can try to get the slides as well yeah i can i can try i can decide yeah i'll give him a second can you perfect it yep yeah we can see it yeah how to start it you know oh scishow yep oh yeah uh the the button to the right uh slideshow oh yeah there you go i couldn't see it like no worries so yeah let's start uh the history of indexer like it has begun in year 2000 uh i call this year as a near

11. 5:45

    stone age because at this moment near has only json rpc and everything like and nothing else uh at this point uh the project explorer like i hope everyone is like seeing and using it uh the explorer was the most like the only consumer of near data at that moment which uh like showed everything to people and at that moment uh for all as like an inventor of explorer uh created a pool model so-called pull model so

12. 6:20

    explorer was like asking jsonrpc every second for a new block and trying to save this into the like internal database uh for further like show up uh in the explorer pages uh it was like uh like it was working solution but not the best uh like the next uh step in the history it's like indus industrial uh age uh it's year 2000 still uh but i have joined near at this moment and uh for all

13. 6:54

    like with all the tricky and uh like sneaky way he actually uh like can he propose me to do some rust and at that moment we have started to create indexer framework so like i will tell you what is indexer framework a little bit later but the main point i wanted to emphasize here is that after we have like built uh indexer uh it was like uh used by indexer for wallet which is deprecated currently and we were like entire 2000 like year 2000 we were building an indexer for explorer

14. 7:32

    because it was and it is the biggest indexer out there uh at least for now so uh a little bit what is index or framework like uh um on the left you can see uh the like a representation of near core node so it is uh totally closed in itself like environment which doesn't like to interact within like outside world uh with any other opportunities like and uh possibilities except json rpc but we have managed to like convince it to uh

15. 8:07

    like give the data out of it and like this was an indexer framework we integrated there in the near core node to make it possible to read and watch for something from from there uh so like next step in the history like uh it's a future it's a late uh 2021.

16. 8:31

    uh we like we ate enough with indexer framework we saw like it's like like oh sorry i have missed uh the slide that i i'm thinking like something is weird so uh during 2020 first year we have used indexer and running it and we like we found out that it brings a lot of suffer uh for anyone who is using it not because it is like a bad software by itself but like it because it is a

17. 9:04

    complex software and uh like when you need indexer it doesn't mean that you need to become a node operator and it's something you uh have desired before running but you have no other options so the software it brought like it was very resource consuming and very expensive uh like uh the hell of uh sinking nodes uh everyone who was running like who who is currently running uh those know that uh like when

18. 9:36

    you start the indexer and you see this like downloading headers and this percentage is like endless the process is endless uh i remember we had a call with george like in the uh 2020 i guess and uh josh asked like can you show how to create an indexer i said yes of course but like we won't be able to start it because like waiting for note to be fully synced and working it's like a madness and like another step like

19. 10:06

    another software is uh like i i'm calling it like an update maintenance hell because every time your core is releasing a new version you need to upgrade your indexer otherwise it will stack you will need to update it anyway and go through the node thinking process like again and maybe again and a lot of times so uh in the late 2021 we decided that we need to change it somehow we

20. 10:38

    have come up with a brand new name we decided to call it near lake and we put the goals for us like we wanted uh to create something that is uh like simple to use it requires the minimum resources it almost require like it requires almost no none or minimum maintenance and of course we tried to come up with the idea how to avoid the sinking hell and we wanted to make it like a fast start like quick start like just like starting and

21. 11:11

    you're on the go so uh we have created this uh in the like early uh 2022. uh we have created a lake ecosystem we come up with the like we came up with this so lick echo system consists of two pieces like it is leaky indexer which is still an indexer framework uh stuff uh but like like the end user doesn't care about it we're running late indexers by our own we're saving all the data about every

22. 11:46

    block to the aws and late and we have created a lake framework which is like for the end user uh you're using analytic framework and you're getting the streams from aws uh with like with a minimum boilerplate uh so like what stuff we have achieved with the lake ecosystem like uh it costs like somehow approximately 18 bucks per month uh it uses 145 megabytes of ram and it is not optimized yet so it's not the like not the minimum and it is always sync it which is like

23. 12:24

    the most uh like the biggest achievement uh we've got and uh if i'm not mistaken i had some yeah i have some uh example of the code yeah uh i hope it is you can see it so it is like oh yeah this one yeah so uh it is a rust example uh using lake near lake framework uh rust version i have marked the boilerplate part you can see like right here and here is the logic uh you're putting it like it's

24. 12:59

    your logic of what to do with the streamer message uh and we have like a bonus a small example of a new lake framework javascript version uh like maybe it's probably to say typescript version but nevertheless uh the same stuff i have marked only the user logic here so like as you can see it is like minimum code uh and like all this code i have like brought as an example it is a working

25. 13:31

    solutions like completely so yeah uh and uh recently we have decided that it is time and we have created a dedicated website with the docs about indexers uh like everything indexer related uh some information about near core about your data about indexers so we're going to fill it up with a lot of tutorials uh we have already created like a immediate like i don't know explanation of uh how the data is flowing

26. 14:03

    in your protocol uh we have created like uh some articles with explanations and stuff and like more to come so please uh like visit our new indexers uh like share your thoughts and stuff and i guess it's all from me so i'm passing it to and i will be your like host for like next slide please and stuff i'm not sure like what is the best time to answer some questions like immediate or yeah we can we can we can definitely answer questions as as they come uh i

27. 14:39

    don't see any that have come uh in so far just i hate one from mario saying uh congrats on a great project lake is very uh needed addition to near um but yeah uh yeah please feel free to continue to answer those questions or i'm sorry ask questions oh we have another question we have a question from benji how does the speed of the lake indexer compare with running your own inductor node if i was running and node using the index of framework where sorry like framework okay i got the question uh so like yeah

28. 15:13

    uh there is some uh like i would say between indexer framework and lake but it is not noticeable it is like uh just a matter of milliseconds and that's all yeah and i have a question from mario uh why s3 uh it's a tough question actually like uh we uh we went uh for the first we went with the uh how it's called kafka with the kafka and we tried it but we actually our goal was to save history somewhere

29. 15:50

    and uh like the and goal of the lake ecosystem is not to provide uh completely decentralized the like uh solution it is uh uh like for complete decentralized you can use uh indexer framework and nothing we can do to speed it up and we wanted to empower like web 2 developers or newcomers or someone with like not a big budget uh with something that will allow them to get the data from them like from the

30. 16:21

    blockchain as fast as possible uh even a historical one so that's why we end up with uh the storage like s3 so i i hope this answers the question yeah one more one more thing i'd like to um just you know before we move on to the next topic uh yeah frol uh mentioned it in the comments but also yeah i'd like to also highlight that yeah before this we needed you know to run a node or an indexer you'd need an eight

31. 16:52

    cpu one terabyte solid-state drive um you know and you would yeah it would take like days or weeks to sync up uh in the beginning and then the rough estimate was like what 500 a month um you probably want to run two of them for redundancy issues and as you know bogdan uh talked about earlier yeah as releases are coming out indexers would stall you'd have to update it you lose a

32. 17:14

    little bit of time it was just yeah uh not it was not an optimal experience uh and now going to something like this that costs very little in comparison uh the boy the code compared to what we were doing before is so small and it's awesome that now we have a javascript the little implementation there um it looks yeah very easy uh to use and and super yeah and just to justify like uh the boilerplate plate code is almost the same uh we just use another package uh

33. 17:46

    like your logic can be as big as you want uh and as you can like optimize it but uh uh the way it gets the data like uh olga is currently working on the solution and she actually like i i believe she is the happiest person because she is building on lake already and she like doesn't participate in this sinking hell uh and uh like the most annoying thing that you can't develop indexer properly uh with a real data from testnet or mainnet on your local machine anymore

34. 18:21

    like the data is so huge and uh like your machines are so like weak that they can't even sing and this is a problem uh so and currently like if you need to stop it just stop and go back to that block like immediately without any delays this is like a lifesaver that's true i i mean uh our previous version of indexer had this downloading headers uh uh line you see it and you just want to die

35. 18:52

    and now you can you can test anything on mainnet on your local machine and it's not a problem at all you you can just receive a box from the point of time that you want and it just works it's brilliant really yeah the long headers is not the most like the yeah that's the killing part after you get it when you're finally awakened and then downloaded yeah it's it's the amazing stuff yeah

36. 19:24

    downloading headers 100 yay downloading blocks 60 yeah yeah um one more one more question we had from mario um what's the approximate time delay between the block written to the blockchain and the event received in the message listener in uh link framework yeah yeah it's a very interesting question because it depends on what kind of data you're expecting because like if we're talking about uh how like in comparison with

37. 19:56

    indexer framework like how fast you got the data in the lake uh like it's uh about like milliseconds i would say like the the most uh long delay you can get is the two seconds because we make some pauses so if we don't get like a new block but uh if we're talking about like real condition then we have a lot of stuff happening on near core like transaction is sent to rpc rpc is like um routing this transaction to the node uh where it belong and like where the

38. 20:29

    account belongs to and we're starting to execute it there and and so on and so forth and for all uh remind me what's the numbers you you've got uh so let me sum it up uh basically there is a delight before you get the transaction into the blockchain but the question was about once the block is minted uh how long will it take to arrive to indexer framework or lake framework so the time is actually a finalization stay is like the the longest time you wait is for

39. 21:01

    block to be finalized it takes three blocks to finalize and currently main net runs uh roughly 1.3 seconds per per block production so this means in four seconds you will receive it on indexer framework side and then we we take like 100 milliseconds to to to go back and forth through the network uh to put it on the s3 download it on the uh framework side so in comparison four seconds for a delay

40. 21:36

    of like finalization stage and then just roughly 100 milliseconds to get it on the lake side awesome yeah i think we think we're good on questions again oh you're broken do you have uh something to follow up with that no i i wanted to say that we don't have any questions and we can pass the ball to olia oh one last follow-up so the difference is only 100 milliseconds between uh lake

41. 22:07

    and running index of framework i would say this this difference is artificial like we don't have real number for that it's mostly like network uh latency that we count here awesome thank you okay so passing the ball to olga uh so we discussed uh this nice near lake and now i want to discuss with you next future steps and in order to start at first we need to discuss who actually needs this indexer data

42. 22:48

    because we want to to serve it in a structured way and be able to to query that uh so at first our first uh like a client is a near explorer and uh as you all know uh it's possible to to search there for any historical data check the state of the block the account any transaction receipt anything you want secondly we have different wallets our official wallet and the third-party wallets also they want to show

43. 23:26

    all the transactions all the stuff about nfts the information about taking pools and so on and so forth we also have some third-party tools uh other explorers other wallets uh many different startups that just want to to serve the data to communicate with data to collect the data about their for example smart contract we also have analysts that want to provide people with different analytical stuff and run heavy queries for that

44. 24:06

    next five please so as a result we need to run both transactional and analytical queries and they are really different ones and it's hard to find one good solutions for both of them and we also need to give the access to the third party tools it means that they should have the credentials to to run the queries and they should be able to pay for this access uh because it needs to be so the solution needs to be hosted somewhere and it's not free and we need

45. 24:44

    to maintain it somehow uh next slide please so our current solution is postgres and actually it runs it runs pretty good uh next slide it solves many different uh tasks from our side uh at first uh relational model of phosphorus suits best for us because our data is highly structured and it's really comfortable for us to to use all these relations tables uh so we use the primary keys for many

46. 25:21

    different foreign keys unique indexes it really helps us to to check the state and if something may go wrong we will find it at the really early stage it it also helps us to to explain the data to to the user uh i need to say that uh transactional queries run pretty fast even now whenever when we have five i guess in in our biggest table we have 500 million rows now and we still can

47. 26:02

    join it uh query i don't know order by filter and it's really run pretty fast even in in a tables of such size finally postgres is supported really good in different orange we use application written in rest and it's not like a default option and even in rest we have at least two good wear ramps that support postgres let's go to the next slide but as you may notice we discussing all all this stuff and it means that podcast is not

48. 26:41

    ideal for us uh so it has some disadvantages uh maybe the main one for us right now is the speed of insert statements uh it's so limited and we are really close to its limit uh i know how to uh how to boost it a little bit more and we can live with uh with this boost for a couple of months more but it's not about living for two or three months more we want to have a sustainable solution for next few years and that's why it makes no sense to try to boost it

49. 27:21

    right now uh second problem is that now it's absolutely impossible to run analytical queries i have a story here when i just joined near it so it was uh more than a year ago it was a brilliant time to to join here as a data engineer because uh the biggest table was maybe two million of rows and it was possible to run any query that you want so it was a really funny time for me

50. 27:54

    because i i created absolutely crazy select statements i knew everything about the blockchain and i could run anything and then our data started to to grow uh in an exponential way i guess so it was a disaster and and after that i needed to think how to speed up the queries uh and so i i need to say right now that it's just impossible to run queries that i run at that point of time so um but we still need to to

51. 28:34

    to draw the charts as we have for example at a statistic page in near explorer uh you you can see a small star here so now we have a workaround uh we are uh computing uh so each night we are computing uh the statistics for for the previous day so it's just not possible to compute the statistic for the whole uh blockchain uh lifetime so we need to to find this workaround um another disadvantage is that we can't

52. 29:12

    truly give the access to the third-party tools we also we we are giving the access uh but it's not super powerful machine and the connections are limited and so if you are a startup and you want to compute uh complex queries uh and you are ready to pay for that uh sorry but we don't provide any solution right now for you so it's just it's not possible to to run complex query if you're a third-party tool finally postgres is really expensive for

53. 29:51

    us we're running on gcp we have more than two terabytes of ssd we have several replicas to to maintain all the stuff with explorer wallets and other applications so uh it sounds like a big problem next slide please uh so for for these lines with the small stars uh i wanted to provide the links uh so our analytics calculated at the second link and the first link is contains shared public access so it has

54. 30:31

    credentials if you want to to connect to our database and query some simple queries please don't abuse it second slide next slide please yeah so as you understood we need to go further and find another options uh and since uh it's not an option to burn everything so it actually should solve our problems but it may produce a new ones next slide please so we need to go further and find uh

55. 31:07

    other solutions with uh try in other databases maybe other relational databases could suit us better maybe we need to look at columnar databases sharded databases we actually started to look at them we also are thinking about map review solutions the next slide please what we want to do is we want to ask you ask your opinion here so do you have any ideas what do we need to look at at first

56. 31:47

    maybe you know any good solutions that you can suggest us because we are in a in a stage of uh for research and we're trying anything awesome and we have um we just presented a poll to the audience that's currently uh in the call do we have something for those that are going to be watching this recording later so that way they can participate in this poll is there like a page that they can go to yep i'll send in the typeform um also for to share it later

57. 32:22

    cool so we'll just add that in the description the video description below cool so um this is all from my side so i'm really interested in the results of this poll and i can pass the presentation to i'm not sure um so last bit from the presentation is our roadmap for the next three months so going off of from our objectives that we just mentioned at the beginning of the call um for the next three months we're going to focus on really you know we already have the rust version of the

58. 33:03

    near lake framework and we'll continue to work on improving that and also deliver the jazz version which i believe also um is shared initially with the mvp and we'll also be working on that as well as what olga's been just presenting on the warehouse db and analytics um you know database that's fulfilled that needs for our developers and community and the second is we want to create updated content documentation examples

59. 33:29

    tutorials you know you've seen our team our uh you know we have really fun people who really want to um you know benefit our community with you know to answer questions to present really clear simple easy documentation and explanations on how to use our tools so we will be working on improving that documentation link that was presented earlier and the third one is provide a stable and extensible data infrastructure to support for pakoda

60. 33:53

    products so we as pagoda will have a lot of exciting products coming along and you know as the data infrastructure platform team will also be supporting all of those exciting initiatives going forward next slide please and secondly enable product-based decision making um so we'll be posting uh you know community sessions uh with with all you guys and really value this opportunity to have this direct feedback and another one is we want to improve our project management tooling so we can you know

61. 34:25

    share with you more and even you know further on what we have been doing and receive more you know real-time feedback there and we want to create a council of users for research and feedbacks to you know to validate our our assumptions to to you guys to experience the beta and you know get all the feedbacks from there and really improve our products and next one please and the last one is um

62. 34:49

    it's all about really maintaining and improving what we have and make sure you know it doesn't break and you know make sure uptime is up and um you know and another one is identify gaps of data needs that has been happening um so we might be missing something and we really want to know what it is and we want to design a solution with the brilliant minds here um together and to improve your experience and fulfill all the data

63. 35:13

    needs that you have and that's it for the room maps and i think if we go next was it was there another slide i think there is a slide on on our linking to our stack overflow page so just to mention that also um as our data platform team really hope to use stag overflow as our center place for you know questions and answers because we see a lot of them might be uh very you know like multiple people will probably have share some similar patterns of questions and we really want to answer them and

64. 35:50

    you know also ultimately have this self-serve question and search and you can get it real time really fast just searching by yourself we'll always we always be happy to answer questions so you know we have the discord channels um our amazing deferral team helping us with that as well um but yeah so feel free to submit any questions um to the stack overflow page we have and we are really looking forward to answer them

65. 36:20

    cool i believe this is the end of the presentation now we can open it up for any kind of cool uh questions that anybody has so yeah again please feel free to drop any questions in the bottom i see some rolling in uh also i think someone asked a question earlier arvin asked something about a schema being published somewhere yeah boyd already provided the link but i wanted to take

66. 36:46

    it live as well the link that olga shared during her presentation pointing to github page near indexer for explorer uh in the readme file you can find the public access credentials and at the bottom of the readme you can see the schema representation but they also can use any db viewer uh for postgres and it will also provide you with this nice uh relations because we have foreign keys and setup in place

67. 37:18

    so it should be more or less uh intuitive for you but at the same time our representation is a bit more aligned and like layout makes a little bit more sense and the default ski like a layout that your db viewer would present so once again near indexer for explorer let's see how repo you want to find uh to see that schema cool awesome another question yeah okay yeah like uh um about the comparison of the like i don't know of the speed of javascript version with the rust version of your like framework uh like it's a bit early

68. 38:00

    to say about like a speed but i guess it's comparable like any other the same tooling like in written in javascript and in rust like i i don't know how to properly answer it we haven't measured uh i'm not like i don't think there would be any uh delays from the data getting uh part of the work but it might introduce uh like additional uh overhead on the data like processing on your logic like i guess like any rust logic you can write

69. 38:41

    uh to like process and index the data will be faster in comparison with the javascript version just to add here so gs version is um is not the performance sensitive part of the story it's more about consistency and reliability stability of this whole implementation and from our experience rust tools once they are written and deployed they never cause any issues on our side whereas but but a lot of people prefer javascript like why not so that's why we have

70. 39:19

    created it uh and we have another question in this regard i guess like about planned release date uh i can't say we have a planned release date uh currently javascript version is under review uh and it might be released like this week maybe not this week but like next week maybe in the week after the next one but like very soon like it's soon enough yeah it will be just early version of the like early version that we want uh to give to early adopters just like we

71. 39:51

    did with the last version initially that's why but we uh want to so our water plan is to get a rust version and js version out of the door by the end of this quarter um before we do this you you'll see some pre-releases as well another person yeah other question where can we follow the discussion on what the analytics transactional x will become so i'm not sure we have a proper place to to discuss uh which solution we will prefer uh in the

72. 40:32

    end but i have a repo where i'm playing with the different databases i'm also trying to think about a little bit new structure so it will not be like a super another database then we have in a near indexer for explorer so we have some fields renamed we also will add some new columns there where can i share the link to my repo and then yeah maybe maybe the github discussions portion of that repo would be a good place to yes yes it will be i guess it will be the best place to

73. 41:18

    discuss anything and if you have any thoughts please share them we will highly appreciate that uh i guess someone has marked as answered the question the wrong question yeah the wrong question there is another question are there any plans to integrate client actor or view calls and also the form interruption enums for indexers they utilized those tools and we passed this ball to role

74. 41:58

    and for all has frozen i'm i'm still integrate client actor for view calls and also from interruption and in amps for indexers i'm not sure what the question is yeah let me elaborate a bit more yeah can you elaborate or like rephrase it somehow because we can't understand why client actor can be used in the view goals like it's completely different stuff also it is an animus question so like it might be bowen who is trolling us out for example

75. 42:46

    uh well there were a few other comments questions in the in the chat so i'll just shoot them here so there was a question regarding suggestion regarding using s3 json data lake type solutions is a like solution to analytical type of queries and that's exactly what we are exploring as well but it seems like it will suffer from latency over the response you get with any mapreduce solution you are usually in realm of

76. 43:25

    at least several seconds in the very best case uh usually it takes minutes to to get some result from scanning the whole data set so let's say like a backup solution for now we tried to hit something in realm of seconds of delay so the queries would take like milliseconds for very simple ones and maybe up to tens of seconds for complex ones that that's the ideal goal um okay there is a question yeah

77. 44:06

    i i guess so it is an elaboration on that question about like dropping the view client actor from the lake indexer yes uh because uh like a view client is a part of like from the new york world some sort of like undocumented uh stuff that it is possible to use from indexer framework but not from lake one and uh like it is intended uh to drop this like because we would try to avoid new york war as a dependency in the

78. 44:38

    first place uh so um i i can't like i believe in most cases if you need something uh like um your use case involves some sort of querying the blockchain and you don't have like a hard requirement to to query like your own node uh you can freely use the newer json or pc client address uh which is uh like an api uh for the json rpc and you get the answer uh yeah and like something about uh interruption enough i i would say we should probably read the whole question just so that way yeah

79. 45:24

    yeah questions yeah um so he followed up he said as for the uh interruption enums when starting lake indexer you need to manually input the block height but there is you there used to be an option to start the indexer from interruption without having to input the height is that the pipeline as well uh like in the pipeline we have uh um a task to make it starting uh from the latest block like uh the from the index or framework it's uh the

80. 45:56

    start from latest uh from interruption uh might be implemented on your side uh like we don't have it in the pipelines and it's actually the first time we like got the request so you please like uh feel free to open the issue with like this kind of request and we can consider it but i'm not sure we actually wanted to add anything uh like storage additional storage to like to the near lake framework uh but it is discussable uh i really encourage you to open the ishon github and

81. 46:35

    we can like collect uh different thoughts like with like indexer framework you can easily now you can easily implement many of the things you could not implement with indexer framework original one um and this particular one start from interruption is very easy to implement on your side when you process your message you're in the block after you finish processing you just wait for this support height somewhere in like on disk and then

82. 47:10

    uh interrupting you i i suggest like we can create some sort of tutorial on this regard uh and put it in docs and it might be useful uh for those who want to like this feature so i guess we can add this yeah another question is how will dynamic resharding affect the pricing of running in lake indexer as the number of shards increase will it ever become more expensive to run the lake framework versus using the index or framework uh yes

83. 47:41

    it will uh a little like it it depends on the number of shards uh in the readme section of uh cost estimation we put a formula and our estimations were done for the four shards like uh and you can easily follow the formula and change the number of shards and got the like numbers we can even implement some sort of calculator but it will never uh exceed the cost of running excerpt framework because for indexer framework you would need to run a node per chart

84. 48:19

    and 8 800 500 usd per month is unbeatable uh challenge here and with with the with the lake framework it's in realm of adding maybe five bucks per chart or something like that in terms of like read and cost on from history um one more question are you guys working on a dimensional model design for the near block chain like dimensional user dimensional contract and fact transfer tables if yes can you share your design if no are you aware of any web three-dimensional modeling blog

85. 48:58

    or reference material so we can make a near-dimensional model ourselves according to the polls and the silence i guess we are not even aware of what dimensional model design is so the answer is no we're not working unfortunately can't share anything because we have heard it for the first time and we really encourage you to actually like do this if you want why not like even more cool projects based on you cool um i think we're run out of questions uh

86. 49:41

    if you have any more we'll give a minute or two to um let you drop any more questions yeah and while you're dropping like i really want to thank you for like some ideas for tutorial and encourage everyone uh to like visit our github repo of the near indexers docs and put any requests you actually want to see as a tutorial or article uh for the documentation because there are a lot of like topics we can and want to discuss

87. 50:13

    but it's really hard to like prioritize uh and i guess we would like uh to prioritize it based on your requests so please like don't be shy posted like acquiring issue yeah absolutely uh can i find the video the presentation deck of the of this session somewhere i'd like to share with my team yeah absolutely after this rule ends hopefully today or tomorrow at the latest uh this will be uploaded at on youtube youtube forward slash near protocol we'll have this

88. 50:48

    presentation there as well as the slides and all the links and where you can participate um yeah and again just to really reiterate what was just said we really want developer feedback community feedback you are the reason why we are building this and we want your inputs on what we should prioritize how we should build things where we should go um so yeah um arvines does the data team have a discord channel or forum to discuss scaling issues

89. 51:20

    i would say right now um just come to the the um near dot chat and come to the near um channel we have a validator section uh in there but mainly under the engineering aspect if you have any questions there's like developer support post there um yeah you don't have a specific data platform team channel maybe if that's something that people want to have we can talk with the team

90. 51:43

    about it um to have something specific uh to that um but yeah feel free to jump in there and also we have office hours every day twice a day if you go to near.org forward slash office dash hours um you can see twice a day at devrel um having live sessions that you can ask questions we'll answer the best we can and forward what we don't know to the smarter individuals on this call for all you got

91. 52:09

    something uh yeah i wanted to add that we some somewhat deliberately avoid using chats because we found that they are not scalable so it's better to open up the conversation on our issues or discussion boards and on github and that this way we can always get back into conversations regarding certain aspects later on and revisit our decisions and also keep several tracks at the same time discussing analytical data solutions versus event-based ones

92. 52:50

    so yeah um i guess just one comment thanks so much for doing this an amazing project running running lake indexer has saved my team so much time and money i'm super pumped to see how this project grows if you need any beta testers you know where to find me go team like indexer so if something might went wrong it would be for benji so awesome cool well thank you so much um oh one more is there a specific github discussion link to the postgres scaling

93. 53:36

    issue discussion did you guys create one yet i guess the one that we already shared okay uh near lake flows into sql base uh issues cool awesome uh yeah and again once we post this on youtube we'll make sure you have all these links available for you um cool word yeah just about out of time i want to thank everyone for joining us um yeah really stoked to have these community sessions get this live feedback uh and engagement so yeah this meeting

94. 54:10

    will take place every last thursday of the month and feel free to join us every thursday we're having community meetings uh first week is gonna be dev console slash explorer second one is the protocol anything nep related or for the core team join third week is tooling so anything developer tools come join and then yeah again this one is the data platform team so yeah excited to

95. 54:35

    have these continuing every month uh and yeah feel free to to let us know how we're doing give us comments suggestions we're we love feedback so thanks so much for joining really appreciate it thank you everyone thank you have a great rest of your day we'll see you next time bye