Tools Community Call #4 - near/dev/hub

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:08

   hello everyone uh I am very happy to see you around and to have you all around here my name is lud and I am Dev Dev contributor at near Dev Hub uh today um we are going to have our fourth to community call so if you haven't attended one of those calls before those are informal sessions that we do regularly that we talk about tools the community and the ecosystem and in general so if you would like to present or get some updates or news or suggest

2. 0:45

   uh uh discussion topics then I highly recommend you to join our telegram group that you can see below it's the near TOS community group um on today's agenda we have I have the pleasure to actually be joined by heat from Jutsu Uval from laanet and Eugene from fast.com and yeah today we'll be having uh those three presentations and then we will have some uh time for open discussion I I I hope uh before we

3. 1:18

   actually begin with the heat I would like to give a quick heads up about the devhub newsletter the devhub newsletter is an awesome place that uh you can get uh we just that we just sent out the second um second part of the second issue of the newsletter it's a great place to get like tldr versions of about what's happening around the developer ecosystem around near so I strongly

4. 1:45

   recommend you to uh sign up for it and another quick shout out I want to give is about that uh the survey that we are running we just recently um put it on so the dev Hub server is basically we are trying to gather some data in order to see how we can further support developers and tools and the ecosystem in general so please after the call or whenever you have some time just take five minutes to fill we know that it's boring of course but then again this is

5. 2:19

   like one of the ways that we can improve for the future so yeah uh the hits are you with us so without further ado we can start okay um all right then uh take away so for those of you who missed the last call uh zit actually gave up last minute updates that there were like breaking changes of what changed to to Jutsu II and uh so on this session we'll be getting updates of what's happening with it and what are the news and yeah what what what's uh what happened post it Denver and what

6. 2:59

   the plans for the future so Z okay um so those who know uh who don't know Jutsu so at Jutsu what we are trying to build is a web3 co-pilot where you will be able to build a full stack dab just using prompt uh we are not there yet so right now what we have is you can build a um frontend using prompt and then you can deploy that front end as a uh boss component uh you you can you can check

7. 3:34

   our website in Jutsu doai uh but the um the the uh generative UI part where you can build a front end just using prompt is Beno dojutsu doai where you can see like we launched this thing uh this platform during Denver and already we have more than 400 uh component people build so it's like a super interesting um so you you can see these all these UI are kind of just made up using prompt and this is not just uh one prom so you can have a

8. 4:20

   collaboration and kind of a improve your uh improve your UI so these are uh like if you come here you you can type whatever uh apps you want to build uh these are just few examples let's say like how many of us created some sort of registration form uh so you can create a registration form using like a create a user registration system with username password and date of birth then when you click uh it uses uh large language model

9. 4:54

   to kind of a create the uh create the frontend code and in few second it will show you a basic version of the of the UI then from this basic version uh you can keep up uh updating the uh the um update updating the UI so let's say I can say like a can you uh add can you change the back from color to I don't know what is your favorite color but my kind of favorite color is one of my favorite color is green so in few second it will kind of

10. 5:48

    fit change the background so you can see like it changed the background now um maybe this uh this um this card is not centered can you Center the user registration div let's see uh if it can Center the user registration div so the idea is like you will have a series of conversation and then uh it will kind of keep up updating the UI so you can see this version It's centered the UI I don't know about you but like I'm not a backend developer I'm not a frontend developer so using flexbox and

11. 6:49

    uh like a moving deep into Center sometime it takes takes me some time but you can see from here to here is like just a series of conversation and then when you want to deploy as a boss component right now we generate kind of a pure H JavaScript CSS HTML because of the status of um of the boss VM so when whenever we settle with the web engine probably will generate the code that's uh that is uh kind of equivalent to web engine uh but right

12. 7:27

    now in one click we we have that uh in the boss uh in the Jutsu U editor and then I can I can just deploy deploy this component so that's the first part of generative UI um that you can generate a UI and then deploy in Boss the second part is um let's say um sometime uh you might want to uh kind of a explain some code or like this is a code like I have no idea what is happening here so so we

13. 8:06

    have a kind of a co-pilot feature it's more like a GitHub co-pilot where uh you can ask question about um about your code so here I want it to explain explain this function so if you ask it will it will explain this function so if you don't un if you don't know what is going on in this function so the idea for this is kind of a uh help you understand and kind of write better code so it's not just

14. 8:45

    explain um so if we go back you'll see like it can explain it can it can generate code it can write a documentation for for it can suggest a documentation for your code um so there are a lot of things it can do so yeah these are kind of a two big feature uh update uh uh uh from the generative UI side and from the co-pilot side moving forward um we want to support kind of um uh as you know like a the claim was to build a full stack that

15. 9:25

    means like a both smart contract and and the boss component so the future goal is to support smart contract so that you can build a full stack uh web three app so yeah that's kind of the demo um so if you have any question I'm happy to kind of answer yes I do uh you can unmute yourself and have a question do you have a question or yeah aumn so like I uh I had like while you saw the demo I saw that you can explain code do you plan on I'm sorry do

16. 10:11

    you plan on uh actually incorporating that as a learning tool as well I mean like would JSU would be used as a learning tool as well for those newcomers to web comp like both components or smart contracts that's a great uh great idea so that's that's in our like like a kind of our pipeline but like a the the next next step is kind of a uh you probably some of you probably know we have a learning platform so probably in there we'll incorporate this

17. 10:43

    kind of a uh learning feature yeah amazing um any other questions if I see in the chat for zahit let me know thank you uh then would be I wonder if would also help Implement fast out Matias is what is fast out yeah okay well I guess not well fast out is a tool that helps us actually for users to sign in and make use of like actually get to uh using different uh different gateways like near. org and so basically a way for users to start interacting with the

18. 11:29

    blockchain without actually having the wallet themselves yet so yeah this was not really a question from Matas but he was wondering uh anyway so if there are no question that we can see I guess we can continue on with youal from laanet yal has been with us on I think two calls before uh he gave a brief overview of what lava was doing and basically they're helping to to scale Services uh as well and they they do have the

19. 12:00

    partnership coming up with well actually undergoing with near Foundation where they're incentivizing RPC nodes and yeah so like I guess that we'll get some updates and maybe the upcoming menet you have you or am I mistaken so if you're ready to present then take it away yeah so I'm sharing the screen hope you can guys can see yep we can um yeah nice to meet everyone I'm yal I'm

20. 12:28

    leading Partnerships at NBD uh at lava Network now lava network is generally um building a modular Network um of not providers uh we're building the modular infray for web 3 um essentially generally how it works is we're a cosmos based app chain and lava enables anyone to get access to a very large network of top tier providers uh and that starts with RPC um it moves on to Archive data

21. 13:00

    then Advanced apis indexing oracles um IBC uh Rel layers Etc so any kind of uh web three data structure that you can think of can be added to Lava and the cool thing about lava is that we have a very large network of providers so whenever something is added to the protocol we're able to onboard a um very top quality group of providers to support this API um um and we've been working with near Foundation to

22. 13:32

    essentially uh create a more Diversified more um smooth RPC experience for users and developers on near um the more redundancy of providers um the more reliable The Experience becomes so we we're trying to on board um we actually onboarded like a pretty large amount of providers I think the number now is around 60 um and the endpoint is already live we've been kind of working together for a while um generally kind of how public RPC usually looks like it's a lot of you

23. 14:12

    know if you Google near RPC uh you'll find probably like the top five six providers and as a developer or as anyone who wants to access the chain uh you have to kind of do your own research you have to try the top one and then maybe you want to find a cheaper one so you try the second and the third and then one of them fails so you try to it's it's a very long journey until developers find a reliable endpoint that they actually can use in the long term and that's a pretty

24. 14:46

    painful experience um that can sometimes make it hard for developers to uh focus on developing excuse me um and what lava does essentially is it Aggregates to to Providers into a single endpoint so the developer um has only one endpoint to choose from and the endpoint connects them to uh the top providers in their geolocation uh now it's not like a standard proxy server that you just uh

25. 15:16

    send a relay to and then it will load balance it and direct it in the direction of some nodes um it will actually uh give you what we call pairing so it will pair you with uh 12 top providers in your geolocation and the service will be completely peer-to-peer uh and and off chain um and then the lava blockchain uh acts as a settlement layer for providers um which are incentivized in near tokens um on the uh incentivized public RPC

26. 15:49

    initiative so you can see here in the image there's one endpoint many providers and the top ones are chosen for uh the user uh so as I said it's already live we have uh mayet and test net endpoint uh let me actually send a link uh no it's it's depending it can be for the user as well depending if it's an app or a wallet um just to bring for those who are joining us from the recorded session uh there was a question by Owen asking about EU apps and not EU users so this

27. 16:31

    is what you was referring to yeah so I send the link uh for the end points um in the chat um so we actually launched in the beginning of February and we've seen some amazing adoption from the near ecosystem we've been working with a lot of teams and developers um so you can see we've saved uh 200 42 million relays in the last month um compute units or cus is the breakdown of relays um so it's a bit it's a more accurate measurement of how much compute uh resources are invested

28. 17:13

    um to support those relays um so we you have for example you have RPC relays and you have archive relays which are way more heavy and expensive to run so that's one example also by the way AR archive data is also supporting on the laa protocol um we have uh 56 AR active providers I made this deck like yesterday so I think today I saw like um six 59 or 60 which is cool to see more

29. 17:43

    people joining every day um and another cool thing that you can see is that we've added um near to what we call magma so essentially uh lava as I said is a blockchain we're currently in test net but we're getting very close to launching mayet which is very exciting um and we launch what we call so as you know magma is lava while it's still underground and once it erupts it's it's h called magma uh lava sorry so we're

30. 18:17

    currently in the test net phase uh right before eruption so magma is a gamified experience where any user can interact with lava essentially you log in with your wallet and we're working on a partnership uh we already announced send a wallet and are going to announce uh and launch here wallet as well as a partnership uh so you you log in with your near wallet um and then you get an

31. 18:46

    RPC endpoint for near test net and Manet you put it in your wallet so now every time you um active onchain every uh onchain action that you do is sending RPC relays through uh lavas point and for every onchain action that you make you get magma points and the magma points in the future might make holders eligible for any kind of incentive uh structure that is decided by lava foundation so essentially it gives anyone in the near ecosystem the ability

32. 19:16

    to engage with lava prior to Manet which I think is really cool and it also kind of um engages user users to be more active in their wallets in their dabs and we're working on on a few more Partnerships with uh some cool DBS in the ecosystem I know it's number four so it might be confusing uh but that's kind of like the coolest uh thing that uh I think uh the new thing that we're bringing to the new ecosystem this uh month um let me actually drop you guys the

33. 19:51

    link so that's points uh some other stuff we we've done this month uh so we working as I said with near builders on on some future collaborations uh to be announced uh we actually um met up in East Denver and we did a talk uh with near devhub we're also participating in a lot of other cor marketing uh initiative so I really encourage you guys to check out points um you you just take your endpoint you put it in your wallet and uh from now on

34. 20:24

    every time you interact with near on chain uh you will get lava points um which is really cool and yeah as I said we had a really good month we're going to publish a monthly report uh that details um those details and more about uh the traffic on the endpoint uh the health of the endpoints the providers their activity Etc um so I'll share that with the team and the community when it's live and yeah I think I talked enough let me know if you have any

35. 20:57

    questions amazing uh thank you uh so uh before uh we continue are there any questions for youa going once twice or three times yeah so yeah I got to say that there is no better time that uh you guys like you from lavet and also eghi have joined about this call so we are all familiar that for the past like few weeks they have been some like issues with the rpcs and generally like

36. 21:34

    the nodes on the network and people experiencing rather issues so there is definitely like a problem or it gets like traction points that have to be that have to be solved so such Solutions are gladly welcomed so this is not only uh but for isolated chains but in general into the space if you want to bring like more interactions for the future if you expect to have like the billion users uh aspect so yes scaling more so the scaling the uh rpcs and

37. 22:07

    services in general and access to data is uh pretty viable so yeah I think that there is like no better timing to have you around so I'm very happy to talk and continue the continue the discussion on it so since there are no questions for you at the moment first of all thanks for like also like thanks you all for the update thank you guys and yeah I look forward to see like more data probably on the next one the Kobe FS like how actually are people using your

38. 22:38

    service uh without further Ado uh I also like to welcome GUI more known as Eugene who is one of the biggest contributors to our ecosystem and thanks to his efforts and his time we have uh like quite a big amount of interesting projects that have been uh initially his undertaking um and he's I believe that actually for the past month or something he has been working on Fast near uh

39. 23:11

    which is going to as like he's going to share some updates with it and like some news with with with us now but uh essentially what I understand of it is that uh they're working on a new way which is uh to like essentially a new way to run nodes and provide services with them uh if I'm mistaken I'll be happy to be corrected SOI if you are ready please take it away hey uh sure so I'm Eugene um I we started fast near

40. 23:45

    pretty much to um uh to scale some some of the apis that were like struggling and we realized that there's huge demand coming in from um C wallet and like new apps that launching and the traffic that they have will probably like exceed like we will double the capacity of um archival data probably within a month based on the traffic that we receive from them so SWAT and kink will

41. 24:20

    probably just uh kind of isolated instances that were like High number of users but low volume on number of transactions so with what happens with hot wallet we get ton of transactions with like ton of users so that will increase the demand on the infrastructure that we have and a lot of the stuff that um previously was run by Pagoda including the explorers and some of the apis uh we'll have hard time to

42. 24:52

    scale properly so we decided to step in and to fix some of the pain points very fast and realized what else we need to build to support the scale of this and then I personally developed ton of projects on here and I know what's the pain points that developers face and that includes stuff uh from the indexer and um like subscriptions to you and just getting row data it all costly

43. 25:28

    right now and like Lake indexer for example requires S3 payment and that's fairly expensive to get the whole historical data around like 2K or more um so we're trying to fix some of this problems so developers can come in and get um access to the data as long as they are willing to pay for the traffic on their notes not on our side so some of the apis that we provide will be completely free free and enabled for

44. 26:00

    the community so long as they don't cost us that much uh some of the apis uh will be probably paid at some point and we hope to provide as much as possible for free um to the community and second one is all of our work will be open sourced and available for people to replicate if they're willing to run the hardware so we are still very early and like most of the data and like apis and documentation

45. 26:34

    is fairly flaky uh but some stuff is kind of getting better so uh let me present a quick overview of what we have so far um I just wrote it like right now to explain um so the idea is we currently provide a single API and this is replacement for Kit wallet API um that returns uh foring so it does a reverse lookup on public key towards account ID and that information is actually

46. 27:15

    um should be available on the noes itself but they are not so um we basically provide it and then it provides you a list of fible tokens contracts on a given account nfts and validators towards you an account delegated and that's pretty common information that when you open a wallet you need all of this info updated uh in real time when you open it so it has to be fast so every wallet opening is required it and that's pretty much how

47. 27:51

    we built it so we have near notes on myet that just send R indexing data in Json uh towards kind of message CU we use radius for this um and we store for example last 10,000 blocks in in this radius CU um we have a bunch of consumers that subscribe to this R node to consume roblocks so they don't need to run their own RPC noes or read this from Lake indexer so they read directly from this

48. 28:30

    uh cue and there's a bunch of them so first one is just dumps data to the dis so we have a historical view of all the blocks once they kind of disappear from the queue then the second one it's updating table tokens nfts and validators on a live separate live already this instance which we call kind a master instance and it just saying okay this user touched this particular fungible token in this block um reset the value to nothing to empty

49. 29:09

    so that data is later and the third one we have a click house database that we use for quick analysis and some of the back fi of the data for example like give me all fundable tokens that user ever interacted in a Hulk history so the master data Rus instance is being replicated to multiple machines um and they on on the same machine running an instance of the pass near API this just a rust server that

50. 29:47

    reads data from the uh local instance of reddis and serves it to the client so when you need a fun ible tokens of a given account that account have interacted it makes a query uses um a set on redis and it retrieves it from memory so that's less than a millisecond on a local machine and then it just serves this to the client on a request so behind it we have a in front of it we just use the simple DNS based

51. 30:25

    load balancer right now because we are not sure how much the traffic we'll get from the wallets um so we are not really using the actual load balancer but depending on on the load and demand we can consider this so and we have I believe six instances right now that we able to serf about like 5K TPS RPS each um hitting from like five machines so that got us to around 30k RPS um but it probably was limited by the single machine hitting it so we might be able to serve

52. 31:12

    more and um yeah we ask wallets to basically switch to us and we probably very reliable right now so instead of actual red we use dragonfly DB which is multicore um red this alternative and uh that's what we have right now uh you can go to uh the GitHub um uh link I will send it to chat and you can um see what apis are available so it's live and should be fine to scale up to certain point and that setup doesn't

53. 32:03

    cost us much um on the way we run it right now so we can provide this data for free and scale it more horizontally pretty much the same way we scale this this amazing uh yeah I wanted to ask like meanwhile if you have any questions for or uh Eugene OR fast just put them into the chat I actually had a question myself because in the recent past I think the past week I have seen users um

54. 32:39

    like uh reporting that they are missing well the the the wallet applications are not sending like they're not showing their Stak amount or something like that which is like it is only like is it the issue was it that most of the the wallets at the moment were relying to uh pit wallet so basically that's why there was a congestions with the networks and is fast going to be able to mitigate that in the past if a lot of the wallet applications currently available near

55. 33:10

    switch to fast near yes uh that's the um the hope that once small wallets switch to fast near it's still it still requires RPC right now to to retrieve the balances the actual balance of the tokens or like the actual Stakes but we are exploring a way um that would just do this at the moment when we update the data So the plan right now is pretty much once we calculate all the updates that needs to happen on in a given block we instead of

56. 33:51

    writing an empty value we will write the actual value for like balance of the fishable token of the given user by hitting an RPC node at this time so that will pretty much remove a lot at the run time and move it towards the indexing time which would be very small comparing to like actual demand from the users um um I know Pagoda is developing a read only RPC that is based somewhat on a similar stack except the they use some other database no

57. 34:30

    SQL um but it does work very similarly you can scale the database you can scale the machines you don't need to scale the compute and that increases the throughput of the read notes dramatically nice uh thank you very much for for your answer and like another question that I got like since your code is open source that means that of course everyone can replicate your service but uh where like where would you encourage people to do that when would you

58. 35:04

    encourage like third parties to do that and replicate what you're providing so for for this particular service it's probably not as important to to be replicated because we will provide it but some other services that will require more scale for example like click house um is a bit more expensive to scale than like this type of approach just because it requires you to be heavy

59. 35:32

    on compute when you making a query so um that stuff we want people to run on their their own and that shouldn't be that expensive so we can provide a pretty much setup and recommend Hardware requirements so our also idea that we don't want to use cloud and prefer to run bar metal just to to have a predictable cost and um that's just a philosophy of this yes amazing thank you thank for your anwers so are there any questions for Eugene and fast near just Dro them

60. 36:14

    into the chat so I can bring them up I have a question umad Eugene uh do you see like um or do you see other uh people also run the same stack or let's Jutsu run same stack or use um FME as a service um it's obviously up to you but okay so our setup should be barly cheap to run for other companies so our goal to basically at some point we'll write a proper documentation how you can spawn all the stack and that will give you flexib IL so if you want to index something custom that would be a really

61. 37:01

    good option um that requires some extra pieces such as like historical data to be um provided as well so right now our other ounder fast near vad graina he provides some of the cool stuff like web 4 and um near Lake torrent where whole historical data is just indix or Torrance mhm so you should be able to get the data on your machine um backfill the data that you need and um Spa the instances or like let's say you want to

62. 37:41

    index all the social interactions and move it to like K said for quick retrieval you will be able to do this um pretty much just foring the stuff slightly modifying the processor and doing it we also plan to provide some of the consumer end points for the blocks stuff like just providing Ro blocks and subscriptions through like web socket or something like this um just Ro blocks right now are fairly

63. 38:14

    large they're like couple megabytes per block so it just requires extra throughput depending on the number of consumers but if we don't expect uh actual user to to hit it on only developers then we probably can handle it easily if it's like users who would want to subscribe to this then it will need extra um extra scale and how how much it is different from the the query API that um pagora created kind of the query is kind of um

64. 38:55

    very cool check that they have it's a I believe JavaScript um updator Works somewhat similar that what we have in Rust um and then it writes it into AAL database so it's somewhat similar to what we have except um um depending on how the back filling works for the query API and whether you need historical data is it sequential or not you you will have a different time creating this okay thank

65. 39:37

    you yep amazing thanks very much for answering the questions Eugene uh we have another couple of minutes which we can use for open discussion and see like if it Sparks any interest to you but one of the things and I'm as I mentioned earlier I'm super happy that uh Eugene and also yal has have joined this today is that like I would like to know and how we can for example the a lot of the tooling that has been built in the recent or medium to recent past is

66. 40:09

    relying on the same rpcs for example and how we can how do you see uh not like a plausible solution or in the future to avoid such kind of congestions where all the tool link is hitting the same rpcs which were the previous like uh public publicly released and like how do you guys see that like to actually mitigate that essentially to make a load balancing against the rpcs and different services in general because yeah you're both in the same space and providing uh

67. 40:45

    more or less uh something into a similar uh similar goal so what do you see the future would be like youal of G so anyone well this is not a question this just an open questions to anyone so if you have any input uh that would be like to to be honest me personally I have seen because we have seen all the tooling that has been produced by b pagod or the foundation they're hitting the same rpcs

68. 41:12

    and therefore a lot of congestion is happening so I've been thinking that maybe we should provide a way so like this tooling can actually provide ways for people to either select different rpcs or like custom rpcs some of the tooling that we Curr have already started doing that we saw that recently myar wallet as well now let's you select an RPC server for example that you would like to interact through so I think this is also amazing but like what what other

69. 41:42

    Solutions or ways do you do you think that there there possibly can be well yeah it's like um selecting it like buil it into the wallet selector or align the RPC somehow towards the account or just like when you log in through the wallet it should retrieve return the RPC that you're willing to use all of this is kind of good Alternatives so ideally you on a wallet side you you select um

70. 42:27

    the RPC and then whenever you sign up to any of the applications through this wallet through limited access key at the same time it would pass the preferred RPC endpoint to the app and app just okay can let you select um the RPC as um as desired by the user so this way you can build subscription models on the account level where a user actually have access to this RPC by paying the um paying like monthly subscription to

71. 43:11

    the RPC so this way RPC providers can actually monetize their stack and provide the best RPC so in this lava net seems to be a fit for this if they can handle that type of U data that requires slightly modifying the RPC or challenges identifying that the request comes from the given user so that might be settled through either cookies or us or something un like bear talking in the headers of the read

72. 43:50

    requests mhm nice yeah uh yeah I because I totally agree with that so that's why I was happy to see that what my new wallet did and also there is a way that you can select in the RS there is also a way that you can add like custom rpcs so we can avoid congestion in general but yeah apart of that I my I personally haven't come up with anything else maybe I don't know in future we would have like one load balancer in general run by

73. 44:22

    Community yeah probably that can just switches to our to different rpcs which is similar to what laanet is doing but yeah so it's just on on another level of it so you Val if you have any input on that I would also be happy I'm assuming most of the people in this score as well so if you have any inputs that would be interesting to to hear yeah I think definitely you know RPC redundancy and diversity should be

74. 44:52

    encouraged I think making it easier for users to change the RPC end point and customize it is very cool so what my near wallet is doing is really cool and we're actually working on uh some similar collaborations with different wallets as well and at magma points is also kind of an incentive for people to get more engaged with you know and get more familiar with how they work their wallet is working on the back end um be

75. 45:24

    more aware of what RPC means and uh the effects of RPC and you know the fact that you can even change it most users probably have no idea um that it means anything or that it exists um so yeah I'm not sure if there was another part to the question I'll be happy to yeah no for me not uh yeah that wasn't a part like from another part of my question I was just like trying to get inter to it because yeah I think

76. 45:54

    that's uh in a way the easiest way would be yes to most end user facing applications to provide them with such kind of of a Service Such kind of a way to avoid congestions so yeah that might be the way so maybe in the future we have like a meta app for LPC selector or something like that so uh that's pretty much what I had so is there anyone that wants to add something or if not then maybe we can wrap it up

77. 46:29

    I assume not so well then uh for yeah I want to thank to Eugene yal zahit and everyone who joined us today uh before we go I just want to uh share with you again the devhub developer survey check out our newsletter that was going into the pinned message and the slides before if you would like to present uh your project you're currently uh working on tooling or you would like to start up a

78. 47:00

    discussion or present what you're currently working I strongly encourage you to join the near tools community group and where we run these uh calls regularly you can get to know better the community and what we're doing on devhub pages and also our telegram group and with that I guess I would leave you around here and I hope to see you on the next one thanks thank you everyone thank you

79. 47:32

    thank you Eugene thank you see yeah thank you bye bye