NEAR Blockchain Concepts (1/6) -- Introduction to JSON RPC APIs with Explorer

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:07

   yeah okay so um froel thanks again for taking the time to walk us through the rpc api and this time from the perspective of the near explorer so i think what we've agreed is we're going to talk about um how do we build up this interface of near explorer actually and you'll give us a tour of this uh kind of sweeping through the the different data that we see where it's coming from to help us make sense of how we're using

2. 0:32

   the rpc api to produce this interface here yeah sure no problem we have if you open the website you immediately grid it with this uh nice page with the dashboard and the search and main bar let's go from from the top to the bottom just to understand better what other tools that you can use here here we have the network selection current currently the explorer is deployed for mainnet and testnet

3. 1:07

   and whenever you switch you just select the network and this their interface is the same it is just the data that is different based on the network uh selected on the right you have uh more deep down a breakdown into different uh pages that you could be interested in uh blogs and transactions unnatural to uh near protocol accounts is aggregate about like what are the accounts essentially are currently active or deleted uh

4. 1:44

   and the notes is the page where you would navigate and see more about the network itself about the archival nodes validation nodes um other types of services that are also in the peer-to-peer network of near so um keep keep keeping going on the um on this main page you can see that the torch is the main part of the page and that's on purpose it's very hard to navigate and find to a specific block or specific transaction in the list uh it's much better if you have your account id

5. 2:33

   or any transaction that you want to explore you just dump it here click search you're there exploring it um so search is a very core feature and we encourage people to use it or there it's just a lot of nice graphs and overall stats about the state of the network um let's go one by one here so the nodes uh is essentially representing the health of the network itself it's very important to have as many

6. 3:11

   validator validating nodes as possible they are ensuring the security of the data and that's why it's great that we have so many of them right now online notes it's all those for leading nodes plus those who are using a near-core network near-core software to essentially get access to the data uh one of the examples like a few of the examples would be json rpc uh nodes they are not like validating nodes does not expose json or pc to the public

7. 3:59

   just for the security reasons so instead what is suggested is to always have rpc nodes in front of your validating node and in other instances would be archival nodes which keep track of all the history from very beginning of the network from genesis also another example is indexer nodes which we one of them is used for explorer itself so a lot of them are here we will open up them uh once we cover the rest of the page so sorry so just just to summarize what i heard you say there because

8. 4:39

   this wasn't obvious to me when we first talked about it so there's 60 validating nodes and best practice is each one of these 60 validating nodes should also have an rpc node sitting in front of it so that's that's 120 of the 186 right there if people are doing what we're asking them to do right then there's the balance of 60 65 66 other nodes that these these are indexing nodes

9. 5:03

   including you know these are the kinds of nodes that are listening to the network and letting us query it maybe using a database or something and then archival nodes are like an indexing node but it's not for querying purposes it's just for archival purposes so so the the nature of what the node is doing yeah it's a backup node and it is that are those basically the three sort of three four classes validating

10. 5:27

    nodes that are actively maintaining security and integrity of the network rpc nodes that are sitting right in front of them by recommendation indexing nodes and archival nodes are there any other nodes that are running on the network not that i'm aware of but they're not limited to these kind of usage so it's up to so anybody could modify near four source code and build some sort of tool there could be other nodes that are you know registering here listening to the network

11. 5:57

    conforming to the if you need to clarify to classify them i guess these four are the majority of the whole thank you sure blogs is in core nature when you blockchain you produce block every second and here is just a simple stat exposing us to what is the latest blockade right now and whether the network is uh doing fine so and usually you you would see numbers are almost identical all the time here uh next the transactions this section is

12. 6:38

    quite interesting in terms of like to to understand how the network is used uh because blocks are produced no matter what but the transactions essentially represent how the network is like used by the end users or other entities it would be programmatic use sometimes it's just ping or oracle request to the network sometimes they are transfers so all kinds of transactions are floating in the system so here the latest 24 hours total

13. 7:17

    and and also 14 day history we exclude today from the history just because it would be always a drop here that makes sense to display it and it's useless um and gas prices here is essentially to represent how what is a current price to to to pay for a gas on the network make sure that you see that it's in terra gas metric so it means it's a terra prefix here but we have an article we have an article in the documentation that talks about you know thinking and gas

14. 7:57

    what this number means why we say terror gas it's a little bit easier to reason about than you know uh some some of the large numbers that we're talking about you just use the prefix terra gas and it tells everything you need to know um i'm surprised to see gas price uh uh you know uh kind of featured um so prominently like this but i think it's because of of the way that we we price gas it actually fluctuates

15. 8:22

    doesn't it i well it it does not right now it will eventually with a greater load of network right now it's constant and actually i'm considering to replace it with the average transaction fee which makes more uh sense to the end users after all and it just hasn't been done yet that's the only reason i also agree it's it's not the best metric to be uh so prominently exposed but i mean it's it's mentioned that it's still useful

16. 9:00

    yeah sure it's it's fine but i have another question about this how do we know what the capacity of the network is so so we see here ten thousand ten thousand five hundred transactions in the last 24 hours are we are we close to capacity are we nowhere near capacity we are not not even near to the capacity if we open the blocks you can see the number of transactions in in a block you can see that rarely you you can see one transaction sometimes two

17. 9:33

    in my experience i like on the on main net i only observed like 10 transactions in a in a single block we have a special network deployed just for benchmarking and we could reach uh 5k of a 1k transaction on a single chart and it it depends on the transaction obviously so maybe you're talking about yeah transfer transfer transaction that's right but with the number of charts it could be linearly scaled and

18. 10:13

    here we are very far from like any any capacity problem right now and that's why we essentially take take it with caution to release more shards we currently run only a single chart network on mainnet it's uh it's stable it's well tested and we don't expect any unknowns whereas the sharding is a bit complex thing and it needs to be more thoroughly tested and i know that core team is currently

19. 10:57

    making sure that it's solid and stable and it's a very important part of future development let's talk a little bit about how this data is getting rendered into the interface here so the the if we if can we go back to the home page uh just the main kind of view of this aggregate so just to get an idea that the search bar is uh is using the indexing service right because we have the historical data there and in fact a lot of this data that we saw maybe in

20. 11:28

    the the other page as well is coming from the indexing service so so the the search bar when you search this you're actually searching a postgres database that the indexing service is filling with with data as it's listening and so all that makes sense and um can you tell me a little bit about some of these other numbers like where are you getting the number of nodes online where are you getting the number of validators those i think should be available in the rpc api right

21. 11:51

    yeah this number is available in rpc and api you can query your validators status and the detailed validator status here here yeah the valve validator status you will be returned with current voltage released and if you count the number uh of this records you will get the number 60. uh if you open the page you will see the same list essentially displayed here render it with all the

22. 12:24

    detailed information how many blogs they have produced how many blogs they are expected to have been produced all of that is here so this is a presentation of the response from calling that method exactly okay so that explains the 60 validating you know new upcoming nodes that that's the data returned by the method where's 186 coming from yeah so every node optionally reporting

23. 12:48

    telemetry to explore we we've decided to make it optional we also decided to make it reporting to explore explicitly so this number could be even potentially it could be zero while we have 60 volt leaders why because all leaders may choose to opt out of the telemetry and uh that's the reason then yeah so there's at yeah the nodes are are telling us that they're online so we we know one there's at least 186 nodes online there may be many more

24. 13:31

    right uh and and so um that's but that's being reported to what exactly to uh some some other endpoint that that explorer expo exposes or something like explorer dot near the torque slash telemetry something something something like that and they just they're they're just communicating against that saying i'm i'm i'm here uh supporting their version they are agent uh their latest blog and

25. 14:02

    uh all all the rest info and we just record when we heard from them and update us in our database in explorer got it got it and and then the 60 proposal nodes is coming from the detailed validator uh rpc endpoint as well right that's like an another collection below the current validators there's like the upcoming proposals probably right yep current proposals yeah uh over here

26. 14:28

    so yeah okay so so that's how we're getting 60 186 so at least 186 online maybe more 60. and then what's the nodes map here uh it's essentially a representation of where the nodes are located uh on the the map how can you do that where's that information coming from when when the online nodes report to us uh we can part of the token where yeah where they accept actually yeah part of the telemetry so here you can so essentially we

27. 15:01

    well while leading nodes they are not reporting uh not not of them not all of them are reporting telemetry so you could probably find less than 60 here because they are not reporting we couldn't really display them on the map uh but online notes all 186 we could we can represent them though some of them are off sometimes they're reporting uh some weird locations or default ones uh where it's it's it's a bug and we should

28. 15:39

    filter them out just keep in mind uh okay it's just a for representation point of view like to to see how uh distributed is a network uh we currently don't see anybody on the on this hemisphere nor in australia nor in south america and africa is also underrepresented it's it's about africa i guess it's mostly because the data centers and like big factories of like computer power are not really built in in africa due to

29. 16:24

    like very unstable internet connection and all other reasons uh so but okay overall it's distributed pretty widely across the globe yeah i mean this makes sense so if if we go uh back to the main page we've now covered excuse me the search nodes um we looked at blocks uh briefly there can we can we dig into a block can we see um in more detail about a block in this case let me find some block with at least one transaction so it's not temperature so here's block uh every block

30. 17:04

    so blocks are producers said no matter what and we can see that the transactions here are in the block listed here you can create this data through json or pc but we still but we prefer to use our own indexed data anyway since it's already in our database why would we uh request it again and again from the rtc so this whole page is being rendered from the indexing service as well from the indexing server as well yeah i believe so uh maybe well we

31. 17:38

    what what we check is the status so to understand whether the block is finalized we essentially have this info like we have this block in the database okay but we don't know whether it's finalized to make sure that we display this finalized we created the status page status endpoint on rpc and see if it's uh oh well another status sorry uh i'm querying the uh block the block details maybe the block details with the finality

32. 18:12

    uh final as a parameter and in response i will get all maybe with that maybe with the block id actually and then you get the details of this block right not really if you query the block id you will have this info all this info is very useful but it has no information whether the block is finalized essentially i say to in order to make sure that the block is finalized we query the final block

33. 18:44

    we research for the final and we compare that the height of the final block is greater than the block that we are currently looking at i see okay and that guarantees it's final obviously if that height is greater yeah that makes sense um there's nothing else here being rendered about this block besides the list of transactions but but the response the payload that comes back from this block details it's quite extensive there's a lot going on there

34. 19:17

    um is is there is there more to talk about there do you think or is it out of scope for this conversation well mostly it's uh references to next and previous in the state and the various other parts of the network uh timestamp is there uh when well random is like internal thing it's quite interesting topic by itself how we deal with random numbers but it's a very different one i don't think it's available on explorer side

35. 19:49

    gas prices is there yeah overall i wouldn't say that there is something that is missing we could provide more you know advanced features just for the sake of uh like blockchain developers themselves uh but from the user perspective i guess that's all that is really interesting uh maybe chart and trunks could be exposed but we decided to uh ignore the fact that like no not ignore but uh essentially uh height and fact that we are having

36. 20:27

    charts in the system what we believe that uh people mostly care about blocks even even that is already controversial whether they really need to know about the blocks and about the transactions themselves everything else is implementation detail that's why we decided to just skip it and unload the end user from knowing all the details about the implementation itself okay yeah right on yeah so and just to finish up the core concepts

37. 21:05

    uh if you go and open the list of transactions page you can see that there are transactions here in a ordered by the time that they occurred in the network again as we had already covered it it's all sitting in the database you can't quit it through json or pc so we needed to index this data through going through every single block and save the data in our own database uh specifically for exploring needs

38. 21:39

    and this way you can answer the question like give me last 10 transactions here and here we go the same goes for whenever you so when we open a transaction itself it's uh all great uh it's it's still under construction here uh we are building better ui to represent the internals so let's not focus on this right now uh as it will uh change in this quarter quite drastically and represent the data better

39. 22:17

    but what is essentially about the transaction that the core thing is that every transaction has assigned already the signer account id and receiver account id we estimated status it's a bit of a different topic and it's not focusing right now uh transaction fee is probably what most of the people might be interested in terms of like estimating how much it costs currently it's quite a small

40. 22:48

    number uh even for very complex transactions if it's just a transfer call it's even smaller uh and the like well deposit value is uh also quite interesting thing when you call a function uh method sometimes you can you want to pay for for the transaction here the deposit value that represents the attached balance and all the rest is essentially the internal repository representation or

41. 23:20

    internal details about the maintenance of the blockchain itself and what's important here probably is when the transaction happened hashes are useful but you as a user as a human did not really make a lot of sense out of them they mostly are useful for um you know the reference purposes and to make sure that you are looking exactly on the same thing that you expect to look at if somebody is like sending you a link or

42. 23:59

    or whatnot so that's all about the transaction itself and all the all the info about the transaction if you have a transaction hash and know the receiver id uh you can look it up through json pc that's a bit unusual usually transaction hash is enough to to look up the transaction in other systems but given the sharding nature of lanier the transaction will end up on the chart where the account belongs to

43. 24:41

    and that's why to look up the transaction you need to specify some some you need to have some way to specify which chart you need to go to uh that's why whenever you look up through json rpc uh tx status uh send nope um transaction status here here we go uh you need to specify the hash and the uh what's this hash of fruit it's a hash of the transaction of the of the body of the transaction itself yep

44. 25:21

    so whenever you submit a transaction do we talk about it much not really so when you construct a transaction uh on on your side and sign it you sign the hashes transaction hash itself your signature confirms that the hash represents the data that is passed next to it and this hash can be later in throughout the history can be used as a reference to the transaction the this hash is unique and you can always

45. 25:59

    refer to it so you don't receive this transaction hash as this identifier from the server you get it when you have on on your side when you just create the transaction so i'm i guess i'm i'm i'm a little bit confused why you would also need to specify the receiver account i mean i understand in a sharded environment that maybe center.testnet would be on its own shard separate from you know some other account uh but you're saying that there's no way to

46. 26:33

    kind of globally search for this transaction hash you can only search for it on this shard because because not shards don't know about each other actually they're they're essentially running uh in parallel right that's how we're getting uh this this ability to scale linearly with shards exactly so the transaction is i i was uh i was misleading uh previously i was uh telling you that the transaction uh ends up on the receiver side essentially it's now it's

47. 27:06

    the the receipts are on the receiver side but introduction are uh always belong to uh to the signer that's where it starts its journey to start executing the transaction you need to first deliver that this transaction to the chart where the signer account is stored to verify the signatures yeah we verify the signature and the budget to actually conduct the transaction right before we generate the receipts yeah that's why the transaction is ended

48. 27:40

    up recorded on the signer account chart and this way we can look it up so internally to look it up we need to specify the sender but in explorer as you can see you can navigate to the page uh only providing the transaction hash it is possible only because we have our own indexing backhand which goes up through every single transaction we store all the transactions in a single database is there some way to change the representation random thought

49. 28:29

    uh by uh or have we considered changing the representation of of this data that's rendered on this page by appending something like dot json to the url uh so so that when we request it from the explorer it sends back a json response of this data as opposed to the rendered html is there anything like that that we've we haven't considered that it's a good feature request we may consider to build it i'm a bit uh you know uh concerned with such features

50. 29:02

    because it may create uh unnecessary um incentive uh in terms of like people will rely on explorer too much as opposed to having their own indexing service first yeah as opposed to using like the decentralized to comply with decentralized you know network and i'm a bit afraid to uh let people down that way and if explorer will be so much in like central service for every for for everything we may

51. 29:41

    lose the whole app layer or it goes down sure so what else uh we covered transactions i feel like i feel like we've covered quite a bit here um yeah we can also go to the last piece probably the account account page is essentially uh also relies on the data that we collect throughout the history through analyticser we have transactions uh you you see i open the account and i immediately see the transactions that happened

52. 30:21

    months ago it's not possible to go over every single block right now on demand and know the data it has to be somewhere and explore an indexer is what is helping us to have it and yeah we have uh different types of balances here i guess we cover them pretty extensively on documentation yeah we we talk about them in the docs for for your account balance and your validator stake sorry why does it say native account balance that that seems surprising to me what's why the word native

53. 30:59

    oh yeah it's well we had a lot of discussions here and people are getting more and more confused with the balances and it's not no surprise even internally sometimes we refer to the same thing using different terms and what people tend to um want to see whenever they navigate to their own account uh is the uh their balance it's on the one balance right why do i have several ones i i just

54. 31:34

    i just want to have one but naturally and near is built is a low level platform with uh high level concepts built as a like extensions from the application level level uh so it's we provide certain tooling and the balances that are in here in in near protocol they are tied to the single account so these are native near protocol accounts this one is you can spend it immediately you can send it to uh you can transfer it you can attach it to function call this balance is

55. 32:24

    about the balance that you take to as a validator i see so so it's we're saying native account balance because uh there's there's you've broken this out into three places right so there's the the two parts the what you can spend today what you're staking and then some aggregate of those two those are the three views here well it's not the aggregate of these two it's an aggregate of

56. 32:53

    potentially more other contract more other accounts which are uh in the connection for example lock up contract is a as a sep it's a separate account right very i see so it's the aggregate is the of the balance that you can control from this account that's what yeah you see uh if we open another account which has a lockup account we display the lockup here uh you can navigate there you can see that it has its own native account balance

57. 33:31

    uh we what what people tend to expect is to have a sum of this value plus this value plus uh the the account could have delegated certain amount to the staking pool then it means that we need to sum also the tokens from the sticking pool and it's it's very complex to pull together and that's why we tried to do it first on the explorer side you could have seen we had quite a lot of balances here it was very hard to

58. 34:09

    reason about them people were always confused they they only referred to the one that calls total but we knew that at some point that total will not be total because more concepts will be introduced uh like more lock-up contracts or more you know like other tokens nft tokens what not could be also attached that's why we scaled back let explorer only focus on the native thing that we expose through the

59. 34:48

    protocol and uh wallet uh deal with uh like all the rest like you can see here that the uh nexus account has this amount of tokens some of them are reserved uh then the same goes for lockup again so it's nice breakdown a much cleaner representation and we couldn't make it on this on this page uh with the current design it should have been redesigned anyway uh to something like this to have a breakdown which would be

60. 35:29

    which would make sense to the end users after all yeah so i think i i think i get it in the end you're saying from the wallet perspective you're seeing the account that you control and any other sort of uh you know con contract level assets that are controlled by this account so that's the view from the wallet from the explorer you're talking about sort of the native uh token that this account is holding

61. 35:52

    and i guess maybe also a lockup account but there's no room here for multiple lock-up accounts i don't know if they would show up as a list or something so so it it makes sense that we're sort of keeping this focused on the account as you know some um you know from the account perspective you know on the network an account that engages in transactions and transaction history that makes sense you can see that this account has sent 72 transactions or signed 72 transactions maybe received one

62. 36:23

    something like that whereas from the wallet it's about asset management you know what what lock up accounts you're connected to and all that sort of stuff that makes sense yeah yeah so that's all what i wanted to cover with the um the balances also as we might have seen in the docs when you view account uh query you see the balance so this is a native balance essentially and this is a state balance and validator validators take balance

63. 37:04

    these are the two values that we are connecting from json rpc response to the explorer ui uh storage used is here storage usage uh essentially yeah it just bytes and it's uh 300 358 kilobytes and it's coming from the contract that we deployed on the account um what else and that's why we need to maintain 36 near in this account to to offset that the through you know yeah that's what wallet talks 360k there yeah as you can see three five eight zero seven

64. 37:52

    three nine and it's exactly the number we see here so correct at least currently uh we are uh the new version which may lower the cost and it could be just a different different number but it's right now it's very easy to observe how the storage usage is connected with the minimum balance uh yeah yeah and that's basically uh reserved until you reduce the storage on the account then it's unlocked you can do whatever you want with it yeah that makes sense

65. 38:31

    okay i feel like we've covered quite a bit here is there anything else that we should uh include in this i mean we've kind of gone over all the interfaces maybe in the explorer touched a little bit on the wallet as well yeah i guess that's all we don't need to like it could be a very different conversation about various balances and contracts but overall the explorer side is i i think it like

66. 38:56

    covers pretty extensively nice okay great yeah thank you thank you