NEAR Blockchain Concepts (2/6) -- Genesis, Epochs, and Validation

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:07

   i will probably use explore to navigate us through so basically exploring some terminal basically the blockchain consists like the from something from the name you may assume that blocks are the core piece of it but it's it doesn't help i would say most of the time so let me start from where the like how the blockchain builds up and where it starts and how it grows so what we have is our near core is a piece of software which defines the behavior and the the rules uh how like the system will

2. 0:49

   work and there is some configuration that you need to start from and this configuration is uh is called genesis so basically genesis is a configuration that all the parties in this decentralized world agree upon and they reviewed main so they should review it let's put it this way i don't believe many of them will really go into every single detail but overall there is a like an

3. 1:26

   opportunity to do so uh there is a time when this configuration was created uh there is a name of the network that we use for acute human readable needs there is a hate uh of uh like we will talk about it a bit later but uh what i want to mention here that uh blockchain does not need to start from like completely scratch from like zero and there are a lot of different obscure configuration at least in here in your core uh there are the the

4. 2:04

   important part is the that um we have costs for different operations on the network and these costs are set in this file as well we have a separate tool to estimate those calls to essentially compute collect statistics on how long usually such operation takes and record such metrics for the future reference so like how much would cost to create an action uh or like to do to deploy something to create new account uh to call a function

5. 2:43

   like on on the base uh on a basic cost and provide whatever like a lot of configuration we don't need to understand them all right now the the configuration also the initial configuration also needs need to state who will be the initial uh parties to bootable network so these are set industrial leaders uh config so these are these four were for machines for servers that were uh as it's easy to uh

6. 3:20

   a reason nf states for near foundation with controlled by near foundation and that's because when we started the main net we essentially did silently uh we didn't uh announce it widely and we wanted to make sure that it it runs for a while and then we onboarded our validators to the network so that so in initially our plan was to you know collect all the distributed parties and aggregate the genesis file and boot the network from scratch

7. 3:59

   at this like at the same time but uh last year during like our rush to release we realized that it would be too much of the effort and it would be uh like quite hard to follow uh for to to make to make uh in like in time whenever we had any like deadline we always like missed it so instead what we decided to go is to launch main net in phases and you can hear that we refer to some like phase two for example uh thing

8. 4:38

   which essentially enabled what was the last phase to launch the main net it unlocked uh transfers and that was the last piece and there was phase one at phase zero those three phases starting with zero were happening last year and like all the configuration you can re like try to read about them in a way but i covered most of the important settings uh over there and there is also the important piece over here is the records

9. 5:12

   uh which is essentially the state of the network uh which the the nodes will start with without these records like there is no way so if for example blockchain does not have an account you any accounts uh you cannot create a new transaction because you can confirm that your account belongs to the network so it's natural for a blockchain to have this initial state there is the thing called total supply so it's an amount of near tokens uh available on the network when it starts

10. 5:51

    and there are certain balances uh on each of those accounts by the way it's in yorkton year so it's a 10 to the 24th power so it's essentially like something around like this is five near i believe or something like like that so it's not like ridiculously high amounts it's quite small but uh these contributors uh that near and foundation.near uh were quite uh loaded uh balances i believe and also these uh

11. 6:27

    nf validator accounts as well so you can see that there are like records of type account and if we scroll down a bit uh you can see that there are contracts deployed one contract deployed on near account that's uh i i will refer to this later when we will talk more about the count system but basically you can see that it was not deployed by anybody it was right in the in the genesis file and uh another was pulsed appear

12. 7:05

    i guess it's a bot uh account or something and you can see there are access keys again without access keys even if you have accounts without access keys you can't claim that it's your account just providing the uh you know account name account id you always need a signature and to have a signature you need to have a key player and one of the public key should be recorded on the blockchain so

13. 7:35

    we can see those accounts have uh access keys and this is uh this is basically all on the genesis side we have also the data uh record which is essentially a state of that contract uh that we just saw uh linear account and there are like some other configuration options just for near needs are there any questions so far nine all right so basically this is the how the blockchain starts you need this file and you need the piece of software

14. 8:17

    that understands it and can operate on top of the protocol with whatever like we we call it we just have numbers here so protocol number 29 uh is what's uh required to run uh genesis so it's near protocol version 29. uh whatever it means and then you have these four machines nf validators uh so these are servers where we run near core they boot up and start operating they have a routine so it's it's it's alway it's an ongoing process

15. 9:03

    which is called blood production so they essentially need to well block production and chunk production chunks i will cover them later but overall validators are in charge of creating blocks and chunks so they do it regularly every second or roughly every second uh in near core and you can see on explorer's side that there is some average block time production um and you can also see this number it's changing so essentially

16. 9:41

    look up the latest uh i don't remember exactly like uh one minute i guess a time frame and see how many blocks were created during that one last minute uh and it's roughly 60 blocks uh created in the last one minute uh so that's how we how this number uh is counted uh block 8 is essentially a number of the the block number uh that is produced on network uh one thing i want to mention that it's monotonically increasing but it it does not need to like

17. 10:21

    not every block exists sometimes due to validators being offline they skip their uh their turn and they don't produce uh block and then another validator will uh come in and produce their block that is scheduled to produce uh for them so they they the the numbers the blockade is not sequential uh meaning that there could be some like missing blocks in like in general they're quite sequential and usually like most of them exist but there are

18. 11:02

    some hiccups here and there when you can observe that quite many of them are missing so every second we produce a new block we started from 9 million block number almost 10 million blocks this essentially means that we had mainnet network running before that which started from zero but at some point at this point i believe uh we had some i i guess it was a bug or some behavior that we didn't want to deal with uh like to do a proper fix so instead given that the network was

19. 11:41

    not under heavy use we decided to dump whatever was in the main at that time and that is what we can observe here in the records actually so records uh is the latest state of the network essentially and at the point of 2 million or 9 million 80 thousand twenty yeah what whatever this hey we had that state where we had just a few accounts yet and some of them had some like balance changes

20. 12:20

    and like some balances on them and this happened last year in august 21st since then we arrived to 36 million blocks 36 million blocks is the latest blockade so basically uh 20 26 27 million blocks later uh we arrived to a different state we still can actually dump the network uh as we did back then but uh unfortunately it's naturally this uh records array will be enormous and

21. 13:06

    like too big to to you to to review and it will and given its in json format it will be like completely unreasonable to load that's uh what happened around a month ago with testnet actually uh testnet is not that heavy i would say but still the current genesis file this one for testnet uh with records is five gigabyte json file uh we call this process hard fork even though it's like it's it's a wrong term i would say it's a hard reset so we essentially reset the network to

22. 13:52

    the to these values uh we can edit it we can edit the json file what whenever whatever like to whatever state we feel is needed we could remove accounts we could uh remove access keys like anything like that so it's it's not safe from blockchain a point of view to have hard resets but for testnet we decided to go that way because well it's a test net not a huge deal seems though i i'm still

23. 14:31

    i can't say that it was the best decision but anyway main net so we don't want to make this such hard reset on mainnet because we will lose the history and we cannot really prove that these records will be correct so it will it would be quite a quite a major incident if we would need to do hard reset that's why we have a lot of migrations we have a lot of quirks here and there and that's why we sometimes introduce new protocol version just to patch

24. 15:10

    certain behavior and uh bringing missing data that for some reason we like didn't process uh during the like operation of the network before so we have quite a few bugs that caused us a few inconsistencies in the network and we had to implement some patches with the protocol version upgrade so that's how we upgrade the network uh but let's get back to the core concepts we have those validators

25. 15:48

    we started with four validators over here with nf validator one two and three four and they operate and they produce blocks they produce chunks that's fine but how do we onboard new validators we could we could have like a essentially a process where validator would just come and say well i want to start valeating and we would allow them to do so but that would be quite challenging to implement because those validators need to have history those all leaders need to be uh ready to

26. 16:26

    to operate also the other like the schedule needs to be changed as i mentioned there is a schedule uh where each validator is allocated to produce certain block uh there is no other validator can replace and like produce a block that is supposed to be generated by a validator let's say nfl later too so if the new validators would uh immediately be onboarded to the network we will have a mess so we introduced this concept of epochs and

27. 17:04

    essentially we have uh it's like a voting mechanism for like for a president or like a prime minister where instead of going through it every four years or four five years whatever we go through it every 12 hours or rather not even 12 hours but epoch length this number of blocks this is like if you if we have a block production uh every second and so we would assume that it's a number of seconds it's roughly it's it's exactly 12

28. 17:44

    hours uh long i have a question go ahead uh does it include the missing blocks or not i mean i mean the number it does okay yeah so if if there are missing blocks they just like okay we keep going so basically they let me there's no good instrument right now but oh actually it is there is the risk of the instrument come on so there is a script which is just a cron job um so we started with this block uh over here so the script uh

29. 18:44

    saves historical data uh for each epoch the what exactly was the the stats for the epoch and as you can see uh the blockade is like almost this like it's as you can see it's exactly the epoch length variable here however sometimes this specific block 13 at the end will be missing and but you still need to have some block produced to conclude the end of the epoch and that's why at some point you can see 13 13 13 13.

30. 19:49

    okay it was fine but then it turned to be 17. uh what happened is essentially the there was there were no validators to produce those four blocks and only like uh three three blocks and only on the fourth after the thirteen uh on on the seventeenth block uh we finally produ the exam validator finally produced a block and the epoch was ended or rather started a new epoch uh or rather ended yeah i guess it's a last block in the epoch so it's it's the

31. 20:31

    end of the epoch answer your question yeah great thank you um so we have this uh mechanism of epochs they happen roughly every 12 hours and while they just produce blocks we can open any of the blocks uh they they could be like they could we call we call them like empty it's it's hard to find to block these days so there are no transactions in the block but the block still is produced have timestamp of when it was created well it's finalized they like

32. 21:23

    what i want to point out here is that validators produce blocks no matter what even if there is no transactions at all blocks are produced that is because you cannot prove that there is like something does not exist you can only prove that there exists zero transactions when you produce a block so that's the reason and that's why we have so many blocks which we could potentially

33. 21:54

    hide and or or rather even not have information about them in the first place but we the blockchain needs to create those and uh actually spend some resources to do that uh um yeah and as you can see the currently it seems like we have quite sequential block production so there is no missing blocks over here uh but it's uh it could be that uh like some validators would be offline and at that point

34. 22:40

    they will not produce block uh we are still in the works with the uh on the explorer side to have this similar page where you would see how many blocks were produced by each uh validator so you can see that this validator uh was expected to produce 400 blocks but only produced but but did not produce out of those for hundreds uh it did not produce uh one block so it uh it's i guess it's a

35. 23:18

    it's a weird statement uh on like it's it's easier to look over here i guess actually this 17 is a good place um so yeah so our adjacent rpc returns the following structure about every single validator and there is no non-produced blocks and num expected blocks so this validator so we control it uh was expected to produce this amount of blocks but only pretty well produced one less

36. 23:59

    than it was necessary what what is even worse this nfl leader three uh was totally offline during the the whole epoch so for some reason we had outage on that server the network operated fine we had three out of four validators online to progress network needs two thirds of the nodes online her question in this regard like for example in this situation we have four validators a specific amount of blocks like it's constant and in the statistics we have

37. 24:45

    like uh one validator completely missed the epoch uh so it doesn't mean that uh we actually miss that amount of blocks and how come in this file like if we sum up all the expected blocks numbers we won't get this uh 43 020 blocks yeah so actually i i just realized that it it's it's an interesting thing but so if you sum up these numbers uh 2 plus 1 plus 1 you will see that it's four blocks more

38. 25:29

    than is needed and that's exactly why there is this uh push to the block number so basically uh what what this means that throughout the epoch they produced uh enough blocks uh the for for um forty three hundred for the three thousand uh two hundred blocks and then it seems like it was turned for uh nfl data to produce a block then for an evaluator two was uh was scheduled to produce a block

39. 26:08

    and they they missed those can i ask you to like uh make a font bigger because like three minutes oh okay so there is completely different numbers yeah okay okay so okay so it's it's not about this team but about previous team i can i can add the information so when you when you are looking at the the blocks and looking at the number of transactions that were included in this block even if there is zero uh it does not mean that uh nothing was uh uh nothing

40. 26:55

    was making at this block because uh it could be hard right now uh apart from transactions we have receipts and it's possible i just checked it it's possible that this block has a receipt process correct yeah i i guess it's something like more deeper and it should be covered after the receipt so and the receipts here are very yeah receipts are very hard to understand uh even for

41. 27:34

    like uh for the guys who knows something about blockchain so receives a very weird guys it's it's it's not that hard actually but uh so the only thing that i wanted to to say is that if you are seeing zero it doesn't mean that nothing was made in this block yeah uh i i will cover it right now then uh that's a good point actually so here here is the json rpc endpoint that is that is called block we can provide params to it we provide the

42. 28:12

    block id we can specify either blockade or block hash over here and what we have in response is the author so they actually the validator name to produce the block we have a list of a chunks a list of chunks i will cover them a little bit later and we have a header of the blog that is produced so we have like a list of leaders who approved uh the block so they essentially synced up and they confirmed that um the the consensus and these

43. 29:02

    validators conclude that the block is final uh there are some other useful information over here uh what's interesting is there is uh like uh prehash which is the previous the hash of the previous block uh there is a hash itself where is it yeah it's it's hash over here it's a block hash and there is a hate so i just queried this number and so it's no surprise that we see it and what else

44. 29:41

    um conversion timestamp uh is the uh is a number in nanoseconds where the block was produced uh there are quite quite extensive info over there but we don't want to focus on that right now i want to focus uh for for just now on chunks so the nature of near local is that we have uh scalability in in our system uh we we support uh horizontal scalability of execution meaning that we

45. 30:23

    don't need every single validator in the network to validate every single transaction every single block every single child every single uh list of transactions let's put it this way we built sharding into near protocol design and blocks are essentially a virtual entity which does not include transactions by itself blocks contain like records the chunks that are included in the in the block and they don't care what's inside the chunk they the block only

46. 31:09

    care about which chunks are included and then each chart tracks uh different charts charts are are just like there is a chart id chart id is a is an integer uh from 0 to n minus 1 where n is the number of shards so basically right now near operate midnet on here operates with single chart um which means that we don't leverage scalability yet uh the reason is we just don't need it given that we only have like three

47. 31:47

    transactions in a block it's not a problem to handle uh for near protocol uh but overall in the future we'll enable uh the feature the the support and configuration essentially it will be configuration change to have more charts and each chart has their own produce their own blocks in a sense of regular uh blocked other other blockchains uh but this uh blocks on a shard we call them chunks so the parts of

48. 32:26

    uh the final block that will be produced and every every chunk then has sort of the same property as a block and chunks are the entities that collect the transactions inside of them and that's where the execution happens one question so every chunk so like every chart has like their chunk blockchain and then the blockchain has like the block has n number of chunks yep okay okay cool though some chunks can be missing

49. 33:14

    like if like as let's let's get back a little bit to to the block production that i covered like blocks could be missing because the validator is like not online uh blockchain will not get stalled uh it will progress uh there will be a hiccup but uh it's like okay we'll produce block not the next second but the second after that so there is no problem for the operation of the network the same goes

50. 33:46

    with chunks chunks are produced by like different validators as well and if chunk is missing it's fine block will still be produced in the case of a single chart this could mean well then does this mean that we can all like have an empty list of chunks uh in theory you can view it this way like there could be blocks without chunks uh in practice in year protocol again just to prove that

51. 34:23

    something does not exist you need to prove that something exists and to do so uh when there is no chunks at all in like a new there is no new chunks to be included in the block uh block producer the validator that is in charge of producing block will include the chunk from a previous known block so it will send from the json like from the rpc point of view uh you will see that from the rpc point of view and from the internals point of view from the traffic that goes through you will

52. 35:02

    observe that uh some blocks include the chunks that is already included in it like in a different block what you usually want what you usually want to do with that is just to ignore duplicate so you ignore the chunks that this hate created it does not equal to the blockade um i hope that that makes sense i have another question while we're here uh i thought that uh like why uh especially with the data wants to be a validator is because a reward like uh producing

53. 35:48

    the block is uh rewarded by the network like by producing a new coin if i'm not mistaken and this coin goes to the validator who produced the block uh and currently we can see that uh field validator reward is zero i guess it's duplicated it's just duplicated field okay but uh like uh do they have uh rewards uh on every block producing or uh they got a reward after the epoch like they are rewarded on the

54. 36:27

    epoch boundary so when they well on the second sec only at the end of the epoch they will be some sort of virtual account and accounts how many blocks uh each other data actually produced and how much was expected and after that they were rewarding yeah so let me put one comment and then we'll go to sandy uh i uh i want to mention that after the epoch ends all the validators perform the same operation

55. 37:05

    they count how many blocks were produced by each one validator due to throughout the previous epoch uh and if the number of expected uh so they compute the ratio essentially uh of how many was expected and how many was actually produced so they they see at this uh to the computer's ratio and if it's less than 90 percent validator does not get any reward if if validator was online for more than 90 percent of the time so it has great online

56. 37:51

    uh they are rewarded according to this ratio and also according to the stake that they uh they have so if you look here uh different values have different stake so according to this number uh the the higher the stake the higher is the reward the reward is uh constant but uh it's it's computed based on the stake uh that each validator has and this uh on offline uh can affect them and i i i thought that the number of seeds like

57. 38:35

    number of blocks uh which should produce a validator is computed based on the stake but i can see it right now that it's also this like uh so yeah but not number of seats is is a different metric number of seats how many blocks you will uh need to produce so you can see this validator with so high stake expect is expected to produce 400 blocks uh while this one is expected to produce only 145 yeah but a little bit at the top

58. 39:15

    one of the top can you scroll top please that's actually kind of weird yeah like 10 millions of steak and he have to improve it's weird let me see maybe not normal numbers follow the bug maybe we could uh so basically these i guess they're sorted let's see yeah now they're not sorted but as you can see some voltages like bison trails they're they're quite big they expect it to produce so many blocks while others are expected to produce

59. 39:55

    much less i guess near staking has some sort of bugger yeah it could be yeah i think uh i don't think the value is blocks i think it's uh it stays in feather that's the number of how many epochs uh they they were present the present as a validator maybe online ebooks yeah yeah thank you mario thanks yeah it's sorry about the confusion in fact yeah it's a it's on uh on explorer side is where we see this number over here

60. 40:37

    so it's it's quite a you know obscure metric unless i told you you would never guess what what this numbers then the new design will be better yes uh like everyone reminds us and you want to yeah oh every time when you produce a block you get all the rewards for that block from the transactions or not no you are allocated so the reward is completely decoupled from the transactions so no matter if even if there is

61. 41:16

    no transactions in the network at all uh you will the validators will be rewarded so what does stop me to just create a validator who creates blocks which are empty all the time others can uh like decide that you you are doing bad and like destroy you well you you will need you will need to implement first you will need to implement in your core in a way that will just yeah ignore produce like including chunks second

62. 41:53

    uh you will need enough steak to join the litter pool and to do that you will need uh three million tokens uh even more yeah but like what if the current validator is just like bison trails figures out oh instead of just validating i can just change the codes to produce empty yeah but there aren't multiple validators they definitely can do that that will make the rest of the validators working harder meaning that

63. 42:30

    they will need to process the transactions i'm not sure how exactly we mitigate such a behavior if we watch for that at all no not now i don't think we do it now that's why i was like i know there was like some penalty thing but like it's not working right now yeah um from my perspective uh having uh the proof of stake is actually like uh i don't think if you have a validator node like

64. 43:10

    which enough to run near core uh that it's too hard for you to produce not uh not them not empty blocks like it's not so hard for you as producing empty blocks like we're not proving the work so it's kind of not very heavy stuff so i guess it's useless to hug the node and risk your stake to get this like this hearty thing yeah but like as long as you don't get slashed you don't care and other people

65. 43:46

    don't care they take you because you just kind of uh yeah i guess it will be once we get with slashing mechanisms in place that would be considered again because well we we don't have slashing right now flushing the mechanism which is essentially banning the node and getting their stake from them currently it's not enabled we have some code but it's not very well tested and it seems to have some

66. 44:20

    bugs when we can see consider uh multi-shorting sharding so that's the problem so uh currently you can delegate your tokens to any uh validator because you don't care and if something went wrong you will get your uh delegated tokens back slashing yeah one slashing will work uh you are risked as well to lost your delegated tokens it's actually debatable like there was a debate when we had like before maine at launch and there were different ideas whether

67. 45:01

    we should slash the delegators as well or should we shouldn't uh i can send you a link if necessary i'm sorry i gotta run i have back-to-back meeting here uh so you can maybe share some of your experiences uh among each other yeah i i i guess this topic is kind of big enough uh to split so i i guess we will we'll better continue next time sounds good to me to be continued see you yes see you guys thank you thank you

68. 45:39

    bye bye you