[NEAR Protocol engineering interview series] Introduction to State sync (March 29th, 2023)

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:02

   nice so yeah stating uh please give us the context what stating is why we generally do it in our protocol what is the purpose of it let me share some documentation yes this looks right so answer to this question is mostly documented we have a page called how sync works it describes all kinds of sys we do all kinds of s a node needs to do if it starts when it is far behind or not so far behind and in one other case

2. 0:50

   which we will cover in a second so if your Note is very far behind first it gets all the headers so that it has a complete chain from Genesis to the head and it can verify that all headers are created one after another without skipping without inserting anything fake because blockchain next block contains the HH of the previous block so step one header shink relatively simple Step 1 a not implemented we will not cover right now step two

3. 1:32

   staing this is needed for the case when you have a lot of blocks to when you are behind by a lot of blocks namely a few EPO and to avoid the terrible work of processing multiple tens of thousands of blocks or even hundreds of thousands of blocks you can just download the some welln you can just download State corresponding to some wellknown point in time and continue processing the remainder which is not so many blocks from that

4. 2:10

   well-known Point what state syc does is it lets you download State at the beginning of the current ook and then you only need to catch up in that EPO by processing a few thousand blocks which is relatively fast cool that's scenario one and scenario two you are a Chun only producer who exists only to generate produce chunks you configure the note to not track any particular shart so you kind of start to know that

5. 2:49

   DRS nothing but as you join and stake the protocol will tell you not to produce chunks for a subset of shards and the next EPO you will need to have the state of those sharts available and maybe a few epics from that time you will need states of different subset of sharts and how do you switch you don't want to go back in history because you explicitly said you don't care about state of sharts you only want to produce your blocks your

6. 3:20

   chunks and so you download well-known States well States corresponded to well-known points in time which is the beginning of the EPO you process blocks for that subset of sharts and you have up toate State when the EPO switches and you can produce chunks when they are expected from your Lo okay so the second scenario seems like crucial and unavoidable the first scenario is still uh not happening on at least archival nodes and can be avoided on RPC

7. 3:59

   nodes and that's what we're doing right now so uh right now State s is disabled why so archival nodes don't need it because archival nodes need to have all state after each block and they needs to have each block yes RPC nodes avoided by downloading a recent snapshot we provide on this three internal modes get snapshots instantly from gcp very convenient and we disabled it today because State Sy is not in a good State

8. 4:34

   because our state is like 100 gabt on test net maybe 50 gabt on Main net and to make all this state available to a single peer that wants to sync multiple hours if not tens of hours of computational work needs to be completed by its peers and we know that making the state available is such a computational burden that validators actually see impact on their app time that therefore they throw to requests for State syns

9. 5:14

   therefore you can't possibly get main net state today if you ask validators to give you any bits of state and testet it's just huge it's just yeah okay to show you how big it is I have a graph I have an experimental Noe which has dedicated threats to each chart which constantly creates all the parts of state and writes them to S3 doesn't matter where exactly right St but what what matters I hope you can see the

10. 5:52

    graphs the orange line is chart three left side is test net right side is main net and you can see that this orange line is moving up very slowly on both graphs and it shows that it takes you in main it maybe three epics to dump a single state of shart three on test takes you I don't know 10 EPO maybe to dump state of shs three yes you cannot expect your peers to do all this computational work to SN a single a single

11. 6:31

    and my understanding because State on Main uh main net like State column on Main net is much bigger than State on test net and it is not nope test net shart two and three are 60 50 gabyt and all of main net is like 40 50 gabt all sharts I said 70 but okay like plus or minus maybe 70 I don't I don't remember exact numbers and they will change today is March 2023 okay uh so I think uh the problem with shart three on testnet is that it's

12. 7:13

    sweatcoin shart right and and I don't know what's the problem with SH M but even it's also coin oh okay so we just cannot fix this problem by recharging even they uh like having a lot of smaller shares because we always will have a huge sweat coin right I don't know maybe but this is not a viable the feasible solution that we can easily yeah I understand it's just we have we will always have an unproportionately huge shart is what I'm saying and we cannot expect all sharts to be

13. 7:58

    even I hope it will not be the case necessarily but we have to work with unproportionally big sharts we have today okay so talk a bit about the solution that we have right now that we are working towards implementing okay we have a design doc called go State syc V2 and it says that to make staing work make it great make it revolutionary new more scalable performant and reliable what we need is to make the dumps of sharts take not 70 hours but

14. 8:39

    maybe two three hours that would be fantastic and I see two Dimensions vertical and horizontal to make it to speed it up by vertical I mean let's use flood storage to generate dumps of State and there is a prototype from marim who left us that if you read flood storage it actually takes you maybe 10 times it happens maybe 10 times faster which is understandable because all those nodes you need to do a lot fewer random

15. 9:20

    lookups to get the states corresponding to sub trees you wants to export and all this concept of State parts that need to be sub trees we can also examine and reconsider but flood storage opens up a lot of possibilities experting sub trees was natural when the whole state is a three why not split it up into sub trees and then just merg it with flood storage we'll think about it

16. 9:51

    again but good that we have a prototype which is 10 times faster that's nice what is the horizontal horizontal Dimension um why not use multiple nodes to dump the same state of the same shart to some well-known place like S three so you could roughly use 10 machines maybe 100 machines and make it I don't know 100 times faster that would be nice in in on top of making the process 10 times faster by

17. 10:31

    itself so that lets us reduce the Vol clock time until the dump is complete and noes can use it to sync and using flat storage we let us decrease computational work needed to generate a dump okay I for some reason s that it was either our scenario where we either approach like vertical scaling or horizontal scaling but yeah it doesn't it it doesn't make sense not to PIR it yes so can you show us some code that illustrates stat in algorithms M main

18. 11:11

    stages what the node goes through just to sing one sh okay so part one client actor determines if a note needs to think it chooses if it needs the header sync blog sync State sync all of that mhm somewh somewh anyway yeah exactly exactly this so if syn is needed and I'm not currently sying let's see what is going on header syn happens always when the the node starts unconditionally then we check if State sync is needed and to do this we go to

19. 12:15

    the yeah there is a function which tells you when it's happening and one of this the condition for triggering stat it's an interesting one because it's not a fixed amount of blocks MH it's this it it says that header head and head are at least in different blocks mhm and if I read this correctly headhead is at least sorry not in different blocks headhead is at least in two different epochs like the not in the next EPO and not in but at least in the next next EPO

20. 13:14

    that's what I'm reading okay yeah so they are not the same and not even not even the next ook because sketching up to the next EP may be a matter of a single block so you need it needs to be at least one full EPO behind to trigger St Sy but there is also a condition that I maybe your EPS are artificially short five blocks and then there is an extra constant on top of that to make sure it

21. 13:45

    doesn't happen too often yes that's the condition for triggering State Sy so then how it works if it's needed the state sync object is created in the client then client actor I don't understand why client actor is driving this part but it is maybe there is a good reason I haven't thought about it so it checks what's the status if it's state sync then okay let's proceed with State sync and state sync itself is a state machine

22. 14:31

    it has several different states no intended yeah sever several different states and you can dly see them represented this shart sync status so if I need the header if I need parts if I'm scheduling the parts to be applied the difference between applying and scheduling I don't understand and then complete and the extra step for to instate for to instate SP if rearing happens in that part I haven't looked that is complicated and also if you're changing Shard layout you cannot

23. 15:22

    do stting the next EPO and I totally understand why not because it's a corner case which doesn't happen often but hard to handle yeah may maybe we should handle it but let's get back to it so State one what I didn't mention is that the state needs first you need the header the header represents the state the root of the three of your Shard so that this well know point in time that you dump and that you apply

24. 16:01

    changes to and contains also the last chunk of the previous epoke and some proofs so that you could verify that this is all legit and not made up MH but mostly as soon as you all you need is State Ro and to believe in this state root second part you get the part second step of this state machine is get the parts which combined give you the correct state currently State parts are

25. 16:36

    hardcoded to be 1 Megabyte in size so shart three of testet requires 50,000 parts to be downloaded to they that's a lot of parts yes that's a lot of even simple Network delays to send the request and get the request yeah MH so status of every part is tracked separately um GitHub GitHub no this one yes then for each part it's trct when the request Was Made Who was the target of my request and I believe only one request for part is sent at a

26. 17:24

    time and only if everything is correct we're moving to the next stage which is scheduling so in scheduling what we say so I have all the parts let's start another thread that only works on applying those parts one by one yes and it calls some task Schuler and to see what the task scheduler is you need to to go three levels up the stack but it basically uses one of syn chop actors to run a loop over every

27. 18:07

    part to apply it mm and so after you're done applying you have the state and you can skip all of that you're basically done uh do we validate saying after we download the state like just just the hash right even when we have the part with validate that the part represents at three such that from the given route I can reach all the nodes contained in that part the nodes contain sizes so you can at least verify that if

28. 18:54

    you can take if you have the full sub three that that size is of that sub three matches and because e are heres of the nodes you kind so hash of the node kind of hases also the children it is very hard to fake some data in the three so it does validate that the three itself is correct and when you have the global you combine the full State yes you can verify that the root of the

29. 19:27

    three is the hash and that hash indeed matches what you were expecting mhm and and if it doesn't match then you delete all state parts and you start from from the beginning and if you restart you also delete all the state parts and start from beginning right if you restart what if you restart no no no no if you restart the load the progress is carefully tracked per part so if you restart the progress is persistent but for a long time we had

30. 20:02

    an issue that it looked like there is a database corruption but there is no it was just a matter of unwrapping some Network code which expected some block to be available but as soon as you remove it you restart you have the progress per state part and you can happily keep downloading parts or in case of what happens if you restart when the parts are being applied I haven't tested but it seems that they will be

31. 20:34

    rescheduled and reapplied but yes I haven't tested that part but downloading parts right now is a matter of downloading them from S3 will be matter of downloading them from S3 uh as a single part or like as like a a huge pile with all of the Parts but the moment it is files of one megabyte size so sharts can require 60,000 parts to be objects to be downloaded that seems like it's going to

32. 21:17

    break something when something triggers many notes to restart the same so M maybe several nodes maybe we have a huge database migration and several nodes will undergo this huge database migration emerge from this very late and go to a three requesting villion files each roughly at the same time what will happen then we'll have three handle it if it's the same files I know that this Tre is like a powerful outside

33. 21:56

    Corporation of Amazon but still I expect it to Cle it I also expect pagota to pay a lot of money to three for all this agress traffic We already pay them a lot of money for storing and ser serving all our database snapshots so that should work and sometimes S3 BNS you for a me saying you have too many requests but we can introduce time outs red R with after some I don't know minute or so should be enough to make

34. 22:35

    reasonable load on a stream that seems like solvable problems solvable by paying a lot of money to Amazon okay and in the future it would be really nice if we could decentralize this approach to somehow yeah incentivize the community to run to obtain State generate State Parts in advance to make them available and to serve State parts to other nodes which need syncing or to chunk producers which want to switch

35. 23:07

    from one sh to another okay last question I know that the dream is to have an RPC note start from Genesis and in the matter of 10 minutes or I don't know how many but like minutes be uh able to produce everything uh new blocks being see but what is an achievable dream for like next six months even what you describe is not Out Of Reach if we Implement EPO sync and we don't plan to implement it in the next six months unfortunately EPO sync is

36. 23:47

    basically very fast head syn because you don't download even 1% of headers that would really make it faster a work around work around would be to provide a snapshot of database which contains only headers right so then you can skip other syn beautiful and you just download State and go where you need to go that would be achievable in maybe three months maybe six and so in three months I want a note which is configured to direct no sharts

37. 24:27

    which acts as a chunk producer to be able to freely switch between Shard zero Shard two Shard three not caring much which sh at is and not avoiding any shart because our state our Dum process of State sync should be fast enough to handle it okay there is also a question of whether our application of State Parts is fast enough maybe not but we haven't tested it much because we cannot generate dumps

38. 24:59

    of shart 3 this is a blocker to test an application of state and yeah to apply shart 2 on testet takes a bit more than an hour so basically in three months we don't see uh downloading and producing uh of State Parts it will not be a bottleneck in production should not be the Bon yes okay cool thanks for your time I got to know I got to ask all all that I wanted to ask I now know more

39. 25:45

    about code so that's very very nice and I hope it will be useful for anybody who is willing to watch it I don't know if somebody is that clinically insane in our company but maybe there some people other than me okay thanks for your time thank you very much nice interview okay