NEAR NEP-639 Walkthrough: Automatic Shard Splitting

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:01

   All right, start the recording and uh yeah, go ahead and get this underway. Uh welcome everyone uh to NEP 639 discussion around dynamic resharding. Um this proposal is going to introduce dynamic resharding that enables automatic splitting of shards um based on their state size um without requiring the protocol upgrade. uh it's replacing the previous static approach um that previous NDPs had where layout changes were hardcoded uh into the specific protocol versions and required those

2. 0:34

   coordinated releases. So super excited about this um to get us going. Today we're going to have the NDP author Adam present an overview of the proposal. Um we'll then have uh subject mat matter experts um Andrea and Tran um give their technical analysis and some comments that they may have. After that, we'll open it up to community Q&A um to discuss or to ask any questions that

3. 0:58

   they may have and then um we'll wrap it up with a vote. Um so yeah, without any further ado, I'll go ahead and you should be able to have um presentation capabilities. Adam, if you want to go ahead and take over. All right, cool.

4. 1:42

   Do we see like the editor mode or the Yeah, we see the editor mode. We see the editor mode. We don't see the slideshow mode. [clears throat] It's interesting. It's grayed out. Do you have another uh do you have it another tab or another are you sharing just a tab or uh let me try once again weird it's grayed out I don't know what happened no problem if if if you just want to just share the the the editor mode I'm

5. 2:24

   sure that'll work let me open a a new tab and try to do it that way. Yeah, no worries. Maybe that that should do the trick. Sorry about that. It's okay. Let me out present. I might need to rejoin the meeting. No worries. No, you're all good.

6. 2:58

   suspense and oftentimes, yeah, I feel like every, it might be me because every time I do a presentation somewhere, I always have HDMI issues. I just I have the the dongle doesn't work and then I'm sitting there on stage like trying to do a demo.

7. 3:16

   It's not working. Oh, all right. Yeah. Okay. Works. All right. Let's go. All right. Adam, floor is yours. Okay. So I'm here to present the dynamic resharding NAP as it has been already introduced. Uh so let me first present the reason why uh are we even doing this. Uh the current as of now uh deployed uh version of resharding uh is manual pretty much. I mean every shard split requires a developer to identify which

8. 4:00

   shard needs to be split, what boundary account should be used. Then it has to be hardcoded and this requires a protocol upgrade. Um and it is a process uh that takes multiple weeks uh uh because preparing a release is is not that fast. So both the latency and the frequency are impacted and it also uh results in a lot of uh unnecessary manual uh work which might uh not be necessary. Uh so in order to change this

9. 4:40

   situation we propose a protocol level mechanism that automatically splits shards based on the state size. It will no longer require successive protocol upgrades once it is enabled. Uh thus the resharding in the future could happen much faster and without manual intervention.

10. 5:07

    uh the um overall steps uh of the mechanism are as follows. So at the end of each epoch uh during chunk application for each chart uh the chunk produces uh check whether the state has grown past uh the configured threshold and if yes then there is the uh computation of candidate triclit uh which I will talk a little bit more later. This uh candidate split is embedded into the chunk header. Then during block production, the block

11. 5:56

    producer if there is more than one proposed split picks the best to split and the priority goes to the shard with the highest memory usage. Uh this is then embedded in the block header. Uh then during epoch finalization which happens on each and every node uh there is step to derive a layout for for the next uh epoch I mean for the epoch after the next one technically and this is done based on the split selected during uh uh block production and then comes

12. 6:38

    the actual executing which reuses the exist existing uh resharding uh mechanism that we have in place since resharding V3 was released. It's just that the trigger condition is is slightly different. What's important to notice is that that proposing selecting uh of the chart split. It's all deterministic and it could be validated uh via state witnesses. uh so no malicious actor could forge uh any of this information.

13. 7:16

    Uh a few design decisions uh first how the split uh how the boundary account is found. uh it's based on the state size and what we do is uh a binar binary search like algorithm that distends down the mem uh trying to find an account ID that would result in a [clears throat] possibly best split.

14. 7:49

    uh that is possible because the mem nodes already store memory usage. So that this is already part of the state and it's subject to consensus. So this allows the boundary search mechanism to be deterministic and verifiable. Uh then uh we also have uh the timing constraints which say that uh we we only do one split per epoch and there is a configurable cooldown. Uh we're aiming for like two uh epochs interval.

15. 8:30

    So there will be no possible interactions that might result in splitting two shards simultaneously and also there will be enough time for uh for the nodes to preload the mery when a when another split is is uh upcoming. And last but not least, there is like a companion protocol feature which is technically independent but uh tied to this one. Uh there is a new strategy for assigning validators in case of resharding which tries to preserve as

16. 9:04

    much of the existing uh validators uh uh as possible. uh the parent validators are split into the child shards and and then uh some validators are pulled in from another shards in a roundropped fashion. This is supposed to be released together thus reducing the excessive state synchronization that uh usually happens after resharding.

17. 9:34

    Uh now about some impact of this feature on the operation of the network. Uh in in uh on RAM provisioning we might say that approximately twice the largest shardsman try needs to fit in the validators RAM. Um this uh includes like some space for the currently tracked chart, the parent of the upcoming split and there is also some headroom for uh orphan nodes resulting in the frozen layer of the parent I mean

18. 10:17

    the parent ment try being frozen. Uh there is a bit nuance here. It's all described in the in the net. So I don't want to talk too much about this but twice the largest shard is is a reasonable uh uh is a reasonable uh limit. I believe uh mantra preloading uh this is something that is done in the background. uh uh it oper on on a single threat not to uh uh steal the CPU from other operations. It causes some state deltas to accumulate. It

19. 10:59

    might take up to 30 minutes. uh but the impact of these accumulated uh deltas on the performance of the node uh has been assessed as uh uh non-critical. Uh one caveat for any external tools which uh uh need to read the layout or present the layout or are based on the layout is that they must from the deployment of of this uh of this feature start to rely on querying uh the layout actively per epoch instead of of of

20. 11:42

    relying on epoch configs. And uh one uh consequence of this as well is that uh mainet and test net layouts will diverge over time. So the test net layout will no longer be the same as mainet and uh that's an acceptable consequence. uh but something to be aware of and that's really uh all I have but I'm open to questions in case you are eager to learn more.

21. 12:15

    Awesome. Thanks Adam for the uh overview. Super appreciate it. We will go to our reviewers, our subject matter expert reviewers, and then after that we'll do a little bit of Q&A uh if that's okay. Um so yeah, why don't you maybe we'll start out with um Andrea and then we'll go to Sharia.

22. 12:36

    Sure. Sure. I I can start. Uh yeah, definitely uh in my opinion a good proposal. It solves missing piece of resharding. There are several advantages. Uh for one, we reuse V3. We run it I think twice in mainet. Uh so the resharding itself it's pretty proven technology.

23. 13:02

    Uh even on top of that we will ship uh this improvement on the stickiness for um assignments of shards which should make uh transition to the new ebook with new shard layout even smoother than it was before. uh uh [clears throat] then for the benefits uh definitely [clears throat] it would be good uh now to have automatic uh resharding instead of manual process. Um I remember when

24. 13:38

    before we had to run uh some tooling on the mainet state uh taking some developer efforts to figure out what would be the good split and then code this split into a code base. Make a release. wait for the release to go through and by the time release is actually going to mainet you find that the split you found one month earlier is not ideal anymore. So all of this goes away with dynamic resharding.

25. 14:14

    Um split will be found very close to the epoch boundary. So no chances that it's like on a stale um view of the shard memory usage. And besides uh what else? Uh uh concerns. Maybe my only concern was about memory usage for uh nodes uh throughout their operational window when we don't do sharding.

26. 14:51

    Uh in the past we could say um look there will be a resharding at this date. You have to add some memory. Uh and then you can kind of go through the resharding and maybe resize dynamic resharding it won't be possible anymore. Um there is a limit on how many shards we have on mainet. So you know that if the max is 10, you are at nine there will be one resharding event and then after that your node is not supposed to do any resharding. So you can they

27. 15:28

    provision the memory but still like you don't know exactly which epoch we are going to to change shard layout. So we need to keep this twice the biggest shard headroom. And I had a question for Adam on this. Um 2X is for validators. What about RPC nodes that don't have any me try?

28. 15:56

    What about RPC nodes? Uh well if they don't have any me try uh I think they will still need to uh preload the the the me try of the parent shard when the split comes because that's how we split the shard but then they don't need the uh extra space for the currently tracked shard right so I would say it's um half of the memory required from the validators. So it's just like the size of the biggest shark. That's that's

29. 16:37

    probably uh what would be necessary. Okay. Okay. Thanks. Yeah, that's all for me. Yeah, just just wanted to mention that like we force load the mere even for nodes that are technically not tracking it or have explicit instruction saying that we don't want to track it in the configs. It's it's a requirement and this overrides it.

30. 17:03

    Cool. Shant, do you want to go ahead and give your review? Yeah, sure. Uh reshading um dynamic reshading it's it's great. Uh it's a step in the right direction. And I feel like Andrea had mentioned we in the previous implementations of reshing we did have to go through a lot of like manual process of figuring out what is the split, how do we do it, schedule a protocol upgrade. It's great we are

31. 17:28

    getting a mechanism um that we can use to like automatically do the resharding and I think that's sort of like a requirement for any system that we consider scalable that hey they should be able to like automatically split and manage the the memory usage or the gas usage or whatever uh concept we may have for uh the sharding uh itself. Um we do have a bunch of u built-in um we can say like um what's the right word over here like um

32. 18:07

    fallbacks no not fallbacks like basically um we have like all all sorts of configs and stuff that say that hey we should not be like resharding too often so like from the 9 to 10 limit that we have like limits and checks in place yeah I meant checks um and that's it. So we can like keep track of like what the situation is like uh at least for like a couple of resharting events and then slowly we can like slacken the

33. 18:33

    system further. Um this gives us an opportunity to see what are the potential proposed splits and uh see whether the system works as expected. Um I think the current design also takes into account potential extensions that we may have. So for example, right now we have the memory usage uh as one of the uh as the main um uh criteria that we use for splitting. Uh in the future we might want to have things like gas usage or anything else and and I think the

34. 19:06

    design the way it is built right now is uh extensible to take that into account. um couple of things that um are slight concerns I think not too much but all of them are more or less like future facing I would say um so for example we don't have shard merging yet although that's something that we can plan out um ahead in the future and in the growth phase it makes sense to like

35. 19:36

    focus more on the splitting side right now um then there's also So I think it's more of a design thing but the symmetric the the splits are not like super symmetric and that's more of a consequence of um how the accounts are like structured and how they are like grouped. So like you can't split an account in the middle. It always needs to be on the boundary of accounts. Uh but again that's like a reasonable limitation. That's also a limitation of current reshing and uh it's a different

36. 20:09

    problem to solve in case like accounts get like too large. Um what else? Yeah, I think uh overall good proposal uh good direction to go in uh good feature extension possibilities. So yeah, sounds good. Awesome. Thank you so much.

37. 20:29

    We will now open up for anyone that has um questions from the community. If uh you have any questions or comments, feel free to raise your hand. All right. I No, thanks. No. Yeah. Go ahead. Thanks. No, no, no, no. Any question? No.

38. 20:56

    Well, you're just saying no questions. All right. Yeah. Cool. Awesome. So, yeah, for so from my understanding it's like we have um you know we're going to have like a memory threshold like once it receives that it's like all right now it's going to propose a split from the specific shard and then it'll take effect two epochs later. Um but we don't have a mechanism currently for it to like say oh actually it's not using it was just a spike in or traffic or usage or something or and it

39. 21:25

    won't go back. So say like we don't have that current mechanism to go backwards just just to go out is what you were saying right? Uh yes what I mean is like right now once a split happens then there's no way to like merge two tries together and I think regarding your question about potential spikes. We had done calculations behind the scenes in terms of like what is the amount of that needs

40. 21:51

    to be uh staked to actually have a potentially malicious split or well malicious is the wrong word but like force or split um I don't remember the exact calculations but from what I remember it was quite a large amount of NEO that you need to have in deposit um so it was fine uh gas usage is a bit more uh susceptible to these things. But that's that's kind of why we didn't use that as a criteria for splitting and we went with memory. Uh but yeah, if Adam wants to like add more

41. 22:24

    details, so she can do that. Um yeah, as you said, uh memory usage is not uh very susceptible to uh temporary or like even artificially generated spikes. it requires some P2 uh to use storage and it's not something that fluctuates all the time. It's it's a rather stable uh stable growth. So that was exactly one of the reasons why we rejected the idea of basing it off uh gas usage.

42. 23:03

    Gotcha. Perfect. And then I guess my only other question I had in my head was is there Is there any um yeah that's probably unlikely but maybe because two shard you know it's per shard that is u making this proposal would you ever see a situation where you have two shards that you know we're at we have nine shards already we're capped at 10 two of them are proposing that they need to split which one gets you know which one wins that race or wins

43. 23:33

    that proposal yeah that's that's in the select uh phase uh on the block production because like block production encompasses all shards. So the block producer if there are multiple proposed split would pick uh the shard with the higher memory usage to be split.

44. 23:57

    Okay. And there's also some nuance to that because we have added to the configuration uh something like a white list and blacklist for either forcing a split or blocking a split of of a particular shard. So that actually takes precedence o over the normal criteria. Uh but uh we're not planning to use it anytime soon. It's just for the sake of some possible future uses. So the answer is high higher memory usage wins.

45. 24:33

    Okay, cool. Cool. And currently like that the memory usage we're configuring as like a hard-coded value and if we wanted to change that in the future that would be required other protocol upgrade but for right now we have a a set limit.

46. 24:47

    uh the split threshold over which we split it is a yeah it's a value in the in the app config and updating this would require a protocol upgrade that's correct okay cool those are my only only my my questions uh yeah if anyone doesn't have any other questions in the group I guess we could move to a vote um be between our twomemes so Andrea thumbs up thumbs down.

47. 25:19

    Thumbs up. All right, we got yes. All right, Shan, thumbs up. All right, looks like this NEP is approved. Um, so yeah, we'll go ahead and update the proposal uh in the GitHub or yeah, if you want to check it out, go to github/neps.

48. 25:36

    This is poll number 639, dynamic resharding. I'm really excited about this. Um, and yeah, thanks so much for all the work you guys have done here. And um yeah, without any further ado, thanks so much and we'll chat with you later. Thanks. Thank you everyone.

49. 25:52

    Yeah, thank you. But