Why NEAR went from 4 to 6 Shards w/ Bowen Wang

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:04

   uh the serut uh of near is in a way determined by how many shards uh it has right so uh previously we have uh four shards and um that was sufficient for quite a long time and I I think in fact until the end of last year we had a pretty low utilization rate uh across the board right so I think we were like at maybe 20% utilized across uh all shards uh until the end of last year but then uh near starts to get more traction more adoption uh there's you know a lot of

2. 0:38

   users coming through through kijing uh and also with the hot launch this year get a lot more usage and what happens is that um on some of the sharts and it become it starts to become full and what that means is that um you know us when you send a transaction you get processed right away uh but the starts get full it get start get conjested means that when you send a transaction it will not get processed right away you might wait for a period of time before it can get

3. 1:09

   processed uh and uh that that is really a suboptimal experience for users because you know if you're like you know telegram user just get unboarded all to hot without you know knowing like much about blockchain and all like you're like oh I click this button but like it doesn't do anything for like 10 minutes what what the heck is going on right um so yeah so it is important for us to uh

4. 1:35

   improve the seput or like capacity of the of the network uh and uh yeah so essentially you can think of by adding more charge we are improving the serut of the of the network uh it's not as simple as that but like you know conceptually like you can think of like well we we added two more shs so that's like 50% uh capacity in compared to what we had before it it is a pretty uh pretty significant we had we had anticipated uh sometime in

5. 2:09

   the last year that um uh there may be as maybe in the future there's going to be this uh uh this this increase in uh in utilization and and we might need to do something about it so uh we started working on uh this project called rearing uh so basically it's the ability to break one Shard or like split one Shard into two shards right so the idea is like uh you know when the network

6. 2:37

   gets more uiz usually what you will see is that on on some sh maybe there's like popular contract or maybe a set of popular contracts so some sh might get start to get full and uh what you want to do there is generally to break it into two Shar or like more Shar then uh actually like you you kind of spit the usage uh and then uh and and also like uh incre increase the capacity right so

7. 3:01

   let's say you have like two contracts on the same sh that are both uh heavily utilized then because they all live on the same sh then uh you know other people people generally start to suffer because the transaction cannot get processed in time uh so the way to to to handle that is to let's say split that into two shs which uh each of the contracts a different short um so that

8. 3:24

   that's kind of actually what what happened uh we had uh Kai Ching contract and one of the cing contracts and the hot contract the main hot contract on the same Shard and that you know that's like uh two of the yeah the like top two deps across web three in the same Shard that it's like not not not sustainable so actually SP that in two shards and that was actually so the restarting project took us maybe like a few months to I mean started before all this you

9. 3:56

   know crazy adoption happened uh so we were actually so it's interesting because like uh what what hard was first launching is actually what we were like about to roll out the first rearing and then it's interesting because back then uh when we first started working out the rearing um shards three are like the origin the old sh 3 which had a threat coin contracts on it was the one that was more heavily

10. 4:20

    utilized so that's why we first um in that release we actually spread shs three because that's what we tested was and we want we want to make sure that we can get everything uh correct uh and so basically what first start is rating Shar three uh which actually by by the time we actually WR all the releases already not the most contest Shar uh so that's why like basically right

11. 4:43

    afterwards right right after we see that you know it goes uh fine obviously there are uh some issues we observe during the release uh then like we quickly address them but basically right after we see that okay everything can go through we can actually do the sprit of any Shard uh into two shards then we right away spread uh Shar to in to sh as well no it's all life yeah yeah yeah I think that's one of the you way design principles we had from day one is that

12. 5:16

    we want the broad chain to uh keep running like through uh throughout all the upgrades we do right so we have done a few major upgrading the p as well including the uh the simple nice shate upgrade uh which Chang from Shard to four shards uh but that was an easier uh thing to do because uh it's like yeah it's starting from one so there are some like um things that are easy to handle

13. 5:41

    from the engineering point of view uh this restarting is is much more difficult uh because we already had full shs um and uh yeah so so but anyways yeah we we we did not want to shut down the chain and because that that would mean that you know people not be able to use uh whatever app they they want to use uh during that time period so that that I think uh and and then we really value the the US experience and and we

14. 6:08

    really want to make sure that um you know sharding or like you know just like upgrading channel uh do not affect users and they they don't even know that they are not like six shards or like they don't even know we had four shards and then and I think the truth is people people didn't know if you're just regular user you don't you don't like maybe you don't even want to know like you're just using your app and have good time I mean it's a pretty cool feat and I think so I want to think the developer

15. 6:33

    perspective a little bit and so some developers might know the concept of sharting from databases so if you're coming from non-blockchain you that it's not like a brand new term necessarily uh how would you compare this though to like a database sharting setup versus how this is actually set up yeah I think in a lot of ways it is similar uh but as you probably know like starting in a

16. 6:55

    distributed database is also a difficult thing like um and and not not many people have have done right um so and then in the blockchain world it's even more difficult because in in this distribute database the advantage you have is that you have like centralized control of the entire system and I mean this this mostly covering uh in two aspect one is that uh you won't you won't need to think about like malicious activity yes you need to be like fall tolerant but it mostly you're tolerating

17. 7:29

    uh know that are like machines that are like offline because like they so something happen with your like battery or whatever like this is like gone or something but but like no no machine is like going to M attack others like no sh is going to be malici attacking other shs uh and you don't need to think about what what will happen when that happens uh and and then actually that is why

18. 7:53

    like starting blocking is a lot more difficult is because of this um uh you know games like in the security aspect of like what what we need to do to to get to design uh things that will work well uh against those cases and also like the game theoretic incentives that you need to design to make it work and I think the the second aspect is more I would say like uh engineering related

19. 8:18

    right so because you have the centralized control of the entire system it is actually you can like know what's happening every single machine like you can like design like centralized logging J system to analyze what's happening on every single machine where you need to debug something and you do have that full view of the entire system um but actually in blockchains you don't have that luxury and uh it that that makes

20. 8:42

    like certain things like debugging very difficult uh for example like something happens on M that you want to understand what's going on you only have like partial view like you don't have the view from like the the nodes you wrong like the RPC noes you wrong uh and to to kind of try to analyze trying to like Cate what's happening so that also makes things very difficult one thing that I

21. 9:04

    know in previous implementations for other chains you have to be Shard aware um but you do not at near um how does that you know what's a simple way to put that for a developer yeah so that yeah I think that that again is like one of the design principle with that in mind which is that uh it should be as user friendly as developer friendly as possible so when you deploy a contract you just

22. 9:28

    deploy this on the account you don't like you don't really care which sh that the count lives on uh and and this is um because we had this U Grand Vision in mind that you know uh in uh in the future we will't be able to dynamically uh size the number of shards to cater to the Demand right so let's say uh you you are like some Shard is congested the the system would uh be intelligent enough to know that okay this not congested so I need to split into two CHS then it would

23. 10:00

    a automatically happen uh and then let's say two shards are both underutilized then I can merge them back into one so then then in this way like as you as like developer you don't really need to care about which Shard uh you deploy your contract out because then you know that there's underline guarantee that yes see there could be congestion but then uh the the system will be able to react to it and and you you won't you

24. 10:22

    you won't need to worry about it yourself in in the future as we start to add more shards is it always going to be a resharding process or is there a way that you can just add a new Shard to it yeah yeah pretty much right because that that's because the how the address space is divided basically like you know essentially going think of the uh all the all the possible accounts out there they're divided into shards and it has to be a complete division because

25. 10:46

    otherwise because pretty much like and like we need to make sure that when when the account is created people like the system know which sh they is on can cannot be gaps right there cannot be like uh like some account is uh falling through the cracks right that that's not acceptable uh so yeah pretty much like uh it just like you kind of s of like today divide this way then you need to

26. 11:10

    add more more s you kind of change how it's divided but still they have to divide the entire um address space of all accounts uh so yeah basically it will all be uh some kind of rardy uh but uh we we we are going to make it EAS uh we're going to make it easier to do rearing and uh hopefully the overhead of doing that will become uh lower and lower over time and it'll become a more like regular

27. 11:35

    currence that's yeah yeah I mean it makes total sense so just kind of my own curiosity so then when a new contract is deployed how is that determined of where that goes is it just totally randomized or is there actually a mechanism that's actually Distributing those no it's not totally randomized uh so basically there today the the the boundaries are like hard code this this because we don't

28. 12:01

    have Dynamic re rearing enable so essentially today every time we need to do a rearing we we do need to compute the uh new boundaries ourselves and then we we do like a code change to say well this is now the new boundaries and U uh yeah then then we do this release and then that would actually take take effect that would you know trigger restarting and then the new boundar will take effect but we imagine that in the

29. 12:26

    future it won't be this way the the system will dynamically calculate where the boundaries are and then Shing get triggered a so we have like homogeneous Shing right so I think Shing is not loaded term like there's different interpretation of it uh but what we're doing is that uh all the shards are homogeneous they have the same ground time uh same like you know account account structure and and and all of that and so basically this allows us to

30. 12:51

    to achieve this scalability uh while preserving this uh composability and and the the same user experience right so so from like user perspective you you don't really need to care about which Shard your account is on which Shard the app that you interact with is on whereas like if if you're in a Ser that's not the case right you do need to care about which L2 you're using so with the way the protocol is designed in you know

31. 13:16

    both where it is today in your future plans can you just briefly talk about um why this is the like unique opportunity for the concept of chain abstraction and chain signatures that maybe CH couldn't do yeah so I think uh one thing that uh um a lot of people don't don't realize about uh the uh the CH I mean obviously the the Chan obstruction CH signature is to uh unify the experience of cross rep 3 make things uh chain agnostic and and

32. 13:49

    users just use uh multiple different chains at the same time and so on um but but like in order to to achieve that uh you need a very scalable uh infrastructure that other being should let entire experience right uh imagine like you have you know 10 million users using like you know in one day and you need to have this highly scalable infrastructure and and and in our mission that essentially like near

33. 14:15

    because this um this whole this like entry point that and the the thing that off Street all of their experience across stre that means like all the transaction would originate from here right that means you need to have this extremely scalable uh blockchain that that that actually serve user uh this great experience you get uh through uh chain signature and and chain obstruction and then allow them to to aggregate across all the chains um and

34. 14:41

    it's not and also it's not only just stable has to be uh very flexible and allows uh you like allows users to have this uh granular control of like different accounts they have on different chains through some kind of smart contract on year and ability to uh you know this highly customizable control that you can Implement in a smart contract and all that is uh it's as in critical for uh having that unified experience across street yeah that that's that's a great

35. 15:16

    answer no I really appreciate that