Sharding vs No Sharding Fireside Chat: NEAR Protocol & Solana

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:05

   ok let's start ok sharding sucks but ok so so so they want me to start my opinion is that sharding is a really hard computer science problem and I think it's solvable but that's what it is it's a really hard computer science problem but clearly no charting is so simple you have to bring super super good minds from Qualcomm to tackle it right I don't think that we would get anywhere I don't think we would be near the finish line if we try to build a

2. 0:50

   shortage a but that's that's the difference right yeah your breath talent from quality because you're building hard core low level optimizations we brought people from REM sequel we should charting before right but I have to have certain arguments again towards starting I think that for for certain use cases right if you look about payments or taxes I think I think something like Solana

3. 1:12

   was great right it's it's a it's a very homogeneous workload right and you can easily of like process like thousands if like hundreds of thousands of transactions on the GPU concurrently right but if you're running an arbitrary smart contracts on a platform you can only get that much of parallelism right so you will be bounded by the whatever number of course you have on those

4. 1:37

   mainframes running Solana so for arbitrary code it seems to me that you're doing you're charging ultimately like at some point you will reach the capacity of the of the mainframes if adoption grows faster than this every two year you know doubling up CPU I think you're actually compute is not the problem here like a computer is just so easy to get right like if there's actually enough demand to fill capacity people just add more computers like

5. 2:08

   hardware is like actually really cheap and easy to go get I think like the benefit of sharding is the thing that you can't scale which is bandwidth right we're limited by the bandwidth of the super majority of the network but anything more computers is charging no no like you have my validators now like Baraka you know 42 inch rack worth of GPUs right I can add more compute easily right that but I can't like run another fiber-optic cable

6. 2:38

   to everybody in an hour so that sort of works right so so one example is what you gave where everybody every validator runs like a huge mainframe and that mainframe has so much CPU on so much disk space that they can fit whatever it can fit all the information the world has or there's like another school of thought where instead of doing that they they make every validator to run like a set of machines right but it's internal communication so but that

7. 3:10

   that does appear like at least to me it feels like it hurts hurts decentralization significantly because the requirements for validator now is pretty high so it's only for a select group of professional enterprise great validators who can run them so like my argument against that is like every two years Hardware gets cheaper and cheaper and we're just unlikely to see that much

8. 3:36

   usage in the next eight years right like to where it fills up one gigabit worth of dps that's like almost a million transactions per second right like one gigabit worth of data one gigabit is available worldwide you can process one gigabit worth of transactions and a 5k machine right now so like the the the imaginary like world that we're thinking about where we have 42 inch racks of computers is like the world where we have I don't know a terabit or the bandwidth connected to everybody it's

9. 4:09

   just it's just so far away right like but were the networks actually getting that much faster like it's like one gigabyte has been around for a while right but we're not getting much much further than one gigabit but the bigger thing is that we can get out of CPU but you do need you do need to write our right you need to write to this disk is not getting any faster over time that was kind of SSDs are getting faster

10. 4:33

    that's definitely true both SSDs like any anything that silicon is getting basically twice as fast every two years not latency but bandwidth like in terms of how many not sequential but concurrent operations you can do per second all that stuff is doubling bandwidth doubles like network bandwidth of doubles every four years so it's grow slowly like 5g standard is one gigabit point-to-point any phone any two phones in the world hmm so the amount of fiber that needs to

11. 5:06

    support that across data centers they're all gonna serve traffic to this like three billion one gigabit devices isn't saying right yeah so like we're gonna live in a super connected world with a ton of available resources so if you have a network that utilizes all those resources to their full capacity I think it can deliver the cheapest price for the security of this trust minimizing computer and so in your in your ideal world how many how many relatives do you

12. 5:37

    think are going to be running Solana court no well me we designed the protocol like to 20,000 validators like we thought that was our simulation but where are we gonna find 20,000 validators I mean like where are you guys gonna find validators for like I don't know how many starts do you plan on launching one well let's say eight right okay but you see the beauty of shorting right is that you can run on a like on the cheapest instance well not on the cheapest but on a very cheap instance and that's gonna be more that

13. 6:09

    that's gonna be enough to process a short so so here's my kinda argument I don't think sharding is actually scaling solution it's a decentralization solution and I think people think of it as scaling because it on the surface increases the capacity of the entire network but the price that you've made per transaction is going to be based on the capacity of the single shard because what what that price is is the spam resistance if you lower it if you lower it below that then your

14. 6:41

    full shark get spammed out of existence right and you get and you can't afford that you can't afford a single shard to be at capacity because that means all those users all those applications are dead they have to know to this longest migration thing to another shark right that's like a huge attack vector so it increases decentralization it doesn't actually scale but wouldn't you agree

15. 7:05

    that the cost of a single transaction in a short it will be it doesn't need to be any higher than the cost of a transaction is centralized in a single in a single chain system right because the cost of stemming will be the same if it's running in the same hardware right so if if it's running and if it's a network that is like optimized like ours it means that we give you 50,000 TPS

16. 7:28

    worth of capacity our price for spam is that much lower right so that that's kind of like but I think the main trade-off it's a it shouldn't be called a scaling solution it's a decentralization solution which is fine decentralize you know to the help I mean that that's an awesome problem to solve so that's a good argument however I think I think once you start thinking about the same

17. 7:52

    about transaction fees in the context of block chains being wealthy adopt it in in the wild one of the I think one of the requirements for block chains to actually gather gain adoption is that the final user of the service does not play any turns on any transaction fee they used service of free and as the service provider the smart contract owner who is paying for it right so once

18. 8:16

    we once we do that now spanning even if they cannot swim the protocol they can spam the the owner of the application right so you need to have some other way of protecting against spam right and it is it's an open network dude as soon as the price is below my might cost to mass for that I'm going to mess with it of course for me to lie to like sending requests to Instagram is practically zero right I can create a DDoS like a botnet very quickly and yet Instagram

19. 8:48

    doesn't go down right so people seem to have solved the problem of but Oh are your validator is going to answer transactions that are paying in the the right amount of fees but they just like we don't like these because that's what you're talking about right it's like in the brand in the right in the bright world of crypto future which I envision majority of applications are are using this approach with the with application provider paying for the fees

20. 9:16

    so it's a blicket so people use the protocol through those gateways of through hosted wallets of the application providers they paying their gas fees application provide application designers application runners are people who paying for those fees and they correspondingly through their gateways protecting against DDoS if you actually need to use network directly if you if you don't want to use like a host of all you need for security well at that point

21. 9:40

    you actually are in a position to pay a higher transaction fee right so if you need full security of the system you can also pay more for it that's make sense sort of but like if you're talking about like basically creating a walled garden spread between like your users that interact with its they're not essential as users anymore right they're not self custody they're just users of some app what why doesn't that I've just run a Postgres so there's a big difference

22. 10:11

    all right so so a in terms of world gardens right if you don't do that if you make users pay for the gas fees it happens that we live in the world and the world is not going to into any different state we're acquiring crypto is extremely complex right and so they if we if require users to pay for for transactions the adoption will never go off limits like like if you go outside and like grab a random person there that person will not be buying crypto in any

23. 10:36

    time in the next five years right so I think making that like this hosted wallet approach is actually making it less of a walled garden right and they still gain a lot of benefits from using the centralized network because yes they do need to trust the hosted wallet provider right but first of all that could be sound like you know people trust coin based people store Krypton coinbase you only do it for low importance transactions right so this is what you guys are building is it

24. 11:01

    essential eyes that work or not decentralized people right but but that is the thing that your your majority of your audience unless unless you capture the audience which does not fully care about decentralization they remain like people will not go to Facebook if Facebook doesn't have ninety percent of the of the population of the world you need to capture the audience which doesn't care about crypto if you want to capture the Kryptonians as long

25. 11:26

    as crypto audience has their access to full to full security full full value of the blockchain people who don't care as much right they still gain quite a bit rights because if they using a hosted wallet provided by the application developer that means that application developer wants that application developer right they're paying a big pile of fees to run to access the chain

26. 11:51

    right they're paying for the users right so in that case they're the most motivated to switch to a chain without chess the lowest fees right for the same level of security which is one that is designed with the most Hardware right per validator but that's how you get to the lowest price the question is is it safe is it a cool easy for them to do that that's one similar like for like a company that's spending like X about it depends on if they're burning a million

27. 12:22

    dollars a year in fees then it's X amount of engineering hours right just to do it so there's a good argument to that argument number one is that all right that what you're saying is true for like a huge company right if a company built a huge service on the blockchain and a moreover for a small number of companies it is most certainly even even if you're a little bit delusional which you don't you not necessarily are but even if you're a little bit delusional on is not as

28. 12:47

    scalable it can definitely host at least several very high-profile applications right so for them it might be reasonable to move however this one of the beauties of the of the decentralized applications is that they composable right it's it's the open state components that that's actually going to be making a huge difference in the next years and hopefully they will be driving web 3 and but but those open station charging

29. 13:12

    introduces unimaginable complexity of to visibility right because you basically have to build early right like to deal with it look you can abstract out most of that right if you go today too if you go to the dating your dev and you're trying to do like a cross contract call it's not it's not harder than just creating a future and then acting on it and that quite a bit of it from you just that this yeah unless I'm

30. 13:38

    living in synchronous course which is a different story asynchronous calls a completely trivial and today most of the code in JavaScript that you write already actually relies a lot on asynchronous calls because historically JavaScript a single try that and if it you have to rely on asynchronous calls right like if you do jQuery call in in your in your website right on your yeah right but those those open state components their state there's going to

31. 14:02

    be coming a lot from it from smaller players debugging and guaranteeing that all the contracts terminate and execute and all in unsafe states that are executing and multiple shards is is like a super complex problem you don't need to care about it most of the time right as long as your logic like you're right your logic as long as it's relatively straightforward and most of the time it is like how many times you saw people writing ridiculously complex logic on

32. 14:27

    the back end like opening a flask application it's just a formula yes there's gonna be no surprises right yet so compile it to spear V rather than GPUs so the so the point here is that as long as so for the smaller players it's significantly easier to go and build on something that is extremely straightforward right so so the open state ecosystem will be coming from the small player so it's not gonna be coming from Duke players nor over the entire promise of web tree goes against big

33. 14:57

    players you don't want Facebook's and Google's to exist right they their money like their their position as a monopoly is what is what hurts the world what allows them to but come on it was like communities equivalent to them to exist that are all using the same protocol right are the same like that are all coalesced around like one thing yeah you want your centers can we just exist here that so effectively you want you want

34. 15:22

    all the primaries all the biggest services you use today you want them to be running by the community right what you think goes against this model where you have services that maps track that away from the user like I think ultimately the space succeeds only when we have you know to 300 million self custody wallets out there they're all doing something but so so let's let's lightly peel what the arguments right and so let's let's say let's say that it is actually the case that running a huge mainframe is cheaper

35. 15:58

    live thousand dollar machine running than running a huge distributed cluster why is it the case that that IBM and Oracle you know like they can only acquire those like huge enterprise customers credit to pay for their huge mainframes but majority of people they use distributed database today they use Mexico they use they use Cassandra but like there's like this ridiculous misconception that it's a mainframe it's literally a gamer box like a five thousand dollar machine and at one

36. 16:24

    gigabit network I have one gigabit in my house like it this is like this is like the bear level of what hardware and the networks are today right and like them in the developed world so first of all is it the keys that you're one gigabit never never glitches right because that's what that's why you have like a few thousand of them and as long as the glitches are not correlated so let's say that that's the whole point of consensus right that whole other thing so let's say everything goes well and within

37. 16:58

    several years people actually move towards community driven services right do you think that those gamer boxes those gamer non mainframe boxes can run the entirety of what Google Facebook Instagram running today well the thing that's actually happening and chain right there is the small straightforward things where you're just switching ownership and maybe like you have one giant state machine for price discovery right you're not running they like the

38. 17:25

    Facebook application on chains right that's right that's running in the front ends but you're coordinating is like data access right and legging privacy and like controls right look it's it's a it's not like this thing that needs to execute in these trusts minimizing Peters doesn't need to be a Java VM running a big pile of shitty business logic but but like why not why would I

39. 17:51

    not run if I can why would I not run those decentralized applications entirely on chain well you can really do it even in the Charlotte system because the resource constraints writer are huge so if you are like sharding isn't gonna help you there right because you're you're talking about really fat applications unless you have one fat application per shard basically but nothing stops you from starting applications themselves right sure so like if applications charge it on its own for as long as the number of charge

40. 18:23

    grows together with the adoption each application can run its own tender but still limited by what a single machine can give you right but what if applications themselves are shorted nothing stops you from running the entirety of the web 3 own chain which would be we like that's the future I want to live in right sure but but you cannot run the entire two web 3 on mainframes I mean you can't but they're gonna be

41. 18:54

    extremely expensive those those are not going to be gamer boxes the the char like the applications you're talking about they're not going to store data and images and chain they're not going to store like the front ends on chain right all right nobody's designing that no one in the theorem is talking about the web three people are not like pitching that right there pitching we store all the heavy stuff and I in ipfs and we load everything client-side and

42. 19:24

    we use the chain to coordinate right but that's also what I'm talking about right nobody talks about storing images on chain what you want to do all do all the logic the entirety of the backend logic you want to do and chain that's what gives you the knowledge that nobody is trying to take advantage so then you have two options right you have a shorter chain which is running low powered hardware where the fees have to

43. 19:51

    cover the spam for the single shard right or you have an uncharted chain is just running standard hardware like machines you can buy at Fry's right that's utilizing one gigabit we're at the full capacity in the network is available to everybody right so that I think the like why would you pick the shorter chain like like why so there isn't there isn't is that it is unlikely that that mainframes which a large number of entities around the world can afford can run the entirety of

44. 20:30

    the of the vision of what web 3 can bring right so if we do believe that Web T will we agree the distance that the web 3 unchain is just doing small about a logic but but first of all that logic is that logic is not entirely homogeneous that that's one right so if you're running many many different applications most of that will have to be happening on the CPU right unless unless everybody is running a separate instance of solana but that's again that's right so majority of the staff will have to be happening on the CPU so

45. 21:05

    you will have to be running all the business logic of all the like imagine running all the business logic of all the internet on a single like it's like it's like as if Facebook itself did not show their execution and was running a single huge machine replicated to another single huge machine for processing Facebook but they don't do that's right yeah ok so so there is like I guess the differences do you think that we're gonna see homogeneous applications but take off the most capacity or a long tail of non

46. 21:36

    homogeneous ones that are hard too hard to call us right yeah so so my so ah and that's I think I think the difference here is defi I think mostly going to be largely homogeneous partially because I think several use cases will be dominating the entire defy market like a Dex for that Solana is gorgeous you know but if but if we're talking about web tea or or any other kinds of applications if we're talking about a whole slew of possible applications right that cannot be fully moved to the

47. 22:06

    GPUs sure but if you have like a game like crypt kiddies that all of a sudden has like a spike in usage right that goes viral right because it's going viral it means all those contract calls are exactly the same taking the exact same branch breeding tcats storing the random set of numbers right yes for particular if but but do you think do you envision that in salon it will be the case that it will always be

48. 22:32

    like one or very few applications that fully dominate the network and everybody else everybody else goes to the cpu everybody else will be yeah I guess we'll see if I could predict that yeah I have no clue nobody does but the beauty of sharding is that it does scale with adoption and so the particular picture which i think is going to be relevant short term is that until adoption comes

49. 23:03

    and it will take time and it will take a lot of work for me you and others who were here today to bring the adoption until adoption comes near will be very cheap right eight charts on commodity hardware is nothing well people in salon already need to run mainframes right it's a $5,000 box it works well like if you look at a theory right you have what is the 32 e your minimum stake which is six thousand bucks right right right and not only that

50. 23:33

    you're running a so-called cheap commodity hardware that will get slashed like with quadratic leaking right like for for being unavailable so you effectively have to run a high availability system anyways but again do you have to build slashing foreign availability near doesn't slash foreign availability right so that's so like although those are contentious points but they are not necessarily something that is necessary for shorting right so short charging it just about whether you

51. 24:04

    do scale out or if you scale up right that's the only argument here really but the reality of it is going to be you have 80% of the tokens held by 20% 20 20 entities right at most so they've all those 20 entities will have to run a validator and every shard at all times no matter no matter what you do so you have like silica right you have the supermajority the stake is running a validator and every shard at all times

52. 24:37

    but that's actually necessarily bad so so the argument here is that if your stake is huge so there actually there is no but you do want that you do want that long tail of participants to be running their small notes and their stake is small but also they don't need to invest on the mainframe like those clusters but they are still providing a lot of security it's not enough to corrupt two-thirds in

53. 25:00

    a system where there are valid a there's an fishermen a single entity in a short who's not corrupted can still go and challenge and slash those participants could yet collude right so for as long as you have that one little like that one person who made a small that whole solution is only a solution to problem the Roli sharding house right like we need a fisherman detector bed like you have a single chain single consensus

54. 25:26

    just right but let's say so let's say these 1,000 people who bought who ran Solana validators right so so that means that these you need 200 people to corrupt or like you're saying the FT consensus but it seems like you need more than three what more than one third right anyway it's very different than when you have a huge long tail of people and if adoption grows right today number of full nodes is small but the fad option grows you will have a very long

55. 25:50

    tail of people who will be staking because it's also the random message removed like the lotteries do move that's a very important property one of the big problems of proof of work today is that there is a lottery right you see if you buy a GPU and start mining Bitcoin you will never get your reward you have to join the pool yeah okay situation in in the proof of stage system such as such the way sharding

56. 26:15

    usually is designed is that the as long as you your stake is sufficiently small is sufficiently large to get into the validator seed your reward is guaranteed right if you have a lot of shorts if this if the the system takes off if you have thousand shorts each one with like hundred milliliters as long as you have one one hundred thousands of the of the total stake you're taking getting your girl so small notes will be launching I mean but people get lazy and just give

57. 26:38

    their stake to a pool or host echoing bass and like okay just take it through the sintered base right that happens but remember what was happening when Bitcoin was my Nobel like in my home which is like it's a rural place of Russia where like most people don't even think that you can live there everybody was mining Bitcoin because it was cool like once you give give that back to people people will be doing that people will be staking and running running notes on there like you know on the hardware and

58. 27:03

    their clothes and and that creates decentralisation yeah of course so I think which is my point is that charting is a solution to decentralization which is a very admirable right but I don't think it's like a good scaling solution is a solution to scalability which has this property that is decentralized while scaling up is also solution to scalability which has a property which it is less decentralized it's more

59. 27:34

    expensive to decentralize yes but not less decentralized by like constraints and number of validators or participants but also I think one last argument that I personally have against scaling up is that in in in a system which processes like you know a hundred thousand of transactions per second there's a certain reliance like you like you really rely on the fact that every two years you will be scaling up by a biased by some factor

60. 28:04

    because if you start a new full note you need to catch up from the Genesis so it's been 10 years you need to be catching up 100 thousand transactions for every second since um so unless unless your hypothesis is correct and it continues doubling up a bit every couple years you get pillaged because at some point my hypothesis is not correct we should just be buying you know non-perishable foods like if if if we

61. 28:27

    stop doubling like how much compute we can do like every two years we're all screwed it's not optimizing what they do that's not what will bring the world to the collapse if we if we hit like an information limit I think that would that would mean the end of the world and in some ways they're another so I can't can't like that's not even an option right so silicon they're taking it the wafer sizes are gonna keep growing they'll be some way that they can keep adding more work transistors hopefully

62. 29:01

    yeah yeah hopefully cool well this was super fun they get to talk for hours we can talk for hours yeah