NEAR Onboarding Bootcamp | Part 3 of 5: Protocol Development

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   okay let's get started so hi everyone uh i'm bowen for those who don't know me uh i'm leading the teams that work on mostly protocol development and uh yeah today i'll be talking about um the work that we have been doing uh and also some of the uh future works that we'll be working on so yeah let's get started so yeah just overview of what we do so our goal is to build the infrastructure that powers developers to build decentralized applications

2. 0:34

   and it is intentional that in this phrasing that i didn't put the word blockchain in because our goal is not to build some sort of blockchain in a vacuum or even to build the building to satisfy some abstract goals rather our goal is to to make this such that we can serve the needs of the developers and uh truly you know be aligned with the vision of near so that that means today we're building a charity pro stick boxing but if in the future there's some change

3. 1:15

   to the technology for example some zero knowledge proof uh become technology become mature enough uh that we can use it uh to replace some of the things that we have today we'll be happy to do that and yeah just a brief overview of the entire development process we started back in august of 2018 back sentencing we started the prototyping in python we spent a couple months doing that but then yeah actually i think that was not

4. 1:53

   the most uh the best use of our time and we actually went down to pursue a completely different uh uh different paths uh and then we launched mainnet phase zero in april 2020 this was a proof of authority uh mainnet that only has values wrong by the neo foundation and then gradually we start introducing more validators from the community and uh eventually around uh well on october 13th 2020 the vote for

5. 2:30

   mainnet phase 2 happened essentially there was a vote by the community to to turn on phase two which essentially means that transfers are enabled on the network so by now mainnet has been running for uh about i think eight months now um and uh it has one chart and i think yeah if you if you have seen froze post yesterday uh we produced more than 40 million blogs on amazon and that's uh that's a great achievement

6. 3:09

   uh so before we actually talk about some of the details of work that we do i also i just first want to mention the overall priorities for the for the teams uh because this would also um be linked to the actual the actual work that we do as we will see later uh so the first priority is of course uh safety and stability of the network uh because you know if the network is malfunctioning or has some critical vulnerability that is exploited are the network just stopped or like

7. 3:43

   kind of shut down or some some someone tax the network and steal like a ton of money uh then obviously um that's extremely extremely bad for everyone who's using the network i will not only cause applications to stop working but also people would lose trust in us so so our top priority is the safety and stability of the protocol and then the second priority is to make sure that our developers can easily build on us

8. 4:14

   because after all we are a developer-focused platform and we want to serve their needs and we want to make it easier for them to build whatever application they want to build um and the priority after that is performance and the scalability but this also ties somewhat into the um the developer uh developer friendliness uh because all uh the problem with ethereum today for example is that uh they have such low throughput uh that

9. 4:48

   makes it uh very painful for developer to to build applications and make it very expensive for them to run any applications so we will also be focusing on making sure that our our network is performance and also has good scalability so that this will not be a bottleneck in the future and finally we also want uh our blocks and clients blockchain nodes to uh to be runnable meaning that whoever's wants to run a node whether it's validator whether it's rpc

10. 5:22

    providers or some other partners or developers for example they wouldn't have no trouble doing that okay let's actually talk about the protocol itself so roughly on the kind of high level there are three components to it um so the first component is uh is this peer-to-peer network and our focus there is to build a reliable and scalable peer-to-peer network that powers communication between blockchain nodes so essentially this

11. 5:57

    is a custom built protocol that enables communication between blockchain nodes this includes message routing and things that we will talk later about but there is no essentially no good existing solution for this problem and then there's chain which is this um you can think of this data structure that that ox trade everything on the on the blockchain uh so this includes how consensus works this also includes

12. 6:29

    uh how how sharding works on the on the high level i mean like how how blocks and the chunks which which are sharp blocks are produced and distributed and then there's also uh runtime which is this execution engine that's uh basically a price brock's the state transition and so on so yeah let's actually uh go into each of the components and discuss in more detail what uh what we do there so yeah as i said about the network it's

13. 7:00

    a peer-to-peer uh communication protocol uh and since it's purely um peer-to-peer means there's no centralized entity control controlling how uh how the nodes uh interact with each other uh they essentially follow some kind of custom protocol on how they establish connections to each other uh how you know they send messages to each other uh what the format of messages are uh so on and so forth so every node has has a unique key that identifies themselves on the network

14. 7:31

    now so the interesting part is that we want to build in such way that we can route messages from nodes a to node b in a kind of efficient and also resilient manner so let's actually talk more about that so in general in terms of like routing on the network there's two types of routing algorithms one is called bank state uh routing another one called this vector-based routing the differences mostly lies in whether uh the the the client or the node stores

15. 8:06

    all uh all the information about the network uh or only stores information about its neighbors so so essentially we what we do is that each node kind of maintains its own view of the network uh and then when the node when for example when new node joins uh it receives that view from from other nodes and then when some when the topology of the network changes for example when two nodes start connecting to each other uh or some node disconnected from the

16. 8:36

    network that essentially means that a new edge is added to the graph or a edges removed from the graph and then that is essentially broadcasted to the entire network um by this uh gossiping protocol essentially like a node will send this message to this neighbors and his neighbors would send to this to their neighbors uh and then uh that's a great that then everyone essentially updates their view of the network well sorry just sorry for interrupting

17. 9:06

    can you just check the participants room because some people say they're waiting oh okay okay sorry about that yeah uh how do i okay okay so sorry about that okay i think i've admitted all of them yeah yeah for people who just joined i'm sorry i didn't check the uh then the notification there um okay so uh let's continue so uh i guess i was talking about how messages are sent through the network

18. 9:53

    essentially once we establish this this entire process every node would have a a view of the network and then it would know for example if it wants to send some message to some specific node in the network how it should be sent um essentially because the network can can grow very large the every node does not necessarily have direct connection to every other node it will choose a a route to send these messages through other nodes

19. 10:24

    and because the node maintains the view of the network it can choose a shortest path to the to the destination and that is exactly what we do so now let's talk about some of the challenges and the future work that we will be doing in terms of the peer-to-peer network uh so first of all we want to make sure that the spirit of gear network that we build is is safe and is robust it means that we need to prevent

20. 10:57

    mauritius attackers from finding ways to cause other network other nodes in the network to malfunction uh this means that uh uh then the attackers would not be able to create some kind of attack such that uh the node would crash because um for example they they run out of memory because the attackers send them some some very large message or uh the attacker somehow tries to spread

21. 11:24

    the network into half such that the the two halves cannot communicate with each other for some reason and then of course we want the network to be scalable as well that means what we have today which is maintaining a view of the entire network uh it doesn't actually uh scale very well when there's a very large network let's say there's a million nodes in the network then maintaining an entire view of the

22. 11:56

    network becomes uh less feasible so one of one of the directions we're working towards is to design new ways to um for nodes to maintain uh kind of local view or some information locally about the network and then route messages based on that and then yeah that kind of is an ongoing work also in terms of performance the current the current routing algorithm doesn't really take latency into into conservation it essentially routes based on the shortest path in terms of a number of like hops it needs

23. 12:33

    to take but it doesn't actually distinguish those different parts in terms of the latency that's uh that's incurred uh by sending messages through that route uh so another another thing that uh um we can so today the net every node has kind of a fixed number of connections to other nodes and that number is actually quite small so we are also thinking about being optimized doing some optimization there

24. 13:04

    to enable more connections more direct connections to to reduce latency yeah so that's pretty much every single network so let's move on to chain uh so yeah for change i'm mostly just going to give a very brief overview of consensus and sharding those so so each of those topics can require like an entire presentation now and so on so i will not uh delve too much into the details here and i will just try to give like a a very high level overview uh of what we need to do um and uh and

25. 13:42

    how how it kind of works uh so yeah the consensus is called doomsdog for people who are familiar with brandon sanderson's work uh fiction skyward uh doomsdlog is this creature well this is actually a picture here that is well without giving any spoilers it is very fast uh so that's why we picked the the name oh that's why alex picked the name uh so dumaslike essentially is this block production and the finality gadget

26. 14:14

    um and uh without describing in detail how it works we can talk about some kind of the property that it has uh essentially blocks require uh two servers of approvals from other block producers to be produced and then yeah if there are essentially three consecutive blocks with three consecutive heights building on top of each other then um that will finalize the first block and also final brush blocks cannot be reverted unless more than one third of the state clues

27. 14:45

    uh so that is kind of the euro bft stuff um okay so yeah so similarly the sharding uh design is called nightshade and yeah nightshade is the the main character in skyward uh there's some very interesting interaction between nightshade and the doom slug in the in the book so yeah for people who are interested and read the book skyward um so the the kind of one interesting design

28. 15:17

    choice that we made in knife shade that's somewhat different from some of the other uh blockchains including uh the original ethereum 2.0 design is that we we only maintain essentially one chain and for all the shards um essentially they only have sharp blocks and the blocks are kind of sharp router which we call chunks are actually contained uh in the blocks on the on the main chain so this kind of um simplifies things a lot meaning that

29. 15:48

    we don't have to maintain separate searching and have to think about uh you know when there are forks on the short chain how do we deal with uh crosstalk transaction because you can have um transaction from one one fork of one char to another fork of another chart and then uh when that kind of changes it leads to a bunch of problems including like uh atmospheric failure and things like this so so essentially we we simplify these things a lot by only maintaining one chain and then the blocks on the chain would include

30. 16:18

    uh chunks from different shards and then chunks are distributed to broad producers um through erasure coding uh well without actually explaining too much in detail how what that is it essentially means that uh it allows uh it allows us to have to cons reconstruct uh the chunk with uh only a fraction of the broad producers so we choose the number so i think in the paper it chooses one

31. 16:44

    one six and as here uh in the actual implementation we'll use one third uh so essentially one third of the block producer or anyone sort of the block producer can reconstruct the any given chunk by having the the erasure coded part and then block producers would send approvals once they receive and process block to attach to the validity of the block and also the data ability of the of the block meaning that they have received their their parts for the for the chunks included in the block

32. 17:17

    so that's kind of a very short overview of how chain works um so some of the the future works are those things that we're we're actually working on uh including uh moving from uh where stars or modbus charts so when we first launched main that uh we didn't you know at the time there's no essentially no usage and uh it uh doesn't make a lot of sense to have multiple shards uh but now um just uh eight months later we're actually seeing

33. 17:48

    uh a lot of growing demand on the on the network and uh to the point where uh we think that it is absolutely necessary that we have more than one chart to uh to accommodate for the ever-growing needs from the developers and the community uh so how this is going to be done is that essentially there's going to take we're going to take like a multi-staged approach where uh in the first stage we upgrade to a simple nice shade

34. 18:17

    uh meaning that there will be like eight charts um and there will be separate trunk producers but all the block producers will track uh every shard and then later on once challenges and dynamic recharging are implemented will actually upgrade to uh to the full knight shape that's uh specked out in the in the white paper the problem is that uh challenges and also dynamic resulting is on their own uh requires a lot of effort to to implement uh yeah i will not uh actually talk in

35. 18:48

    details about uh you know what those things are and and why it's hard to build them because that that don't honestly take a very long time to explain um so another thing that we are working on is um to to increase number of validators on the network uh so today we only have 60 uh which is like a relatively small number compared to some other chains like you know polka dots mana and so on so yeah so we're trying to

36. 19:13

    to increase that by uh well there are multiple things i'm working on for example we're working on uh once we transition to synchronization there will be a lot more validators uh who are also trunk producers and at the same time we're also uh thinking about uh ways to increase the number of block producers uh on mainland so yeah that's kind of the stuff that we're working on okay let's uh move on to runtime so as i talked about before the runtime is an execution engine that's

37. 19:46

    responsible for uh state transitions so very very naive or like very simplistic view of runtime essentially you can single this black box that you know you feed it with the uh the current state and then you you give it uh the the current block and then it will return you the state after applying the applying the block so yeah that's kind of a simplistic view of how it works and actually underneath or rather inside that black box there's a lot of stuff going on to actually make it work

38. 20:18

    so just let's just kind of briefly work through an example let's say there is transaction uh from alice to bob meaning that is the sender bob is the receiver so how it works is that the transaction um well no matter what kind of transaction it is whether it's transfer function call whatever will be first sent to you to address chart to be to be processed uh it will charge the transaction fees

39. 20:42

    from the from alex account that's necessary to convert it into a receipt and and after that the receipt is sent to bobster to be executed further and then that receipt of course can create other receipts that go to maybe some other chart so and that process just goes on so as you can see here the execution is not synchronous like a synchronous execution meaning that in each serum uh a transaction would just be executing in one block and it would just be down

40. 21:12

    here a transaction execution can actually take multiple hubs so since we are a developer platform i want to just talk a bit about how the contract execution works so overall like the developers would you know write some contracts and then in in a language that we support and then they would compile the contracts to to have assembly binaries uh in whichever uh comp in the compiler of whichever language they're writing and then they will send essentially a

41. 21:44

    deploy a contract transaction to deploy the contract to some account uh and then uh so we're actually uh so yeah it will be the case that the that uh the contract is compiled when when it is deployed to uh to some vm module and then when the function call is invoked uh a transaction that essentially means that they send a transaction to to the to the network uh to uh to call some method on the on the contract and then the runtime would would take that in and it would

42. 22:15

    start the vm to actually round the uh round web assembly code um yeah that's a very brief overview of how it works yeah so some of the challenges and the future work on runtime so first of all the stability and the robustness of runtimes is very important because it's inner science more important than other components because runtime is responsible for state transition and once there is a problem in state transition

43. 22:45

    it will cause the network to diverge and it would not only cause that but in the future when we have challenges it essentially will cause the network to split into different parts and then each part will try to change each other because they think the other part is doing an envirostate transition and then when that happens it just becomes a completely mess and the state would just be very messed

44. 23:08

    up and then we have to it's very hard for that to recover because essentially then we need to do some kind of hard work to change the state which is not easy to do and uh and things like this actually do happen right so uh for people who are following more closely with the development cycle um uh we recently found some parking in washington for example that would cause this kind of behavior so yeah it's very important that uh we work on this improving the stability

45. 23:40

    and robustness of runtime and uh we're also some other things that that will be working on including like figuring out better ways to to charge for transactions this means that uh want to estimate the better the the cost of different operations and also optimize uh some of the the operations including like host functions like storage read and storage right and also we're working on better ways to price the the gas price of the network

46. 24:10

    and addressing some developer concerns around gas and storage i mean today it is all confusing because gas is not something that people just intuitively understand and also storage um storage cost is high and also today developers have to use some storage management standard to manage their storage once uh when there is some measuring their contract that allows users to increase the contract storage so yeah there's like a lot of work that needs to be done to improve the developer experience here

47. 24:43

    okay so finally i just want to mention briefly about uh node experience because this is essentially um a kind of horizontal thing meaning that it crosses uh cross paths with all the team other components that i mentioned before uh essentially here we want to make the experience of running near those better uh so some of the kind of feature requests and the performance improvements are working on including things like you know for example validator wants to have a longer time between

48. 25:14

    when the protocol upgrades is voted and when the protocol actually switches to the new version uh and also things like uh separation of different types of nodes today they all use the same binary they all read essentially the same kind of data to the same database but that's actually essentially a waste of uh variety their storage in in a lot of cases that they don't actually they're storing some kind of data that they

49. 25:38

    don't actually need to store and also we will be working on further optimizing cpu and memory consumption of near nodes i think today um the cpu usage is mostly good i think some still some issue with memory consumption uh and also we'll be working on testing of the stability of the of the near nose through more integration tests and larger scale tests so yeah i think that's all what i wanted to present i think yeah

50. 26:07

    also it's almost exactly 30 minutes yeah i think i think we still have a time left to answer some questions right cindy yes you have 10 minutes 15 minutes um okay okay yeah thanks um yeah any any questions about the yeah yeah austin uh yeah i just had one question about the you mentioned uh in the presentation and then also in that government or the governance post that you wanted to

51. 26:39

    increase the amount of validators is that a protocol or technical change or is that just onboarding more validators to to get them on the network yeah so it's kind of both uh but today we're both enact on the protocol side that there is a higher requirement the hard description that there can be at most a hundred validators um in the network and also because of the validator how

52. 27:04

    the variator selection algorithm works it's actually very difficult to reach that maximum amount because the essentially the uneven distribution of stake would lead to some violet are not being able to get a seat and as a result uh we kind of plateaued at around 60 validators right gotcha thanks yeah anyone else has the questions or something i can explain better i don't think i explained anything very well oh actually i was about to ask whether we can post it on youtube we can share

53. 28:01

    with you would love to have such material uh which goes on with over over you and hello and i find they talk pretty well uh defined in in this specific area of like going through everything in just half an hour yeah thanks for out here can you also double check the participants if you're waiting for somebody i don't think they're no i don't think there's anyone in the waiting room disable waiting room how do i disable

54. 28:48

    waiting room um yeah i didn't find how to do that oh okay okay okay disabled with me so i guess if nobody has any questions and we can make a break