Whiteboard Series with NEAR | Ep: 44 Adrian Manning from Sigma Prime

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   okay we're live now hi everyone i'm marcelo from near i'm very happy to have here with me adrian manning from sigma prime we will be discussing about ath2 it used to network layer so can you introduce yourself and give an overview of what we're going to talk today sure um so i'm a co-founder and director of sigma prime and we're building an ethereum two client in rust um so the last two years uh we've been building the lighthouse so that's our implementation of the

2. 0:37

   ethereum two client um and i guess so today we'll just be talking about the the networking layer of ethereum seeing as we're going to mainnet hopefully pretty soon next month or so okay so let's start with the components of ethereum network and like we can go as you as you prefer you can start in the earlier one and i will be asking questions in the way is that sorry okay um so i guess i just start at a

3. 1:10

   very very high level for people to get like a high-level overview before we kind of go down to some of the deeper layers um so in ethereum two i i guess i'll be talking about phase zero so theorem two has multiple phases um i won't go into those but phase zero is the one that we'll be launching within a month or so so in phase zero there's a concept of the beacon chain um and essentially what happens in a beacon chain is we get a

4. 1:33

   chain of blocks um hopefully you can see this and the change blocks gets kind of grouped into an epoch which is uh 32 slots long each slot is 12 seconds that's where a block kind of gets fit into um so essentially we're forming this beacon chain of blocks every um uh you know so a block every 12 seconds and 32 of those form an epoch the validators uh if you're a validator on this network you perform some extra actions then somebody that's just a an observing node or something that's just participating on the network so they're going to have different

5. 2:11

   networking implications between somebody that's validating and somebody that's just running a node on the network so for validators in this 32 epoch window a validators get split into committees um so they get kind of subdivided such that every validator uh has to pretty much do an attestation which is kind of a vote um on uh one of at least once per epoc so once per 32 slots so if we have 20 000 validators we group them all up into

6. 2:43

   into um into committees each slot there'll be one or up to 64 committees and each committee for each slot uh performs some attestation so the networking challenges then uh that for each node on the network if you're a validator you need to form these little committees you need to be able to and and send your votes to all the other people on the network uh of the 20 000 validators one randomly gets selected to propose a block so the person who's supposed to propose a block like for example the first one or the

7. 3:16

   second one they need to get the votes from everybody else uh they need to kind of group them all up put them into a block and then submit the block the block then has to get submitted to everybody else on the network so fundamentally the challenges are we need everybody on the network to make a vote once per epoch and we need the proposer the person that's going to create the block to receive everybody's votes

8. 3:40

   that have made a vote so far and we need to um he needs to he or she needs to propose the block and then submit the block across the network so they're pretty much the main areas of what i'm going to talk about how how we've achieved that and what protocols we use to do it so uh i guess to start so we've got any questions so far that's just kind of like an overview of the ethereum too very fundamental thank you

9. 4:04

   yeah that sounds great explanation so let's go try to go deep inside each component okay so i think the easiest way to start is let's talk about somebody who's not a validate somebody that just wants to run an ethereum 2 client on phase 2 without any validators attached to it so this kind of network or this kind of client or node on the network needs to pretty much keep track of which blocks are being proposed

10. 4:29

    they don't need to kind of group them or collect them all they just need to kind of participate in the network um so let's let's start the like a life cycle of a node that kind of just wants to passively observe the network so an ethereum 2 node let's call it this guy has fundamentally two kind of transports built into it one is udp so it's got like kind of two output ports we've got udp and tcp it's pretty much what we're using at the moment the tcp one might change we might use a

11. 5:01

    different transport at a different layer but there's two fundamentally ports that are going to be exiting on your node at any given time um and ideally if you're if you're behind a nat or a router you want to kind of forward these two kind of udp and tcp ports so inside an ethereum two client the reason that there's two of them is because the the protocols that we use are kind of split up for different reasons so inside a

12. 5:26

    insider beacon node there are three fundamental i guess protocols that we use to communicate with other nodes so the first one is called is discovery or disk v5 discovery b5 b5 uh the other one is an f2 rpc rpc and the final one is gossip sub so let me so these bottom two are grouped under a framework called lib p2p and these use tcp and discovery uses udp that's a d so hopefully you can see that

13. 6:15

    okay so discovery itself the discovery protocol that we use is users it uses the udp transport everything else uses tcp so when a node first gets on the network the first thing you have so i'm going to start i'll go down down the list from the top to the bottom so we'll start with discovery so when a node first joins the network this happens for a node that's passively observing the network or a node that's also a validator uh when we want to join

14. 6:39

    the network we need to find pretty much the network of peers that we have uh we need to find everyone else that's proposing on the chain that already has a long chain we need to kind of discover these guys hence the hence the word discovery um so the protocol that we're using is called uh discovery version five it's a an extension or not an extension or the next generation of discovery version four which is used in uh ethereum one so this was kind of like the brainchild

15. 7:05

    of felix and it there's a number of different reasons and um which is to why we've used udp and haven't kind of combined these things uh mainly because discovery b5 is chatty so the way that it roughly works is we we communicate very uh very often and very quickly with many different peers to try and um find more peers i guess on the network which doesn't kind of lend itself all that great to tcp

16. 7:30

    which requires a long-lived connection so we can just quickly send udp packets uh it's lossy across udp so implementations should be robust against packet loss so the the udp choice in discovery v5 kind of uh kind of works well for us in a particular in a particular case it's fast and it's quick and we don't have to keep track of lots of uh file descriptors i guess if you're if you're doing lots of tcp connections

17. 7:55

    so if and if you're familiar with kedemia uh disc before the application layer works very similar to academia uh and kadamlia kind of builds up a uh a distributed hash table of of peers so essentially what happens when our node first connects to the network we have a set of boot nodes uh so these are trusted nodes that we initially connect to we form initial udp connections to discovery v5 has its own uh different kinds of wire or an encryption protocol

18. 8:27

    across the wire so everything's encrypted um uh maybe i'll go yes i started the application over here thing like when we were trying and we were implementing the network layer we consider using cadence but one of the issues we see with that is that the diameter of the network like the distance between two nodes might be somehow larger than we expect so when we were rooting message it may take more

19. 8:59

    time than expected so is that uh like how how fast our message being propagated uh so when you said that the distance between the two nodes you're talking about the the xor kidney metric distance so that yeah that defines what buckets they go into right yeah yeah so in so locally let me kind of rub this thing out yeah so every node has uh so ink down there every node has a local routing table which gets split up into uh depending on the metric you use 256 different buckets so

20. 9:37

    you've got like these buckets like this um go from like one two three four and these are these are labeled by the the metric that you use between two peers which is not geographic in the sense that you know how far they are it's just based on um the an xor between the node identities so if you have so um the way the buckets are designed in academia is that like if you were to randomly sample

21. 10:07

    50 of any node on the network 50 should go in the first one 25 should go and then this into the second one 12.5 everything kind of has and then as the buckets go it's very unlikely that you get in in the lower buckets but very likely that you get in in these buckets so if you have nodes that are often greater distance than you they're usually pretty good because that that helps you to kind of avoid

22. 10:30

    um eclipse attacks in the sense that uh if you have nodes that are in lower distance buckets if you were to try and fill up that bucket to to replace you know a legitimate node you would need to generate a lot more it'll be it's it's harder to generate a random id that would fit in that bucket and replace that guy because it's a it's a it's a greater distance so in terms of like

23. 10:54

    network propagation and speed of doing those things the the metric doesn't really my knowledge play much of an effect there it's just the the the difference in in the distance helps you um i guess it depends on how you're doing the node lookups right so if you if i go and ask you know and say hey always give me the first two buckets and there's a and there's a there's a whole heap of

24. 11:16

    peers that are in the lower end buckets i'm not going to find those beers if i only ask them in these buckets so yeah i'm not sure if it's worth while going over how academia works and find node works but we've got the time to yeah um but maybe just as a high level overview for people that haven't kind of seen this stuff before uh what happens in discovery it's over three nodes on the net on a network all right so they each each of these nodes store their own buckets i guess they're called buckets

25. 11:49

    which store peers that they've seen so you can only fit uh inca down there at 16. you can only fit 16 piers per bucket so if you've got 100 pairs um some of them are not going to be in your local routing table because this one's usually mostly full this one uh it's it's it's less likely to get into and it's it and it's kind of harder these guys don't get all that often so usually most people's local routing table kind of look like

26. 12:14

    this where the buckets are filled with peers that they know about so you go and you ask um one of your neighbors hey i need to find some peers so the way that you do it is you pick some random identity some random node so essentially you randomize which bucket you're going to ask them for to getting peers there's a there's a bit of theory as to why you do this but this is just fundamentally how it works so i let's

27. 12:38

    say this is us here where a is b here's c we go and we ask b hey what peers do you know in these three buckets for example and they'll come back they'll return these lists of peers um one of them may be c because c because c is in this bucket for example um then given c's uh c's distance from us which is just an xor um an xor thing of of their identity versus identity uh we we fill it into our bucket so it could be re it could be

28. 13:12

    in the lower back it could be in a higher bucket um and essentially we fill our local routing table the same as everybody else which gives us that kind of a list of peers that are on the network um that's like a high level overview of roughly how how we do discovery um well one question regarding ethereum is do you need at any point to send a message between two peers that are not directed

29. 13:34

    and connected directly like to root some message between two peers that might be several hubs like other than requesting a message yeah is it necessary no yeah yeah so um yeah i'll lift list these things here so we have just b5 uh rpc and gossip sub so these these are the these are the protocols i was going to talk about i'm trying to leave p2p so the rpc gives direct node to no

30. 14:10

    communication and gossip sub does uh kind of like a network-wide propagation um so that that handles some of the other things so let me just talk about instead of yeah hopefully how discovery works i just talk about the theorem specific things about the discovery that we've implemented um because yeah i think the other stuff you can kind of just just look up quite easily on the internet so for for discovery we have signed peer

31. 14:35

    records so essentially we have a container which which is what all peers kind of use to identify themselves and this is what gets stored in the buckets that i was talking about in every in every node uh so for each individual peer we have this thing's called an enr an ethereum node record and it has associated with it like an id which has a public key associated with it and this thing is signed so um i'll draw a key now so just this whole

32. 15:07

    thing here is is signed with the public key that represents the identity of this thing so so that prevents um if i'm asking somebody for a particular peer id that prevents somebody in the middle just you know making changing changing the the values that's stored inside here because it's signed with a peer id um so the signature would break if if someone should sacrifice doing it so inside here you can store key value data so for

33. 15:31

    example you can store your ip uh udp port udp tcp port so every every peer record on in discovery so we can search for these particular things and these are the things that we get back so for every uh when i'm looking for peers and i get a list of 16 in return i get this kind of signed container which comes from the original p and it tells me what ip i can connect that pier on whether it supports discovery over udp and the tcp port which means that it

34. 16:01

    supports uh the rpc and gossip sub so that we can connect via by these protocols to it it also ethereum specific it gives us uh we have a field called f2 that field represents the fork id that the node is on so we could have multiple test nets or or different networks where um the dhts could kind of be combined so if i ask it i could ask up here and find uh you know 16 16 peers in return but those peers could be on a different network than what i'm looking for so i

35. 16:35

    can filter those out based on their on their enr there's another field which i'll have to get to a little bit later which is called attestation nets back to station next and essentially we subscribe to these subnets in ethereum too and it's useful to be able to search for peers on those particular subnets so um inside an enr if you want to search uh the ethereum two network four peers for a particular topic you want to put um that kind of information inside the inner so inside enrs we can put information

36. 17:10

    that allows us to search repair so the two ethereum specific ones are the fork so we can search for peers on a particular fork or network that we're looking for or chain and the other one is the annotation nets field which i'll talk about later but it's represents what subnet they're on so if we can look for different kind of subnets so back to just give a recap so far what happens is when a node first starts on the network

37. 17:35

    uh it runs discovery discovery has a set of boot nodes um from those boot nodes we ask them for a set of peers we get back a collection of enrs and we try and connect to those enrs on the iptcp ports if they're on the the correct fork id that we're looking for so that should give us initially a connection to a range of initial peers that are on our network for ethereum too cool so that gives us discovery and we use yeah we use udp to do that

38. 18:07

    so once we have a collection of peers on our network we connect via tcp and we do that using lib p2p this is the framework that supports the rpc protocol and the gossip sub protocol that we use so when we connect via tcp um the essentially what happens is we start the negotiation framework that exists inside the p2p and the way that works is there's a protocol of sorts called multi-stream select which uh for every i guess every protocol inside lib b2p that you support you have

39. 18:49

    an identifier name or it's called a protocol id so when we connect we we establish an initial connection to another peer we start this negotiation process where i say hey i support this this this this and this basically i support this this business and and amongst the protocols that they all support we choose which ones we both support and try and set up the initial connections for

40. 19:14

    um for those protocols so for ethereum two the the handy thing about this is that it doesn't just support high-level application protocols it actually can support different transport protocols so if somebody supports tcp and quick or web sockets you can choose the preference or the order of which transport you actually kind of connect under um uh yeah uh so sorry i should rephrase that um you get connected on one of the transports and the pdp can then

41. 19:46

    negotiate uh the higher level applications from the transports the the i guess the three or four fundamental parts of lib p2p um i go from top to bottom this is from like raw transport connection transport so once uh somebody connects to you on a particular transport that you support you then try and establish an agreed-upon encryption scheme so you want to have your traffic encrypted for ethereum two we're using noise

42. 20:18

    so in the lib b2p framework we uh in the negotiation phase we say that we support where there's a protocol id we support this protocol id which represents the noise version and everything that we support uh if the other person connecting to us or the people we're connected to also support that we establish a noise connection which is our encryption layer uh once we do that we then try and set up a

43. 20:42

    multi uh multiplexer so this allows us to set up uh across one kind of uh connection we can we can we can multiplex a number of different protocols along the top and set up different kind of substreams um the the two that we support one in ethereum two we always support uh all client support um mplx is is the stream multiplexer that uh that everyone needs to support and optionally we can support yamax so depending on the clients and whether

44. 21:13

    they support that we negotiate uh the multiplexer and then after that we we start negotiating the application layout so for ethereum two as i said there's two protocols uh at the application one's an rpc and one is gossips up so at this point the p2p framework we've got connected on a particular transport um the we've we've negotiated a an encryption layer which is noise we've negotiated a multiplexer either mfx or yamax and now we're negotiating which

45. 21:47

    protocols we support at the client level most clients i think currently should support rpc and gossip sub so this will form two connections uh i guess uh we could we can now form substrings over rpc and gossip sub okay so just as a quick recap of what i've just said then when a node first runs on the network we use discovery we find a set of peers that give us a chunk of vnr's from those enr's we can connect via the transport that we

46. 22:17

    support which is tcp because that value is inside the enr from that liberty allows us to negotiate an encryption layer a multiplexer and then we can start forming connections with the two protocols that we support which is rpc and gossip sub sorry what is the motivation for the encryption layer isn't all the data like public that can be passed transparently um yeah so the the data that we've got in the application layer um

47. 22:50

    isn't isn't entirely encrypted um at a at a kind of a transport layer um in general just for for privacy reasons and for other people kind of putting in the middle i think we we always want kind of transport that uh we always want encryption at the transport layer just to prevent any any application layer from um from from looking at the ethereum to traffic so so you can see kind of tcp information but you don't actually know exactly what

48. 23:21

    what ethereum two traffic is actually going across it i see it that is to avoid certain type of attacks probably sorry to avoid certain title of attacks um yeah so in in um even in just in discovery v5 we decided to make even the headers uh all the information is encrypted um from and the way that it's encrypted based on the the node id you're connecting to so that prevents um things like uh deep packet inspection so for routers that decide oh

49. 23:53

    i'm gonna i'm gonna stop ethereum two by blocking its discovery protocol uh i'm gonna look at i can kind of inspect the packets figure out oh this is an ethereum two header packet but if it's encrypted at the that layout the headers then then even it's very difficult for a firewall or something to actually block specific traffic associated with uh associated with ethereum too yeah okay so

50. 24:15

    uh let me briefly just talk about make sure i'm not going too much over time okay let me briefly talk about the two different protocols and their primary use so the rpc protocol this kind of answers your question about uh the connectivity and i guess the topology between the peers uh the rpc is a direct peer-to-peer communication so for a node on the network once you've directly connected to um to the peers that you've discovered so let's say this is us

51. 24:46

    try and make it bigger and we've got four beers one two three four we form you know two-way direct communications with these four peers uh okay so there's two kind of let me let me talk about what we need to do so as a passive observer on the network we've now connected to four peers that are on the ethereum two network we need to firstly um make sure that we're we're up to sync

52. 25:18

    with the with the current state of the chain so if we're a new node that just joined the network um we've just connected to four piers we we don't know whether our current view of the of the chain is up to date with uh with all the other peers on the network um so we have to directly ask them essentially when we first connect what's the current state of the chain that you're looking at

53. 25:42

    and we ask each of them individually so we use the rpc protocol the rpc protocol has like four four or four or five different methods so it's essentially a request response and we use that to essentially kind of figure out the current state of the chain uh the status of these peers and get just kind of simple information directly from them it's primarily used for syncing so uh let me go through an event where we're kind of a structure where we're not in

54. 26:14

    sync so we use so the rpc has let's let me just go through the different request responses that we have in the rpc uh for ethereum two so there's a status message uh a status message is what we use when we first connect to um when we first connect to appear so a status is essentially asking them what's their current state of of the chain that they're looking at uh it tells us in response is uh the current the current

55. 26:44

    head slot that they're looking at which is uh kind of where that where where their chain is there the head of their chain in ethereum two we have a concept of finalization so it also returns when was their last finalized slot and also the root hashes for both of those things so if so there could be two piers that have um you know the same head slot but they're they're different blocks so they're on they're on different forks so with that information we can determine uh essentially

56. 27:16

    the current state or the current head uh or the or roughly the you know the head view for each of the peers that we connect to of of the beacon chain um the status also responds with uh the fork id so that we can tell if any of these peers are connected to are on a wrong fork so then we instantly disconnect so uh i i won't give him the time microphone go into all the syncing logic but we have uh so we

57. 27:42

    could say this guy is up to block i know 10 this guy's 11 this guy's 10 and this guy's nine so when we first connect we look at the status and each of the pairs give us you know their current view of the chain uh if the current block that we know of uh if we haven't seen any of theirs and our current block is let's say five then we know that we need to download the the blocks that these guys have seen to be able to make an informed decision on what is the current state of the ethereum two beacon chain

58. 28:13

    so that's when we start to invoke another rpc method uh which is uh blocks by it's called blocks by range i'm just going to go bvr and a blocks by range rpc method allows us to essentially request uh up to a thousand blocks in a range from one slot number to another so in the particular case that i'm using here we want to get blocks blocks from slot five to ten from this period 5 to 11 from

59. 28:45

    this p 5 to 10 from this p we can do this a little bit smarter where we don't double up and and duplicate the requests but fundamentally that request will uh the response to that request will will be that these peers give us the blocks of the beacon chain that they know uh we collect all of these uh we kind of um we process them and then we decide using some fork trace algorithms what's

60. 29:07

    the current state of the beacon chain um and then and this process this presence happened only this process happened only between nodes connected directly with each other yes yeah so yeah so yeah so of the peers that i i connect to i get uh roughly a view of the network um uh based on that so the algorithms behind all this are a little bit more complicated than what i'm doing i'm kind of simplifying it but essentially from the peers that you're connected to you get a view of the beacon chain you can

61. 29:41

    use the rpc to download blocks directly from these peers from from your immediate peers there's a there's you know you can get more heuristics or extra information from the network using the second protocol but i haven't kind of thrown that in yet because i haven't described it i haven't talked about it but yeah so fundamentally we connect to all of our peers we find them from discovery um given that given that view of uh using

62. 30:05

    the status request we can use the blocks by range request to to collect all of the blocks um the rpc has some quirks associated with it that allow you to stream blocks from peers um but i won't go into those technicalities just to save time and to save the interest uh the other status requests that exist uh uh there's a goodbye message so when we disconnect from uh we disconnect from up here we can gracefully go we're leaving and give them a reason why um one of the main goodbyes that you'll probably see on the ethereum two network

63. 30:39

    is that peers have a specific kind of number of peers uh like a peer limit and usually there's a lot of peers trying to connect to everybody else and you'll get a goodbye because your note on the network kind of has too many and we're kind of pruning you there's a blocks by root uh blocks by root which allows you to uh request individual blocks um i'll explain that guy it kind of goes goes hand in hand with gossip sub

64. 31:12

    um there's a ping protocol so that for all the directed all the peers that you're you're currently connected to you can inter you can ping them uh intermittently to figure out whether they're still live kind of like a liveness liveness check uh and there's a metadata rpc request so each of these peers store a small amount of data called metadata which contains these adaptation subnets

65. 31:37

    um which was in the enr the reason we have that is that if a peer connects to us directly we don't have access to their enr so that that attestation nets um value that's inside the enr we we would not otherwise know so we need we have this extra protocol to request it often if we don't otherwise know it um and i'll explain that uh in a second with gossips up okay so if i if i move on to gossip sub that's all right with the rpc roughly that we have direct it communication make sense

66. 32:08

    yeah okay so uh gossip's up so the next thing that we need now is let's imagine we've connected to peers so i just do another recap just so that everybody's kind of still with me uh when the node first gets network we use discovery we connect some minions other people can connect to us at the same time we connect via a different transport tcp uh using the olympiad framework we negotiate these protocols and we now have connected peers we use the rpc protocol to determine the current state of the chain

67. 32:41

    and figure out whereabouts we stand relative that chain we also use the rpc to download blocks and to get up to up to the current state based on all of the peers that we're currently connected to once we're in sync let's say so that we are we have the same number of we we we know about all the blocks that our peers know about we call that being in sync once we're in sync we we need to for an observing node every 12 seconds there's a beacon block being created

68. 33:12

    and each of these things need to propagate across the network to every peer every 12 seconds so this is where the gossip sub protocol comes in so the gossip sub protocol is a publish subscribe protocol pub sub for short which means that you can subscribe to a particular topic and you can publish to a particular topic and messages should get routed to all the peers on the network on that topic so let me explain there's there's a number of uh very good

69. 33:43

    talks and articles by the pdp on how gossip sub works and uh what it's uh what it's designed to do so i probably recommend checking those out but at a very quick five-minute inter overview of how close-up sub works is if you have peers that are kind of connected to each other in some kind of you know network hopefully you can see this you've got all these different kinds of connections um the the very naive way uh quick way of

70. 34:16

    sending a message to every peer on the network is you send a message to all your peers and you get them to send a message to all their peers and then they send their message to all their peers and you get this message getting propagated really rapidly across the narrative because everybody just sends the message so if one of the peers in the network says oh i've proposed a block he publishes a total of his uh they publish it to all theirs they publish to all theirs and it just kind of rapidly

71. 34:40

    propagates across the network the the downside of that is that there's massive um message amplification because if every peer every peer is connected to every other peer um you're you'll obviously find that the message gets duplicated um quite rapidly almost exponentially so the there's huge bandwidth costs associated with that the next um semi uh semi improvement to this is instead

72. 35:05

    of sending to all your peers you randomly send it to a subset of your peers so instead of if i have 50 peers that i'm connected to i don't just send my message to 50p as i send it to a random 20 of them and then they send it to a random 20 and they center a random 20. and depending on the probability that you use uh to randomly select pairs as to how many that you select you can you can kind of tweak the amplification factor that gets kind of sent but the downside is if the probability is too

73. 35:31

    low based on the network let's say you know use one percent you only send it to what one key and then listen to one p there's a chance that uh the message doesn't get propagated fast enough or to all the peers on the network so gossip savvy is kind of an improvement on that where instead of instead of sending it to a random subset of the peers that you're connected to you create

74. 35:52

    an overlay network called a mesh of a smaller subset of your connected bits so if i'm connected to 50 pairs i depending the the configuration parameters are uh configurable to set up the mesh but for ethereum we use roughly about eight is the ideal number of peers that we connect to and we say we form what's called this mesh and everyone else on that network also form connects to eight of their peers and they form this overlay mesh network so instead of connecting to every single pair we form like a mesh which is a

75. 36:28

    subset of of the connections i guess uh then when we when a message gets forwarded to us we don't fold it to every single appears we only forward it to our mesh pairs so that way we if you design the mesh depending on the the number of connections that you have to fit kind of the topology of you know how many how many nodes and stuff are on your network you can minimize the amplification factor but still get quite good propagation across the network um to counteract the the the probability

76. 37:02

    that uh a message doesn't get sent across the entire network there's the idea of gossip so if a message comes to me i send it back across all of my mesh peers which is only a subset appears that i have then i randomly select the remaining pairs that are subscribed to this topic and i tell them hey i've seen this message and the message is uh identified by uh an id and i and i randomly tell people hey i've seen all of these messages over the last few seconds then so those are the peers that are in my match so then if those peers

77. 37:37

    get that uh get that gloss up and go hey i actually haven't seen this from my mesh they can ask me directly to get that message and then and then they fold it on so it's kind of a correcting thing for any messages that don't get that don't get propagated across the network so it's a five minute inter uh overview yeah that was very very interesting about gossip sub it has its sound healing

78. 38:00

    mechanism but i'm wondering like it's uh fully organically the way it grows like or it also has a mechanism to guarantee that it has some nice properties about fast propagation yes um so there's a number of uh there's a lot there's a few papers that i probably recommend having a look at um specifically uh about how its performance goes and how it grows and scales with number of peers um and the propagation and latency associated with

79. 38:30

    that um specifically as i said the mesh is very configurable so you can configure the mesh to be the size of the number of pairs that you have in which case it you know it's like a slider it'll it'll go down towards the flood sub-level where you'll get very rapid propagation and very small latency but huge message amplification then with the configuration parameters you can scale it back to the other to the other the other side where you only have one mesh pier or something and then a lot of

80. 38:57

    it kind of gets properly go through gossip sub but you get an increased latency the the lib p2p guys that invented this had a uh upgraded this to gossip sub 1.1 uh in the process it was mainly security updates to to pretty much address a number of different security issues that attackers can have when they kind of join the network uh in that process they built uh kind of a simulation and testing framework that allows you to simulate

81. 39:21

    these networks at large scale and based on the parameters that they have and the networks that they set up they you can see how this thing performs what it's what its general properties are and how it scales um so i'd encourage to kind of have a look at that for ethereum two um the as i said it's a it's a pub pub sub system so you you subscribe to specific topics um and in ethereum too there's a

82. 39:46

    number of different topics and they have different network properties so the parameters kind of the scoring parameters at least change a little bit based on the topics that we're subscribed to um but for our use cases this fits the need for for the block propagation so uh so okay so yeah so for block propagation fundamentally um one of the nodes on this network will say i have published a block and then i'll send it across their mesh piers it should all get propagated across the network if it doesn't the gossip sub the gossip element of gossip

83. 40:19

    sub should should pick that up for peers that haven't seen that message um and then they they they request it and propagate it across their mesh so this has from from all of our tests has worked quite well for for what we need so so that roughly makes sense so far um so i think that's all the as i still had a high level i was gonna potentially go deeper but at a high level that's roughly for an observing node an observing node they connect to the network they use discovery over udp

84. 40:50

    they find a number of peers which are enr's the enr's tell them how to connect them what fork they're on um once we connect to them we use the rpc protocol that allows us to get a view of the current state of the beacon chain and download any blocks that we think that we're missing um from that we use gossip sub to then subscribe to uh the topic that publishers blocks and then because we're subscribed on that uh on that protocol we'll regularly see blocks that get published uh through this propagation

85. 41:23

    mechanism okay that was great so it's part of the interest yeah so the the next one is that's for a node that's just observing on the network for a node that is um that is a testing they have to do a little bit more and it's a little bit more gossip sub uh explanation so so for a node that's a validator essentially the same thing happens they go through and they find peers and then they get their current view of the network but they have to now do this extra extra step which is voting so

86. 42:02

    they get every epoch let's say this guy is one epoch there should be 32 of these so i can put a little dot so every epoch a single validator you can have many validators per node so if you've got a thousand validities you need to multiply this this step a thousand times um but for every epoch we get shuffled into a committee um and we need to attest to a particular to a particular block and

87. 42:34

    and form essentially an attestation so for every committee we get shuffled into for each slot they can be up to 64 of these things uh depending on the validate account so i'm going to so i'll do these these are committees that we kind of get shuffled into so they so validators kind of get grouped into these little sections per slot so every slot there could be let's say there's 10

88. 42:58

    um that would mean there's a lot of validators but but very slightly get shuffled into these kinds of committees uh the idea is that we don't want uh the person who's proposing a block shouldn't have to verify if there's 20 000 attestations uh to verify the signatures of which which represent the vote of every valid on the network because the load of that is going to be way too high the signature schemes that we're we're using in ethereum 2 is called bls and it has a

89. 43:32

    a very useful property that allows you to aggregate signatures so people on the network can let's say if you get if you get a thousand validators and you group them all up and you get them all to vote uh if they all vote on the same thing you can kind of group all those uh those signatures into a single signature called an aggregate signature and then you can give that to the block proposal and the block proposer then only needs to verify

90. 43:56

    one signature not a thousand so that's the idea of these committees these kind of like sub-committees we group all the validates into these sub-committees and get them to vote uh and i and then the validators get grouped into uh there's on average it's a it's a probabilistic thing on average we get uh 16 of the people in these committees to collect them all and aggregate them so they collect them all for each of these for so for each of the committees 16 of these people are

91. 44:26

    randomly chosen uh and so everyone kind of votes they they grab all those votes they group them all up into a single into the vote that wins or the vote that the sixth that the person that's aggregating them voted on and you group them into essentially one aggregate vote so instead of and this happens for each of the committees then the person that aggregates them all sends them on a

92. 44:51

    kind of like a public gossip subtopic similar to a block so a block proposer who needs to needs to create a block just listens on a gossip subtopic which represents the aggregate aggregate signatures so all these committees kind of group all their signatures and then they publish the grouped signature of the group vote on the gossip sub aggregate topic so the proposer just watches that gets all of

93. 45:16

    these aggregate votes in uh checks them all throws them into a block and that forms the next block fundamentally so the topics that we have in gossip sub uh there's a beacon block i'll just talk about the beacon block the main ones so there's a beacon block so if we subscribe to this topic in gossip sub we'll get regular blocks that come through as i was explaining before there's a beacon aggregate topic it's called aggregate and proof and this is what uh block proposers

94. 45:52

    should listen to and they get the aggregates from all of these committees the there's um there's a few others uh so each yeah so the next important one is that the from a networking layer when you form people into these committees you don't want the each of these individual votes being sent across the network to every single peer on the network because uh only these only these aggregators or the people in these committees really care about um those

95. 46:23

    individual votes to kind of group them right so each of these committees get split into their own gossip sub topic which we which we call subnets um so i won't write out the name it's a it's beacon beacon after station underscore subnet but we can just have like for example topic one topic two topic three uh and there's up to on mainnet there'll be up to 64 of these things these kind of subnet gossip sub topics so uh so if you're if you're on subnet one

96. 46:55

    for example that's that's for every slot let's say this is one two three four you get shuffled up you find out you're in slot one so you need to subscribe to the gossip sub topic one and you publish your vote on the gossip subtopic one all the other people in this in this uh subnet i guess or or uh committee uh group if you will um they will also subscribe to the gossip sub topic one so that they so the messages that get sent don't get sent uh to you know the entire network they only

97. 47:27

    get sent to all these people here the aggregators then kind of group them all up as i was saying into a singular vote and then submit it onto the beacon aggregate topic in gossip sub so that as i was saying the block proposal just listens to these two main ones and they get the the gripped one um yeah is that amazing so far these topics like they run on top of gossip sub like they they are

98. 47:54

    yeah so eat so you form a mesh as i was saying so this mesh of pears there's a mesh per topic so um yeah so to simplify it let's um if i'm connected to 10 peers of those 10 peers two of them could be connected could be subscribed to this beacon block topic so if they subscribe to that beacon block topic and i only have two peers i'll chuck them into my mesh of beacon block topic but if i have 10 peers and they're all subscribed to beacon aggregate

99. 48:25

    i'd probably chuck them into my mesh but then prune them off to to have summers left over and some that are in the mesh but there is a mesh per topic so when i subscribe too okay yeah so you form these kind of like dynamic overlay networks for each subnet yeah yeah there's a bit of complexity in there in the sense that um if peers are like let's say this is the this time period that you you need to shift between um sub next is 12 seconds if you have all these peers shifting every 12 seconds then the stability of

100. 48:59

     whether you can find peers that are on on those subnets very quickly within the kind of the 12 second shifting is is difficult so we have um kind of like if you have a validator you you subscribe to a subnet for a very long period of time so we have these kind of backbone peers that kind of sit on that network that you can form meshes with um so this i guess brings me to the part that i was talking about way back at the start where there was i said there was an an at

101. 49:27

     nets agitation next um key value thing inside the enr and also in the metadata so this thing uh is a bit filled so it contains a whole heap of up to 64 for mainnet lots of zeros and ones which determine uh which tell you which subnet they're subscribed to long term so you can use discovery to find peers that are subscribed to these subnets in advance so i know i know one epoch in advance so

102. 49:58

     i know 32 slots in advance when which one i need to subscribe to so i can use discovery to search for peers that are already subscribed and will be subscribed when i connect to to for example subnet 3. so i find those peers i connect to some subnet three that allows me to form a mesh and then other beers do that as well to to in order to do this uh small individual votes they can get grouped into the

103. 50:22

     larger nitro voting system i think so that covers it like a kind of like a high level the the dynamics of um how we're how we're splitting up the the individual votes so instead of the proposer having to verify if we've got twenty thousand validators on there to verify twenty thousand votes we can split them up into these subnets um the subnets get propagated uh the individual

104. 50:46

     votes get propagated using the gossip sub protocol and they get aggregated by a subset in that committee the resulting aggregate gets sent to the beacon aggregate topic which uh most observers and stuff will just listen on and you can see the aggregates kind of coming in and the block proposer whoever's whoever that is can kind of chuck it into a block and then all the votes kind of get included

105. 51:10

     in in the block as we go okay it sounds great i have like a lot of follow-up questions but yeah too little time left that's okay so one of the things we are struggling at uh near is about upgrade regarding the network how how do you handle that in ethereum like for example when there is an upgrade that requires some network changes is it something that happened and how do you handle it

106. 51:45

     yeah okay uh so upgradeability so i guess for consensus level things um as i mentioned through i i kind of skimmed over it because it wasn't a huge important detail but inside the enr is a fork id so we can upgrade the fork version essentially of the entire chain and here we can then find piers on a new fork uh this fork uh uh i guess number if you will like if the if the chain hard forks for example um it's built into ethereum two that all the signatures become essentially

107. 52:19

     invalid on on an old fork so uh all the new signatures and everything will kind of shift across um for peers that agree on on a new fork the rpc status message when we connect to appear also tells you what fork that they're on um from a networking layout i didn't actually mention this but the firstly in libpetterp when we negotiate the protocols so actually let me just split this up logically so the fork

108. 52:49

     id that i'm just talking about handles consensus kind of level upgrades so the signatures change um we could change some core level parts of the specification and nodes will be able to identify other nodes on the network and signatures kind of become invalid the next part is we need to kind of split out all of the networking communications on a different fork as well so we can do that based on so discovery usually wouldn't change we can still

109. 53:21

     talk to other peers on old forks or old outdated code but we can discover new peers for a specific fork based on the enr for lib p2p the way that we do the negotiation of protocols that i was mentioning earlier is that we have essentially a protocol id so protocol id is a string that says which protocols we support if we want to upgrade one of those protocols we can change that string inside inside the protocol id for

110. 53:46

     all of our all of our for our rpc protocol and in gossip sub has a version number so if we upgrade the version number of for example one of their rpc messages speaking i want to go like for the status message for example then we can support both versions if we want so when we do the negotiation through lib p2p we can if if one node supports status version one and another one supports version one and two status

111. 54:13

     version one will get negotiated and we'll use that but if both nodes have upgraded and let's say only one supports two or one supports both you can you can order which one you prefer which one has a preference so we would negotiate uh based on the version and the ordering preferences in libya p we can do that with gossip sub as well so if we upgrade the rpc or any of the rpc methods or gossip sub then libya people want to go share that for us

112. 54:36

     the the final part is for each of these topics that we're subscribing to um i only gave the name of the actual topic but uh the actual topic contains uh the fork id that we're using which uh and the encoding that we use so if we decide to change the encoding then the topic itself fundamentally changes so you would subscribe to different topics for different encodings or different versions so i think uh based

113. 55:00

     on the lipid p modularity and the way that it negotiates protocols we can easily kind of upgrade and change versions for old code and it kind of shifts everything across we won't be able to we won't communicate with other peers on different versions that we don't support yeah okay sounds nice it's very good this uh all the way you have the versions inside the topics or together

114. 55:24

     so and about sharpening well you use use the topics to like for example when you are rooting and you're submitting new transactions and you need to route them to new block producers is it handled and any meshes yeah yeah so um so there's there's some there's further issues especially in the networking um which when we introduce sharding there's there's more challenges that kind of come in uh one thing that they're that we're kind of working on at the moment is that um there's a data availability for it so we need to do kind of like random

115. 55:56

     sampling at a network layer um but the shards should fundamentally come into kind of subnet similar to these committees so we can still uh split up uh i guess we're calling things subnets but it's based on we can segregate nodes based on gossip subtopics so that network communication or propagation only happens to people that subscribe or are interested in a particular area so so the sharding and

116. 56:21

     some of the other networking challenges can kind of be subdivided based on using gossip subtopics okay nice okay i think we are on time that was a great explanation a great overview of the theory negotiator thank you for being with us today thanks for having me um hopefully it was somewhat useful to get a general idea of kind of how these things work and whether there's some crossovers we can have with near that that make things applicable

117. 56:52

     or if you you guys solve some problems that we're having maybe we can uh you can browse some of those ideas as well yeah okay goodbye okay cheers

118. 1:00:30

     you