Whiteboard Series with NEAR | Ep 43: Alexey Akhunov from TurboGeth

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:01

   hello how are you i'm good thank you and we're live right now one sec one sec there we go so uh hi everyone thanks for joining us today uh we have a whiteboard session with alexey working on turbo gas and uh joining us is danny michael uh misha and eugene who would be asking all kind of questions that they have after they watched the recent video without by alexey about the turbo gas so i gonna let you guys kick it off

2. 0:39

   and just don't mind me from now on awesome okay so i'll get started so alexi i know we talked about how you've been researching this uh for ethereum for the past three years and you found really some proposals for improvement regarding constructing state in a better way in order to avoid having to deal with issues like state rent or you know staking fourth said state brand later on the line just getting the design

3. 1:13

   right and so uh we talked about uh kind of learning from those best practices so i wanted maybe if you could tell us about those best practices for the group or like what were your findings and that can be a jumping starting point for us here yeah so thank you there there was a can you hear me okay yeah i'm just kind of muffled you know okay can you hear me can everyone else hear alexi yep yep it's not good to me okay so because i just i just come across

4. 1:47

   muffled to myself anyway so it the conversation i guess started when we were thinking about the state rent and you a lot of people probably are aware that we had the state rent project in ethereum and the reason why it started is because there was a realization that the state is very large and basically this the size of the state was creating a lot of the issues uh for example

5. 2:11

   synchronizations and processing and so it looked like a very kind of crucial very urgent problem um and nobody really knew exactly how big the state was and that was actually interesting because depending on how you store it the number would be different so if somebody asked for that moment but how big is it like what are we talking about and some people say well it's a hundred gigabytes some other people say no it's 60

6. 2:43

   gigabytes so it was already the first sign of that we don't really you know this is not very well defined so and in the trooper guest very recently we have experimenting with different ways of laying down the state and until recently we had a representation which currently would be where we stayed currently would be 50 gigabytes which was already quite a kind of good achievement but then we found another representation which is as efficient uh to access as that one but uh but

7. 3:15

   basically it's 10 gigabyte so and then made me thinking that uh if that is possible then a lot of the worries that we had uh back then in 2018 when we kick started the stage project were actually not as urgent as we thought they were and we can address them in much better way instead of you know we we do still have time to think about those things better and whether the stage rent is actually does have to be coming right now or does it could it go could it come a bit later

8. 3:48

   because it does come the state rent does come with a very um hefty price tag if you introduce it on a live system of course if you design a system from scratch and maybe you can do that much easier but trying to put the estate rent on ethereum where there has never been a state trend it's actually quite hard and it's taxing so yeah that's basically the what we were um talking about before right michelle do you want to like ask

9. 4:24

   questions that you had uh yes so actually the most relevant uh part of the presentation for us was their flat state the flood stage so yeah so this is a very relevant problem for us especially with wasp execution so in in our case contract execution is very fast because of compiling international code but uh storage access is still really expensive so we are very interested in optimizing it okay well to be fair

10. 5:12

    actually um i don't know if you've seen that the on friday there was the uh i mean there was my another talk on friday at least online but there was another one which is called evm 356 384 which is basically this is this is a project which also has a long story and essentially the moral of the story is that the evm you can actually make evm interpreter very efficient i mean the the perception of people people had before that evm is a sort of evm interpreter has to be

11. 5:44

    super slow actually is is not really justified because you can write a very efficient interpreters and actually there are already efficient ones and in this particular talk they demonstrated that for certain things you know if you had a certain operations added to the evm it actually becomes more efficient than web assembly um and simply for the fact because evm has the long word operations like 256 and and this proposal is for example introducing 384 bit operations and if

12. 6:17

    assembly has essentially 64-bit and it has to assemble older more complex operations out of 64-bit operations so it's not fair to say that you know so the web assembly is by definition faster than evm it all depends on your interpreter but coming back to the state problem yeah definitely uh so the flat state is exactly where the flat representation state is exactly where the stupid yes project started

13. 6:49

    from the observation that when we do state access in the traditional layout when we when we represent the state as a radix tree essentially so i can start drawing if you want so some people kind of know it as the patricia miracle tree but actually there is much more uh standard definition of this radix tree that usually if you if you had them if you had like a job interview uh sometimes they ask you a question how would you

14. 7:19

    um like if you had a dictionary if you want if you wanted to have a take a dictionary and put it into a data structure in your program what would be the the very efficient way of representing a dictionary in the dictionary essentially you have to look for whether the word exists or not and if it does exist you need to let's say find our definition or translation to a different language

15. 7:42

    so usually the the way can i oh why is it not oh because i don't have a pen so usually the dictionary so the the correct answer to this interview question is the radix tree where you essentially build for so the radix tree consists of multiple levels and at each level you can basically have let's say that we have only six letters in the alphabet and so the first level is that all the words that start with letter a they go in this way and all the words starting will let the b

16. 8:17

    go this way and and so forth right and and then sort of like then the second letter then the next level is the second letter of the word in a dictionary let's say that if we have a word apple then it of course will start with letter p and there is no letter let's say that there's no word with starting with two letters a like there's no a a uh this is like a b for example so that thing would start with the b so we have words starting with a b or like abacus and then we have the word starting with

17. 8:48

    ac which is like what is the starting something i don't know there definitely exists or something like that right and so you get the idea on the second level you have all the kind of letters here and so forth and then in the third level you get the what is the third letter in in the word so abacus would have a again and so forth um instead of abba for example if if abba wasn't a dictionary then it would be here and so forth in the end you get all the through the levels and somewhere in the

18. 9:21

    end you will get either the the definition of the word or the translation uh so here you get so that's the leaf of the tree definition of translation and so basically if you're basically building a dictionary in a job interview this is what you have to say you know the the radix tree um and so that's the similar idea to what the patricia merkel tree is or any other kind of radix tree but so in order to map this problem to the dictionary question is that in ethereum for example

19. 9:56

    imagine that instead of the words in the dictionary you have addresses of the accounts right and that's why if the first and here essentially in the first level instead of the first letter of the word it basically you look at first four bits of the address or or something and so there's only 16 variations of the first 40 bits so you know that in the first level there will be at most 16 elements and then in the second level at least at most 16 elements

20. 10:26

    and so forth and all it goes all the way down to a certain level where there's no more um you know as you can imagine that with the dictionary for example um at some point you don't really have a lot of branching anymore because for example if you take the the same example of a abacus after you've done a b a abba there is no many words that actually start with the a b like maybe a b a ba or whatever so imagine that there's no other words

21. 10:58

    which starts with a ba and then so in this case your abacus could actually be straight here so let's uh let's uh sort of like re erase this and so let's imagine that there is no other words that start with a ba so what we do that's in ethereum then we say we put the kus in here is like some sort of special element and then we map it to the to the definition or something like that

22. 11:31

    and the the reason why we could do that is because it's essentially like there's no other words with this is such prefix and this is a you know the way to kind of compress the this is how you can compress the tree um yeah in ethereum essentially the definition instead of definition you will get a balanced nonce um and something like this and of course is not only in ethereum and that's basically that's the structure and it's all good uh

23. 12:03

    but there are a couple of nuances to it um when ethereum had one of the audits uh the guy who did the audit andrew miller from least authority but or at least he was at least authority he basically noticed one thing is that it is possible if you don't if you just put if you just use the simply the addresses of the accounts as the words in such dictionary then it's pretty easy to construct um a very

24. 12:34

    kind of deep so because we we're not really like here it's not a dictionary it's the contract sorry the addresses and you can generate the new addresses very very uh easily so what you can do is that you construct uh the structure which is super super deep and and you can see how you can do that what you need to do is that you need to find two addresses which are which have basically really long prefixes two different addresses but they have very long common prefix for example if you can if you manage to

25. 13:07

    find two addresses uh for which uh which actually it's not very hard uh because because basically you can just say fff f f and then on the very end you get uh basically um let's say f so and so because the address is um 20 byte long so the depth of that thing is going to be 40 40 uh nibbles what are they called nibbles and then you generate another one which is going to be exactly the same but with the a with the e

26. 13:47

    in the end or something like this so yeah and then you're you're gonna have a very deep tree and there could be other things like oh so he was actually suggesting to do uh something like the spine structure so it's not just you do it on the very top bottom level but you actually you branch it on every single level so you're creating some kind of spine structure like this on every single level you do this kind of branching

27. 14:18

    so basically by doing this you create in a very deep uh structure like it's 40 levels deep and and then basically the reason why you know usually you say we can on what like okay 40 levels deep and what's the problem like okay that's thing that's fine um it becomes a problem precisely when you start thinking how you store this in the database and this is where you get tonight yeah yeah sure so i just want to

28. 14:50

    summarize kind of my high level understanding so the goal of this tree is the ultimate goal we have is to have a state road no not it's one of the goals but it's not actually the only goal so okay so what are the goals of actually this building so the goal the the the from my point of view the most important goal of this tree is to be able to add things easy easily and find things easily like is any it's essentially a representation of

29. 15:30

    a a stored sorry sorted map it doesn't have to be sorted it could be kind of just a map or something like this uh yeah but you need to do it in such a way that you can actually calculate some kind of commitment to this entire map and it happens to be that people risk decided that the commitment to this entire map is going to be a miracle tree of certain way of certain uh construction but the the the problem of computing the

30. 16:01

    hash route in a specific way wasn't there in the beginning it has been constructed the original problem was to be able to have a data structure which where you can put stuff in and you can get stuff out like a mapping sort of so what i mean is that if we don't have state route it doesn't have to be part of the protocol it could be any key value storage correct yes so yeah if you need to so where do we come to the state route thing

31. 16:35

    we come to the state root because so when we say okay we need to make sure that everybody has the same mapping right in the memory or somewhere in the database how do we ensure that well first of all if you look at what bitcoin does let's go let's step back from ethereum because in bitcoin you can imagine they have the same kind of structure but it does not um explicitly specified in a protocol because in bitcoin um there is also a state

32. 17:04

    which is a set of uh unspent uh utxos right and you need to have a state in order to be able to verify whether a certain transaction is valid or not so it's the same kind of thing but the difference is that in bitcoin they don't construct commitment to the state and they don't put them in a header so what they do is that they assume that everybody is having this same code or code does the same thing

33. 17:28

    without bugs so that if they run all the blocks they will arrive to the same state they're assuming that that everybody could do that and if somebody doesn't do that means that they're incorrect then they they sometimes can do wrong things in ethereum essentially for for lots of reasons the designers decided that okay we we don't just assume that everybody has the same mapping but we want to somehow compare these

34. 17:54

    mappings i mean of course miracle trees were known for a long time for such purposes like if you take the products like cassandra for example it uses miracle trees all the time to reconcile uh the lab replicas so if they have two replicas and one then data missing in one replica they use miracle tree to quickly find out you know what is missing and just reconcile it so they call it repair

35. 18:18

    so here is exactly the same idea okay let's uh have a miracle tree and then we basically going to have a way to reconcile the replicas because basically it's a replicated database okay that's uh in the replicated state so and somebody chose this particular form of merkle tree for for the for the task yes so ethereum as i understand uses this radix tree with hashes of keys for keys to make it balanced correct yes correct basic projects have uh

36. 18:56

    different ways of building the trees such as like balanced binary search tl correct yes the notable example is cosmos right yeah i've looked at that structure as well so yeah that works too have you looked into trade-offs between them at all yes i did so uh the trade-off is this um so one of the main properties some one of the main differences can i uh create a new page here is it like a disk plus button right

37. 19:37

    yeah but but but but you can also like scroll down uh with you know you have the controls on the right and the right panel you can just click on this cross there and scroll it down or you can create a yeah you can click plaster a pan you mean the punting okay fine yeah so essentially the the trade-off are quite interesting so the trade-offs are uh in the balance tree uh let's say it's ivl tree or it could be a white balance tree or whatever red black tree

38. 20:09

    so the avl tree are probably the best ones because they they have a much better bound on the on the depth um so what happens in the balanced trees is that if you take a one tree so they basically binary and you um imagine that this is like some kind of algebra right so in this algebra your trees are your elements and then you get some operators that operate on these trees so for example one type of operator is that when you want to add a key value pair

39. 20:41

    which is basically adding some sort of key and then value at the end so that's one operator another operator for example is deleting the node so another operator is removing certain key value pair um and so forth so there's two different two main operators you add something your delete something so the main property of the uh radix tree is that all the operators are they they commute with each other like it's this particular kind of in

40. 21:13

    this particular algebra these operators are commutative so it you can or you can basically add elements in any in any order and then or remove them in any order in the end it doesn't matter in which order you've done it uh what really matters uh sorry but uh in okay one correction the the additions are community but the um uh and deletions but they do not commute with between each other so basically additions commute with additions deletions commute with additions with additions but the uh additions and

41. 21:46

    deletions obviously did not commute um so you have to know in which order to apply them but for the balance tree it's not the case in the balanced tree these operators are not commutative so it does matter in which order you add them so essentially if you have it two elements you add them let's say a and b and take the balance tree you add a first and b next and you can have a

42. 22:11

    different tree than if you actually add b first and then a uh next so that is a bit of a challenge um and if you need this property if you do need the property of uh commutativity then you have to solve it somehow you need to i mean there are solutions to that but that basically that's the main uh con cons i mean uh the disadvantage of the balance tree but there is advantage of course in the balanced tree you don't have to do these extra hashing to uh to

43. 22:44

    produce the balance because the tree balances itself and it has a it simplifies quite a lot of things i would say uh for example because you have this hashing stuff for uh some implementations initially we also had it but now we don't you've got to have these pre-images which are quite annoying you know we when you look at when you look at the tree and it's it's hashed and you have to always like um let's say basically if you go back to this tree uh

44. 23:17

    if you imagine that these um um you know these letters like these paths in the tree are not actual addresses but the hashes of addresses and you've somehow found one of these hashes how do you recover what the address was well it's not really possible without really knowing without having another database which will map the hashes to the addresses which is a bit of a you know trouble anyway so i think the main thing about the balanced trees the main thing i like about balanced

45. 23:50

    trees is that you don't need to do this extra step but as we say uh it's a bit of a yeah it's a trade-off yeah so then if we're implementing a client so the client would store for this try structure it would store my pin from a hash of trinode to the node object essentially well this is how the yellow paper tells you to do so but that's not how you should do it well yeah let's start with this naive implementation okay and then that would be for that it would

46. 24:37

    be using some kind of key value storage that yeah that's what everybody essentially is doing um they are um so if you go uh go back to that structure or i don't know it doesn't matter which structure it is it could be this one uh if i just uh let's see let's make it a bite of course what is it i'm just going to get them confused let me see let me erase this thing and this thing okay

47. 25:07

    okay let me just have a little tree here okay let's three levels will be enough okay right so the way i usually think about this is that imagine that you have um the um the nodes are pointing to each other they're kind of pointers and each pointer is i i draw it as an arrow here so usually when you have those structures in memory um you basically have this element on the top um okay so how do you actually point to something here is there a way

48. 25:45

    to point like uh there is we can see your cursor my cursor well i don't have a course or the ipad because i um yeah yes i don't that's why i i chose ipad because it's easier to draw but and then i don't have a coarser anyway it doesn't matter so i'm just going to say so you know even in this top element where i drew the two arrows from it imagine if you store this structure in memory not in a database what it would mean is that

49. 26:17

    this top element would have two pointers right and the pointers usually i mean if you have a 64-bit architecture the pointer is essentially a number which is eight byte number which has the address the memory address of the thing that you're pointing to so that you can go there and look it's very similar how these naive would say implementations do it but instead of the pointer being 8 byte memory address the pointer is in the

50. 26:47

    case of ethereum 32 byte hash and in order to dereference this pointer in order to find the things it points to in the in the case of our memory we simply need to you know ask where what is inside this at this memory address in the case of the database representation the referencing the pointer means looking up that record using this hash as the key so it's basically the same process so that's why if you establish this connection then you can basically think about this as the pointers in memory

51. 27:20

    and then you can very easily reason about what happens when you access things and you can obviously see that the depth now matters because if you always start your search from the root which you basically have over time and you need to find something the deeper it sits the more hops through the pointers the more dereferencing operations you need to perform and then if you remember that in a

52. 27:45

    database dereferencing means doing a lookup then you realize okay you know if i go eight levels deep i do eight lookups if i go to 40 levels deep i do 40 lookups and another feature of that is that you cannot parallelize it because you they are data dependent you cannot hope let's say in this picture you cannot hop straight from the top element to the second level you have to first go to the first like to the first level then to the second level so you can't skip the levels

53. 28:15

    uh because you need to find this it's like a it's like a you know a quest you know you first you need to find the next point or the next point or the next point and that's why the depth really matters and that's why the uh this uh security audit i was talking about that's why it was a problem if somebody could uh create those spines as i was pointing out those okay where's this thing again oh no it's the wrong button sorry yeah the spines that i was drawing here

54. 28:46

    right you see the spine um no actually no anyway so you can't see the spines because you somehow moved this thing around um i don't know um but the spines that i was drawing yeah the the spines here on the right so the problem with those spines if you manage to construct them then um you force the tree to be very deep and then uh you your all your queries inside those the area of the street are going to be super slow um so you can specifically like if you

55. 29:18

    don't like a certain contract if you want that contract to become slow you will create spines around this contract by by basically sending you know one way you know a transaction with one way to those addresses and you don't even need to do any kind of address mining you just um do that and of course if you now hash everything with the sha whatever 256 before you put it in three it's still possible but it's uh it's much more resource intensive so you need to buy um some sha i6

56. 29:52

    sarsha 3 a6 to be able to pull that off i mean you could still uh probably go a few levels deep but um i mean it's not that potent anymore and yeah that's actually uh the good point is that this prehashing before putting it into the tree is not a perfect protein protection against uh this this balancing attacks whereas the cell balancing trees are pretty much the best protection you can get theoretically so it's theoretically the best protection

57. 30:25

    yeah so let's start with the naive implementation and then try to optimize it okay let's do that in any case we always have to compute state road for each block correct not not correct um not in any case because so what you can see now happening in ethereum is that a long time ago i don't know which year was at 2016 or 2015.

58. 30:56

    most people basically gave up on the idea of computing the state route at each block and that came in the form of what you know as a fast sink right you're probably familiar with fasting so with the fasting essentially instead of trying to go through blocks through starting from genesis and all the way in computing we stayed and executing everything and hashing it every block which is super slow what we do instead is we we're choosing a certain recent block which we call pivot

59. 31:29

    and then we download from some peers the the the these the pieces of that tree and once we've done that we compute the state root on that tree and then we verify okay that matches up with the head but notice that we have skipped the computation of the root for every other block before that so from that yeah from that on of course we we start computing the stage for every block when we're validating new blocks we always have to compute

60. 32:03

    status again this is not true again because basically i want to open your mind to other possibilities basically the one the whole process of discovering trouble geth is throwing away the dogmas that you might think are they have to something has to be true and the reason why i'm saying is that uh because in turbo gas for example sometimes we skip the verification of the root uh the for the for example if we if we think on

61. 32:34

    hard drive on hdd it cannot do it for every single block but it can for example do it every 15 blocks which is i think still fine you know because if there's some problem you will notice that anyway um after 15 blocks so i don't see this as a as a completely dogmatic thing that we have to validate uh state root average after every single block i mean depending on what you're doing you might sometimes skip that and in fact a lot of people skip that for most of the blocks when they do

62. 33:08

    fasting yeah that's a great insight okay um so so then uh then my question is now we could go uh now we have two different things we can do to optimize it we can either change the way we store the state completely or we can store a separate secondary version of the state yeah which would be better okay so what you just said the second option um okay the second option where we say we start building a secondary structure

63. 33:57

    which has basically kind of flat-ish store this is what i believe currently go ethereum is doing so they also realize that the flat uh representation has advantages and but they don't want to kind of completely redesign the the model at least for now so they basically started building this thing on the side which is the snapshotter and that's uh one of the approaches but from my point of view if you maybe that's the only thing they can do i mean it depends like what situation you're in if you

64. 34:30

    are if you want to get the best results obviously you have to be change the the data model entirely because there are a lot of things that you simply cannot do with the if you still have the try i mean basically we're not completely so thing to understand is that we're not com completely throwing away the try as the concept but we're throwing it away as a physical thing where uh so essentially another dogma

65. 35:02

    that you have to get rid of is that the try is does not have to have a physical embodiment it could be just a logical concept and as long as we can produce the state root in some ways which will match up with whatever the try would produce that's fine you never have to have a try actually even in memory and in fact we don't we don't we used to have a try and memory in true beget at

66. 35:26

    some point but now we don't even have it anymore as the as a physical embodiment of the of that thing so yeah you can go like as far as you want like you can go baby steps but then you you it might take you a long time or you can just like rip everything apart and then create a new thing of course in our case we didn't repair i mean i didn't rip everything apart i was discovering lots of things on the way like as we would you probably plan at this white boarding session is that we're

67. 36:00

    going to start applying optimization a bit by bit and eventually we will come to the final design we could definitely do that if i remember all the steps but there were a lot of steps uh we could try to directly go to the final design but then let's talk about problems we have with uh okay if we would try to implement it right away so one difficulty i see is that there is one thing that this

68. 36:34

    naive way of storing the tri gives us which is accessing any version of the state right so if we have blocks coming on top of arbitrary blocks and we have to compute state transitions we need to access every version okay so this is this is what i want to question in order to compute the state transition you don't need access to all the versions you only need to access to the previous well basically you only need to access to the current version

69. 37:06

    that's it you don't but which one is current okay so when you're when you start your state trade transition you need to make sure that you are in the correct state and then you apply the state transition and you change your current state so in case of ethereum where we have proof of work blocks does it like when a new block comes in does it mean we have to roll back current state a few blocks and then apply blocks from indication no no okay so again yeah

70. 37:37

    i think what what we're kind of missing a bit here is that again in um naive implementation you're right you can think that you can have multiple uh states at the same time you can hold on to because basically every you can point um you can point to any version of the state by just simply one single pointer which in your case is the root hash yeah that's fine so you don't actually have to make a choice which one is the current one

71. 38:08

    you can say oh you know i've got few of them around and you can use any of them in the if you start using the flat model you have to make a choice you have to always know which one is my current state right that's my current state and you only you you can only have one so now you have to be doing things slightly differently so if you basically receive multiple blocks which are the same height and you need to make a choice or you already made the choice but it was wrong choice you have to come back and reapply the

72. 38:40

    other thing so you always have to maintain your your chosen state so you cannot have it like you know in multiple ways so what triple get does it's actually it has only one current state and it moves it back and forth mostly only forward sometimes it has to move it backwards but that's not a problem you move it backwards and then you go forward again if you're if you know the other path is

73. 39:06

    better and yeah that's another realization you have to do is that you always have to know what is your current state yeah so does turbo gaff try to make try to make current state the most recent blocks or if kind of a few block break and storage if separately the most the recent blocks so we are so we always the current state is always the state that has resulted from the whatever last block we have received uh i mean whatever was the best last block received yeah

74. 39:41

    we never try to be a bit behind we we always try to be as far as possible right in the in this process um so okay and then you probably were asking how do we go back right is this what i assume we store diffs yeah okay so that's the that was the case some time ago we started divs but actually now we store reverse dips uh uh which is there's a slight difference but it makes a big uh impact later on yes so we store dips as we process the blocks every time when we uh process the block

75. 40:22

    we remember which uh bits that we have changed and uh we obviously during this execution we have information about what was the value of the thing everything before the change and after the change right and so obviously the value after the change becomes part of the new state and uh the value before the change this is what we remember right and then we remember that for every single block as we go we call them change sets and so we have them

76. 40:54

    and that means that whenever we need to go back what we simply do is that we take that change set and we're applied to the state because it will basically just because it contains the the things that were before the change so essentially it rolls everything back you can also call them maybe roll back sets or something like that so they basically allow you to do to do to go back uh and and then you can also go back pretty fast because uh you know if you want to go back let's

77. 41:19

    say three blocks you read all these three things you combine them and you roll them roll back i mean you can do it in many different ways or you can like the merge sort because they already pre-sorted you can you know there's quite a lot of things you can do so the flood state is a certain map right sorry what is what map flat state is assorted map of keys it's a it's a shorted map yes um i mean it doesn't strictly have to be sorted but being sorted

78. 41:54

    um helps a lot of things along down like down the line um because basically having them sorted allows us to um do some kind of map reduce things you know uh you know what i mean like uh the the basically you can uh set up this kind of flows of data which are which have to happen in a certain order um yeah it's sorted because basically it's a database and the database we use is the b3 and it's sorted yeah um oh actually that's interesting uh the

79. 42:34

    one of the things that haven't happened yet but let's say that if either scan let's say um or whatever block explorers started using uh turbo gas what they could actually do i mean we could introduce the rpc request so because we actually store the state as the pre-hashed version let's say before the oldest hash is applied so we essentially store it as the mapping of addresses to this to the values not the hashes of addresses what you can also do is that i don't know how the user scan does it let's say

80. 43:04

    that you have this um you know the uh the the search bar where you start typing the address and it shows you the you know what are the addresses where this prefix exists in the state right let's say you you you say type in abc and it already shows you the you know the these are the existing accounts with the abc and they probably uh pre-you know pre-made this specific table but in our case you don't have to have a special table because you can simply search uh through that state

81. 43:38

    and that kind of helps to be sorted for that case for that reason so you don't catch the addresses we do but we don't hash them until so basically we don't we only hash them until sorry we only hash them after we have executed the transactions so it's all separated so evm works on unhash state once it's finished then we convert an hash to hash and only after that we compute the state route

82. 44:10

    it's basically a separate process but are you worried like if your back-end storage that you use i assume you use some kind of that's excellent question i know what you're going to ask but it's an excellent question so why do we store the state twice right once unhatched one hashed right okay that's kind of a waste isn't it yes it is the waste and it was even more waste when we um so when we actually introduced this extra thing extra

83. 44:42

    separation hashtag so obviously in the beginning we only had hatch state and we executed avm on a hash state and that means every time you access something you just have to do another three calculation which is not maybe a big deal but it actually would have become a big deal later on anyway so we separated them you know and it's obvious the state was 50 gigs or whatever 60 50

84. 45:07

    60 50 60 gigs so then instead of one table of 50 gigs now we have two tables of 50 gigs bad isn't it but it wasn't actually bad because we also had another table which was more than that tiny bit more than that which is pre-images remember i told you about pre-images so guess what when we introduced the separated the hash and plain state we could get rid of pre-images because we didn't need them anymore now we have all

85. 45:32

    the plane state we can run all the queries from a plane state within net preimages so essentially there was no net gain in the storage then what interest another thing happened when we managed to squid squeeze the state into from 50 to 10 then it was actually net positive we ended up because we already throw away the pre images and we wouldn't be able to squish the pre images that way so essentially we ended up with 20 gigs instead of 120 gigs so it as a result of these two

86. 46:04

    operations i see so at that point could we maybe make uh kind of go back to to go directly to the final implementation okay yeah so you probably you probably imagine what the final is it scrollable or for me or not when i scroll it do you see it scrolled okay uh it's very it's a lag well well i'm scrolling it i'm sorry you can also share your own screen and zoom if you want so that i don't share

87. 46:47

    my screen you want to do it ah this is what no the thing is that i no no it's okay because i i basically have an ipad here on my right and i have a screen here i uh it will take me a while anyway so but okay no just scroll it back so to make sure that they um pass that the binary tree because i'm going to be drawing here look i'm going to be drawing right here here can you see that under the binary tree where i'm drawing

88. 47:18

    okay yeah this will make it like on the top of the screen or something like this i'm going to start from there okay so let's say that this is address one address two okay so what basically we do is that we optimize the data layout for evm because evm wants to have instructions like a balance let's say and the balance takes the address as an argument right and it wants to have a balance or if you have essentially a slowed instruction then it gives you two things

89. 47:59

    it gives you address and it gives to the location one something like this and it wants you to find um a storage item or if it doesn't exist it gives you zero right and so that's basically this is exactly how we map this in a database so we make it easy so we map in our database it's the same table but we map uh addresses to the very simple structures which is basically balance

90. 48:30

    nonce something like this uh maybe cold hash but we we we actually wanted to remove hashes from there as well but let's say there's some sort of hashes there doesn't matter and the same thing here um balance nons and some sort of hashes for the all for these things of course uh we have some kind of concatenation there is a tiny bit um complexity here which we called incarnation incarnation that's the i would say it's an advanced concept

91. 49:04

    um but that's basically how we managed to get rid of get around the create two and sell destructions uh because that creates a bit of a trouble for a flat model so essentially for the for the contract storage the keys in our database com consist of three components not two first component is address 20 bytes then there's incarnation which is currently 8 bytes and then location is another 32 bytes

92. 49:35

    okay the location is on is not is not prehashed and the value is essentially the value and that's basically the model so if you want i can go deeper into what incarnation is if you really want to know but do you want to know yes we do okay so the incarnation essentially is required because there is some really odd thing about self-destruct and and and even more odd things about the combination of self-destruct and create two so self-destruct

93. 50:08

    is this operation so this truck so what it it's very unique operation because what it does it for the very fixed amount of gas it allows you to delete a lot of data theoretically like if you have a contract which has like million entries of course it doesn't really happen in practice because such contracts usually don't have self-destruct instructions but theoretically um you can have a really large contract

94. 50:39

    and then with one in one go you do self-destruct and all that data is gone it's not visible anymore and of course if you store everything as this tree model that's easy right you just remove the pointer and everything and you expect there's some kind of garbage collection will be going on and then everything which is not referenced by the pointer will be collected right that was the still thought but for turboget because we use the flat model

95. 51:10

    it means that um we can't do this thing uh we have to remove it genuinely remove data and that wouldn't work because that would be a dos attack vector if you're requiring to let's say for this fixed gas cost remove a million entries from the database right you know it could take quite a long time so that's why we decided to get around this problem is to um so first solution will be to just ignore it because we said okay

96. 51:42

    let's if do we do self-destruct let's not do anything with the storage let's remove it at all okay fine because because i thought the only way to do self-destruct is first to do create or create or create a contract using let's say a usual transaction to a non-existent address or doing some sort of create but if you remember how the address of the contract using create is calculated it calculates from the

97. 52:11

    um creators so it's basically calculated from the creator's address plus nonce and some sort of hash of it so essentially the idea was that it's not it's it's theoretical or it's basically practically not possible using create or this the old way of doing things to uh to land the contract on the on the same address as as the previous one so that's why the our strategy of simply not removing the storage worked

98. 52:45

    we just simply didn't remove it and it was fine until the crate 2 came along and with crate 2 it is possible to land the contract on exactly the same address as before and that is a problem like what do we do with the self-destruct right now okay so that was a big issue and that's why we started to introduce incarnation we still did not want to remove the whole storage straight away maybe later during the pruning but so instead of deleting it we wanted

99. 53:10

    to mask it to make it invisible and this is how we introduce why we introduce incarnation so every time we create a new contract on the same address we increase we increment the incarnation so let's say the first time a contract created on this address it has incarnation one then if you sew this track we still remember that there was something in there and when you started create it again it gets incarnation two and then so forth so basically then we this way we separate the um

100. 53:43

     the storage items of the different incarnations of the contract and we don't let them to be mixed up so that if you're you know we basically make them invisible and then the idea is that when we implement pruning which we haven't done yet but it should be pretty easy we will figure out which incarnations are dead and we just clean them up uh but that's how we get around the issue with create two using this incarnation thing

101. 54:07

     which is kind of weird but it's one of these things that we have to figure um sort of figure out and incarnation also uh crucial when you do um unwind because basically let's say that if we didn't do the incarnations and we did it delete all the things right imagine that you did delete everything self-destruct but then you said wait a second i want to you know i want to do rework can i just go back and then

102. 54:42

     do the block which did not do the self-destruct so that means that if you do want to do that in the in the rework you have to reinsert all this million uh things into the database again so that actually kind of convinced me that we we should not never delete them basically because you want to you want to go back and you have to reinsert them and then you might remember maybe you haven't seen this but basically i was saying

103. 55:07

     that the self-destruct was probably a bit of a design flaw in the evm because it's a it's this kind of it's this kind of operation which could potentially have a very large impact but has a fixed cost and then reverse of this operation is even weirder because i think now from my point of view the inverse of any operation should also be quite cheap to execute but the self-destruct is very weird

104. 55:37

     basically its inverse is even worse than the operation itself okay makes sense so basically that's the flat model it says that's it there's not much more to it uh just the way you compute the okay stage root is computed in two uh steps so can we scroll it uh can i scroll by the way let me see does it work nope nope nope okay you scroll it okay so um so step one way you're going to scroll it under i want you to write under the

105. 56:29

     self-destruct thing okay thank you so first step one is that we obviously need to convert the unhashed to hashed which is easy right it's it's pretty easy i mean it takes a while but it's easy so i'm not going to get into the details of that you can figure that out so in the end we are we're sorry okay so at the end um after the stage we have address hash at one mapping to something an address hash to map into something okay uh to some values values

106. 57:07

     okay so then what's next um and the next you want to compute the miracle root okay um so what we do um so the problematic bit about the flat model is that in initially um i didn't even bother uh having some extra data structures for for for this particular operation uh because i would just say okay we just recomputed this whole thing uh from scratch all the time but obviously and it at that moment the state wasn't that big and it took about you know sometimes it took

107. 57:40

     about half an hour on some devices but maybe one hour on other devices and so it was okay but then i i thought i'm gonna solve it somehow differently so what we do currently is that it's also not very uh it's not very uh difficult so imagine that radix tree again let me see it's right x3 and so what we're going to do in this radix 3 is that we also going to have a values here at each point which are going to be

108. 58:14

     inter we call them intermediate hashes so with each with each node on this radix tree let's imagine that we have these intermediate hashes okay so that's basically the meaning of this intermediate hash is essentially it's the it's the hash of that sub tree right and that's exactly why we like why we like these um explicit representation of the smerkle tree because at any point of time we could um would say that okay so let me illustrate

109. 58:55

     this problem um okay so let's okay so let's say that we wanted to modify uh a bit this bit right this bit okay so we wanted to modify this bit and as you know what will happen is that will be some sort of ripple effect right so if we modify this then the hash of this whole element will be modified i'm going to circle it then because of this is modified then the hash the hash of this is modified right and then eventually the hash of this is modified

110. 59:33

     right but as you can notice is that although the so when let's say we're computing the hash of the root element uh because the other hashes are not modified we can just use them as they are and this is exactly where um how we uh we basically do the auxiliary structure so um we store the mapping of let me just um it a little bit it's very simple actually it's a key prefix to intermediate hash and that's basically it that's the whole magic so for each node in the tree

111. 1:00:20

     we simply store the key prefix uh mapped to the intermediate hash that's it and that structure is so turns out to be pretty small uh i mean i think now it's about two to three gigabytes for the entire state and that taking into account that it contains all the intermediate hashes for all the for all the basically uh non degenerate prefixes i'm just gonna call it okay so basically there this is this again this is not uh the magic basically is in the way that we are so it's not actually again this is

112. 1:00:56

     not a magic the the bit that where you need a lot of interesting engineering is that how do you update the structures and how do you essentially walk through them because when we are when we are updating the state route we first we start with this change set which we're talking about or rollback set where which basically is the enumeration of all the keys that have changed and then we're starting with the structure and then we

113. 1:01:26

     we know we basically walk through the structure and we we carefully using some sort of hash stack we just walk through this whole tree and compute um compute the result so it is a kind of like it just it requires a careful algorithmic work it's not uh it's not like something out of ordinary so basically because it's it's it's like um imagine that you have to do some kind of merge two um

114. 1:02:01

     two structures that already pre-sorted right the merge swords you know and one way to do it if you if you basically do it in memory then it's easy but if you let's say do merge sort of with the two huge files which are pre-sorted then of course you enter you open two i would say we call it core source or whatever one of them goes through the first file or iterators go through and another one goes through the second file and then just at each point you compare them

115. 1:02:26

     and you just move them um in a lock step with each other so the same mechanism here but uh what you do is that your um these two structures one the first structure is this uh mapping of address hashes to the values and the second structure is the uh mapping of the key prefixes to intermediate hashes essentially we have this uh kind of merge iterator going on through those things so so basically this uh preferences are

116. 1:02:59

     only if there's a note right so so let's say you have abba and then you have like a b is the same abacus all right so then you will have prefix a b yeah that's okay so yeah so basically we don't store all the possible prefixes but only the prefixes that are non-degenerate that are that are kind of have something in them and i guess this is actually this is this kind of comes to the point is that um people before

117. 1:03:36

     before us uh did not essentially just attempt these things because i think they should have because somehow they assume that all these things will be super hard and and it will take a long time but if they tried it would be fine you know like did somebody try before us to just take the entire state and mercalize it you know even for this like you know we we tried it it worked and then

118. 1:04:02

     people other people started trying it and said okay actually that's not that bad and then um and then they started trying other things like oh what happens if you yeah when you happen when we do this prefix mapping oh actually it turns out we're not that not that large you know uh you know if you probably thought it would be like uh tons of like tens of gigabytes actually it's a two gigabyte

119. 1:04:25

     or three gigabyte um yeah but from their complexity point of view when you're changing the storage like you also had to change a bunch of other stuff but presumably client it was relying on the fact that every version is available and now you have to do switches between yes that's why it took two and a half years to essentially retrofit all the functionality using this approach because basically we broke

120. 1:05:05

     i broke everything i have to fix it back again and i just keep kept breaking it and until until basically this year the around you know when we did the alpha release in july i sort of realized that okay we have unbroken enough we have fixed enough stuff that we've been broken yeah so that's basically the thing that you do i mean that i did just broke everything completely and radically and then try to fix it again

121. 1:05:38

     so what yeah so the question is like let's say you need to from rpc perspective return some historic state um is it even possible in ethereum or like you don't really need to worry about for this particular implementation good question yes that's very good question so there are two um ways you can look at the historical state one is i think more important one is less important so um you can look at any um so you can ask the questions about

122. 1:06:16

     okay so what was the balance of such-and-such account at such and such block right that's quite easy with a flat structure right so you just look at the i mean you basically because if you're if you store the state in a flat structure you can also state the history in the flat structure in our case we store the history as these those rollback sets and then we index them and that's another story i think

123. 1:06:39

     actually the data structure which stores the history is more complex than the data structure that stores the flat state so it's we're basically making an index over the robux sets but we can pretty efficiently figure out you know which uh block this certain account has changed uh you know around this sort of basically if you're asking what was the balance of such and such account and a block one million then we go through our index

124. 1:07:09

     and we find okay when was the this account changed uh first after the block one million and we we use the index to find it and once we found it we look at the robux set let's say that it was changed in the block one million and fifteen that was la the first time it has changed since 1 million we then open up the rollback set and we look okay what was the value before that and the value before that was actually the value at the block one one one million and this is what we do

125. 1:07:41

     for such queries it's still much quicker than going through all the trees right um right and so so how how do you start this index to support this type of or about operations okay so we have we have a one way stored indexes but now we're actually going to change it to a different way so the way we change it is so index is uh essentially a series of block numbers so index has one record well logically one record per per account or per a storage item

126. 1:08:14

     and imagine that the value is basically the string of block numbers so from and those that the block numbers all the block numbers where this account has changed its value and you can pack it in different ways for example we we currently store them as the actual numbers and when split them into the chunks but uh we have now we're now moving most of the indices into the bitmap roaring bitmap indices so essentially just going to be a bitmap with a certain representation and it

127. 1:08:44

     turns out to be even more efficient than than than this just stored as a sequence and i think we're going to change that soon to to the bitmaps but even without bitmaps it works the index works pretty well and the index is not actually that large either well it depends on like how active an account right so let's say well oh yeah you know i what i mean by not that large i mean in total for all the accounts in this in

128. 1:09:16

     the system uh i mean he um let's say that i just let me find if i can find the the latest data i mean the latest that was in like september uh of course so okay so the update on it it seems to be like incremental constant time because you only need to like update accounts that were touched in the block correct yes so updates are incremental lookup is just a binary search on the

129. 1:09:48

     top of this right now potentially yeah lookup is a binary search and so everything is kind of searchable we don't do full scans so yeah the oh this seems to be quite quite so it seems to be 19th of august the latest chart okay so basically the i just have a data from 19th of august how big that index is for example from that date the rollback set or we call it the change set we probably should remove it

130. 1:10:21

     for accounts it was okay for accounts it was 50 gigabytes for for contract storage it was uh 65 66 gigabytes and then index on top of that the index for accounts was 16 gigabytes in index for storage it was 40 gigabytes uh so that's basically allows you to um go back to any account any storage item in history reasonably quickly and figure out what the value was and this is what we use for uh historical retro like re-execution so what you can do with trooper get is that

131. 1:11:06

     you can pretty efficiently re-execute all the historical transactions and my recent test was about 38 hours i mean it takes just 38 hours to rakes get everything back using the history i mean this is basically where you don't keep updating the current state you don't actually do any rights you just basically run through the history and you're re-executing everything and i need this for because i want to add some extra indices like for code traces and stuff like that and i decided instead of just rerunning this

132. 1:11:39

     whole whole state calculation i'm just going to try to do this historical thing and that's also quite useful for other analysis that people do it's very useful to to be able to run this through in about couple of days you just get through all the historical stuff um yeah that sounds sounds good i think most of the stuff we understand well okay i would probably want to ask michelle michael if they have

133. 1:12:12

     um any follow-up questions on top of what we do that might be unique to us um so one thing we do is like garbage collection and that seems like might help especially with indices so we'll just clean it oh um actually so the garbage collection is going to be trivial in this case because what happens with the rollback sets for example because rollback sets are keyed by the so the keys are the block numbers so and the the beauty of the robux sets

134. 1:12:49

     is that if you remove the older robux sets it does not affect the functionality of the newer ones because they all kind of they never look um back they always look forward so if you if you uh basically think of history uh if you think about history as represented as the miracle trees right in the miracle trees the historical references go backwards in history like they you know they go back back back and

135. 1:13:19

     that's why the garbage collection becomes quite tricky because as you try to delete something old you have to first figure make sure that this old thing is not referenced some from something which is which is still current so you have to look uh before that however if all your references go always forward so this is what this the property of the of the rollback set is that all the references going forward never

136. 1:13:45

     back that's why when you chop something up it never invalidates anything in the more recent history so that's why the rollback because basically becomes deleting a bunch of records from a database and for the uh history indices it's also quite uh uh quite tricky whoa you still have to go through every record but basically you truncate at most your you essentially delete some records and then you truncate one of the records you just throw a bunch of numbers out of it or if it's a bitmap

137. 1:14:17

     you just basically truncate the bitmap so in all these cases i think we had we used to have the pruning implementation but we'll just throw it away to rewrite it again it's pretty trivial and so that's one of the reasons why we have rollback sets rather than like forward div sets because it makes the pruning trivial or because basically we don't need to run any garbage collection

138. 1:14:42

     at all right so michael wanted to say something yeah one thing i was interested in that we haven't touched on yet but you mentioned in your talk was the concurrency changes that you had made where you were switching from this you had parallel streams where each stream was you know entirely processing the block to more like stages where in parallel you were doing all the signature verification and all the transaction execution you know et cetera like ten times faster and i was

139. 1:15:14

     wondering yeah uh you know more details about why specifically that kind of a change made made things faster and like what uh what what the parallelism underneath looks like okay so there's um there's two reasons why it made things faster uh kind of first reason is that uh when we if we manage to to pro basically like uh first of all if you zoom out of this whole what does the ethereum client actually do right

140. 1:15:47

     and what it does really is that it receives some new data from the network right that ingests new blocks or something or and then it basically goes through the series of data transformation it transforms this block i mean transforms the the current state so it's all kind of data transformations until you get to let's say to the data in the stores history dating in the state and then you figure out the state route and so forth so it's basically all data transformation and

141. 1:16:18

     so the first realization that uh uh the first reason why the stage sink is faster is because if you now take this on mass it's basically a large scale if you have to do a lot of transformations um then you can um and all these transformations end up taking stuff from the database doing something with it and putting it back into the database so when you're actually putting stuff back into the database it matters quite a lot in which order you're putting it in so if your if your database is empty if

142. 1:16:50

     your table is completely pristine empty as it what happened in initial sink if you pre-sort every all the data with the sorted level of keys then inserting in in this way is about 10 times faster than if you're trying to insert them in some kind of random order essentially instead of using database as your sorting algorithm which is basically like ins and it would be some kind of

143. 1:17:14

     insertion sort right uh basically on the disk which is actually a bad idea instead if you're if you did the if you said what we do is that we do quick sort of the memory chunks then put them into the files and do merge sort out of the files and into the database and then database doesn't have to do any sorting it just puts the data in the disk on the disk so essentially we're utilizing

144. 1:17:39

     much more efficient sorting algorithm to put the data in and on that large scale it matters quite a lot i mean you would not think of this but it doesn't matter the sorting algorithm matters um and that's the first reason and the second reason is slightly and more unexpected is that when you uh separate this work into these kind of much more homogeneous stages and you start looking at them first of all your profiling before it

145. 1:18:11

     becomes much clearer you start seeing things that you haven't seen before you start looking you basically all your bottlenecks become clearer and then are you literally running like is there literally a piece of the code that says you know we are currently in stage one and we've got right now worker threats there is a there is a directory in the code which is called east stage sync and then it lists all the stages by the

146. 1:18:36

     names that stage heater stage this i mean we don't call them by the by the by the numbers because we sometimes insert new ones but you can see each stage essentially has two functions one function is move the stage forward another function move the stage backward when we do the unwinds and and at each point it gives the you it is receives the handle to the database transaction which is already pre-opened so it needs to do its job take the data from what it needs to

147. 1:19:07

     put the data where it needs to put and then job done and then the same happens on the other direction and at some point the the overall kind of driver thing will decide when to commit that transaction because for example one thing we do is that we're we don't commit every every stage we just commit in cycles like when you're so to make sure that if you're running like a block after block your your database is always consistent up to the up to the like latest block you never

148. 1:19:38

     see the state of database in the middle of processing one block you always see it at the points where you know the block is completely processed and committed and you know if you're external reader that's what you see you see the very consistent snapshots and that's actually quite important for um you know for our pc uh requests and stuff like this so you can simplify things quite a lot if you have this assumption

149. 1:20:02

     um but yeah the second reason is that as you can imagine the code organization becomes different you know you have these much clearer chunks of code which does do one thing and do them well and you know you can think about okay can i do so you you say sort of like you unleash the the creativity a bit more of people who working with this code because they can now uh it's sort of the the problem space is much smaller you can simplify things and you can profile them much better you can

150. 1:20:32

     like oh yeah like this is the bottleneck and we had these moments where lots and lots of uh optimizations actually happen after we have separated stuff out because it becomes basically obvious okay this is why it's slow or this is why it's still slow right yeah yeah okay that that makes sense and then that actually leads us into the the last thing we talked about was the architecture right you wanted to do some pretty heavy architecture changing oh yeah components

151. 1:21:05

     and like most of them on that slide it hadn't started yet but you do you want to just like talk a little bit well we actually if you if you watch the one that i did last friday the you know 23rd this is where i went more into detail about the architecture components but i'm going to shortly explain you what i meant is that it started with rpc daemon when we split it up and we decided to ship turbo get as already two components in

152. 1:21:31

     the beginning so rpc demon was already separated and it's a moment it's running through some grpc connection but actually what we will do quite soon and i mean it's already working but we don't advertise it yet if you still want the sort of the old way of running everything on the same machine through the same database you can still do that using shared memory it works but we as i say we don't advertise it

153. 1:21:56

     yet because we need to fix some other uh we need to make sure that migrations happen in exclusive mode uh so and so forth um so there's some technical details but now um we have an appetite for more component separations because actually i think this is important and i think it would be crucial for this whole project to grow and become stronger because i think the the sort of the death of other projects or maybe all certain like uh

154. 1:22:28

     not not a death but a considerable difficulties of other projects were that over time you know the the the development team cannot grow very fast and uh so over time the the team becomes overburdened by all the feature requests and all the stuff because it's been pulled into all sorts of different directions and uh the core team which still understands the code cannot basically satisfy this growing demand and like oh we need this we need that and

155. 1:22:57

     we need to fix that and we need to fix this so i'm expecting that by splitting things out and actually making other people code owners let's say one of the good examples would be transaction pool we want to split it out and make other people code owners so that they can run their own releases they can so we will have interface we will agree on interface we will run the the core part and they will be running

156. 1:23:21

     uh you know they will be developing transaction pool and it will be compatible and at some point i think when we fill the functionality gaps i would like the rpc demon to also be a separate project run by different people uh and then right so so the idea here is not about scaling you know the the software itself it's about scaling the organization around the software because exactly exactly because what you also

157. 1:23:45

     it's sort of it's it's only natural that this should happen if we want uh the system to grow like because okay of course it becomes more complex for the users to say okay now it's got six components it consists of like what do i do like i need to run a century transaction pool uh demon this demon dad like uh here like six different things how do they talk to each other yeah that

158. 1:24:08

     becomes more complex but like you then there will be another tier of people who will be whose job will be to package this all together like that node for example they would they already made the package of trooper yet but i expect them in the future to make packages of selected bits that people want like do you want the component with these things or like the the packaging gives you more more freedom to customize your note to suit whatever your needs it's like it's like a

159. 1:24:39

     different linux distributions and stuff like this so there are people who are specializing on providing certain distribution with certain flavors and and i think that's kind of a sign of the healthy growth of the system where you have different tiers of people doing different things like not necessarily like core developers doing everything um yeah yeah for sure okay cool yeah i think at near actually we do have a pretty good uh separation of components into different modules and

160. 1:25:13

     in the code anyways it all ultimately gets compiled up into the same monolithic binary at the end but like well this is actually if you watch this presentation i started with the kind of discussion about code ownership and why it is important for the code to be outside of your binary even i think it's quite i mean for some people it might be just a semantic difference like semantics but not it's yeah it feels like an implementation detail right like whether or not it's running in the same

161. 1:25:47

     process yeah but it's not implementation detail because it it's much more so basically i was explaining why it's very hard to like um get the changes in in some of the implementations and pretty much all of them is because people feel ownership they don't want to just you know they need to personally verify a lot of things like most of the things to make sure that if something bad is you know it's not going to affect the quality of the product because they're going to be responsible

162. 1:26:18

     for the result in in any case not the people that you know you can't blame the people that you accepted contributions from for all these things yeah of course i mean you i mean you can but it doesn't you know it doesn't look it doesn't work very well if you do that no it does not no it does not so you're essentially but the the proper com componentization with the proper interfaces very well

163. 1:26:42

     defined documented uh sort of stable interfaces they only these things give you the the ability to separate the code ownership not some kind of clutch inside the code base that you're called a plugable something like sure okay yeah and i guess the other advantage right is when you have separate binaries you can have entirely distinct release cycles right like if you're all compiling up to the same binary then you have you're all on the same release schedule ultimately at the end of the day you cut one release you know who knows what features got in

164. 1:27:13

     there exactly and then you're you have to be much more disciplined about the versioning of the interfaces you have to be you can write in different languages of course you know if you if your interfaces are and that's actually already happening as i pointed out my presentation we already have two implementations of i mean they're not production ready but we have two implementations of p2p sentries one in rust one in go uh we have two implementations of the

165. 1:27:42

     the core execution component one in go and one and c plus plus uh i mean the this whole thing is we we we're actually probably going to have uh two or three implementation of transaction pool one in go one in rust and one in uh whatever in python so i think that's uh this is the future right cool well thank you so much uh i think out of time yeah right no it was a pleasure so i um i hope that

166. 1:28:15

     it was um useful and um yeah yeah thank you so much i appreciate it all right thanks guys bye