[Protocol technical on-boarding series] Storage (February 16th, 2023)

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   oh come on what should they do you go slide show yeah can you see it yes yeah so hi guy um yeah hi everyone my name is Alex and uh I will lead this uh on boarding session of our near core storage I'm the core engineer in our protocol department and uh let me tell something about my prior experience so the key points is that like I found and tried to beat like undercharging for function call fees it's like some problem which we have like longstanding all the time the thing

2. 0:47

   is that like our Fe doesn't actually reflect the computation time for ex executing our functions then I was working on the Chan cach for storage node to reduce the costs for well charging for trodes another interesting experience was to work with our again like for the cash of the nodes for coin contract to make like all the data items fit into memory and my current work is the flat

3. 1:25

   storage it's like a big big project I may or may not talk about it now if I have time uh the major goal of my presentation will be to go from some high level view of uh like all accounts contracts like everything we know about near core like from this Explorer view like you can see that there is some account as near it has some near tokens it uses some storage but uh I assume that like you

4. 1:59

   the viewer doesn't know anything about like what exactly is stored on disk how exactly all this data ends up on our storage so from this picture we will come into the actual codes like this is one of the samples of the code we use for storage so after the presentation uh I hope you will be able to understand what is this like flow from a single transaction to the actual that the base operations and uh yeah so like I divided

5. 2:35

   this presentation into two parts the first one is uh more like a description of storage model we have like how we use it how we interact with it and another part will be like a big deep dive into the codes and we will actually look into all the important points of our code like you see what is happening I try to EMP iiz the most important things there and uh so like all of this will be done

6. 3:04

   to get this understanding to um to achieve this goal so let's start from this uh like overall the storage like what do we have so we use oxb to store all the data like all the data which is persisted into discs is St in O DB so this is like implemented Now by Facebook but started as level DB in Google and is actually on C but R okay so we use some like Port just called R and it was actually updated recently which is quite nice and um you might ask the question

7. 3:49

   why H DB and it seems that we don't have a really good answer to that like the like best answer which I really liked was given by Bowen for thought that it was b as and dark dependency from pity crates which means that like we fored some like open ethereum crates I guess and they already were using rdb and it worked well for them and it now it also works well for us so we just stick withb this is some like Fast key value store which is enough for our

8. 4:22

   needs we also like made several like small attempts to migrate to different storages but we didn't get like any amazing results so we have what we have you can look at this uh like overview like we they have really nice documentation for that like I will just say that it is some set of columns but this columns is not like a traditional column like in this like SQL databases and whatnot like instead like our column

9. 4:52

   is a key value mapping and like you can see all these mappings like in like DB call structure in our code and it is also nicely documented thanks to Marin or maybe like other guys who participated in that and uh I mean it really STS everything like there can be all kinds of data you can imagine about the blockchain like block headers transactions State part State updates I me anything and uh like like the consensus logic really uses be in like

10. 5:25

    different interesting ways and uh but I mean I already say I will already say that uh I will mostly focus on the state like the state colum will really be the main contributor of main customer of the O DB I also tell what happens behind the scenes so like when you create rdb when you write some values to it so like what actually happens on the operating system like it stes uh all the updat dates in

11. 5:59

    the right Ahad log W and once it gets big enough then these contents of w are merged into utable SST files so if you go to the storage folder which is like near slata you will see lots of SST files with really different sizes the model how it actually works is that like it's like that's why it's a one of its names is level DB like uh all the data starts on the like first level level zero and once like each SS

12. 6:32

    file gets big enough then like they got merged and go and then they go to levels below like to L1 L2 and so on number of levels is configurable and the really nice uh property of that structure is that on each level each file responds to some contigous range of keys and all the ranges don't intersect which means that if you want to find a value for some Q on some level then you just need to search uh only one file which is great if you stop the Noe you can also

13. 7:13

    analyze the structure using tools like ldb also maintained by like hdb guys and we also use like per and Flame graph for analyzing performance if you like R perf against like some storage cave operations and then like you look at the graph you can like often see some RB related methods like they can do something with RB block cach or like in index blocks or something like that I mean you can have some interesting

14. 7:41

    experience related to that okay um you have a question Yun hi um I I think I would like to know a little more about why we are doing this leveling and the how it really works because it seems like it is not same as like cash levels or you know like it looks like or it doesn't look like each layer provide different types of performance or does it not sure I follow could you sure like what why why what's the benefit of

15. 8:23

    leveling and how how does it change the way we are interacting with the database uh this leveling is happening inside Rock DB which this is how rocks DB DB is storing the data so we're not doing this that's not our code oh okay okay um so one more thing to that is that like these files are immutable so like if you want to add something to the file then you need to rewrite the whole thing so that's why like this is like postponed and put in two layers of aggregation

16. 9:04

    basically I see thank you thank you I guess to add a bit more the reason why it's why it's batched is to make SSD uh this work well on SSD this drives that don't like overwriting uh the same chunk of data many times that you try to minimize amount of Rights you issue towards that the disc oh yeah and also to answer the question of Andre yes like for to get the key yeah we need to do exactly one Lo up for each

17. 9:44

    level I see another question for Yun or oh it's not the case okay I don't see any R strengths um thank you guys for uh qualifying all that stuff and uh so yeah another thing is uh everything in RB is configurable you can also have fun looking into different parameters we set and we actually also invested some time into like try and to tune or be better but it turned out that our current uh figuration is quite nice

18. 10:23

    but I mean there are many things you can configure like the five size multiplier for each level for example like and in the previous slide I showed the structure of all levels that like they grow like let's say from like one megabyte like 10 100 like one gigabyte and so on and like you can configure this like exponent like you can also configure cach sizes and we will like look into

19. 10:48

    caches into the second part of the presentation and yeah like lots of other stuff I will not go into details like that just want to kind of give some some understanding that we we do we can configure lots of things and uh also like the way how we actually interact DB is this like custom DB operation it's like basically like what we can do like we can set values insert them like delete I mean just simple like all you can imagine like uh we also like

20. 11:22

    can sometimes said new operations like we recently added G range operation because works well with range then you can kind of more or less easily remove not a single entry but a range of entries and another interesting operation here is update rep count and it's actually a great example of merch operator so in RB I mean like by default if you set some value then the new value i writes the old value but actually you can also Implement custom merge

21. 11:54

    operators which mean that you can not just replace like like value a with value B you can also treat like this set as a well binary operation with A and B and this can be any operation so you can say you can say that if you set a to B it can end up into like um adding this values if you set like custom operator like what you do you need to just Define some function for Like A and B and if

22. 12:25

    you define it as like plus then it will sum this values if you add like a multiplication it will multiply them and updat account actually tracks the reference count for each key value pair and uh we need this for State this is needed because we like store our state as a big three which can have by Design lots of the same nodes like some not industry can appear like 10 times and we don't want to duplicate this data

23. 12:56

    instead we store a reference counts for this data and like it can happen that we already have in our database like reference count equal to four and you we want to increase it by two so we like add this DB operation of update GI count like please update GI count by two and then it will like behind the scenes INB it will add two to four so we will end up with six if that makes sense I will go to the

24. 13:27

    next slide yeah and yeah actually I mean it's almost the end of the model we use so as I told the main customer of store is the DB call State so State stores like all the data you can imagine in our uh blockchain mean accounts access Keys contract CS contract data everything the key is a calized try key and also like St structure will be very like relevant for you if you want to

25. 13:59

    debu something to related to State data and uh like value is realized value specific for each try type also in this state we store the same trodes but also not uh think about this right now because it can get too complicated Let's uh talk about that later and uh yeah also there is a very nice uh picture made by Jacob describing the whole flow of um a storage operation from a wasm contract to like actual OS and xdb from down to up here and I mean

26. 14:38

    I don't want to uh talk about each uh item specifically here because it will be too messy I mean this is like exactly the point where I want to like jump into the code and like show exactly like each of these structures like why do we actually need them how they implemented so that we will have a general understanding of how the storage works but overall this picture and some other docs can be found on the link

27. 15:11

    which I mentioned here yeah at this point uh I stop the presentation itself and uh we will like go over like each of this structure from up to down respect but um if you have any questions at this point feel free to raise your hands can you see the cine yes okay um so let's start from the

28. 16:19

    transaction execution transction execution happens uh in the r time apply function like it's quite big but uh let's start from the like uh this part so as you can see apply receives some set of receipts and transactions and that's what happens in each block and chunk like chunk responds to like a single shart and uh what like should happen here is that like some like receipts and

29. 16:53

    transactions should be taken and uh executed and uh like what generally I would expect there is that I mean it kind of my initial understanding is that it shouldn't be like that complex we like we need to just execute each receip just for example if you have a transfer right we just need to Le some account data then modify it like modify the balance and then write it back to storage but it's actually not the case Let's uh for example look at

30. 17:31

    some execution of some receip code so to do that we need to jump into POS receip function then we need to go into apply action receipt because I mean each like transfer connection we see then we go into actual action because receip can have many actions and transfer is one of these actions and then you finally get into a list of all possible like actions like you can create account deploy contract execute

31. 18:20

    function call and so on but we this transfer and uh yeah so here you can see that like we actually like already have some account and we just set it some out like to like deposits like makes perfect sense but the important part is that we don't actually write anything database at this point like the account is just uh like in memory structure which we use for convenience to like maintain the current

32. 18:51

    balance of account because it can be changed in many actions but set amount doesn't do anything like we just if it and the account itself is read in some different ways yeah it happens here in get account function and uh how we read like each account like it this logic is happens in like yeah it's a bit hidden behind the trade trixis but uh it's actually implemented by the structure called Ti update so I mean that's how we like get into the first

33. 19:31

    part for some reason it's can find it easily some reason my code navigation doesn't work good enough tdate new yeah in general triy update is the structure which uh tracks all the current updates made for the current chunk and the stores uh all the values which we

34. 20:35

    modified yeah it's really confusing the comp oh interesting yeah I found it so Ty update is uh yeah I tried to search but I couldn't find it I mean because I I don't don't know maybe I misclicked something and it has a the blue symbol before yeah anyway uh so yeah TR update is a structure which we talk to we need to get some value like get exess key or get account so for each uh kind of f value you can have we have a separate

35. 21:31

    function but what all they call is like the generic get which just delets a key which is a try key makes perfect sense and then it calls try get and then when it gets the value it actually deiz the B Vector to a structure which we want to get so either it can be a account or say ke or anything else for example like the account start here like account has some balance some of it some of them is

36. 22:02

    locked it uses some storage which is also track in account has of the codes account version so all that is stored in storage so okay we understand uh how we get a single value from an account um get a single value for a key now let's understand how the new value is actually written to database because as I told it doesn't happen during receipt execution and I mean what is more it

37. 22:37

    doesn't uh happen in an apply either like here you can say that you can see that like yes um we take some V transactions then we do some like Au operations then we process transactions then we process receipts like do some other stuff like balance checks and uh you actually will not find any like operations which write something to storage we just return some function called return some structure called apply result and uh the answer why is that

38. 23:17

    actually we don't trust to the person who send us a block SL Chun I mean how we like actually received all the rece transactions and receipts is that we just got some block for some guy and we de realized it we got some like set of transactions we started to execute them but actually each single transaction can fail the verification process I mean anything could be wrong and uh if uh a single like receipt or

39. 23:49

    transaction was failed then and we need to do something with it and the general approach we use the way is that we need to actually drop all the changes all the intermediate changes which we made so like you see that like if everything uh was right we return okay but uh if something like failed verification we return actually an error and uh so that's why kind of we don't commit any database right now

40. 24:22

    instead we keep all the changes in the tri data structure and this is called State update but I mean anyway and uh only in the very end we kind of prepare a package of changes which have to be committed later and this package is prepared by the finalized function which uh generally iterates like over all the updates and like create this like try changes and also State changes structure and these TR changes we later used to be committed into

41. 25:02

    database okay y uh when things are in a TR change and um let's say uh we find out that uh something something needs to be reverted where something cannot something is not right and then do we have to throw away the entire Tri changes or can we actually cherry pick specific part of that Tri change in revert only that part I think we need to Define mostly like what exactly is not right but I think like the answer is no that we

42. 25:39

    don't d drop a part of try changes we are not able to do that okay do you mean do we mean that like uh about possibility to like if some if some reip isn't right just like drop this receipt and continue the execution so I I guess maybe maybe my question is like um like you you mentioned that uh the possibility of um like we cannot trust the uh the this receipt right um as in like so and when

43. 26:12

    in this try changes do we um store multiple receipts uh execution of multiple receipts at the same time or just for yes absolutely okay then now it then now that also means it is possible that like only one of those receipts are malicious and need to revert it right I mean yeah that's exactly why like if one of receipts isn't correct we like yeah re everything right then then does it mean that we can only revert that receipt from the tree changes or should we throw

44. 26:51

    away the entire tree entire things entire thing recorded in the tree changes as a whole I mean I mean yeah we re everything I mean in particular we don't even um return anything we just return some runtime error like there's some like validation error B much or something else we just it on empty package you can see like that okay thank um I have a question about the finalize function I'm curious if it has

45. 27:30

    anything to do with the block being final or is it just just database okay yeah yeah there will be like many concepts with similar naming and this is indeed confusion but this finalize finalizes okay so this apply and finalize all happen within processing of a single block yeah I also see the Andre's question in the chat I will repeat it for the record so would it be fair to say that each receip application is atomic either fully applied with all actions or

46. 28:09

    failed and I don't actually remember that I think like it can be the case that we can apply on set of actions but some of action can fail because account doesn't have enough balance and that's okay and I I think I mean the best uh idea is to look into the code is take Does this answer make sense or I need to elaborate more yeah makes sense my impression was that the receipts uh should be atomically executed but I'll check the

47. 28:53

    code yeah but I mean like um I can't fully agree because like at least some actions can fail because we don't have enough balance and we can't really control that yeah so I would guess that the other actions within the same receipt would be rolled back that's that's kind of I guess the question are they rolled back or not yeah that's fair but I in this scope I will not confir what deny that yeah I think there is two different types of issues or errors that

48. 29:25

    can happen right one is the like consensus or us receiving invalid receipts and this is one type of error where we need to revert everything and then another one is receipt fail like a correct receipt failing due to like us running out of gas or not having enough balance I suppose and I would call that a valid execution of a receipt just that ended up with a user error if that makes sense

49. 29:59

    and in that case we can just you know uh it's not an error we still apply all of the receipts in that block as long as all of them are val yeah that's a great comment and uh but I I I actually refuse from like commenting there because I kind of this fall out of scope of the presentation and I didn't really look into that code um okay anyway um we stopped the point when we so now we know that if we

50. 30:30

    need to read some data we talk to TRD dat and if we want to like write something we need to prepare some like package which is called try update oh sorry try changes okay then now then let's uh talk about like what actually happens in Tri update so and as you can see also kind of non-trivial so for example when you look at this get function or let's look at just get function for Simplicity so

51. 31:08

    you also could expect that well like let's just go to database and read something but no we also have like something called perspective and committed and this is like not really convened which is not great but anyway I will explain that um and hope that we will add later so what actually happens there so the thing is that um we like assume the case When We Touch some account multiple times like you like uh some money was sent to account T

52. 31:45

    near like twice in a single chunk and uh when you do this at the first time then yes like you need to go to database but if you to twice if you already do it then instead you would rather keep some like intermediate value in memory and use that value so that's exactly what we do here so perspective is the set of changes which were meant to the state in the scope of a single receip so at first in this G function we

53. 32:25

    talk to this Prospect field and if this contains a value we just return it and uh you can see like the kind of the opposite picture when we talk to the set so we not here let's uh come back to the action transfer yeah um yeah we know actually set amount is not a great example here yeah let's better look at setic key because okay it's more straightforward so when we call setx Key we call this again it's specific for each value

54. 33:18

    because they are cized differently and when we call this set we kind of convert to a bite vector and then we call the state update function set which puts this new value into perspective so that's how we work in scope of a single e like that's exactly what this kind of this this like Atomic thing like when in some scope we make some change the perspective and then we have a chance to read intermediately change value from

55. 33:49

    it but that's not the whole story then like uh when some is actually executed we we need to somehow say that this is like complete set of changes so we actually commit these changes and that's why we have like this committed field so committed stores all the State updates which were made by already successfully executed receipts and transactions and uh how that it works like At first we talk to perspective to see what happened in this particular

56. 34:24

    receipt and then we talk to committee we see what happened in previous receipts and to put a change from perspective to committed we use this commit function let me try to find the best example here we execute some actions validate receipt yeah so yeah as if results of all the execution is okay then we commit all these changes and say that like yes we sucess process the receipt and like

57. 35:14

    that's what causes the perspective changes to be made uh to be moved to committed and otherwise we roll back the changes which simply means that we clear all the perspective changes any questions so roll back is only for a single receipt then not for the whole chunk yeah yeah for a single receipt exactly okay like if the whole chunk fails we just dop everything I mean it's made on Tri update scope like Tri update stores all the updates for given chunk

58. 35:52

    if chunk fails then we just discard everything okay so now we understand how try update works and uh then it's it's finally time to understand what happens I mean actually one level um one level below what happens if our value is not stored neither on committed changes or prospective changes so if this is the case we actually go to our like underlying TR St uh try storage like yes why exactly I

59. 36:45

    mean it's called try because like try is a data structure like it's a prefix three over all the keys from storage and uh yeah so if we didn't find any values in try update we go to try which I call like that probably should with state but anyway and uh then we go to TI get and uh I will not like mention this but uh this calls getf like before reading a value we actually need to read a

60. 37:20

    reference value reference which is a hash and then we read a whole value by its reference and uh this get ref is also kind of not trivial like the important thing it does is this look look up function and U I will tell a bit more like what exactly is this so in the state we don't really store a hashmap like the try is a prefix three which means that we need to get into the value by by like starting from the root noes of the tree and going to some Leaf

61. 38:01

    no using uh single like bites or like nibles of this key one by one and like this is what happens here like you just like read some notes pass this notes then like understand what is the child note and you go with it and in the very end you find the leaf and you return the value reference then difference it and you the actual value we again we will not talk about like what happens in Lo up but the

62. 38:33

    general idea is that this up should return an actual value like so for a user like for you if you want to talk to try if you want to talk to current state just like call this function and it will like get your value you mean you shouldn't care much that like it does any pups so that's F yep just reiterate Whenever there is a safic REV in a type does it mean that it's really a hash

63. 39:07

    of whatever it is yeah um that's I mean that's a good question let's talk a bit more about it so do it to several reasons we can't just return a value right now like instead we like almost everywhere return like a pair of value hash in storage and its length and this actually happens because of like our cost model like for example someone could request a 4 megabyte value and then like attach very little amount like one like y to near or something so like they can request this

64. 39:48

    but they don't have enough money to pay for it but uh like our storage can't really and the on that level we don't really know how big is the value so like At first we have to make this get and then we read it and then we say to the user oh like you read four megabytes so please pay for it but they say oh but I don't have enough money so like I don't care and that's what we want to avoid because I mean we want to make this like charg and

65. 40:17

    fair that's why what we do is like the fast we get this length and then like we say okay like you need to read this like 4 megabyte value and then we charge the costs and then we finally get this value by its has does it make sense so this is like this Val is kind of generic think which we use everywhere at this moment so like all the GFS return the value and all the gets are like wrappers around GF which then it'ss

66. 40:50

    like a value by the soes is actually like this kind of the and uh at this point we can jump again one level below from try try storage so like this try storage is like a general tra which can be implemented by many structures again let's not talk about that the most important function is retrieve bites the most important implementation is for TR storage like TR storage like with is this it's again it uh can read

67. 41:30

    from database but uh we also need to apply a lots of question here because uh like we have lots of keys which read which have been read very frequently so it's better to keep them in memory than on the database level and in this retri byes I mean again like it's kind of generic function I mentioned on the slides that this function kind of it's used not just for reading values but also for reading nodes but I talk about this it would be

68. 42:02

    too complicated so let's just to keep it in mind a generic value not just like a state value per se but uh if you just think about value then this uh like logic um it does the following so at first it looks into different caches so like so-called Chan cache and then shart cache and then there is some Perfection logic which also like some like great thing which we really need but it's another it's

69. 42:33

    another topic and then like when we get some Val we put it on in all the Cates and then we finally return it so let me like At first let me show you the actual database call we finally got into it like it's Le from DB yeah so it's read not from DB and yeah like that's where we actually go into store and call this St get from a state Column Way so this also contains some like reference count in logic which you can

70. 43:05

    look into but again like let's not care much about it so like this kind of actual value is a like reference count plus like the state value itself so we need to like STP this reference count and then we return actual value that's what happens here and uh yeah then successfully get from read from DB and also we apply like different kinds of cing like the most important thing to keep in mind is the trach or shart

71. 43:36

    cach like this is like implemented by trach function and this is I mean basically the lru cache for well most recently used uh nodes and uh we allocate actually a huge like chunk of memory for it it's 500 megabytes I remember correctly and uh this is somewhat crucial part of our on time because if we don't cat this noes then uh our performance deterior deteriorates significantly and this

72. 44:08

    actually want to fix with the food storage project I mean currently our I mean we really heavily depend on this like C of no of specifically try nodes like and this is important I mean like if you come back to this try story that like the TR is the prefix three and we need to read the root nodes all the time I mean when we read like any single value we need to read the same notes all the time that's why we kind of it's much

73. 44:40

    better to keep this in memory then go to dis each time and um so that's mostly it so that's how you like from a transaction let's do this kind of mental roll back like that's actually what happens when you like make some money transfer and uh change some account balance and like you go this whole way just to read initial balance we from DB and um I yeah I don't have much time so let me really quickly tell how it's

74. 45:23

    written database just to set some pointer because I have only two minutes so uh so I mean we remember that uh this like package of updates is called try changes so you need to search for try changes where they exactly where where they exactly are added to the database and like so I believe mentioned that like the store talks to the database itself and again like uh similarly try

75. 45:57

    update we have store update which contains all the current intermediate database operations which we will commit later so GB transaction is simply a vector of DB Ops yeah we finally got into it and uh if we want to commit something database we simply prepare this transaction and commit this transaction and um what actually happens like onas level is that we kind of get this like at first

76. 46:33

    when we apply some Chunk we get try changes then we gather all the try changes for all the chunks and then kind of we like convert all this updates into like a big database transaction and can this transaction at once okay this is still update yeah oh yeah okay um now now I remember so this whole logic happens in So-Cal chain store update so like kind of the TR update was a small structure which

77. 47:35

    prepares only run time updates and Chin St updates like updates all the data relevant to a block or a chunk and it's really a lot of changes like you can for example look at yeah it's G logic not really important okay yeah so like like the whole DB transaction is prepared in this like finalized function I again like briefly touch it so it updates like all the needed chain data so like a block

78. 48:10

    process can change the chain head we update the head like it can the head head the header head then like it's updates like our mapping from like blockes to height we also update that and then like we go all the way down and finally we have the TR changes like they were like return from runtime some time ago and then like this changes have some like insertions some note which we convert to which we need to add to State

79. 48:41

    and then like they're finally applied I really like jump this and again so this actually like what happens inside so like when we add some trod we incremented it count and like append the separation to a set of transaction database Ops so like this is like where you actually finally write something database and uh once this all is finished you actually get like this kind of package of updates like this update contains the set of

80. 49:20

    transactions and uh in this function then then like you commit it finally and by commit kind of U all operations we have some nice tracings in that like yeah hey like you set this value you update this value and then you call this right and like this is like right some data in batches but this is again like not that important for us yeah so and that's mostly all I wanted to say

81. 49:53

    about our storage to uh have like some general understanding what happens in our storage yeah you don't have a question uh yeah so if I go to let's say if I let's say I initiate some transaction let's say I'm like I send some near to you Alex right and if I go to near Explorer to see like what the current status of my transaction is right then where do we read that data from like of oh your transaction is completed or it is not for

82. 50:32

    example um if if I remember correctly this is like a separate indexer project so I mean it's kind of complex question depending on like what exactly data you want to read but again I think the indexer has its own like postgress database or maybe they migrated to separate solution and indexer is essentially a archival near node which just like like in parallel with regular like block Chun

83. 51:06

    processing it does all that but it also like tracks all the changes which we ever made it also like stores all the blocks all the chunks and provides some content for it right so I guess I guess that that that pretty pretty much implies that a user like outside external users doesn't have any access to intermediary state of the transition until it is actually finalized and uh recorded into Rocks TV right because all this indexing will happen afterward I

84. 51:38

    assume that's correct okay this and technically it depends on the actual indexer code maybe they they can add some like hacks checks like intermediately I mean no no one restricts them but I think that they like do this like actual indexing when everything is indeed committed to database toxb yeah but there's also column called transactions in our database do we also store transactions there um I think we do but I don't really remember the purpose of it I

85. 52:19

    think that can be like a transaction pool yeah it doesn't say it just says it's signed transaction and um meanwhile the question how big is uh our current RB state for FPC nodes and other any dashboards awesome question so I don't remember exact values but RPC node is something around 500 gigs and uh archal note is several terabytes like three terabytes like can be changed if we will be able to compress something

86. 52:56

    thing and we have some stats for that if again if I remember correctly the main like the biggest column for FPC node is block headers because we store block headers for all the history but stat only for last five epex and for not it's actually the state because we St like all intermediate States from the Genesis and for grafana we do have uh lots of nice metrics um I will uh add links to the presentation later in like put when it

87. 53:30

    will to confence any other questions okay so to conclude that I hope after that you will be able kind to like understand what actually happens on like our like database what exactly each of nodes does to disk when someone submits a transaction and it's process later and also like if you have some awesome idea about storage like some projects you want like to optimize something then kind of I hope you like understand some like key points which could be like interesting to you and like which kind of interesting

88. 54:16

    consequences your idea may have and um that's it I hope you enjoyed it thank you Alex thank you thank you thanks lot