Whiteboard Series with NEAR | Ep: 45 Joel Thorstensson from ceramic.network

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:04

   welcome back to whiteboard series it's been a unfortunate break due to corona but we're back now that things are getting back to normal and i have joel here from ceramic to really dive in into how ceramic works so joel please introduce yourself and let's dive in cool thank you uh so my name is joel i'm co-founder and cto at threebuckslabs and we're really kind of the creators of ceramic network

2. 0:28

   and yeah i guess let's dive in so maybe before this like which problem ceramic is solving that's a good point yeah so ceramic is a protocol for mutable data objects where you have objects that live under the centralized network and they have like a controller some user that can change and mutate the object over time essentially like get a history of like how the the object will change over time and you get like kind of like an immutable kind of uh log of like what has happened um and

3. 0:59

   these objects are independently verifiable so you don't need to kind of sync the state of an entire blockchain to validate the state of like one object and so that's kind of like very high level of ceramic decentralized object store that people can use but then with kind of auditable log yeah exactly you can rely on uh-huh all right cool all right so we're gonna dive in yeah let's do it right so we have a log is made up of these events sometimes called commits and so we start

4. 1:30

   creating an object by creating a genesis event and this this event might be just a date object that contains okay how to best officialize this we have like a controller and this is basically like the user id so this we might dive into this deeper way of representing a a user in a in a decentralized network and then it might have like just some other metadata um all right so so once we have this we've generated the the stream id based on this this is kind of the unique identifier of of an object in ceramic

5. 2:16

   so now a user can go about creating a an update uh that is a signed commit and that is also um so so all of these objects are stored in ipld and iple is essentially a way to represent hash linked data so this object has a pointer that points to the hash of the genesis object and this commit contains a signature issued by the users and uh that's essentially it then it contains a patch for like how the object is updated so in in

6. 2:57

   ceramic right now we have support for something we'll call a tile document which is essentially like a json object and the patch is just a json patch so you have a json patch and then these sign commits can be you know added over time and they link back and create this kind of like event log and then in order to kind of get some kind of real sense of time into the system because without this you need kind of need to like either trust completely like if there's timestamps if the user

7. 3:27

   puts the timestamp in there so we have another system that basically creates something called anchor commits or anchor events so this is you can call it a and this also points back to the previous one and this essentially contains some meta data information that's like which chain chain id it was anchored to and like a transaction hash and it usually so so this is usually done by some kind of anchor service that can like take a bunch of updates to a bunch of different streams generate a

8. 4:09

   merkle tree and put the root on chain so usually it contains like a um uh merkel witness as well this is merkel witness across a bunch of object yeah so there might imagine like this this kind of stream there are multiple of these like multiple objects that are updated by multiple users so an anchor server is essentially like an actor in the network that takes requests from users um and takes uh this you know the sign commit and

9. 4:43

   batches a bunch of those into a merkle tree and puts the root on chain and then it kind of gives back this anchor commit that contains like this was anchored on this chain id right now it's a theory minute transaction hash and a mark like a path through the to the merkle tree um and yeah and then like you can continue updating the um the stream like this um so this is kind of the data structure

10. 5:14

    and so a ceramic node um basically either you create a stream on that node so um then you have kind of all this data when you make an update to one of the streams uh you you publish an update in elite pdp pub sub network uh so that anyone can kind of that is interested in a stream can like update and sync that stream um and uh yeah that's essentially it like and and then like the second part of this is um we have a query system that allows you

11. 5:50

    to if you're a new node in the network you want to sync any particular stream you can you know you have you have the stream id you want to find what the latest comment is or what the latest update is uh so if to do that you basically um look into um right now like in in the in this we use the ipfs dht and basically put uh say saying a node that pins this stream basically says hey i have this stream id put it in the dhc

12. 6:20

    and then put there there so the ipfs dht basically works like you know you you say you have an object and then you put your ip address or like your multi address into the dht so we can basically find all the peers that pin the stream and so we walk the dhc and to find all the peers that have this stream and then we ask them like hey what's the latest update and you sync the log

13. 6:45

    and then you kind of verify the history of this log from the beginning to to the latest update and then reduce the state based on that i haven't done any questions great um so i guess let's start with with the last thing you said which is let's say i want to find the latest state of an object um and so how frequently are this like i'm assuming pretty much the way for me to determine if this is the latest state or

14. 7:14

    not is to look for anchor commits like if i'm getting like from multiple nodes i'm getting different states yeah so if you get different states there is a conflict resolution yeah and so i think so that basically is based on the for like uh earliest anchor rule so if you have to uh if we have uh a stream here i'm just represented upwards instead so uh we have one commit here and one commit here this way sorry so camera can see that you can see it good it's okay

15. 7:55

    great and then so so let's say there is like one anchor commit here and then the other anchor commit is like later this means that if this you get these two branches uh and there might be like other commits after here um you will resolve to this um this basically this conflict logic will say like this was anchored earliest so this will win um that's the current kind of current conflict resolution we are also like looking into various

16. 8:42

    ways of like merging uh as well uh using crdts but like this is kind of like this a simple way of doing it um and so you can think of an attack on this might be that okay you you basically publish this thing and then uh you don't reveal this right uh and then you later reveal it and then you just kind of change the state and so in a blockchain where you're tracking like financial transactions that's a problem because you can steal money but ceramic is is primarily aimed at

17. 9:15

    like user-centric data so if you do this you kind of do this and then later reveal kind of the main person that's like affected by that because this this stream id is like controlled by you and like it's controlled by your identity you're basically like basically um revoking things you've done and it's probably mostly harmful to you and your reputation as you built up on this this account

18. 9:43

    well it depends how people are using it right because yeah as soon as you do some data stores people will use it for financial transactions but yeah like for example you know somebody's pop wants to publish like stock prices or updates or whatever like that that's can be a data structure to do it yeah it can be like we're pretty like you know trying to like make people like this is not meant for financial

19. 10:07

    transactions it's meant for like primarily like user-centric data yeah um so okay i guess for me to understand like how this can't even happen like is the anchoring service just literally like it doesn't care and just find some thing and so so we so so we run an anchoring service and we make sure that this doesn't happen but someone else can run an accurate service right okay there's no

20. 10:31

    is it like should there be a consensus on ancient services kind of or right that might be interesting to explore like so um when we decided to sign this protocol we had this kind of like identity use case in mind and the problem we're trying to solve with these anchors is like key rotations so if you have um if they imagine the stream represents a an account the id so the id is essentially a way to go from an abstract identifier to a document that contains public keys yeah

21. 11:03

    and if you um if you like have these two branches like someone has rotated a key if we were to go with this branch someone that steals key those like in in the start in the start they could potentially like you know create a new update in the future and so we need to go this kind of earliest anchor rule um because you wrote it like if the earliest rotation of the key needs to be the one that's valid otherwise the the key

22. 11:35

    rotation part wouldn't be secure got it got it so i see and then for for anchor for this information to get the actual timestamp like you need to fetch the transaction from the chain right yeah exactly all right cool and so so you're saying let me let me just kind of draw instead so you have like a leap p2p on the bottom as your uh networking stack usually pops up here this is where

23. 12:13

    you have pretty much you have nodes which store kind of data presumably anybody can publish yeah so updates exactly so like nodes choose which streams that they want to replicate tracking like is there like as a user like i want to store some data how how do i tell which nodes are tracking this or like how do i tell them to track um so right now like developers in their application configure which node that they use so right now it's kind of like on the developer to make sure that you know they're using it that

24. 12:53

    they uh care about for their application is pinned um yeah so that's roughly it is like right now like in theory like a user could spin up a node and replicate their data and uh so inside node like um this like this data started like you have you have this log and you'd probably have just like the latest like stream id to latest state yeah exactly so this this is queryable from

25. 13:30

    from an http api that the node exports and and you said you have like a dht somewhere which is stream id to hash or id address yeah that's stream id to node peer id right right okay so that's also in in the btp okay so presumably a even a web can you know web thing can connect to this query find ap address and then request via http yeah what do you mean by web thing like a web page yeah i mean so

26. 14:07

    yes but like the doing like the like javascript implementation of the pdp is like fairly limited but in theory it's possible yeah i guess like yeah generally i guess the lipid p implementation can expose like a websocket so you could connect to that and the crawling the dht in in web is kind of fairly expensive but it's possible well i mean you just need to this is phd right you can just query the dht with some peers that you have connected to yeah um

27. 14:40

    okay cool so yeah i mean it would be good yeah to probably dive into the id here um and i'll show you how like you mentioned kind of the the the id use case using ceramic on top right that would be interesting as well yeah so we actually like use this stream to represent the dad that's actually also like an author of a stream okay so you're going to okay yeah yeah exactly this is the id actually sgt

28. 15:09

    yeah yeah so it could be like it can be like we support internally we have we have a dnd method called 380 and that's based on on this um but we also support other dad methods they're like based on ethereum and whatnot um okay yeah i mean i guess so in general like the id is simply like uh yeah perfect it's a little bit loose i think do um want to erase okay yes oh we can keep ah whatever yeah okay let's go

29. 15:49

    so so a deity is a very simple you can use that uh you basically have um a an identifier so this is a url that's defined as did colon and then a method and then an identifier and so this is a url that you resolve to a dad document um

30. 16:50

    and so this document contains like verification methods so that might be you know a signature key like public public uh or a key exchange key so it's essentially a way to have like an abstract identifier that resolves to some document that contains the public keys that are in used by this vid at this point in time so an example we have the id method called the the id three and then it's just like i don't know like that's a stream id

31. 17:58

    and so this resolves using a ceramic stream to a dnd document that contains this type of kind of thing um yeah so that's that's essentially it like basically there are also like the id methods that are very simple like the id key and this is basically just how uh just a key public key pub key in here so that supports like um you know sick p uh k1 k1 it supports r1 and it supports adjacent current uh

32. 18:45

    yeah it supports like bls yeah his voice is like most of the useful ones okay there's also a new one that we worked on with spruce id that's called the pkh and then it contains something called a 10 account id and what this is is essentially a way to represent a blockchain account on any blockchain so an example is did p e k h um eip one five five colon one four theorem main net and then zero

33. 19:36

    x one two three a b c and you could do something similar with nier i think uh near we have like p cage near colon main net colon i don't know how a near address is usually different here or illegal.near i mean there's like it's pretty much a stream okay i just put you in the dust but didn't i yeah okay cool yeah i mean you can create any type of the id method right they can so this

34. 20:27

    kind of resolves to like a dd document that basically says like how to verify that yeah essentially all right and so but like do you have like actual code there or how does it no it doesn't have actual code that's kind of like an annoying thing this this you can specify verification methods that are like different things so uh you have to for each kind of like curve in each thing you have to have like a specification somewhere yeah and

35. 20:51

    it needs to be like okay this is how it's verified uh so that's like go officially like it needs to go through the w3c like standard process full stack the id standard but that is jc yeah okay um and so this is actually something we're uh pushing through like or working on with the uh credentials community group in w3c um so like make that into like you know a more official thing uh did key is like i guess one of the main kind of

36. 21:24

    references but i think the dad community in general like a little bit doesn't really like the key because it's like oh but you can't rotate keys but it's like super useful primitive and i think i think the pkh is going to be like similarly useful to like that key yeah i mean it's definitely useful like even if it's not i mean the reason why they have named account is that you can

37. 21:46

    rotate keys well and have many keys etc but yeah yeah um okay no that's interesting and so so right but like right now for for for this to be you need to implement it all on uh yeah on the node pretty much for every single verification method and then like even for pkhs right you will need to implement every single like sub yeah that's right um so it would be nice if there was like an abstract way to just like find a

38. 22:20

    uh you know see a d80 method and like find the reference implementation like fetch that code yeah there are some projects working on that type of like dynamic code loading uh but none are like super solid yet as far as i know yeah i mean i think like this that assembly is probably like the way to do it yeah you have the same modules yeah definitely yeah i mean it's it's easier for these kind of things which is kind of like static yeah um i think i don't

39. 22:45

    know like so these are like public key hashes but i guess basically based on like this is a hash of a public key so you don't need to resolve anything you can just like verify it uh yeah just you need to actually fetch the current set of keys right uh so i don't know if like this pkh would actually work for near accounts that's something that we probably need to explore yeah i think right now we

40. 23:08

    have like this is closer to this actually right because yeah this actually resolves the set of keys that you can yeah yeah and so so there are like standard interfaces for like resolving at the id but yeah we need like uh it would be ideal if we could like dynamically resolve these types of things um but ultimately it kind of becomes like if the node has support for like a lot of

41. 23:32

    dld methods that kind of puts a lot of like external dependencies uh in terms of like trust yeah yeah so i guess i guess it would be interesting to talk about so this did pkh the the reason we are interested in that is we're building an object capability system uh to do authentication into ceramic because right now you kind of like need some kind of system for our did you know if it's a very simple case we have a good key you need to store the private key somewhere yeah and so you

42. 24:04

    could generate a private key forwarded key and started in the browser um and that's fine for like short lived things if you have a did three like we want to keep those keys secure and right now we have a system that involves an iframe and involves like signatures a little bit like uh involved in like using an iframe is obviously like not like the the best security practice the

43. 24:26

    most secure thing um so what we want to do uh and let's maybe clear some space here no i don't think so so we want to use the pta just like a root identity or like as a base to issue capabilities to to create streams in ceramic and so imagine we have like you know wallet up here this is like and it has an address so that's represented as it did pkh um so what what we want to do is like signing

44. 25:12

    things with this thing is could be done but like that requires the walls to pop up like a signature and do that every time uh so if we have if we have the dot right i don't know but no need for this have the app and so the dap can the idea for or for the object capability system is that we have that it can generate a dead key and then it requests a capability from the wallet and that's basically a signature that includes

45. 25:51

    um like um domain uh dead key i realize this might be super small uh let's do this um and maybe some like uh ttl and then and then caveats so so let's say this the app ones get access to stream a and stream b so this is signed and then returned to the top right so

46. 26:57

    they get the signature and now the the dap has this capability this object capability ocap they come from here and if now the dap wants to update stream a you can simply create a commit so let's say we have stream a over here so here's the genesis and this is stream a so now when i create an update it's signed and this is signed by the dead key and in the header of of the signature or like the signat the things that's signed we include uh the ocap

47. 27:58

    so now like when when a ceramic node is verifying the event log you can see like oh this was signed by by this dead key but it's including an object capability and then see that okay this object capability was actually owned signed by this yeah exactly so this is like you know the pkh um and so this this will allow us to like have these session keys that maybe it gets multiple access maybe it can we

48. 28:30

    can have like other types of caveats that like include this is you know you can only write to this particular subsection of the stream or only make this particular update uh so you can have like more granular permissioning inside of like the data that you give the apps access to um and one cool thing about ocaps we're probably not going to have we're not going to have this in the first version

49. 28:55

    but it's that in theory this the key could live in adapt or it could live on someone's back end or it could do whatever uh but the the wallet gives access to um the dead key here this key could potentially delegate it delegate it and say like they could attenuate and say like only access to stream b and not access the stream so you would just like sign with so you'd sign with actually this key yeah it was this key and you add mix

50. 29:30

    sorry and so you add like you know extra caveat on top of this you know stream b right yeah so create like a yeah and so this would be signed by the key yeah yeah i mean it's somewhat similar to our model but yeah our model is like this and then dab or or whatever backhand yeah has a key and and so you sign that you want to add key right key you add for which contract so it's kind of the main and which methods you can call

51. 30:18

    and you have a gas limit much gas allowance as you can you know just send it down to transactions and then so this goes on chain and now you can use this to send transactions with this so so in this in this model you have like this access list of like all of the keys yeah all the accounts that have access yeah whereas in here we don't need to keep the state anywhere this dap is responsible for keeping this all cap around and only like displaying it once once it's used like including it

52. 30:49

    once it's used yeah i think the problem is like if if your wallet displaying who did you give their permissions to yeah like across multiple devices that's right yeah so one thing you could do is actually like store a cap like this ocap could get get like permission to also write like write itself to like a to another object store yeah yeah and you could have like potentially other caveats just like

53. 31:14

    only if this is false so you can have like revocation list for this and that could be like you know in in theory you could make all caps like very generic like general uh i think i think ocaps is actually when it was invented like used on an operating system level where having these kind of axis control list is like really expensive because you you kind of need to check them and keep track of them whereas here you've kind of like only have this object

54. 31:45

    where wherever you need it yeah i mean for sure it's it's cheaper if if i mean you still may have like multiple like your pkh here is itself maybe complex to verify like your did here maybe complex verifiers yeah um but you know and like if if you rotate the key here this overcap have yeah if you rotate the key uh this this o cap would be invalidated unless you know there's an anchor here

55. 32:16

    so this is anchored before this the pkh that or some public key that supports key rotation then you can see like okay that was that particular version of the d80 so it's still valid uh but any if they tried to use this o cap later it wouldn't be valid right so let's say like does is pkh able to pull like pretty much how does this jd pkh or others that pull external information work right because in this case you're saying like if i'm revalidating

56. 32:49

    and at this point i need to verify that this is signed by this you know did pgh but if the speaking h is external you mean if it rotates keys yeah and yeah yeah so the ph is static we can't rotate there but yeah so so we have like g3 d3 yeah uh and then just like yes yeah definitely whatever and then so what we actually do when we sign thing is like we actually include the the id that signed it and it actually points to

57. 33:21

    the specific if you remember in the id document there's a verification method you might have like multiple of the keys so we actually like uh have a query param here that's like version version id and this version id says that this is the particular version of the did document that was used to sign to sign it and that's included in here so um what you like this version id we need to check that it was valid uh

58. 34:02

    it wasn't invalidated before this thing was anchored so when you resolve the id that has a version id you'll also see like you can get yeah when when this update happened and when the next update happened so you can like check that this anchor that came after um actually yeah this update actually happened in that time period so do you anchor every s like as an anchoring service you run do you anchor every change like no yeah because there's like multiple changes yeah no we don't we

59. 34:33

    don't anchor every change we right now because of like you know ethereum maintenance uh expensiveness we occur twice a day um and obviously that needs to increase as it's like the the network grows um but yeah i mean yeah these kind of transactions here are kind of like not find really final until it's been activated um yeah because if you have you know change the queue and then you don't have anchoring for some period of time then like this version id

60. 35:06

    this points to the specific document yeah this points to the specific version oh yeah so uh if this is the the stream for the 380 yeah this points to the version id can only point to like an anchor so if you rotate the keys here yeah yeah so you need to point to this uh actually in the implementation we don't allow you to rotate keys until it's been anchored and then you can go through again

61. 35:32

    but for the for for so so in 380 we kind of anchor every every update uh but in like generally in streams we kind of like make a bunch of updates and then anchor and depending on how much it's used of course yeah i mean i think this is definitely like where complexity is if you don't if you don't check it in right then yeah no it's definitely this thing uh this was in the d80 core specification but the ability to check like when was the last update and when is the

62. 36:03

    next update that was not something that we we had to add that to the d80 core and we barely got it in for like conversion problems like this is a bug in this thing because otherwise it doesn't really work in trustless systems exactly yeah and like trustless systems what's that like a bunch of like web 2 people uh building this decentralized identifier spec well i mean there is a bunch of like uh projects in the in the kind of web 3d ecosystem it's also like

63. 36:28

    contributing contributing yeah for sure uh yeah that was that was like we got that founded back in like the last moment of like being able to update the spec yeah i mean this is where like not having consensus always kind of creates problems right because you don't have the time so you're trying to kind of checkpoint at least some surface form but yeah um okay no this is cool do you want to talk about kind of what are you guys thinking for the future

64. 36:58

    yeah we can do that that's kind of very loosely defined okay um but this works well for when you have individual user identities and like you want to be able to uh yeah because the revocation is mostly like you kind of slash your own data and your own reputation uh we have implemented an experimental feature or experimental did method called the ide nft and obviously that allows you to like i can update it then i send it to you and you can update the stream and so there this kind of data withholding attack that we talked about

65. 37:38

    is problematic because um people can you know if i create this nft i make a bunch of updates and like now you see like oh that's super cool like i want to have it and i send it to you then i can kind of like i mean you can always look at you can always like look at the history of the data and like the things that will change and like remake it to that version yeah but it's like for for the experience of that it's really bad

66. 38:01

    but also like how does this work with this problem like le because you you change your owner right if you give you no cap with an nft and then you you sell the nft but you still have no cap yeah i mean i guess like an ocap would be revoked at the time of transfer and of transfer but anything that was you know issued before and like used before would still be valid because it would be at the time there's like a lot of complications so ideally would make this kind of like

67. 38:32

    general to any blockchain but there's complications like querying the historical state of like that because like now your version id is actually like some hash or block or something yeah so with version id we're actually using version time now and for we have this implemented for uh a few different ethereum chains and uh it's essentially like you take the version time you use the graph to basically look up what the state of that

68. 38:58

    who the owner of that nfc was at that point in time yeah so uh but yeah so for this type of of thing the the streams that we talked about isn't really kind of ideal and so this is one of the reasons we want to introduce like a a blockchain where you have streams represented just as objects on chain you can still have this kind of like event log that happened yeah and kind of verify the history independently that it's actually like signed by

69. 39:30

    the right user so that you know the if there's like an attack by validators or whatever they can't actually like fake which transaction took place but yeah and that you can and we still want to keep this kind of property of like every stream being completely independent um so you so you can kind of like sync only the data pieces that you're interested in um yeah like i don't know how much it makes sense to dive into that right now it's

70. 40:00

    like very early in in kind of research and development yeah yeah that's kind of the what we're going for there so that would be like that ideally it's also solving a different problem right also because like uh one problem with this right now is that you have to run your own ceramic node yeah or you trust the third party to run a node for most users are not going to do that so the the the blockchain could potentially

71. 40:25

    also provide like a service where you pay and actually have these streams maintained um yeah yeah makes sense anything else we should cover now i think that's actually like everything i was thinking about um i guess well could make sense to cover kind of we have a framework like how you use ceramic on top um called self-id and it's it's a way kind of like to allow interoperability of data between applications

72. 41:06

    so we could just go like a brief overview of that journey right so self id um we also sometimes call it idx it's more specifically like the the data structure of this thing um but conceptually you can think of it as a big user table so here we have the ids so we have like the pkh or the 3 or whatever um and then the columns uh so so if you imagine you're building like a you're gonna build like a user table that is shared across the internet now if someone goes here and like okay

73. 42:13

    developer i wanna have a name for my user so application am i store that as like you know a string but application b my story has you know like an object it's like name string and surname this is difficult also string and so a user that comes to application a like creates a name that comes to application b application b is broken like this is probably anything it's problematic so um we can't have this right this is bad uh

74. 43:11

    so we need a way for developers to kind of claim names or define define things so we have something called data models so data model is essentially um they are represented as a stream in ceramic so we have data model one and it is represented as like a stream id and this might say like okay this is this data you can find in here is you know how's the structure of uh not explicitly specific name string uh

75. 44:02

    your favorite emoji uh i guess that's also like a string you know yeah it will string but like two characters and then now like someone creates goes an application and creates an entry here when they're writing to the stream uh that's one thing i didn't mention the protocol we have like an additional feature that allows you to specify a json schema for forestry as a whole stream yeah so genesis

76. 44:32

    yeah at genesis but you can change it but like you cannot like so you can also like not allow it to be changed um but basically you specify a json schema and then the state object needs to conform to that json schema so this schema can be defined in a data model and the data model also contains like other metadata like you know the name some description maybe um potentially like other metadata in the

77. 44:57

    future and so an application developer that sees like okay this data model one here is super useful for my application okay they can just like import this into their their app and then start pulling that from all the users that come to their application and they can be certain that the data that will get out of this data model is of this format but you said this data model itself is a stream right so it can be updated it can

78. 45:22

    be updated yeah so i'm gonna have it nav so yeah i mean so this is like a problem in general in these types of um of systems where you want to be able to have updates over time but if you're building an application you might not want to you know limit yourself to one one schema so there's actually like a lot of research going into like how can we have schemas that we can change over time

79. 45:48

    that allows us to have the data kind of change with the schema so that's kind of like an unsolved problem still probably like a good first approach to this is just like not allow the schema to be changed um so that's like provides you know solid foundation until those types of systems are more available i feel like yeah data models being more permanent right than kind of but really yeah i guess like the data model itself doesn't need to be permanent like you might want to have

80. 46:19

    um you know the the schema be permanent but maybe you want to change like how the data is backed up or like how more metadata there yeah there might be maybe change the description over time or yeah there's like potentially other things you might want to store in this metadata and it's essentially a way to like for developers to find why should i use this data model this is like it seems useful and so so then you can have like a an infinite amount of like columns here essentially

81. 46:52

    and then you have a discovery problem which is like which column should they use right so then the discovery problem uh so that needs some other type of system but you know we can start thinking about that more once we run into that we have more of those yeah that makes sense no this is cool yeah uh so yeah this is kind of just simple the framework and let's saw so a an individual user

82. 47:16

    a store has a stream that stores the uh their row and then there's an individual stream for each cell um yeah so when you say stores a row they have a stream which or like they just have a json which has like uh dm1 to another stream id yeah so the m2 so it's like just a yeah it's it's just it's just a key value store okay and obviously like if you have a json object like that that's gonna if it grows very big it's a problem yeah so what we want to do is introduce a new

83. 47:56

    stream type that's uh used to something called a hand and hash rate map tree and essentially like a merkle tree that has like a n b or k bucket and just use that as like a key value store actually yeah that's kind of the question i had is like how scalable this all is given current model yeah yeah no so in the in this current model like you would run into problems when you know document is

84. 48:20

    large one document is really large and so that's kind of like um one of our our main focus next year is like or introducing these new types of stream types that allows you to have like a this hand type of key value store doing similar things for like lists but also like i mean obviously this all independent structure so potentially you can shard the hell out of them but how sharp how scalable this is right now uh which aspect well like the number of

85. 48:52

    streams like for example that people can do in parallel and updated parallel oh did an individual user can do a parallel or that a node can handle that node can handle yeah i mean uh it depends on how much resources your node has but like that's a good point yeah yeah i mean yeah as you said like you can have multiple nodes that pin differently yeah different substitutes so that's already kind of like it's kind of on demand charting pretty much yeah i mean yeah you can think of it like that yeah yeah

86. 49:22

    and so if you're getting like destroyed by some streams you can like unpin them potentially yeah exactly yeah yeah cool but yeah i guess like yeah as document grows for you yeah i mean that's definitely a problem that needs solving yeah i mean like it's interesting to think like yeah what what's uh so you're trying to kind of like have a miracle hash table here right so yeah exactly uh so that that could itself be stored in like an ipld

87. 49:53

    representation so you could like as so you know potentially like a live client of this stream you don't need to like sync the entire tree right you sync only though it's just a subset that you need yeah yeah it's interesting because it's not it's it's getting closer and closer to what you know we all do for blockchains where we have like a localized hash table underneath right so right but you might have like for different use cases you might have different yeah i mean different data structures like that the the the the

88. 50:24

    like the merkley's tree is like very useful for or like the the hamped kind of like symmetric trees are useful for uh a key value store but if you're making a list you probably want to have a different structure yeah and depending on like which direction and which way when i iterate quickly through the list you have different data structures for those yeah yeah i mean we right now like each smart

89. 50:47

    contract just has a key value store pretty much for it and so and then link like list etc implemented on top right so we just have like so it's it's yeah it's it's a hash map essentially yeah it's a hash map for for each contract and then they have permissions to write it to mutate it yeah yeah and then underneath it's like a merklized try pretty much to use all that cool cool all right um i think that's all yeah yeah thanks and

90. 51:20

    uh tune in for more episodes coming soon i hope cool thank you all right thanks so maybe before this like which problem ceramic is solving that's a good point yeah