Whiteboard Series with NEAR Protocol | Ep: 47 Juan Benet from FileCoin

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:04

   hello everyone we're here in austin after consensus with juan founder of protocol labs and we're going to dive in into filecoin but before that juan please introduce yourself yeah uh thanks for having me my name is juan um i started ipfs um and fuckpoint and i'm the founder of protocol labs um and yeah i work on a lot of different network protocols and i've been around in the web through community for quite a

2. 0:29

   while yeah pretty much the most og stuff yeah um great so uh today we're gonna go through the falcon protocol and uh in order to get there we're gonna learn a little bit about lipitorp and ipfs so just what we need from lippy to be and ipfs to get to falcon and then within falcon we're going to cover the just the core proof system and how that works and how we build up a storage market out of that

3. 1:03

   um and then we can get into like a lot of the interesting dynamics that that get um to sort of emerge out of the economic structures of of filecoin uh cool so um maybe let's like just very briefly describe um let's revisit the pips and then kind of get to falcon like the goal here is to be able to build a market where you have um storage providers um and clients um and you're able to transact here uh between a network of of participants where a lot of participants are bringing storage and a lot of participants want to buy that

4. 1:36

   storage um and you want to enable this to happen with crypto in a decentralized network with you know high integrity and verifiability of the storage and all kinds of stuff but yeah so how do we get there uh liberty first and won't dive into too much detail but liberty is a modular network stack that lets you uh connect all of the um lets you connect lots of different peers um using peer-to-peer protocols um use using cryptography um that doesn't rely on any kind of central authority or anything like that

5. 2:09

   it's all self-certifying um crypto so that means um you're using mostly public key crypto um and it deals with a bunch of um important self-description constraints to enable many different kinds of protocols to right on top of liverpool this is why now lipidpp is used by a ton of different blockchains um to build build up their their system so it's kind of like the missing peer-to-peer network stack that maybe should have been built in the early

6. 2:34

   2000s but we sort of extracted it out of ipfs so many people could use all of the peer-to-peer magic that makes fps work for a bunch of other systems and kind of what you get out of this is that you can build a network of any kind of topology that you want um and design like the connectivity here depending on the protocol structure that that you choose so you kind of like plug in components um

7. 2:58

   uh into this network stack uh depending on what your kind of user protocol once right so it gives you first of all like transport independence so you can use things like tcp if if you want but you can also use quick or you can use satp or or any kind of reliability protocol that you want or you can even use um unreliable protocols as well um and then all the uh library has a kind of bag of modules um that you can then kind of start plugging in so it gives you things like uh dht's

8. 3:30

   um you know cad emily is a famous dhd implementation um probably the biggest most widely deployed dht then it gives you things like pub sub uh there's a lot of different protocols um that can give you this um and so on there's a bunch of different kinds of network protocols that you may want to use in a peer-to-peer network and liberty just kind of assembles a lot of different interoperable modules that you can kind of plug in like lego bricks um into building some peer-to-peer protocol

9. 4:02

   um and any questions so far well i mean yeah like i am somewhat familiar with the p2p so on on uh i would say like there's you know the transport layer right that it just offers like a bunch of options yep there's uh um kind of the peer layer where i think the important to note is you know that it has a pub key public key private key cryptography which you know what's missing in general yeah communication right and then and then you have kind of this more

10. 4:35

    advanced um protocols i think like i mean for me obviously i'm always interested in dht's kind of and specifics of like when they applied in a protocol like how because of economics they can be broken right yep and because there is you know there's a lot of attack surface on this layer yep that people usually don't mention but we can dive in i guess yeah progress yeah that's a great point you know i tend to think that most blockchains are kind of broken in the in the um message passing layer of like

11. 5:07

    the broadcasting of the of the uh but i guess nope nobody has quite attacked that layer very much yeah um uh one thing to mention uh with a lot of the lippier p modules out there um there are good incentivized analogs that have that that are good open problems to build right so people could go and build an incentivized dht people could build incentivized pub sub and so on and all of those kind of are like

12. 5:30

    really good primitives that could build a much more robust uh web 3 in blockchain systems um and so it's one of these things where like those could be extremely good primitives for people to to then depend on so you're saying this is two projects to go and build and uh yeah yeah exactly and actually many right because there are probably probably a few dozen different variations in the hd's that give you different kind of properties and so out of those you might end up with like three to five

13. 6:01

    different dhts that offer you those different properties that um and you know there's like five instant device networks here or five instead of networks here i mean i know this few of pop sub versions i think that kind of operate but i don't think there is uh well outside of you know falcoin and being one of the ghts and a few others yeah so um out of liberty we get this kind of like um peer-to-peer networking stack where we can kind of plug in a bunch of different protocols and ipfs uses that to build a

14. 6:32

    content addressed data system so think of it as you want to be able to address information by by what the information is instead of by where it is so that's kind of like a really really key point so in the web today maybe i'll like describe the web case and then kind of make the ipfs case um without kind of like going into the details of how exactly it works uh because it's less relevant to falcon but it's over in the web case you have like a url like this um

15. 7:04

    today https um you know we have like foo dot com uh xmi picture yeah jpeg um and so today this part the domain maps to an ip address for the most part and so when you're resolving um this it actually usually is like an ip address on a port by default it's support 80 or you could use other ports https now 503 yeah that's true great great point that's why i'm here yeah exactly so the um so today in the web if you put in a link um at that moment the moment you load this

16. 7:48

    address you want to resolve what food.com is in that moment in time which points you to an ip address and specific server running and you go and connect to that server and you ask the server for this part the path uh what that means is that all of this information whatever mypic.jpg is is not defined in the network until the moment you you fetch it and you view it so you this is like a very kind of weak security um and it's tuned for mutability and so on but but this gives you like an extremely shaky foundation

17. 8:20

    where everything sort of depends on the the party that controls um this domain and the parties the parties that control these servers yeah so what ipfs does is that well and also this part as well which the resolution yeah yeah which is also very shaky super super problematic um so what episode does is like instead of this um you want to map this into a hash tree so that means ideally you

18. 8:45

    want to resolve this into something like this these are like the old iphone sashes kim fu suppose that this is like you know some like large hash um that identifies what my picture.jpg is um and you can build i'm gonna describe how we build up a you know arbitrary content into a graph like this um but the idea is that from here then you can ask who in the network has this content and from there you might get you know

19. 9:15

    the old server 1.2.3.4 or you might get some other server like you know 5.6.7.8 although of course like you want to use cryptographic addresses not just ip addresses so this is kind of like yeah um it's just describing how it maps to each http yeah um but ideally this is kind of like a secure um secure links with like crypto um public key this has a public key as well man this is done because otherwise there can be somebody in the middle who can

20. 9:47

    actually give you something yeah exactly exactly pretend they're one two three four but but you get super nice security properties out of this because if you map just the names to a hash link it doesn't matter who serves you the content you can always verify it it matches to this hash um and this idea of like uh using hash links um is fundamental to most um data structures in the web3

21. 10:10

    world and and and so on it's like the fundamental thing that makes bittorrent work the fundamental thing that makes git work and all of the blockchains so the idea of ipfs is like let's take that fundamental primitive and and address all content on the web this way um and so this is what's called content addressing the other version of hp was location addressing you're addressing content by where it is in this case we're addressing content by

22. 10:35

    what it is you're sort of deriving uh a fingerprint of what the content content is so let's kind of dive into like how we generate that so all content on and this kind of like traditional file systems um stuff and databases stuff but at the end of the day you can have like some um you know uh drive that has um some data on it or you might have like a database this is these are

23. 11:06

    like iconic um systems yeah um and and these might give you like some like block level storage which you know sort of splits up this disk in a bunch of regions that you can address and this gives you usually some kind of like um either relational or key value store model um but either either way in in this case you end up building a file system hierarchy which you know ends up

24. 11:30

    being like it sort of like builds up a tree um where this entire tree is kind of like your directory structure right like a slash maybe slash um users users jpeg or whatever and you kind of explore down and find the specific content that you want and in this case you might have um something like some tables or an index yeah yeah so you know see if this is like a sql uh friendly or relational friendly database you might

25. 12:00

    have a table like um let's call it table very very uh very generative name here table one and you might have you know a set of rows um you know each with a set of values um and so on but of course this is kind of like structured as some data structure of trees as well um and so on like at the end of the day um the way that the that humanity's whole computing platform works is that we store bits and we understand how to access those

26. 12:32

    bits of information through tree structures um always and so the the inside of ipfs is to say hey what if these links along the way um instead of being these like mushy local references that could be changed at any moment in time and so on we're just hash links um secure hash links that give you integrity and verifiability and so on and so you can take all the data inside of disks of all the data inside databases and then expose it to the world through this interface of of hash links

27. 13:03

    we call our hash links uh content identifiers um and there's a bunch of details as to why what's kind of in the cid format it gives you um cds decorate a hash with some other information that's self-describing prefix so you know how to interpret this hash because hash functions might break hash functions might change they have different computational requirements you might want to change which function you

28. 13:36

    use here and so you should just sort of describe the the function that you're going to use and you also describe the encoding of the data this is pointing to this at the end of the day is a pointer uh it's pointing to some data structure over here um and also uh there's like a a versioning identifier to know like how to read this and there's a um a base so i won't go super into detail here but the idea is like the

29. 14:02

    core pointer in in any ipfs system is these cids that give you effectively a memory pointer like in any operating system or any programming language or anything like that but it's a pointer that's secure and that works across distributed systems and that works out of cryptography um so get so a way to kind of the way we think about it um is it gives you kind of access to this sort of like universal memory that you can store any information in and retrieve it later just by knowing the cid of it and that could be

30. 14:36

    a single little you know one kilobyte chunk of data or smaller or it could be as big as the entire file system or as big as a whole database or like a whole um the entirety of a blockchain right so when you think of of a blockchain it's this big you know kind of sequence of blocks uh with like sort of like trees hanging down a bit and all of this is addressed by a hash at any point in time and so like these hashes could all be um can all be cids and you can retrieve

31. 15:07

    all of this content through ipfs so is this hash like kind of is there structure of this hash like if this is this is just a blob of data it's just this i mean a hash with this function but if it's some kind of hierarchy like great question so this is where the encoding part comes uh so the reason we need this this um this this field is that we need a table um that maps the value of that e

32. 15:35

    to some program that's gonna be able to decode what's in there and so the the this mapping is what enables you to when you have a pointer like this uh read this e uh field figure out how to interpret what's in here so that you can traverse it um there's like a you know default one which is we call raw which means it's just a bunch of bits we have like it's a bunch of whites we don't know how to read it um that's like a leaf node it's like a terminal terminal node or something like that but if you include a

33. 16:08

    pointer here to another object um and so on down and of course this could point to some other things and this could actually point back to the same thing um if you if you can run this program and interpret this and be able to read the links within this then you can traverse this whole graph and so this is uh one of the things that gives ep has a lot of power is that this is um there's like same defaults but it

34. 16:40

    enables even older systems that are hash linked to uh come into into this network so for example we tend to think of it as the internet of data structures where you might have a network like git and a network like bitcoin and a network like unix fs which is the default ipfs one and get files sort of like all point to each other and so on and they kind of have local references so it get

35. 17:11

    it's hard for again repo itself cannot necessarily reference objects in another kit repo but you can sort of make sense of that and you can have objects that point into here or point into here or point into here and be able to um refer to each other based on this encoding file here so that means you can take entire git repos and just dump them into our ffs and use all the ipfs tooling to move them around and so on and then pretty much reference from new objects and yes

36. 17:38

    exactly and so for these older data structure you you kind of have to like um understand what you have they make it more complicated because you they don't they're not self-describing like this yeah and so you have to know that it is a good data structure and whatnot and and so on um this is also generating this new world of ipld um which has all of the self-description within it so that you can understand what's going on mm-hmm

37. 18:03

    make sense how big is this library um this is just uh governed on github so you can add a thing by pull request um and where we're headed soon is um there's a really cool codec that uh xp um brooklyn from fission is making called autoencoder where you can define just a wasm program to run so you can point to um so you can have like a like a thing called an encoder here and this can point to a program a program

38. 18:34

    that can read it exactly uh so and and this is kind of um the way we've thought about it for many years is like think of it as building a programming language like um c or people's plus or rust or whatever out of these distributed primitives where you might have you know some like um memory sometimes you have a memory whatever program you have a pointer to the this is kind of like the this point like the v table in serious equals plus uh world

39. 18:59

    um where you have like the functions and the date the you know the program in one that's going to decode the rest of the thing so this lets you build this kind of like a distributed programming language on top of um all the stuff but we sort of like built it up out of underlying primitives um so so out of this like kind of the option of this is that we have a way of um addressing all kinds of content

40. 19:25

    whether it's like in legacy file systems or in legacy databases or in any of the new web3 hash link things um by just decorating all the links along the way with the cids and then out of that we get verifiability integrity checking we get the ability that anybody can service this content it doesn't matter who has it as you get all of the power of merkle trees and all the power of these hashtag data structures but you get it sort of like um across kind of different kinds of

41. 19:57

    systems and across um different kinds of protocols and so on and so like the the the big thing that i enables is this kind of content addressing mm-hmm that's right it's almost independent of exactly how it's laid out on on the network it's independent how it's served it's yes it's just kind of uh exists and so yes exactly um and so right now you know a lot of blockchains don't use cads yet um because i guess people don't know about them yet um but

42. 20:27

    the cool thing about this whole system is you can just you know add a program here to decode this and you're set and so you can you can just absorb all um all the data generated by any hash link system you can just plug in and you can move it around so we have like um examples of like moving around raw get repos or like different blockchains and whatnot okay uh cool so and and maybe another

43. 20:51

    component of ipfs is that so that gives you content addressing um and it gives you the the peer-to-peer networking of liberty p so it brings all of that in um to give you this kind of secure web based on content addressing um and it also solves a lot of you know very pragmatic problems of how do you make this work in os's how do you make this work in browsers how do you make this work in mobile so just a bunch of

44. 21:15

    implementations um of this entire system for different runtimes and different platforms to enable all the things to kind of talk to each other and be able to grab the data and make sense of it and so on so think about this as over time kind of following the pathway of http where instead of having just a single implementation that kind of drives everything instead being a bunch of libraries that let you compose like really small uh servers or protocols or whatever that speak this whole language

45. 21:47

    and that can all resolve um you know the abifas uh link right so dot com or dot um eth or whatever or dot whatever you want and then some some other content and one component i'm missing here is this padding so if we do you know a b c and so on this is just resolution through this graph so we can do something like you know food.com or food.eth get a cid and then traverse this graph by following the links here described here but you still need in in case of fbfs to

46. 22:21

    start with some gateway so that's only it really depends on the platform so you don't have to start with a gateway for example in um you can put the actual hash here but you need but this is an immutable reference right yeah um you could also put in um a keynes what we described as a key name which is a self-certifying you can use a public key as an address to uh for something but you need some name server that's gonna that's going to result

47. 22:50

    so if you use the hash then you're set but it's all kind of um you don't have the immutable references if you have mutable references then you need something that can resolve those mutable references and that traditionally is a name system so you know you can use dns but you can use any anything else there's a sub protocol here called dns link that lets you do that so you can take a traditional dns name put in a hash link as a text record and this

48. 23:15

    works today and it's sort of already in brave and opera and other browsers and if you don't have if your browser cannot speak ffas yet then you can use either an extension to enable it or you can use a gateway the gateway is there only to kind of enable browsers that don't yet speak ipfs natively and so but this goes like if you do put hash here to retrieve it it does dht underneath uh you can use so the the resolution of ipfs um

49. 23:46

    can it sort of depend on the platform that you have there are cases where you want to use a dht there are cases where you don't for example in a local area network um if you're disconnected from the rest of the world you want to be able to resolve that content locally without having to without having to like break the link so for example if you and i are working together on a computer or different

50. 24:08

    computers and i'm trying to look for some content that you have and our computers can talk to each other over the local area network i should be able to get the content from you um and that doesn't mean like we could form a local dht but that's kind of expensive um in a local area network because the latencies are so much lower i can just ask my nearby peers who's got this content and if somebody has it they can they can just give it to me like for example in bra if we're talking about brave and if i put the hash here

51. 24:33

    what how is it going to actually find the content so in braves case if you're connected to the internet um i think it will primarily um uh use the there is one big public dht that the iphone's network uses and for the most part it will use that large dhc but there's other protocols that are we're calling indexers that are serve sort of the same role of dht's um but have a different different topology the problem with dhc is that they're they're very slow um

52. 25:06

    uh you end up with you know long lasting yeah lots of fetching finding the pier yeah yeah yeah um we should do like a whole whiteboard session on each season like what's good about them and and bad about them and so on um hi cool uh i'm gonna erase all of this that's what we got out of ipfs um so the platform sort of assumes you have already a whole bunch of content in this hash linked setting right so that you have

53. 25:36

    um what we call ipld content and it brings in all the networking from from lipidp and so on so you have like uh peer-to-peer protocols um the the goal of my professors to again make this market work between um source providers and clients that want to um hire the storage from source providers and the way we're going to do it is we're going to build um what we described as a decentralized storage network so a dsn and usually like this is a very simple

54. 26:08

    structure where you want to um you want to put some data so that there's like two basic operations put and gets it uh and everything else is kind of like to init to to make this sort of happen so you usually want to put some data and you want to get a key back and later you want to use the key to get the data back and the the the way we're going to use um in our case the key is a cid so here we actually

55. 26:45

    can just write cd but what we want to do is we want to use a set of incentive structures to enable this to happen with certain guarantees like we want to we want um verifiability that the storage is there and for the long term and you want sort of an economic guarantee that that your deal will be honored and you want some guarantees around retroability you want to know that you can get the data later on but somebody's not going to hold it hostage or something like that

56. 27:17

    and um so the way we do this is we use first a blockchain to maybe describe components here we use a blockchain to coordinate let's grab later kind of what we put on there we use a proof system to create verifiable proofs of storage and we use a market um to get source buyers and clients to sort of meet each other make deals um agree on on prices and and all that kind of stuff there's a lot of like complexity

57. 28:00

    embedded in each of these um but i'll maybe uh i think it's probably best to start with um the proof system because that's kind of like a really important sort of foundational uh block so this for us divides into at least two proofs um pour up which is a proof of replication and post which is a proof of space time and the idea of these two proofs uh the proof of replication um what it gives us is the idea that like if i have some data d that i want to

58. 28:31

    um get two replicas replica one and replica two so maybe before this like what are maybe more explicitly the economic guarantees that you're trying to achieve here yeah so um the economic guarantees is that um i guess yeah so an economic decentralized storage network is like you're putting some data with some um with some money um and you can you can sort of get a cid back and in the future um you can retrieve it by giving the cd and some money as well so so

59. 29:03

    the economic guarantees that there's some price where you can hire the network for that price and your data with very high probability will stay in the network for the for the um amount of money that you're specifying and that's kind of where the market comes in we describe this in terms of deals and each deal is kind of a little data structure that has a um the client the storage

60. 29:27

    provider um and then the price and the period of time in the period of time yeah so i think that that's really important pieces like you're storing it for a period of time and then you're willing to pay this price yes so that is great and if you want replication so this is really flexible because if you want you you can build any um reliability uh model on top of something like this so

61. 29:53

    a lot of other networks kind of pick a specific erasure coding scheme or a particular replication scheme and so on and those replication schemes don't fit all use cases what you want to give is a very useful a very kind of simple primitive upon which you can build all of that and in this case like if you want your content to be replicated by many parties you just make several deals um if you want to take some data and erasure code it and create like

62. 30:17

    different objects then you can do that ahead of time on the client side you prepare it and you and you put it here and and also know that the client could be doesn't have to be the end user necessarily you could have like you know client one which uses client two which then goes to the source provider right so this could be like brought processing of some sort on behalf of the user um this we sort of call this like aggregators

63. 30:42

    or on-ramps but so when when storage provider makes this deal because that probably that's why i'm like let's figure this first because it defines what you're trying to get here when storage provider makes this deal what are they promising right i mean time kind of what's the probability and if they don't honor this contract so there's collateral so yeah we'll get to um we'll build up like the the um the way we sort of put in um the content specified by these deals

64. 31:15

    into um into the source system is by using anything we call sectors so um let me maybe build it up um i was going to sort of explain the um the component to the component but but let's maybe go this way so let's start with the blockchain and kind of express it down so um think of the falcon blockchain as having of course some state tree right so like any other blockchain um but then underneath it we're going to accumulate over time a set of sectors and these sectors think of them as lots and lots and lots

65. 31:54

    of storage source capacity think of them kind of like a um a storage container so at any point in time we are adding more sectors to the network every block more participants on board more capacity and so on and so what you end up with is um as the blockchain is proceeding um source providers are committing capacity and they're doing that for the block reward i'll describe later how we're using the block reward but um uh falcon is a storage power consensus

66. 32:28

    so spec uh that uses instead of hash rate like in bitcoin or ethereum or something else um we use the storage power that people are accumulating as the power in the block reward so participants are intended to bring more and more and more storage power into the network um so that they can earn larger and larger fractions of the blackboard so on its own without any kind of deals this blockchain can assemble a lot of capacity that you can then um

67. 32:59

    fill with data so think of these as kind of like storage container boxes that you're going to later put put uh stuff into so pretty much like if if i'm a participant and i'm not bringing new storage capacity then i'm not receiving i mean we can go into block rewards but kind of like you always need to bring more capacity yeah exactly so so the participants maintaining the consensus here um have to bring in more storage power um this

68. 33:24

    is kind of like a like a hybrid proof of work and proof of stake protocol where um we use the algorithms of proof of stake but the the power is not just pure uh money the power in the in the state protocol is the actual storage yeah um and and this could be either capacity or um storage with data we'll go later we can describe later how we kind of incentivize the not just capacity but but actually useful useful data um so this on its own think of the the

69. 33:55

    bitcoin network as using a a function very similar to that to amass an enormous amount of hash rate on the planet um and so falcon has used that same kind of structure with that kind of block reward auction to assemble an enormous amount of capacity we have 17 exabytes of capacity right now which is like really competitive with with a centralized cloud storage network right so this is an and every day somewhere

70. 34:20

    between 20 and 30 petabytes get added so you know a pedagogy is an enormous amount of storage um uh every day like 20 to 30 petabytes more get added to the network and that is all coming from the block reward so the blackboard sort of summoning all of this storage hardware to appear and be ready to receive customer data so once that all of that storage is onboarded source providers can sell the fact that there's space in the in these uh sectors um we call these sectors it's kind of like an old hard

71. 34:52

    drive terminology uh the idea is that you can think of like falcon is a huge disk that you're gonna partition and then you're gonna put some data into it into the sectors um uh social workers can sell the the space in these sectors using these deals on the market um and and part of what you agree on uh in yeah erase the deal i should like bring you back let's see so we have the storage provider the client the time the price

72. 35:25

    and the sector um and the yeah the sector ends up you agree on the deal first and then later you you map it to the sector so the sector is uh um you can have a list of deals um i forgot also to say the cid of the content that you're storing there so it's this this tuple uh and so clients and search writers um find each other i can describe later uh how we get to that they agree on time they agree on a price

73. 35:57

    and they agree on the content and then they the client transfers the data to a source writer the search provider can now store it in one of these sectors and and they will earn this money over time over time one of the other things that falcon does is that falcon prefers content that has deals versus just capacity so um we have this concept of like useful storage where um if you can so this is kind of like a complex economic structure that

74. 36:29

    is that is um there's kind of an uh you have to deal with a you have to solve like a very hard problem which is how do you how can you tell that this content is like usually versus and how how do you know this client is legit and not not um yeah not the storage provider himself putting more exactly like just recycling yeah because capacity is really useful but it's not as useful as as useful storage right so um right now the f the factor network is

75. 36:53

    incentivizing useful storage 10x more than capacity um and so there's a whole kind of uh decentralized protocol of um verifiers and notaries that kind of check that this client is legit check that this content is legit um and then kind of sort of like bless this deal um but you know that's like a candidate construction five is very modular so all of these kind of components and protocols can can be swapped out for other designs

76. 37:18

    later later on um so so the marketplace here um one of the really neat things about about this is that you're supporting tons of different source providers that can offer very different prices or very different kind of other services alongside like um and they might tune their operations for certain kinds of data so some source providers might be storing um very large amounts of data so think of

77. 37:42

    you know 10 to 100 petabytes at a time or something like that and they might have facilities ready to ingest hardware uh or they might um or maybe there might be source providers that offer a lot less storage but they have really good connectivity and so they can um serve it out to end users and so on um and you enable this kind of like market behavior where clients and source providers can find each other by use case

78. 38:06

    all right so you're not i mean yeah like pretty much this model does not prescribe in any way right kind of this relationship right exactly like an off chain you know a paperwork agreement can be done for example as well and this is just a payment yeah we we think that if you prescribe this relationship you um are almost certainly going to get it wrong most of the time um in that most of the time clients and source providers

79. 38:31

    will like um be in some niche and you want to enable the participants in that niche to interact um and you're best served by creating a mark a fluid market where that can evolve over time because also the the use cases and the technologies will evolve over time so you want to let clients and storage providers sort of find each other where they where they need to be kind of the model that we're using here is like this i'm going to build a quick map of the system um

80. 38:56

    so we have storage clients over here that use these like on-ramps to go to storage providers you mean just some kind of registry of storage providers um no let's grab it in a moment um these are retrieval providers i'll describe that in a moment and then this goes back out to retrieval clients sometimes these are the same parties sometimes it's different um yeah i mean like developer putting some stuff on and then serving it to everyone

81. 39:42

    um the idea of this map is that you have storage clients that have a bunch of data these could be as simple as a a web app or a consumer pfp or yeah like your random pfp or it could be a large company with um or a large organization with tens of petabytes of content that they want to eventually make its way into the search provider now depending on the use case the tooling the products and so on that these storage clients are going to want to use

82. 40:11

    uh varies so um storing a picture as part of an nft or something like that is very different than storing 10 petabytes of archival data and so what the storage claim the product the product and the interface the source client needs is very different in those those cases so these on-ramps are use-case-specific products that tune for that ux but that all are aggregating into into uh so so you sort of build on-ramps by

83. 40:40

    vertical you um enable source clients to hire that on-ramp to then um get storage in here so nftw storage is a good example where that's a product specific to the nft market um it enables anybody who wants to store a lot of nfts um to use a product and an interface tuned for storing nfts and on the background it's archiving and storing everything with falcon but can be also you know a b2b business where you go and you say i want to store this many petabytes and then you get an

84. 41:10

    email back exactly yeah exactly there's a there's another on-ramp which is kind of like the connecting which right now is just um kind of like this this uh team that's just connecting prospective clients to um source providers that where they do a little bit of diagnosis of like how much data is it where are they where do they want the data to live and so on and then connects them to a set of source providers that make sense because the

85. 41:34

    moment you're moving around tens of petabytes you have to get humans in the loop um moving around that much stability you need like physically shipping stuff usually yes a lot of times uh the energy turns out the internet is not uh does not have enough bandwidth to move around that that much storage so you have storage clients and source providers meeting each other and source products will go to the clients facilities with hardware um

86. 41:57

    get the data and then bring it back and so you want to sort of enable all of that to happen um the centralized cloud uses um has similar services uh so amazon snowball is an example of that azure data box um and so on uh just in our case this is a decentralized network of participants that's going that's going there now once the data is here you can index it to make it retrievable

87. 42:22

    and you can use these to tool providers um i haven't spoken much about this yet but this is kind of the cdn use case so think of this as the cdn and think of this as like um the object storage so this is kind of like s3 or something like that and this is kind of like cloudflare so so once you have the content here and you have it indexed you want this to be like a like a really fast cache all the

88. 42:47

    way close to the users um this there's a whole set of interesting techniques how you build these that's a whole separate problem yes um when you say index here like i guess i mean we should probably finish with proofs but yeah there's there's a lot a lot of information yes so it turns out like this is a large scale system once you want to store um exabytes worth of data you end up with all kinds of like interesting large

89. 43:11

    scale problems each of which could take an entire you know large scale decentralized network to solve for example this indexing layer today you can um we have several one of them is like the episode issue but that doesn't work to store petabytes of stuff um so we're building these these other um indexing services that are more of one where like the index is going to get rough it's a very large index um but

90. 43:36

    you can make full replicas of them and that itself could be a decentralized or incentivized thing um yeah it turns out that these each of these have strong incentive structures you can create whole businesses here um whole crypto networks or um companies in each of these okay um maybe i'll describe like how to so on the proof system just sort of for completeness um we we've left open how a storage provider yeah proves that they have

91. 44:05

    stuff that they have the content or they just have capacity yeah um so the the and this kind of where these sectors uh come from so i was describing let's go back to um uh the proof of replication um which is that i have some data d that i want to verifiably get be able to get a proof that i'm storing as many replicas as i want so if these are both stored by the same party say alice i want to be able to prove that these two replicas are independent from each

92. 44:39

    other even if it's the same data why do i need to prove that well one many users want to prove that that multiple copies are being stored in multiple places but two it is critical for consensus power that you cannot cheat here and cannot pass off yeah individual storage as as tons of other other storage crucially in in the capacity case if you say generate these randomly um what you don't want is to be able to

93. 45:09

    kind of generate the capacity on the fly um to sort of like pass the proof or like or recycle pretty much that the capacity already presented as a new capacity yes exactly um and so so there's kind of this like deduplication problem where you don't want to allow a storage provider to pretend to have say um x storage when in reality they only have like 100th of that or something like

94. 45:32

    that or you don't want the storage providers to have no surge at all and just be able to to to forge the proofs like they pretend that they have a lot of source power when they don't um and so that's where you know the river replication is really critical and what this enables is um you have a an interactive proof system here that lets you prove that these two are are different and the intuition here is

95. 45:55

    uh very simple which is like you have these uh you can use some um facet of the world to make the encoding you pick some encoding between the data and the replica that is either that takes some time or that is expensive so you can use time or cost um or other structures uh you could have like a some kind of like centralized thing that you know disables creating the replicas or something like that they carry a special signature um so there are many different kinds of protocols that you

96. 46:27

    could make but for example the time one is very kind of intuitive to think about you sort of commit to what the data is at some point in time you give it to the to the storage provider and they have to run some expensive encoding and then once they they run that expensive encoding they communicate the um the proof that they've done so and then some point later in the future you can sort of challenge them

97. 46:58

    generate more whiteboard space here we can scale up yeah exactly you can challenge them and they have a certain amount of time to return a proof so the challenge yields a proof and you can check whether that happened and you can sort of time the amount after some period time that is smaller than than the time it takes to generate the encoding so this is what we call a time based uh poor app where this encoding just takes a specific amount of time and the

98. 47:31

    you you send a challenge and you expect a response in less time that it takes to generate the thing so that way like when you send this challenge and you get back a proof you know that the participant does have that content otherwise it wouldn't be able to to pass the proof so okay so this is pretty much like a vdf type of function that's rooted in data that they're supposed to store

99. 47:55

    um how how is this communication facilitated uh so in so in kind of like the general case this could be any two parties in filecoin this is the blockchain the blockchain is automatically issuing um directions right direction okay so like new block has pretty much a like some kind of you can unique information extract some randomness from the chain to um derive the content you're gonna

100. 48:19

     you're gonna store and then um you know produce proofs at any point in time you here you have proofs and we we run one of the i think the largest snark proving system on the planet um and it's it's all um you know uh large-scale proofs of people actually storing all of this content and here you you um same thing with the challenge you test you know this is there's some uh the challenge issues a bunch of like checks a bunch of points make sure that the content is there and you produce a proof that can normally

101. 48:51

     pass if you have the data so i mean let's just look at like example with single storage provider right um so you produce a block yeah so there's some source of randomness that is used by the storage provider to produce a proof that takes how long so so in this in ceilings case uh this has to take a certain number of blocks i okay we have many versions of the proofs that have evolved over time because algorithms improve i don't know what the latest uh number

102. 49:20

     of log times is but it is it is created such that it takes a certain number of blocks so it's kind of function that yeah it takes whatever x blocks to compute and then it takes the sector as like what what is it takes sort of like what we call an unsealed sector we call this function a ceiling operation so this encoding um we call it seal and it seals an unsealed sector which is kind of just plain text stuff it could be plain text capacity which is all just randomly generated or it could be

103. 49:52

     the actual content that the user is asking for you pass this cl operation to encode that uh whole sector and you end up with a sealed sector and you can you then produce a proof that you've done that correctly and submit it to the chain so what is encoding to seal the sector um so this is where like we enter into like the realm of different poor apps uh there's like a and this was kind of we spent years of work on interior exploring this this space um there are

104. 50:21

     many different kind of pour ups uh the one we use right now is not a latency based board it's just the easiest one it's a cost-based board you can think of the same argument here as an economic one where the transitions here are just so expensive that it's cheaper for you to store the data than to download it again yeah or to generate it so generation here is like this pernicious type of attack where um you could sort of like

105. 50:44

     as a malicious storage provider you could um pretend to be a client pretend to hire yourself or pretend to add capacity that is just all generated yeah from scratch so you want like an extremely hard um constraint that like even if you the data could all be generated um you know it's just randomly generated data um you want to prove that like the it is extremely likely that the the data is actually there

106. 51:10

     um and and there's a very good economic argument here where like the cost of making this computation is uh many times larger than just storing the data what's what is that like right now what are your functionality so so this is just a very expensive operation that where it just consumes a lot of computational power um and and that competition power is bounded to be more expensive than

107. 51:33

     the drives maintaining the drive actually yeah okay um and so the you know as an example of a replication we today use um sdr pour up which is a specific um stacked depth robust graph um depth robust graph porup um uh which like is a very expensive like you you um without kind of like uh going super into into details here um you end up here you take the data and you create um a set of

108. 52:20

     of encodings encryptions all the way to get to the replica and at each stage here you're just running out an expensive encryption that at each of these layers is all interlaced with uh elements in the in the back here so that you cannot um say generate this this stuff until you have generated all the stuff yeah some sequential yeah yeah and then you but you you know you rooted into some randomness

109. 52:50

     yeah exactly and so this is just kind of like a very this takes a long time to go from like the original data all the way to the replica and so you can either use the latency or you can use the cost assumption so that you can use either one or both if you want to be stronger but you end up with this like super expensive process to go from data to replica and once you're here this is what you store and this is what you prove um we kind of like use a challenge response

110. 53:15

     vertical here because it's very difficult to prove that you've done this correctly so instead once you have the replica um you can issue a set of challenges here and you can sort of decrypt these and prove that they're the same ones that were here and you can do that with traditional merkle tree um okay so they store kind of like re encrypted part yes they they store this encrypted part and then so they can like when you're asking them

111. 53:40

     they just need to like decrypt random pieces and give it to you exactly and then and so this is really critical for consensus because this is what enables you to know that you have a very large amount of storage um committed and you can prove that it's all um it's a legit uh capacity and so on and so like the challenges that you should also automatically write pretty much like every block then you know

112. 54:06

     even with the same randomness you kind of have challenge function yep that given the sealed sector yep and the randomness you need to return something approve so how does pretty much the block producers right yep they will receive these proofs like they're receiving from everyone yeah so so this is where we use these post things so important i think of a poor up as a proof that you know goes from the data into a replica and it gives you the facility to sort of call

113. 54:37

     the poor challenge again uh a post just uses a poor app to then you know with some sample with some frequency um to prove that you have like have been storing an arrangement of bits for a given amount of time so you can just keep iterating this and so you do this today with a blockchain so the blockchain issue is a challenge um the random you derive randomness from the block you use that as a as the input to an ongoing challenge function um and

114. 55:08

     you prove that you have uh the stuff you committed um all the storage on the factory network is proved every day so we the way we try to do this is like we so there's a challenge every day for all of the storage so so every every block issue is a new challenge and so it's just that um we sort of like divide up the day into these like um 48 windows and a storage provider storage is sort of assembled it's just dispersed across these windows and they get challenged at every step of the way for some fraction

115. 55:44

     of their storage you want to minimize the overhead so you want to kind of like pack it in here um so you know this is like some number of sectors and so in this window of time they're going to be challenged for all these sectors and so in this window of time they have to like produce this at the very beginning of the time period they get the challenge and they have to submit the proof before that time period

116. 56:06

     every byte of storage gets challenged one kind of once a day for every single so it's one of these um probabilistic proofs where in this time period you get a challenge and that selects um every sector is proved but you it within the sector you just kind of randomly yeah it samples in sector yeah how big are sectors uh 32 gigabytes or 64 gigabytes there's like two settings and that's specific um because of sdr um it's kind of specific to the algorithm and the available hardware so it's one of these interplays where you

117. 56:38

     have to design um the poor app and your implementations of the poor app based on what the available hardware is um and when then once you do that you're going to put an optimization pressure on the hardware to get better in certain directions and so you um because you know four source varieties like this means yes yeah they're spending most of their time most of their yeah so you need a bunch of gpus so in addition to the storage you need a bunch

118. 57:02

     of gpus that can do these proofs but they're not used all the time which is neat because then you can use use the gpus for the remaining remaining time to do other kinds of computation like competition over the data so once you have all this data in falcoin you can then start doing um a bunch of computation over it and that's kind of like what's coming ahead next year it's bringing online decentralized computational networks on

119. 57:26

     top of all of this data that's getting amassed okay so i guess like maybe let's stop by just quickly on the market yeah so the market today um we we're considering doing this as kind of like an explicit market where you put asks and bids on both sides and so on and we might still go in that direction but the um because there's a lot of features here you need this to be a very high throughput system and so for the most part right now um the market is not so big

120. 57:58

     that um you can't sort of like segment it out um there's a few parties for example making a um components where like they a set of storage providers connect to a to a marketplace and then um clients come to that marketplace and sort of find source providers or they act as intermediaries between the two where the clients hire the the marketplace in the marketplace um then hires the source providers

121. 58:23

     but where we're headed in the long term is that this this will likely be a think of it kind of like a dex where you could put bids and asks directly on the market one natural extension that hasn't been done yet is that um you could start having deals without the cid yet so uh you can determine you can pre-buy the the space and what this gives you is um kind of like a futures contract on on spacetime which you know it this

122. 58:52

     really commoditizes um uh the storage uh cloud storage in the same way that you can commoditize oil or something like that and so we think like this would be kind of like d5 meets five coin uh that's sort of uh red limited by um programmability in the chain uh this is where fdm is gonna come in to give uh programmability to the blockchain okay um the so this price and kind of the cost like how does that have set the block rewards and so so right now um the

123. 59:26

     so so the network is going to follow this like um expansion phase where uh we sort of like divide it into three stages um expansion like optim optimization and then sort of like um station maintenance or something um where and we're sort of like here and so during this entire period the block reward is like dominant incentive and so right now the participants get incentivized by storing

124. 59:56

     but by having storage power either with capacity or useful storage and so that means that the client's price could be extremely low so we can be many orders a minute cheaper and we are in fact for these tens of petabytes that are getting stored at a time many orders and actually cheaper than the cloud because storage providers are making so much money just on the block reward alone so this gives you the ability to sort of

125. 1:00:19

     like under um undercut the entire cloud storage system the entire centralized cloud model and bring all the data into following right yeah so what's happening here um so over time the kind of um uh once or the next networks are saturating in terms of the the available capacity and and the usage of the um the usages of the network sort of reaches the amount of capacity that and

126. 1:00:45

     you have more interest in putting data storage than you have capacity then at that point um the prices will start increasing here but i mean this price goes oh so it doesn't go to the storage provider or like when when i'm doing this contract is i'm paying to search for it directly i'm paying i'm burning these tokens uh the those that have those price here goes to the source provider but think of the source right as having multiple revenue flows one is the deal price and one is the the mining

127. 1:01:11

     reward so today so so um today the money reward is dominant uh and the amount of money they get here is very little okay um so they can afford and they just get yeah they just don't they don't care about the price point i don't care about this price today yeah and and that's sort of a function of the saturation of the network so if you have um yeah i mean you have you have like how much capacity versus how much stored and

128. 1:01:34

     then exactly there's more storage kind of the price of this chrome so when you have a ton of like empty capacity yeah you can afford for this to be zero you can actually afford it to be negatively priced i will maybe leave that out as a discussion for another time but when you have a block reward incentive that is um giving you money for storing deals um you can make this negatively priced

129. 1:01:57

     you can incentivize you can yield farms uh people storing stuff exactly um and so that's that's a super powerful incentive structure that's kind of like um you know how like in in the oil market you sometimes get negatively priced oil um it's a similar kind of behavior you need to make sure you're very good at yes you need to make sure that the the um proofs here are legit and and that the client verification well yeah the client verification you

130. 1:02:28

     need to know these are like seriously good um uh customers and and that's where kind of the we call that process falcon plus and it's kind of uh there's a network of notaries that um verify this and and so on can it's a whole other story for another time yeah that's uh yeah there's like this retrieval markets and uh uh not really seems like a few more topics um well yeah i mean that is uh very interesting and complex

131. 1:03:00

     yep so the the option of all of this is we have you know peer-to-peer network that we can content address um all the data on the planet and then we can create an incentive structure to start a masses so you know sort of like we tend to think of it as like three steps one is like a mass hardware then a mass data and then make that computation useful for computation so step one is like you have a block reward incentive structure to summon massive amounts of capacity and so like we kind of have that done

132. 1:03:31

     um now we're onboarding the data so this is kind of like in progress now and so this is kind of like filling up this capacity with um with useful data and then once that's there you can use the available hardware that people have to then issue computational networks and so this is what's coming in the future uh and so you end up with a decentralized cloud storage and computing network um where it's cheaper to sort of like move the computation to where the

133. 1:04:02

     data is because it has gravity like once you have massive amounts of data in one location it's extremely difficult to move it somewhere else extremely extreme extremely expensive so you it's much cheaper to move the computation to the data compute there and produce your and given we have a bunch of gpus this is where we're going to train our ai exactly yep that's where we're headed yeah great well yeah thanks for having me happy to

134. 1:04:25

     dive into detail in any of this in the in the future um but yeah this is kind of like the model of today's just getting tons of useful useful data into uh into this extremely cheap um storage there's an exchange already that's leveraging this kind of negative negatively pricing thing where you can let if you're a legit user you can bring your data to the exchange and sort of sell the

135. 1:04:50

     the right to store your data and so you can auction the sounds like an auction house for useful um so you can implement valuable data storing badge stuff yes well yeah i mean that going into like how to define useful valuable data is it turns out to be not not that hard because um if the data is public that becomes a much easier problem so for for a whole subset of the data where the data is like publicly visible um that ends up being a much easier problem you can just see what the data is and then sort of certify that that it's

136. 1:05:21

     um uh good data that the network wants and because it's public and viewable you can always audit that in the future so you know think of like all of the large-scale public data sets that the web has moved them in yeah yeah and then i guess like just to finance here if this uh challenge doesn't come in time then you get it slashed as a validator and yes that's right so if you miss

137. 1:05:44

     if you miss this proof you get a fault um and you get slashed so um there's there's a and this goes into kind of too much detail but um there's a schedule where you um you're you're you're allowed to miss you're allowed to miss one uh because sometimes there's like uh flukes and so you sort of like miss one um but if you and then you can start getting slashed for for more you can get up to 14 which is two weeks i guess actually we changed that to now six weeks so you can keep getting slashed every day

138. 1:06:17

     for six weeks and then at six weeks uh it's terminated and the source rider loses a large amount of collateral that's been put up for the sector so part of what makes this entire thing um operate is that the source riders have to put up a large amount of collateral promising to the network that they have the storage and the reason for that is um you have to deal with this uh elastic thing where the blocker were in

139. 1:06:40

     the pricing of the token changes the economic return of people um at any sort of like moment in time and so in bitcoin you get this behavior where like the hash rate mirrors the price the price right so if like um if bitcoin is going up then like the hash rate is going to go up uh if bitcoin is going down that the hash rate is going to go down and in fact when you have to kind of build the shock absorber to

140. 1:07:04

     um uh to make sure that like even when you have some amount of volatility in the in the in the price of the token the economic return of participants is spread out over years so that everybody's thinking about averages over long periods of time as opposed to the immediate volatility um and so that's where collaterals play a huge role in that in both adding security to the network in that you know if they if a large chunk of

141. 1:07:29

     storage disappears that weakens the security of the consensus protocol um and you also don't want a bunch of users valuable data to disappear yeah so you want to put up a lot of money at stake committing to store all this data and that's where the whole data structure comes from so am i if for example one of the storage providers uh gets slashed as a user am i getting kind of that um so you don't the

142. 1:07:52

     collaterals are burned always you don't flow them back to the clients if your deal stops being stored you then get the remainder of the price that hasn't been spent but you don't get the collateral back and the reason is otherwise you have this like behavior you where you can sort of convince a bunch of stores you can have like malicious clients that convince a bunch of source products to store data and then um and then you go and like attack them and

143. 1:08:17

     make them miss their windows and then you get a bunch of money back so it creates like this perverse potential so so all collaterals tend to be burned mm-hmm okay well there's still incentive to attack a magister's providers to get collateral burned as a yeah but if but if that happens then you're weakening the utility of the network and it because when you're burning capacity you're burning um

144. 1:08:47

     useful storage or whatever like that that is bad for the network so so the argument is like sure you might be burning some tokens but you're burning the credibility utility of the network and so that's that's probably not mr alliance yeah yeah all right well thanks this is yeah that's really good there's a lot of stuff here and i'm sure there's more to learn uh what's the best place for people to

145. 1:09:10

     learn more um you can go to fatima and you can see a lot of info there there's a number of papers that we published over time as well um yeah and we have tons of video uh recordings of all kinds of technical talks on the falcon youtube channel or the pro collabs uh youtube channel awesome yeah thanks for having me this is really fun this was really good sorry that uh it's kind of like messy going back and forth