{"id":"J-drqD2UebM","title":"Whiteboard Series with NEAR | Ep: 42 Adin Schmahmann from Protocol Labs' IPFS","description":"IPFS Software Engineer Adin Schmahmann (  / ipfs  ) and NEAR Co-Founder & CTO, Alex Skidanov (  / alexskidanov  ) discuss how IPFS works,  including its underlying components, and problems it solves.\n\n------ About Whiteboard Series --------\n\nThe Whiteboard Series with NEAR is a set of video interviews with Founders who are building scalable solutions for Blockchain.  We deep dive into their story, what they're developing and how it is helping the ecosystem. \n\nFollow the latest from NEAR Protocol on, \n\nWebsite: https://near.org/\nDiscord: https://near.chat\nTwitter:   / nearprotocol  \nGitHub: https://near.ai/github\n\n#WhiteboardSeries #Blockchain #FutureIsNEAR","channel_name":"NEAR Protocol","published_at":"2020-11-06T00:00:00.000Z","thumbnail_url":"https://img.youtube.com/vi/J-drqD2UebM/hqdefault.jpg","transcript_text":"hi everyone this is alex from nier uh with me today is adrien schmamann from ipfs uh and we will talk today about uh how a pfs works what are the underlying components and what problems it solves again would you like to introduce yourself and give us the high level overview of of ipfs yeah um so uh as alex said i'm a dean i'm the the tech lead for go ipfs which is the reference implementation uh for ipfs uh and ipfs stands for the the interplanetary file system and you know when i tell people that i work on something called the entertainer interplanetary file system i i sometimes get some some you know some looks all right so interplanetary file system so what do i mean when i say interplanetary right so you know let's say uh you know alice is on a trip she goes to alpha centauri you know which is you know about let's say five light years from earth right and she wants to download warren peace right alice wants the book war and peace right and these days she might go to the gutenberg project right so you know whatever https whatever you know gutenberg you know slash war and peace you know not not the correct url but close enough um and this is going to one this is gonna do a dns lookup right that takes us tina whatever the ip address 1.2.3.4 and then it's going to like actually ask for the data with like a you know uh an http request and and there's and that's the you know sort of the download um and each of these things will take many many years uh i mean even if you have dns caching and whatnot and so you skip straight to the download phase just the simple task of going all the way to earth and then getting your data and then coming all the way back is is too long um but you know maybe this is the best you can do in this scenario so let's let's see what happens if uh you know if we try something slightly different let's say that uh instead on alpha centauri we have both alice and bob they both went together bob was a knew that they'd be in for a long trip and so he brought with him sort of all these books right he bought with him all the gutenberg books and of course earth is still over here still very far away so what we want is instead of alice making this request right which is bad and slow we would like alice to instead make this request to bob um but there are security implications here right you know what if bob lies or what if bob gives the wrong data you know what if the data is wrong um and and this can be there are there are two types of the data being wrong right uh one is we'll call it like malware or spam or or whatnot and the other is but if it's just the wrong addition you know wrong wrong version no just as an example of like a wrong version thing i heard about recently which is sort of funny apparently when uh warren peace was added to uh you know the nook which is like you know a kindle competitor uh somebody seems to have done a a search and replace of the word kindle and replaced it with the word nook which meant that like words like kindled turned into nook and so like it was just not the same text anymore right and that's that's obviously not the same thing and not intentional but if you wanted to make the point that these things were different maybe you were actually looking for the wrong version so that you can make a point right um and alice can fix th can fix this right if she has some function that will verify you know it'll verify the bob data right you know bob's bites right and then it'll be you know either or whatever like true or false um and there's a number of functions that we could we could use here um but what we're going to do is we're going to use hash functions and the reason why hash functions are nice is that the reason why hash functions are nice uh is that you know sort of exactly what you are getting right they actually solve both of these problems right um right so instead of and this will change effectively what alice is asking for so alice is no longer alice is not gonna ask bob right sort of version one as she asks bob for our war and peace right but who knows what war and peace means and how do i know bob sent it to me correctly so this isn't gonna work uh instead maybe instead we're gonna have alice ask for some hash of war and peace right which means that she she sort of knows exactly which edition she wants or you can't hash it um or she couldn't have received a hash of what it might be um and uh it definitely won't be malware because when bob sends the bytes back he'll be able to ha she'll be able to hash it and we'll do a match so it's the hash of the file that that is being downloaded yeah right so exactly so this this thing is called and this is sort of the the ipfs thing right is content addressing i.e the way in which you refer to the content right the way in which you refer to the content is by what the content is itself right with this digital fingerprint this hash function of the data right um this means in a sense there's only one of these things right no matter where the data is stored no matter who has it no matter what system it's in there is only one piece of information that can correspond to this hash um and so now that you know sort of we're on the same page of what we're talking about which is how do i ask for data i ask for it by it for i ask for it by its content you know then ipfs can start to work on problems of how do i actually find how do i find this thing now right and so sort of the other you know ipfs thing is to somehow the other ipfs thing is to take a mapping of you know of a content address and turn it into you know some set of people who have the data you know we'll call them providers right providers of the data of the content right people who have it and are willing to give it to me right this is the other this is the function ipfs needs to provide once we're on the same page about content addressing um i want to back up a little bit on this this interplanetary business um i'm going to sort of make up a concept i haven't seen other people refer to explicitly and we'll see how it goes um which is we'll call it like the the the you know the benefits of this like interplanetary property right so you might say all right well i mean nobody's going alpha centauri anytime soon so why do i care um so yeah let's we'll call this this interplanetary thing right so we have obviously you know we have alice's five light years away here's the way uh we have alice's we'll call it alice and bob are infinity light years away or they're offline right as far as the network is concerned they might as well be infinite distance away they're never being connected right right alice and bob are offline um you've got things just like you know here on earth sometimes latency the speed of light just isn't quite good enough um you know the the ping from you know where i am in boston to australia is like 250 milliseconds uh you know there's some some bad ones from you know africa and australia to each other which are like 500 milliseconds right satellite communications get even worse you know just like long just like long pings you know even here on earth is is a big problem um and also just like bandwidth right some some connections are more expensive than others uh right maybe your your phone is metered uh but your what your your cellular network is metered but your wi-fi is not and so you would prefer to talk to people over over wi-fi than over the cellular network and the fact that you don't really have to care where the data comes from allows you to then optimize on top of this blank canvas and say where do i want to get the data from but but in general if i'm downloading well one file will it be downloading it from a single peer or does apfs provide something inherent inherently that will allow me to stream it from multiple people from multiple peers so this is why over here and you know i guess sorry this is i don't even know why i did this i'm not really like i write and go these days but it's not like uh not we'll call not my not my native language right uh this is this is a set of providers to the content right so yeah ipfs will let you get data from multiple people and this is like a thing that is working how it should in particular you're going to get hit um if you don't if as soon as you try and find multiple people if you don't know them all in advance right if you have to do any sort of exploring and from an application layer you might not you might know exactly where this data is coming it's supposed to be hosted or it might be hosted but if you're going to do any sort of exploring for where the data might be you're going to have to you're going to hit some latency cost right you're gonna have to do a little bit of exploring and that's gonna get some latency but if you can make it up in in throughput then it won't be such a big deal right if it's a if it's a you know a gigabyte file and it takes you an extra like two seconds to start downloading it doesn't really matter um yeah obviously you know for things like video and stuff could be sensitive that first little bit but maybe not yep uh all right so so this is sort of this interplanetary property the next is ipfs is immutable right um right we have this hash function and so right like hash of you know war and peace isn't changing book has been published but let's say uh you know we're working on uh you know working on some text file right it might be version one or working on an academic paper there might be version one of the paper and there might be a revised version two of the paper i this these will result in two different content addresses they're not the same thing uh if you think about this this gives us like you know if you go to wikipedia um it gives us like versioning and specificity right if you go to wikipedia and you you know you look at the references at the bottom and then you see like the references are out of date or they don't really make any sense right that's because you're getting this mutable link to a website right which is in which is in a sense confusing right it's not really the information you're looking for um you know this happens all the time with i mean like twitter for for sure right where people will put links to tweets and then they just like won't be there um but even for things that change right you go to a website the website's not there anymore the website looks totally different now uh but like something closer to apfs would be if if wikipedia was posting links to archive right uh um right now yeah yeah so if archived yeah archive maybe on a particular date right right because they're they're not if archives not going to change anything right they're promising they've downloaded it once they're stamping it then yeah it would be more like that right which would be which which is nice um and it also means that you can start to if you start to think about the immutable actually you know i'll come back to that in a minute before i go too bad done the immutability rabbit hole uh which is gonna take us for a little bit i just want to briefly address that we have ipfs as content addressing which is immutability right so this is fpfs um we can come back to this later but we have also there is there is ipns which is again interplanetary and there's a naming system for mutable data right uh which is from from mutability um but in order to get this you know we'll say these you know like the interplanetary properties uh it needs to be self-certified and so one way to do this and there are a number is uh i but i can't ask to use is author addressing right um again there's probably a more literature generic word but this is what it is it's the author is the address we use the author addressing because we're using public keys for addressing right and the author owns the corresponding private keys um right and these are just examples of you know self-certified right which means that the address tells you how to verify the data you got is correct right which helps if you're letting a random person just send you information right you need to know it's it's correct all right so back on the on the mutability on the immutability train right um once we know we're dealing with the immutable data we can start to think a little bit about um other other properties that help us so another one we'll call it is another one is deduplication right if i know that the data can't change and that this thing with this hash or this address is always exactly the same um i can leverage in a sense these you know slightly more more clever schemes or make things more efficient so say i have like an append only log you know or a blockchain all right and i have you know whatever four three two one and then i make a new operation five right and then five four three two one if i ask for the entire data structure from five having already downloaded the whole data structure from four i'm only going to download this because i know these two things are equivalent because they're they're hash based right but that's assuming you don't load every block separately right if if literally the entire data structure is a single file uh the application wouldn't trigger right um depends yes so you will have to think about how you structure your data to grant some to grant you some of these abilities right um you know if you take a i don't know let's say you have some you have some json file right uh and you want your china you're trying to figure out how do i how do i make any use of that how do i you know get some deduplication out of this you're probably not going to want to just like slice every like 10th byte and turn it treat it as arbitrary text right you're going to want to think a little bit about how you want to structure your data so that it looks a little more graph like right so that it has some of these linking properties that you might want which brings us to to the next thing i want to i want to talk about which is and for all of you out there i'm sorry about all the acronyms i swear um ipld which is again got this interplanetary property link data right which you know if you're familiar with like jason ld and etc is basically i just want i want to be able to describe sort of you know normal things like like lists and maps and you know and strings and integers um but i also want to be able to describe a link to another another piece of data right instead of just having nested structs all the way down i want to be able to like actually have a thing in here that says link here is where the other here's where the rest of the data lives um and an ipld is as core it's it's a very it's like a very generic form for how to describe this information and and how sort of the the generic properties will or how we arrive at this well we'll go into a little bit um but but high level from the linking is let's say you wanted to do i don't know you have some root oh um all right you have some root and uh you know it's got it's got you know number one over here on the left and it's got number two over here on the right right these two different nodes you might you might look at the structure and say that it's something like you know you know left is is you know number is you know number one and right is two right but where where one and two are these linked objects so there's some right sort of it's like the link to one and like the link to two um and so link in this case would be just the hash of data and where exactly where the link yeah yeah is is like the hash is the hash of of the one object and the hash of the two object um so in order to try and make these things you know i i've watched a number of the uh you know sort of great great video series you guys have have put together and uh there's a lot of merkle trees and a lot of different projects a lot of different blockchains and other sorts of uh projects in this you know d-web space um and everyone's doing it a little bit differently uh for pretty good reasons but the the underlying primitive is the same which is why you know when people say merkle tree they tend to just sort of gloss over exactly what it is right but it's just the hash you add up you know you have hashes and you have the objects in the child nodes and then you know uh in the child nodes and then the parent node is just the hash of the children nodes right um so so we want to you know build something that allows you to describe these uh these various different types of you know hash link data structures no matter how you create them right um and to do this as a base we're going to start with a with another project which is called multi-formats um and multi-formats is a really simple idea um the basic idea of multi-formats or is that you know for the low low cost of a few bytes you can know what the data you're talking about actually is um right uh now whatever a few bytes tells you you know what the data is so you know we'll go so one of these is multi-hash um people have different hash functions there are different hash functions people have very good reasons for preferences for for these um but as soon as you try and change the hash function you're using things start to get or you start to allow for the use of multiple different hash functions things start to become complicated because now how do i figure out whether this 256 you know bits of data is a sha-256 is it the first half of the digest of a sha-512 shot two five twelve is it a shot three it's 256 bits like what is this thing so we'll describe it as we'll have a code right just some some integer some some var end some variable length integer that you know will generally be like like one bite or something uh we'll have the digest size and bytes uh which will also be a var end and then we'll have the digest right and you know so i can say that for shot 256 shots to 256 this is you know the code is is you know 0x12 uh and i'm using the full thing which is you know which is 20. uh and then some bytes um next up will be will be so this tells me i can use whatever hash function i want for the links which is good now people serialize the data in different ways um so we need to account for that some people are gonna they're the ethereum blocks don't look like the bitcoin blocks which don't look like json which doesn't look like cyborg which right so we need some way to describe how we transform or how we abstract over sort of the core linking properties and for this we will use a coda a multi-codec um and this is really simple it's just a code it's just the code in some table some examples include you know uh there's one that's called dag sebor which is sort of you know include you know a way of using something that looks like seabor to to encode you know direct directed acyclic graphs uh we have you know we have like a bitcoin you know you can have a bitcoin transaction uh you can have raw data just say i don't know what this is you know you can have git you know and there's a whole bunch of these right and if your project is already using something that is or that is a uh content address format and then you want to start using any existing ipld tooling with it or ipfs related tooling with it you can just write a codec reserve the code in the table and your stuff can interoperate um and then there is there is you know you can combine these things and you have what we call a cid or a content identifier uh which people will sort of people will refer to content identifier as hash they'll sort of use these interchangeably but it's a little bit more um it is you know sort of the version number of the cid format right or the version codec um and then it's the coda and then it's the multi-codec and then it's the multi-hash right we've basically just glued these things together with again this extra property where hey we might need to change this thing in the future so if we just throw in another variant in front we can make sure that we can upgrade this if we need to so why is it a multi-hash or not just a hash um because if i'm using so i can be using say dag cyborg and the links can use sha-3 or shot two right daxybor just tells me how am i gonna like read this this these bytes off of disk and turn it into a graph i see right um but it doesn't tell me what specific type of link is being used in order to turn to access this graph um and so in this case multi-codec and multi-hash that's going to be two arrays of the same length no so some multi so a multi-codec and and maybe i can uh you know okay let's i'm i'm gonna move i'm gonna i'm gonna move on to the next thing and then hope and then we'll let me know if i if i covered it uh one more thing just for completionistic we have something which is which we use in cids which we call multi-base which says hey it's below the screen right now oh no thank you i will i will go down uh we have multi bass and that is the same concept as all the other you know multis which is for the low cost of a character i can figure out what base encoding this thing is is base do you like base64 do you like you know the bitcoin base 58 encoding do you like base 32 do you like hex this shouldn't be related to your data it's really just the way you display it and so we can just reserve some characters in a table that prefix the thing you know so we have like z is for base 58 or the bitcoin base 58 encoding you know we have whatever b is for base 32 and then we have you know we have say like f is you know base 16 or hex right so let's like let's look at an example so like an example hex encoded cid uh i like hex encoding sometimes makes it a little easier to see where all the bytes are right um would be now f for hex zero one because that's the codec for cid version one uh which is the one we're using right now uh there is one that is so if i just do like ipfs added a file what i will likely get is something like this um let's say i have the cd version like making version one i have a there's a codec called dag pb which is 70 and that uses shot 256 which is 12 and it uses all 256 eddie's all 256 bits of that which is 32 bytes and then we have the digest and that would be this is your cid so so what does multi stand for the multi is the fact that it can represent many things right so the multi base says any any base could go here and everything and this should you can substitute any base here and have a valid object right i see right uh multi codec says hey any ipl you know says like substitute any ipld codec in here right anything that will tell me and substitute any hash functions that's what the multi is for yeah i i misunderstood i thought that multicast refers to having multiple hashes there no no no it is a it is a good question but yeah no no that's uh yeah the multi is is for the the sort of the substitute ability uh property um and and so with our you know now that we are we are armed with cids that we can use as links everywhere now so that we sort of generalized this hash thing right for some context there existed cidv0 and cidv0 was didn't have a lot of these multi-things it had multi-hash but it was basically like oh yeah we'll just use a shot 256 and we'll assume that it's always you know it's always dag pb and and whatnot and it's always going to be base 58 encoded and like these things all eventually came to bite so you know to bite us at some level um and so working with primitives based on multi-formats which is like super lightweight and you can just implement the parts that you need right you don't need to implement implement base 32 in your application if you're only using base 16 and base 64. um i i like highly recommend using some of the multi-format stuff because it's super lightweight and will likely save you down the road um we recently ran into an issue where you know we like to you know ipfs people like to use ips and web browsers um and uh people like to use ips and web browsers and uh there is a finite length to dns labels um you can have whatever it's like 63 characters like you know something like that like 60 characters or something and we wanted to use base 32 encoding because it was efficient and whatnot but but for certain for certain pieces of information we wanted to put in the into the url it was just a little bit over the limit it was like a couple characters over so we switched it to base 36 encoding instead and that was not painful right or not not particularly painful um because it's just another way to represent the data and i guess just for for someone completionist's sake hello can you give me a time check briefly because my uh my timer reset over here how we doing i think we're like 40 minutes in maybe a little less okay all right i think we're i think we're good on time um so but we'll see how it goes uh so the ipld data model right so i guess we'll say like the the base thing that you might want to to know about ipld is you know it's going to have all the normal json things like we talked about you know whatever it's sort of you know uh the ipld data model um it's basically you know uh you know sort of you know normal json data model uh plus we want raw bytes because these are really nice for things you want to be efficient and we want links right um and then you can path you can path through things so say i have some some cid encoded in base 32 whatever you know you know baffy abc dot dot that can refer to some content but i can also path into it so i can say like baffy abc slash you know you know field one or whatever you know my slash you know whatever hello right like you know field two right like i can path into it by just you know treating it like it's destruct and i'm going through map elements right like if you're writing this in you know your your favorite coding language you might be using dots instead of slashes here right but you're sort of pathing and traversing through um is that that that makes sense to seem reasonable yeah okay i'm very muted yes yes that makes sense yeah okay cool all right i have to switch back windows double check all right so um a little while ago you brought up this this point which i was very good and i totally glossed over it i was hoping my answer would be good enough to last me until now which i think maybe it did uh which as you said well you know this deduplication stuff doesn't really work so well if everything is all in one file and you're totally right you're gonna have to figure out for it is nice for for other reasons relating to call them like protocol level sympathies and also this deduplication where you might want to chunk the file up a big file you know up into smaller pieces right even if you're talking about something that's just raw bytes and it's not going to have a dag structure and we're not going to even if you're not going to create some graph structure around this which is highly optimized and you're going to use some existing thing like a zip file you may still want to chunk up the pieces or you know or a video or whatever you may still want to chunk up the pieces smaller so you can do things like um so you can do things like allow multiple people to download pieces at the same time right in order for me to download some data from a ran from an arbitrary person and be okay with the fact that i don't trust them it needs to have some hash associated with it right some way i can verify the data if the only unit of verifiability is gigabyte blocks then i have to download a whole gigabyte before i know if that's any good which is one problem and i also can't ask two different people to give me half of the file because what if one of them is lying i don't know who lied i can only hash the whole thing together so i want to make these things into a smaller more digestible unit um so we have one of these things which we call unix fs which is just uh unix fs you know which basically just takes you know file system things file system objects and turns them into dags and it's not the only way to do this i don't think it's not the best way to do this we've come up with other things we'd like to do since then that you know are being are being worked on um right because even you know you have things like you have a file system and the file system you know maybe has has fi has like a has like a directory and the directory has another directory which has files you know and that one has files and so on um and you need to somehow turn it into some dag where everything is a little more broken up somehow arbitrarily right your files are too big you want to chunk them up your directories are too big you need to chunk them up um and there's all sorts of ways you can do this if we talk about files in particular you could do something simple which is you could just say hey i'm just gonna make these all like these are all just like you know 256 you know kilobyte blocks you could do that uh this is okay if you're gonna make like an append only kind of thing right because then you're just gonna like you'll just keep adding more to the end um uh you just keep adding more to the end but it's not great if you're gonna insert something in the middle right because then you're gonna like throw this whole chunking scheme out of whack and your deduplication is gonna go down um so for this you can use there are like content-based chunkers that take into account the data that you are hashing in order to figure out or the data that you are processing to figure out where would be a good place to break it up and those those are a little more amenable to allowing for like extra data in the middle but this is like just you know sort of like one area of exploration and relates to just one particular type of codec or one particular type of not even the codec this is a layout on top of it this is right how do i create my data on top of the existing codec that just helps me do effectively serialization of the dag right right you see these two things are separate like the structure of the data is independent from like the serialization format i happen to be using yes though i think it brings in an interesting point that in a very simple model right so like at the very beginning we started with a model where there's a file and there's a hash of it right and so in that module i think it was implied uh that the hash is predefined uh in that case if there is a particular file that file has only one address right but once we have multiple hashes multiple ways to chunk it up suddenly the same war and peace can can exist in in multiple like if i'm uploading war and peace the probability that it will that the hash will be the same as the same yeah uploaded by someone else is very little now right right so so so totally so this is this is like this is a really i even if we don't really have that much time i think we should spend a few minutes on this because it's like really interesting and and like there's there's some like some some nuance here which is like absolutely so in in sort of in the abstract sense right these two things that are chunked up differently are not the same right like if mike if my machine has like you know two megabytes of ram and the and one version of this had things in one megabyte blocks and the other had five megabyte blocks like i could reproduce one thing like one of these i'd be like i'm having problems with this one dag but not the other one right so these things are not the same but yet they are right like the thing the thing that most people care about when describing this is basically what happens when i concatenate all the leaf nodes is what does that look like right um and that and there's multiple right you can have uh you know there is in the sense there there's a there's a one hat like hash goes to data exactly once you know one this is a one-to-one mapping but data can go to many different like hashes or cids right if we're talking about the set like the maybe this is the backwards way to describe it right but but yeah that's that's totally right um in figuring out whether when right it's the other way around it's it's like it depends on how you think about it i guess whether data means like the data i've received over the wire or data means the concatenation of the leaf nodes that you care about i would as i was writing it i was like i don't even know what the right way is to describe this which is like a little it's a little tricky um but the question is like does this matter um and i think there are certain cases where it does matter but those in those cases are we'll call them like uh i don't want to say legacy because legacy seems to imply like you know we are the future they are the past but i'll say like interoperability with existing systems right um what would be nice to have is if i could go if i go to the ubuntu website and i try and download the and i see the iso and they post the sha-256 of the iso if i could search the ipfs network to say hey i'm looking for the shot i'm looking for this like shot 256 anyone know where it is um but someone could be storing it using the shot three someone like there's any number of things that could change here even without the dag itself right even just the hash function referencing it could change um and so it would be nice if i could do that um and and we can talk about that in a sec but for a lot of cases right no one really wants to interact with us with a hash right like no one really wants to look at these long strings of numbers and letters right these can occur behind the scenes so i don't go to most urls by typing them in the address bar right that's that's this is not what happens uh certainly anything beyond like the root of the website you know you know google.com etc i don't type into the web browser i like navigate to it through some other linking and so the linking is is precise like if i if i upload some data if i add some data set and then people start using that data set they're going to they're not going to be re-adding it to they're not going to be re-adding it and chunking it themselves they're going to be using my graph they're going to be downloading the graph from me and then re-sharing that one right in a sense this is like how the web this is like how the web is built and so it's one of these things that like it it hurts but it doesn't hurt that bad um and there are i think schemes that you could use to get around this um you know you could have schemes some of which could be like kind of really you know you could go like you know full trust and just or you know not full trust but you know high trust and be like okay i'm just going to ask you to send me the shot 256 of the whole file if you have it you'll keep a separate index that has it for for files in particular and then like you could send it to me and i have to download the whole thing before i can check it i could do that but it's a little but that would be a little unfortunate right we'd lose some of these other properties uh you could take advantage of the fact that things like sha-1 and shot 2 are streaming hashes and i could download the file backwards so that i could keep confirming that the i could download it from the end going back so that i could keep confirming that that the blocks i'm downloading can hash into the final results um i could use some sort of you know zero knowledge proof to to prove that two dags are equivalent under some rep under some set of properties that i care about uh i think those are interesting but they're not the they're not it's it's not brutal that they're not there um okay so with some time remaining because got we got a bunch of other stuff to cover here um all right so we've talked basically about uh the why we care about content addressing and then what the content address data structures look like uh there is also like how we how we transport things and how we actually um how we transport data around um what and you know so exchanging data exchanging data and uh finding who has it right uh who has who has the data um ipfs is built on the p2p uh um which is a peer-to-peer networking stack or using it for for all sorts of stuff right the idea is to make it sort of pluggable to what you need do you need tcp or websockets or quick you know or you do you want tls or noise like all sorts of you know what kind of multiplexer do you want it has like all these components sort of in there for you and it also has some higher level peer-to-peer components like um like you know a pub sub network uh and a distributed hash table um so we're gonna for finding who has the data uh all right first we'll talk about how we exchange data so we have two protocols for exchanging data uh the one which is mostly in which is mostly in use now is called bitswap and the other one is called graphsync um not going to talk too much about graphsync because it's not what is commonly used with ipfs right now although is it's sort of it is it is newer and very useful for for reasons that will sort of become obvious as we look at bitswap um and bitswa uh and this finding who has the data is mostly the dht although also a little bit of bit slop uh which we'll we'll come to um so bit swap works like this uh we have alice and we have bob and we have some graph um or we have we have a set of people we have whatever we have bob we have charlie we have dan uh and we have some graph you know uh you know a b c d e f g um and alice wants to download the entire graph rooted at a right so the root note of this thing is a so she'll send she'll ask for she'll ask for a she'll get it back she'll ask for b and c she'll get back and then she'll ask for you know d e f g and she'll get them back and these can be divided up between all of the uh all of the peers right the more peers there are the more i can send off the blocks right so obviously for the first round a i can't chunk that one up i don't have enough information i don't know what else i need uh b and c i can send to different people the efg i can send to three this whole thing is sort of very right pretty simple and straightforward right and allows us given this primitive we can then start to build on some of the more smarts right um let me send let me send requests to people who respond faster and send me data quicker and uh you know let me penalize people who are who are problematic or our connections drop um you know uh things like that of course the unfortunate thing here is like obviously in the case where bob has the data is the only person who has the data right if bob is the only person who has the data um or your data looks like it looks like a chain instead of like uh if your data is deep instead of wide right if it looks like this you're going to be really sad because this whole process is latency bound right latency because like in order to know what the next block is to ask for i have to have already gotten the previous block so i can look at what its children are um and so and so graph sync which is uh i guess in many ways the successor um you can you know alice can send some like selector along with a so she could say like you know a you know plus everything underneath for example right or you could ask for subsets of a you know i want a b and everything underneath and i want to ask someone else for ac and everything underneath so you can you can do more of that um this will probably make its way into ipvs you know in in the future but it requires you might imagine this is a little more complicated to do all of this like uh we call it session management all of the smarts around who do i ask for data when right how do i divide up all the tasks um all right so this is this is transfer protocols um bitswap is just like uh uh so so bitswap is effective i'm traversing the graph locally i'm just sending requests and receiving responses yeah so so bitswap is basically it swaps bits it knows nothing about the structure of graphs there's a thing that lives above it that knows what graphs are and so it tries to traverse the graph and it's like i want a bit swap can you or do i have a locally no bit swap can you get me a sure and then it says okay well now i need b bit swap do you have b and and so on um and again it uses the sessions to keep track of like the various smarts of who and when it should ask for stuff um but obviously graph sync you could you can ask for if you can if you are a graph aware you can make more intelligent decisions around how to download the whole graph right right like downloading downloading a blockchain using bitswap would like not be nearly as fun as using graphsync because you know the latency boundness of this long chain would be could be could be pretty brutal right um yeah now now the question is uh finding data so there are there are two ways in which we find data um no all right two ways in which we find data uh one is uh we just ask everyone we're connected to we are connected to right i mean those are those are like pretty cheap right it's like one you know just like send a packet do you have stuff and they're like yes or no um and if yes they get added to the session and then we don't have to broadcast to everyone anymore right because you've already found the limited subset of people who actually have some stake have some interest in this data uh and the second is we ask the dht um you know ask the dht and say who has this right the dhc does not have the data we asked the dht who has the data right so this is really instead of finding data it's finding who has data um so uh running a little long time so i we can go into some of this a little bit uh here is the crash course to cademya which is the uh dh which is sort of the the the structure of the dht that we use we do some things a little bit differently but that's like the the main the main underlying component is it can never is based on kedamalia um okay uh all right this is um the main underlying idea is that you have all these peers that are sort of server nodes in your dht and you've ordered them right based on some you know these are ordered uh based on some metric below below the screen again yeah sorry these are ordered based on some some metric um right these are going to be ordered by the sha-256 of the peer ids of each node you know and a peer id is basically the is basically it's basically a public key yeah it may be a hash of a public key also if the public key is too big rsa keys we use hashes of public keys elliptic curve keys we just use the public keys because they're small enough um but they all have to be in the same space no matter what key type they are so we shot 256 all of them we ordered by sha256 you know as integer um interpreted as integers okay and now they're all ordered um and what we're going to do is basically treat this thing like a skip list right so in a skip list you are connected to people you know one two four eight away right and so you can get to any you know you can get to like the 70th element you can do all of those in like logarithmic number of hops um but you can't just use a skip list because computers go on and off the network all the time right so like imagine you were trying to build a skip list and chunks of your memory started disappearing and new parts started popping in at like random intervals like that would not be happy so instead um what we do is every peer has a routing table um every pier has a routing table uh and ah sorry um has a routing table where you're basically tracking the peers that are closest to you in that are closest to you in reality we're actually using the the xor metric so um what it'll be is i'll say you know i want to keep track of the peers where you know say uh zero i want to keep track of the peers where uh you know me you know xor you know pierre um is like where we have zero we have zero bits in common right where we have our common prefix of bits is zero um so yes the metric is what happens underneath i guess this is really um you know common prefix length is is zero and then one and i thought you know to 256. although realistically i mean you really only need like 15 of these things or 20 of these things right because the the closer you go the odds are that somebody is going to be that close to you is zero right nobody's gonna have all their bits in common with you yep right it's like breaking a hash function um and and i'll keep maybe you know we'll i'll keep some number of these guys maybe i'll keep 20 maximum in each bucket and so that will allow me to sort of jump around the routing table because when i everyone always knows the people closest to them everyone knows always the the 20 people closest to them because those guys those those will be down here you know in this end and those tables will be sparse i'm not going to run out of slots for them uh and so i can always get traverse closer and closer because i ask this node hey do you know anyone closer to the the target i'm looking for and they say yeah i know this guy and you ask them do you know anyone closer and so on until you find the target um and and in order to sort of i guess the the generic form of this is that when we do a when we do a put slash get uh we will take we'll call like the sort of the key and then we'll we'll sort of we'll shot 256 it to put it into the same space and then we'll say the rule is that you know the some numbers say the 20 closest people 20 closest peers to the key store it they store they they store that record um and in the case of ipfs these records are what we call provider records so there's not just one record they're they're additive right because you know alice says i have the data and bob says i have the data and charlie says i have the data and they all end up in the same in the same place they're additive on the same key is that that sound good makes sense okay that makes sense okay and this is and this is basically kid emilia um has logarithmic lookup time logarithms are are good um they they make this like really small the number of hops you need to find data should be like three at most right um as these networks get large and maybe get up to like four i mean like as these networks start to get like really large they go up but the constant number of hops is like not that big and there are there there if you get if you start to get really concerned as some of the late with some of the latency of these things you're like well this one kilobyte text file i don't want to wait or web page i don't want to wait for three hops three hops is a whole bunch of hundreds of milliseconds there are other schemes you could use uh to to speed things up right the other schemes like coral which uh which focus on like how do i create latency ring based dhts so that i sort of create smaller groups before i go up to the larger groups so that i can find the data closest to me more efficiently right so there's other sorts of schemes here but reminder it does not matter what this routing system right the dht is like not intrinsic it's like it's like a thing that ipfs has to help you find stuff right but you can find content however you want to you can plug in some external mechanism uh if you're using a building ipf as a library it's like pretty easy as long as it fulfills this contract of give i give you a cid and you give me people who have the cid you're good to go and even if you're using it as a binary if you if you have some ideas as to who might have it you could connect to those peers and then request the data via your external system and this whole stuff with bitswap asking everyone will guarantee that you get it too um so the you know and there are people like there are people who are building on top of ipfs in various ways who um you know we've improved the dht a lot in the last year in particular and there's still there's still some really like you know nice big big chunks of work that uh we've got ahead um but for the people who've had like more specialized these cases where they they need to find other ways to do this they've been able to build other routing systems or they are they're in the process of doing so because why not ipfs is this flat space right everybody's hashes they live in the shared space right if there's a c id of data stored in swarm all right because swarm can use cids to refer to data and we have a swarm codec which we do which there is right then realistically i i could find i could look for swarm data in ikfs and if anyone who who was publishing it and you know if i knew where to go look for them i could set up a routing system that way if they were providing and saying i have this data and telling everyone in the public dht that they have it i could find the swarm data in ipfs and that would just work so i would say like the key like i guess the the the key takeaway for me at least with this stuff is like content addressing is really nice because i i do not have to care where the data comes from not just from like a which ip address has a perspective but even from a system perspective right there there are there are blockchains that incentivize data storage and it doesn't matter which one of those you would use right it doesn't really matter if i'm storing the data on amazon it doesn't matter if i'm running my own server um it i can just it doesn't i don't have to depend on sort of the success of other systems i don't have to be tied so deeply to them i can diversify in the same way i don't have to be tied to the success of a particular sterilization format by pld i can diversify same with all of these things the idea is to the stack has been designed to try and make it as easy to swap out things as possible so that we don't have to guess in advance what's going to become successful we can adapt to the ecosystem and how it is evolving um and and i think that's that is something i would i would recommend other projects to to look at because i that's been i think very helpful for us um it also helps with our engagement with with other people in the web3 ecosystem is like we just we just want to work with them because we want to make sure that if they have good ideas that like people who exactly fast can make use of their good ideas yep um right this sort of like collaborative open source thing where like you don't need to use everything you just need to use the parts you need um and i think we are uh we are we are out of time but uh we can talk about naming another time because mutability is a whole other can of worms yeah yeah i think we covered a lot of stuff today yeah yeah thanks a lot for giving us the uh the overview and yeah that was very exciting all right thank you happy uh happy to be here you","transcript_synced_at":"2026-11-04T04:23:01.738Z","segments":[{"start_seconds":4,"text":"hi everyone this is alex from nier uh with me today is adrien schmamann from ipfs uh and we will talk today about uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4s"},{"start_seconds":13,"text":"how a pfs works what are the underlying components and what problems it solves again would you like to introduce yourself and give us the high level","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=13s"},{"start_seconds":21,"text":"overview of of ipfs yeah um so uh as alex said i'm a dean","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=21s"},{"start_seconds":29,"text":"i'm the the tech lead for go ipfs which is the reference implementation uh for ipfs uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=29s"},{"start_seconds":37,"text":"and ipfs stands for the the interplanetary file system","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=37s"},{"start_seconds":46,"text":"and you know when i tell people that i work on something called the entertainer interplanetary file system i i sometimes get some","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=46s"},{"start_seconds":54,"text":"some you know some looks all right so interplanetary file system","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=54s"},{"start_seconds":61,"text":"so what do i mean when i say interplanetary right so you know let's say uh you know alice is on a trip","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=61s"},{"start_seconds":69,"text":"she goes to alpha centauri you know which is you know about","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=69s"},{"start_seconds":78,"text":"let's say five light years from earth right and she wants to download warren peace","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=78s"},{"start_seconds":90,"text":"right alice wants the book war and peace right and these days she might go to the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=90s"},{"start_seconds":99,"text":"gutenberg project right so you know whatever https","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=99s"},{"start_seconds":106,"text":"whatever you know gutenberg you know slash war and peace you know not not the correct url but close enough","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=106s"},{"start_seconds":115,"text":"um and this is going to one this is gonna do a dns lookup","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=115s"},{"start_seconds":122,"text":"right that takes us tina whatever the ip address 1.2.3.4","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=122s"},{"start_seconds":129,"text":"and then it's going to like actually ask for the data with like a you know uh an http request and and there's and","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=129s"},{"start_seconds":138,"text":"that's the you know sort of the download um and each of these things will take many","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=138s"},{"start_seconds":145,"text":"many years uh i mean even if you have dns caching and whatnot and so you skip straight to the download phase just the simple task of going all the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=145s"},{"start_seconds":154,"text":"way to earth and then getting your data and then coming all the way back is is too long um but you know maybe this is the best","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=154s"},{"start_seconds":162,"text":"you can do in this scenario so let's let's see what happens if uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=162s"},{"start_seconds":175,"text":"you know if we try something slightly different let's say that","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=175s"},{"start_seconds":183,"text":"uh instead on alpha centauri we have both alice and bob","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=183s"},{"start_seconds":190,"text":"they both went together bob was a knew that they'd be in for a long trip and so he brought with him sort of all these books right he bought with him all","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=190s"},{"start_seconds":198,"text":"the gutenberg books and of course earth is still over here still very far away so what we want is instead of alice","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=198s"},{"start_seconds":207,"text":"making this request right which is bad and slow we would like alice to instead make this request to bob","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=207s"},{"start_seconds":215,"text":"um but there are security implications here right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=215s"},{"start_seconds":224,"text":"you know what if bob lies or what if bob gives the wrong data you know what if the data is wrong","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=224s"},{"start_seconds":233,"text":"um and and this can be there are there are two types of the data being wrong","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=233s"},{"start_seconds":241,"text":"right uh one is we'll call it like malware or spam or or whatnot","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=241s"},{"start_seconds":248,"text":"and the other is but if it's just the wrong addition you know wrong wrong version no just as an example of","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=248s"},{"start_seconds":256,"text":"like a wrong version thing i heard about recently which is sort of funny apparently when uh warren peace was","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=256s"},{"start_seconds":263,"text":"added to uh you know the nook which is like you know a kindle competitor uh somebody seems to have done a a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=263s"},{"start_seconds":271,"text":"search and replace of the word kindle and replaced it with the word nook which meant that like words like kindled turned into nook and so like it was just not the same","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=271s"},{"start_seconds":280,"text":"text anymore right and that's that's obviously not the same thing and not intentional but if you wanted to make the point that these things were","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=280s"},{"start_seconds":288,"text":"different maybe you were actually looking for the wrong version so that you can make a point right um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=288s"},{"start_seconds":296,"text":"and alice can fix th can fix this right if she has some function that will verify","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=296s"},{"start_seconds":303,"text":"you know it'll verify the bob data right you know bob's bites right and then","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=303s"},{"start_seconds":310,"text":"it'll be you know either or whatever like true or false um and there's a number of functions that we could we could use here","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=310s"},{"start_seconds":320,"text":"um but what we're going to do is we're going to use hash functions and the reason why hash functions are","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=320s"},{"start_seconds":328,"text":"nice is that the reason why hash functions are nice","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=328s"},{"start_seconds":338,"text":"uh is that you know sort of exactly what you are getting right they actually solve both of these problems","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=338s"},{"start_seconds":347,"text":"right um right so instead of and this will change effectively what alice is asking for so alice is no longer","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=347s"},{"start_seconds":356,"text":"alice is not gonna ask bob right sort of version one as she asks bob for our war and peace","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=356s"},{"start_seconds":363,"text":"right but who knows what war and peace means and how do i know bob sent it to me correctly so this isn't gonna work","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=363s"},{"start_seconds":370,"text":"uh instead maybe instead we're gonna have alice ask for some hash of","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=370s"},{"start_seconds":377,"text":"war and peace right which means that she she sort of knows exactly which edition she wants or you can't hash it","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=377s"},{"start_seconds":384,"text":"um or she couldn't have received a hash of what it might be um and","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=384s"},{"start_seconds":392,"text":"uh it definitely won't be malware because when bob sends the bytes back he'll be able to ha she'll be able to hash it and we'll do a match","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=392s"},{"start_seconds":399,"text":"so it's the hash of the file that that is being downloaded yeah right so exactly so this this thing is called and this is sort of the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=399s"},{"start_seconds":407,"text":"the ipfs thing right is content addressing i.e the way in which you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=407s"},{"start_seconds":414,"text":"refer to the content right the way in which you refer to the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=414s"},{"start_seconds":422,"text":"content is by what the content is itself right with this digital fingerprint this hash function of the data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=422s"},{"start_seconds":431,"text":"right um this means in a sense there's only one of these things right no matter where the data is stored no matter who","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=431s"},{"start_seconds":440,"text":"has it no matter what system it's in there is only one piece of information that can correspond to this hash","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=440s"},{"start_seconds":450,"text":"um and so now that you know sort of we're on the same page of what we're talking about which is how do i ask for data i ask for it by it","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=450s"},{"start_seconds":458,"text":"for i ask for it by its content you know then ipfs can start to work on problems of how do i actually find","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=458s"},{"start_seconds":465,"text":"how do i find this thing now right and so sort of the other you know ipfs thing","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=465s"},{"start_seconds":472,"text":"is to somehow the other ipfs thing is to take a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=472s"},{"start_seconds":481,"text":"mapping of you know of a content address","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=481s"},{"start_seconds":494,"text":"and turn it into you know some set of people","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=494s"},{"start_seconds":502,"text":"who have the data you know we'll call them providers right providers of the data of the content","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=502s"},{"start_seconds":510,"text":"right people who have it and are willing to give it to me right this is the other this is the function ipfs needs to provide once we're on the same page about content","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=510s"},{"start_seconds":518,"text":"addressing um i want to back up a little bit on this this interplanetary business","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=518s"},{"start_seconds":527,"text":"um i'm going to sort of make up a concept i haven't seen other people refer to explicitly and we'll see how it goes um which is","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=527s"},{"start_seconds":536,"text":"we'll call it like the the the you know the benefits of this like interplanetary property right so you might say all right well i mean","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=536s"},{"start_seconds":543,"text":"nobody's going alpha centauri anytime soon so why do i care um so yeah let's we'll call this this","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=543s"},{"start_seconds":551,"text":"interplanetary thing right so we have obviously you know we","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=551s"},{"start_seconds":559,"text":"have alice's five light years away here's the way uh we have alice's we'll","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=559s"},{"start_seconds":569,"text":"call it alice and bob are infinity light years away or they're offline right as far as the network is concerned","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=569s"},{"start_seconds":577,"text":"they might as well be infinite distance away they're never being connected right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=577s"},{"start_seconds":583,"text":"right alice and bob are offline um you've got things just like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=583s"},{"start_seconds":591,"text":"you know here on earth sometimes latency the speed of light just isn't quite good enough um you know the the ping from","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=591s"},{"start_seconds":599,"text":"you know where i am in boston to australia is like 250 milliseconds uh you know there's some some bad ones from","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=599s"},{"start_seconds":608,"text":"you know africa and australia to each other which are like 500 milliseconds right satellite communications get even worse","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=608s"},{"start_seconds":616,"text":"you know just like long just like long pings you know even here on earth is is a big problem um and also just","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=616s"},{"start_seconds":624,"text":"like bandwidth right some some connections are more expensive than others","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=624s"},{"start_seconds":630,"text":"uh right maybe your your phone is metered","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=630s"},{"start_seconds":638,"text":"uh but your what your your cellular network is metered but your wi-fi is not and so you would prefer to talk to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=638s"},{"start_seconds":645,"text":"people over over wi-fi than over the cellular network and the fact that you don't really have to care","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=645s"},{"start_seconds":653,"text":"where the data comes from allows you to then optimize on top of this blank canvas and say where do i want to get the data from","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=653s"},{"start_seconds":661,"text":"but but in general if i'm downloading well one file will it be downloading it from a single peer or does apfs provide something inherent","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=661s"},{"start_seconds":668,"text":"inherently that will allow me to stream it from multiple people from multiple peers so this is why over here and you know i guess sorry this is i don't even know why i","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=668s"},{"start_seconds":677,"text":"did this i'm not really like i write and go these days but it's not like uh not we'll call not my not my native language","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=677s"},{"start_seconds":684,"text":"right uh this is this is a set of providers to the content right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=684s"},{"start_seconds":691,"text":"so yeah ipfs will let you get data from multiple people and this is like a thing that is working how it should in","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=691s"},{"start_seconds":698,"text":"particular you're going to get hit um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=698s"},{"start_seconds":706,"text":"if you don't if as soon as you try and find multiple people if you don't know them all in advance right if you have to do any sort of exploring","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=706s"},{"start_seconds":713,"text":"and from an application layer you might not you might know exactly where this data is coming it's supposed to be hosted or it might be hosted but if you're going to do any sort of","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=713s"},{"start_seconds":722,"text":"exploring for where the data might be you're going to have to you're going to hit some latency cost right you're gonna have to do a little bit of exploring and","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=722s"},{"start_seconds":730,"text":"that's gonna get some latency but if you can make it up in in throughput","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=730s"},{"start_seconds":737,"text":"then it won't be such a big deal right if it's a if it's a you know a gigabyte file and it takes you an extra like two seconds to start","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=737s"},{"start_seconds":744,"text":"downloading it doesn't really matter um yeah obviously you know for things like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=744s"},{"start_seconds":752,"text":"video and stuff could be sensitive that first little bit but maybe not yep","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=752s"},{"start_seconds":759,"text":"uh all right so so this is sort of this interplanetary property the next is ipfs is immutable right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=759s"},{"start_seconds":770,"text":"um right we have this hash function and so right like hash of you know war and peace isn't changing book has been","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=770s"},{"start_seconds":779,"text":"published but let's say uh you know we're working on uh you know working on some text file right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=779s"},{"start_seconds":787,"text":"it might be version one or working on an academic paper there might be version one of the paper and there might be a revised version two of the paper","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=787s"},{"start_seconds":795,"text":"i this these will result in two different content addresses they're not the same thing uh if you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=795s"},{"start_seconds":802,"text":"think about this this gives us like you know if you go to wikipedia","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=802s"},{"start_seconds":810,"text":"um it gives us like versioning and specificity right if you go to wikipedia and you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=810s"},{"start_seconds":818,"text":"you know you look at the references at the bottom and then you see like the references are out of date or they don't really make any sense right that's because you're getting this mutable link to a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=818s"},{"start_seconds":826,"text":"website right which is in which is in a sense confusing right it's not really the information you're looking for","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=826s"},{"start_seconds":834,"text":"um you know this happens all the time with i mean like twitter for","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=834s"},{"start_seconds":843,"text":"for sure right where people will put links to tweets and then they just like won't be there um but even for things that change right you go to a website the website's not","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=843s"},{"start_seconds":851,"text":"there anymore the website looks totally different now uh but like something closer to apfs would be if if wikipedia was posting links to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=851s"},{"start_seconds":859,"text":"archive right uh um right now","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=859s"},{"start_seconds":865,"text":"yeah yeah so if archived yeah archive maybe on a particular date right right because they're they're not","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=865s"},{"start_seconds":873,"text":"if archives not going to change anything right they're promising they've downloaded it once they're stamping it then yeah it would be more like that right which would be which which is nice","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=873s"},{"start_seconds":882,"text":"um and it also means that you can start to if you start to think","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=882s"},{"start_seconds":890,"text":"about the immutable actually you know i'll come back to that in a minute before i go too bad done the immutability rabbit hole uh which is gonna","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=890s"},{"start_seconds":898,"text":"take us for a little bit i just want to briefly address that we have ipfs as content addressing which is immutability","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=898s"},{"start_seconds":905,"text":"right so this is fpfs um we can come back to this later but we have also","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=905s"},{"start_seconds":912,"text":"there is there is ipns which is again interplanetary and there's a naming system","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=912s"},{"start_seconds":920,"text":"for mutable data right uh which is from from mutability","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=920s"},{"start_seconds":928,"text":"um but in order to get this you know we'll say these you know like the interplanetary properties uh it needs to be self-certified","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=928s"},{"start_seconds":937,"text":"and so one way to do this and there are a number is uh i but i can't ask to use is author addressing","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=937s"},{"start_seconds":946,"text":"right um again there's probably a more literature generic word but","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=946s"},{"start_seconds":952,"text":"this is what it is it's the author is the address we use the author addressing because we're using public keys for addressing right and the author owns the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=952s"},{"start_seconds":961,"text":"corresponding private keys um right and these are just","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=961s"},{"start_seconds":967,"text":"examples of you know self-certified right which means that the address","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=967s"},{"start_seconds":977,"text":"tells you how to verify the data you got is correct right which helps if you're letting a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=977s"},{"start_seconds":986,"text":"random person just send you information right you need to know it's it's correct all right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=986s"},{"start_seconds":993,"text":"so back on the on the mutability on the immutability train right um once we know we're dealing with","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=993s"},{"start_seconds":1002,"text":"the immutable data we can start to think a little bit about um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1002s"},{"start_seconds":1009,"text":"other other properties that help us so another one we'll call it is another one is deduplication","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1009s"},{"start_seconds":1016,"text":"right if i know that the data can't change and that this thing with this hash or","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1016s"},{"start_seconds":1022,"text":"this address is always exactly the same um i can leverage in a sense","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1022s"},{"start_seconds":1029,"text":"these you know slightly more more clever schemes or make things more efficient so say i have like an append only log","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1029s"},{"start_seconds":1036,"text":"you know or a blockchain all right and i have you know whatever four three two one","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1036s"},{"start_seconds":1043,"text":"and then i make a new operation five right and then five four three two one","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1043s"},{"start_seconds":1052,"text":"if i ask for the entire data structure from five having already downloaded the whole data structure from four","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1052s"},{"start_seconds":1061,"text":"i'm only going to download this because i know these two things are equivalent because they're","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1061s"},{"start_seconds":1068,"text":"they're hash based right but that's assuming you don't load every block separately right if if literally the entire data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1068s"},{"start_seconds":1076,"text":"structure is a single file uh the application wouldn't trigger right um depends yes","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1076s"},{"start_seconds":1085,"text":"so you will have to think about how you structure your data to grant some to grant you some of these abilities right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1085s"},{"start_seconds":1091,"text":"um you know if you take a i don't know let's say you have some you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1091s"},{"start_seconds":1098,"text":"have some json file right uh and you want your china you're trying to figure out how do i how do i make any use of that how do i you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1098s"},{"start_seconds":1107,"text":"know get some deduplication out of this you're probably not going to want to just like slice every like 10th byte and turn it treat it as arbitrary text","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1107s"},{"start_seconds":1115,"text":"right you're going to want to think a little bit about how you want to structure your data so that it looks a little more graph like right so that","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1115s"},{"start_seconds":1123,"text":"it has some of these linking properties that you might want which brings us to to the next thing i want to i want to talk about","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1123s"},{"start_seconds":1132,"text":"which is and for all of you out there i'm sorry about all the acronyms i swear um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1132s"},{"start_seconds":1140,"text":"ipld which is again got this interplanetary property link data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1140s"},{"start_seconds":1148,"text":"right which you know if you're familiar with like jason ld and etc is basically i just want","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1148s"},{"start_seconds":1155,"text":"i want to be able to describe sort of you know normal things like like lists and maps and you know","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1155s"},{"start_seconds":1162,"text":"and strings and integers um but i also want to be able to describe a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1162s"},{"start_seconds":1169,"text":"link to another another piece of data right instead of just having nested structs all the way down i want to be able to like actually","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1169s"},{"start_seconds":1177,"text":"have a thing in here that says link here is where the other here's where the rest of the data lives um and an ipld","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1177s"},{"start_seconds":1186,"text":"is as core it's it's a very it's like a very generic form for how to describe this","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1186s"},{"start_seconds":1195,"text":"information and and how sort of the the generic properties will or how we arrive at this well we'll go","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1195s"},{"start_seconds":1202,"text":"into a little bit um but but high level from the linking is let's say you wanted to do","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1202s"},{"start_seconds":1210,"text":"i don't know you have some root oh um all right you have some root","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1210s"},{"start_seconds":1219,"text":"and uh you know it's got it's got you know number one over here on the left and it's got number two over here on the right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1219s"},{"start_seconds":1227,"text":"right these two different nodes you might you might look at the structure and say that it's something like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1227s"},{"start_seconds":1233,"text":"you know you know left is is you know number","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1233s"},{"start_seconds":1240,"text":"is you know number one and right is two right but where where one and two","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1240s"},{"start_seconds":1248,"text":"are these linked objects so there's some right sort of it's like the link to one and like the link to two","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1248s"},{"start_seconds":1256,"text":"um and so link in this case would be just the hash of data and where exactly where the link yeah yeah is is like the hash is the hash of of","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1256s"},{"start_seconds":1265,"text":"the one object and the hash of the two object um so in order to try and make these","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1265s"},{"start_seconds":1272,"text":"things you know i i've watched a number of the uh you know sort of great great video series you guys have have","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1272s"},{"start_seconds":1279,"text":"put together and uh there's a lot of merkle trees and a lot of different projects a lot of different blockchains and other sorts of","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1279s"},{"start_seconds":1287,"text":"uh projects in this you know d-web space um and everyone's doing it a little bit differently uh for pretty good reasons but the the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1287s"},{"start_seconds":1296,"text":"underlying primitive is the same which is why you know when people say merkle tree they tend to just sort of gloss over exactly what it is right but it's just the hash you add up","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1296s"},{"start_seconds":1305,"text":"you know you have hashes and you have the objects in the child nodes and then you know uh in the child nodes","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1305s"},{"start_seconds":1313,"text":"and then the parent node is just the hash of the children nodes right um so so we want to you know build","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1313s"},{"start_seconds":1322,"text":"something that allows you to describe these uh these various different types of you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1322s"},{"start_seconds":1330,"text":"know hash link data structures no matter how you create them right um and to do this","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1330s"},{"start_seconds":1337,"text":"as a base we're going to start with a with another project which is called multi-formats","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1337s"},{"start_seconds":1345,"text":"um and multi-formats is","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1345s"},{"start_seconds":1352,"text":"a really simple idea um the basic idea of multi-formats","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1352s"},{"start_seconds":1358,"text":"or is that you know for the low low cost of a few bytes you can know what the data you're","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1358s"},{"start_seconds":1365,"text":"talking about actually is um right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1365s"},{"start_seconds":1372,"text":"uh now whatever a few bytes","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1372s"},{"start_seconds":1382,"text":"tells you you know what the data is so you know we'll go so one of these is","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1382s"},{"start_seconds":1390,"text":"multi-hash um people have different hash functions","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1390s"},{"start_seconds":1397,"text":"there are different hash functions people have very good reasons for preferences for for these um but as soon as you try and change the hash function you're using","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1397s"},{"start_seconds":1406,"text":"things start to get or you start to allow for the use of multiple different hash functions things start to become complicated because now how do i figure out whether","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1406s"},{"start_seconds":1414,"text":"this 256 you know bits of data is a sha-256 is it the first half of the digest of a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1414s"},{"start_seconds":1423,"text":"sha-512 shot two five twelve is it a shot three it's 256 bits like what is this thing so we'll describe it as we'll have a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1423s"},{"start_seconds":1431,"text":"code right just some some integer some some var end some","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1431s"},{"start_seconds":1438,"text":"variable length integer that you know will generally be like like one bite or something","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1438s"},{"start_seconds":1444,"text":"uh we'll have the digest size and bytes uh which will also be a var","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1444s"},{"start_seconds":1452,"text":"end and then we'll have the digest right and you know so","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1452s"},{"start_seconds":1461,"text":"i can say that for shot 256","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1461s"},{"start_seconds":1467,"text":"shots to 256 this is you know the code is is you know","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1467s"},{"start_seconds":1473,"text":"0x12 uh and i'm using the full thing which is","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1473s"},{"start_seconds":1480,"text":"you know which is 20. uh and then some bytes um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1480s"},{"start_seconds":1488,"text":"next up will be will be so this tells me i can use whatever hash function i want for the links which is good now","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1488s"},{"start_seconds":1497,"text":"people serialize the data in different ways um so we need to account for that some people are gonna they're the ethereum","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1497s"},{"start_seconds":1504,"text":"blocks don't look like the bitcoin blocks which don't look like json which doesn't look like cyborg which right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1504s"},{"start_seconds":1511,"text":"so we need some way to describe how we transform","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1511s"},{"start_seconds":1518,"text":"or how we abstract over sort of the core linking properties and for this we will use a coda a multi-codec","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1518s"},{"start_seconds":1527,"text":"um and this is really simple it's just a code","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1527s"},{"start_seconds":1534,"text":"it's just the code in some table some examples include you know","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1534s"},{"start_seconds":1541,"text":"uh there's one that's called dag sebor which is sort of you know include you know a way","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1541s"},{"start_seconds":1549,"text":"of using something that looks like seabor to to encode you know direct directed acyclic graphs","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1549s"},{"start_seconds":1557,"text":"uh we have you know we have like a bitcoin you know you can have a bitcoin transaction","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1557s"},{"start_seconds":1566,"text":"uh you can have raw data just say i don't know what this is you know you can have git you know and there's a whole bunch of these","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1566s"},{"start_seconds":1573,"text":"right and if your project is already using something that is or that is a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1573s"},{"start_seconds":1581,"text":"uh content address format and then you want to start using any existing ipld tooling with it or ipfs related tooling","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1581s"},{"start_seconds":1590,"text":"with it you can just write a codec reserve the code in the table and your","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1590s"},{"start_seconds":1597,"text":"stuff can interoperate um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1597s"},{"start_seconds":1606,"text":"and then there is there is you know you can combine these things and you have what we call a cid","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1606s"},{"start_seconds":1614,"text":"or a content identifier","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1614s"},{"start_seconds":1625,"text":"uh which people will sort of people will refer to content identifier as hash they'll sort of use these interchangeably","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1625s"},{"start_seconds":1632,"text":"but it's a little bit more um it is you know sort of the version","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1632s"},{"start_seconds":1639,"text":"number of the cid format right or the version codec um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1639s"},{"start_seconds":1648,"text":"and then it's the coda and then it's the multi-codec","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1648s"},{"start_seconds":1655,"text":"and then it's the multi-hash right we've basically just glued these things together with again this extra property where hey","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1655s"},{"start_seconds":1663,"text":"we might need to change this thing in the future so if we just throw in another variant in front we can make sure that we can upgrade this if we need to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1663s"},{"start_seconds":1670,"text":"so why is it a multi-hash or not just a hash um because if i'm using","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1670s"},{"start_seconds":1678,"text":"so i can be using say dag cyborg and the links can use sha-3 or shot two right daxybor just tells me how am i gonna","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1678s"},{"start_seconds":1687,"text":"like read this this these bytes off of disk and turn it into a graph i see right um but it doesn't tell me","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1687s"},{"start_seconds":1697,"text":"what specific type of link is being used in order to turn to access this graph","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1697s"},{"start_seconds":1705,"text":"um and so in this case multi-codec and multi-hash that's going to be two arrays of the same length no so","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1705s"},{"start_seconds":1713,"text":"some multi so a multi-codec and and maybe i can uh you know okay let's i'm i'm gonna move","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1713s"},{"start_seconds":1721,"text":"i'm gonna i'm gonna move on to the next thing and then hope and then we'll let me know if i if i covered it uh one more thing just","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1721s"},{"start_seconds":1728,"text":"for completionistic we have something which is which we use in cids which we call multi-base","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1728s"},{"start_seconds":1734,"text":"which says hey it's below the screen right now oh no thank you i will i will go down","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1734s"},{"start_seconds":1742,"text":"uh we have multi bass and that is the same concept as all the other you know multis","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1742s"},{"start_seconds":1750,"text":"which is for the low cost of a character i can figure out what base encoding this thing is is base do you like base64 do you like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1750s"},{"start_seconds":1759,"text":"you know the bitcoin base 58 encoding do you like base 32 do you like hex this shouldn't be related to your data it's really just the way you display it","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1759s"},{"start_seconds":1768,"text":"and so we can just reserve some characters in a table that prefix the thing you know so we have like z is","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1768s"},{"start_seconds":1776,"text":"for base 58 or the bitcoin base 58 encoding you know we have whatever b is for base 32","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1776s"},{"start_seconds":1785,"text":"and then we have you know we have say like f is you know base 16 or hex right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1785s"},{"start_seconds":1793,"text":"so let's like let's look at an example so like an example hex encoded cid uh i like hex encoding sometimes makes","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1793s"},{"start_seconds":1801,"text":"it a little easier to see where all the bytes are right um would be","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1801s"},{"start_seconds":1808,"text":"now f for hex zero one because that's the codec for","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1808s"},{"start_seconds":1815,"text":"cid version one uh which is the one we're using right now uh there is one that is so if i just do","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1815s"},{"start_seconds":1822,"text":"like ipfs added a file what i will likely get is something like this um let's say i have the cd version like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1822s"},{"start_seconds":1830,"text":"making version one i have a there's a codec called dag pb","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1830s"},{"start_seconds":1837,"text":"which is 70 and that uses shot 256 which is 12 and it uses all 256 eddie's","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1837s"},{"start_seconds":1846,"text":"all 256 bits of that which is 32 bytes and then","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1846s"},{"start_seconds":1853,"text":"we have the digest and that would be this is your cid","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1853s"},{"start_seconds":1864,"text":"so so what does multi stand for the multi is the fact that it can represent many things right so the multi base says any","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1864s"},{"start_seconds":1874,"text":"any base could go here and everything and this should you can substitute any base here and have a valid object","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1874s"},{"start_seconds":1880,"text":"right i see right uh multi codec says hey any ipl you know says like substitute any ipld codec in here","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1880s"},{"start_seconds":1888,"text":"right anything that will tell me and substitute any hash functions that's what the multi is for yeah i i misunderstood i thought that multicast refers to having multiple hashes there","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1888s"},{"start_seconds":1897,"text":"no no no it is a it is a good question but yeah no no that's uh yeah the multi is is for the the sort of the substitute ability","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1897s"},{"start_seconds":1905,"text":"uh property um and and so with our","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1905s"},{"start_seconds":1913,"text":"you know now that we are we are armed with cids that we can use as links everywhere now so that we sort of generalized this hash","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1913s"},{"start_seconds":1920,"text":"thing right for some context there existed cidv0 and cidv0 was didn't have a lot of these multi-things","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1920s"},{"start_seconds":1929,"text":"it had multi-hash but it was basically like oh yeah we'll just use a shot 256 and we'll assume that it's always you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1929s"},{"start_seconds":1938,"text":"know it's always dag pb and and whatnot and it's always going to be base 58 encoded and like these things all eventually came to bite so you know to bite us at","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1938s"},{"start_seconds":1946,"text":"some level um and so working with primitives based on multi-formats which is like super lightweight and you can just implement","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1946s"},{"start_seconds":1954,"text":"the parts that you need right you don't need to implement implement base 32 in your application if you're only using base 16 and base 64.","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1954s"},{"start_seconds":1962,"text":"um i i like highly recommend using some of the multi-format stuff because it's super lightweight and will likely save you down the road","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1962s"},{"start_seconds":1971,"text":"um we recently ran into an issue where you know we like to you know ipfs people like to use ips and web browsers","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1971s"},{"start_seconds":1979,"text":"um and uh people like to use ips and web browsers and","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1979s"},{"start_seconds":1986,"text":"uh there is a finite length to dns labels um you can have whatever it's like 63","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1986s"},{"start_seconds":1995,"text":"characters like you know something like that like 60 characters or something and we wanted to use base 32 encoding because it was efficient and whatnot but","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=1995s"},{"start_seconds":2003,"text":"but for certain for certain pieces of information we wanted to put in the into the url it was just a little bit over the limit it was like a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2003s"},{"start_seconds":2011,"text":"couple characters over so we switched it to base 36 encoding instead and that was not painful right or not not","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2011s"},{"start_seconds":2020,"text":"particularly painful um because it's just another way to represent the data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2020s"},{"start_seconds":2029,"text":"and i guess just for for someone completionist's sake hello can you give me a time check briefly because my uh my timer reset over here how we doing i think we're like 40","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2029s"},{"start_seconds":2038,"text":"minutes in maybe a little less okay all right i think we're i think we're good on time um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2038s"},{"start_seconds":2047,"text":"so but we'll see how it goes uh so the ipld data model right so i guess we'll say like the the base thing that you might want to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2047s"},{"start_seconds":2056,"text":"to know about ipld is you know it's going to have all the normal json things like we talked about you know whatever it's sort of you know uh the ipld data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2056s"},{"start_seconds":2065,"text":"model um it's basically","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2065s"},{"start_seconds":2071,"text":"you know uh you know sort of you know normal","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2071s"},{"start_seconds":2079,"text":"json data model uh plus we want raw bytes because these are really nice for things you want to be","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2079s"},{"start_seconds":2088,"text":"efficient and we want links right um and then you can path","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2088s"},{"start_seconds":2096,"text":"you can path through things so say i have some some cid encoded in base 32 whatever you know","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2096s"},{"start_seconds":2103,"text":"you know baffy abc dot dot that can refer to some content but i can also","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2103s"},{"start_seconds":2110,"text":"path into it so i can say like baffy abc slash you know you know field one","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2110s"},{"start_seconds":2118,"text":"or whatever you know my slash you know whatever hello right like you know field two right like i can path into it","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2118s"},{"start_seconds":2125,"text":"by just you know treating it like it's destruct and i'm going through map elements right like if","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2125s"},{"start_seconds":2132,"text":"you're writing this in you know your your favorite coding language you might be using dots instead of slashes here right but you're sort of pathing and","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2132s"},{"start_seconds":2140,"text":"traversing through um is that that that makes sense to seem reasonable","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2140s"},{"start_seconds":2153,"text":"yeah okay i'm very muted yes yes that makes sense yeah okay cool all right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2153s"},{"start_seconds":2160,"text":"i have to switch back windows double check all right so um a little while ago you brought up","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2160s"},{"start_seconds":2167,"text":"this this point which i was very good and i totally glossed over it i was hoping my answer would be good enough to last me until now which i","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2167s"},{"start_seconds":2175,"text":"think maybe it did uh which as you said well you know this deduplication stuff doesn't really work so well","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2175s"},{"start_seconds":2181,"text":"if everything is all in one file and you're totally right you're gonna have to figure out","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2181s"},{"start_seconds":2190,"text":"for it is nice for for other reasons relating to call them like protocol level sympathies and also this deduplication","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2190s"},{"start_seconds":2198,"text":"where you might want to chunk the file up a big file you know up into smaller pieces right even if you're talking about","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2198s"},{"start_seconds":2207,"text":"something that's just raw bytes and it's not going to have a dag structure and we're not going to even if you're not going to create some graph structure around this which is highly optimized and you're going to use","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2207s"},{"start_seconds":2215,"text":"some existing thing like a zip file you may still want to chunk up the pieces or you know or a video or whatever you may","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2215s"},{"start_seconds":2224,"text":"still want to chunk up the pieces smaller so you can do things like um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2224s"},{"start_seconds":2232,"text":"so you can do things like allow multiple people to download pieces at the same time right in order for me to download some data from a ran from an arbitrary person","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2232s"},{"start_seconds":2241,"text":"and be okay with the fact that i don't trust them it needs to have some hash associated with it right some way i can verify the data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2241s"},{"start_seconds":2249,"text":"if the only unit of verifiability is gigabyte blocks then i have to download a whole gigabyte before i know if that's any good which","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2249s"},{"start_seconds":2257,"text":"is one problem and i also can't ask two different people to give me half of the file because what if one of them is lying i don't know who lied","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2257s"},{"start_seconds":2265,"text":"i can only hash the whole thing together so i want to make these things into a smaller more digestible unit um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2265s"},{"start_seconds":2273,"text":"so we have one of these things which we call unix fs","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2273s"},{"start_seconds":2277,"text":"which is just uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2277s"},{"start_seconds":2285,"text":"unix fs you know which basically just takes you know file system things file system","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2285s"},{"start_seconds":2292,"text":"objects and turns them into dags and it's not the only way to do this i","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2292s"},{"start_seconds":2300,"text":"don't think it's not the best way to do this we've come up with other things we'd like to do since then that you know are being are being worked on","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2300s"},{"start_seconds":2308,"text":"um right because even you know you have things like you have a file system and the file system you know maybe has","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2308s"},{"start_seconds":2315,"text":"has fi has like a has like a directory and the directory has another directory which has files you know and that one","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2315s"},{"start_seconds":2323,"text":"has files and so on um and you need to somehow turn it into some dag where everything is a little more broken up","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2323s"},{"start_seconds":2331,"text":"somehow arbitrarily right your files are too big you want to chunk them up your directories are too big you need to chunk them up","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2331s"},{"start_seconds":2338,"text":"um and there's all sorts of ways you can do this if we talk about files in particular you could do something simple which is you could just say hey i'm just gonna make","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2338s"},{"start_seconds":2347,"text":"these all like these are all just like you know 256 you know kilobyte blocks you could do that uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2347s"},{"start_seconds":2356,"text":"this is okay if you're gonna make like an append only kind of thing right because then you're just gonna like you'll just keep adding","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2356s"},{"start_seconds":2364,"text":"more to the end um uh you just keep adding more to the end","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2364s"},{"start_seconds":2373,"text":"but it's not great if you're gonna insert something in the middle right because then you're gonna like throw this whole chunking scheme out of whack and your deduplication is gonna go down","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2373s"},{"start_seconds":2381,"text":"um so for this you can use there are like content-based chunkers that take into account the data that you are hashing in order to figure out","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2381s"},{"start_seconds":2390,"text":"or the data that you are processing to figure out where would be a good place to break it up and those those are a little more amenable to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2390s"},{"start_seconds":2398,"text":"allowing for like extra data in the middle but this is like just you know sort of like one area of exploration and relates to just one particular","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2398s"},{"start_seconds":2406,"text":"type of codec or one particular type of not even the codec this is a layout on top of it this is right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2406s"},{"start_seconds":2413,"text":"how do i create my data on top of the existing codec that just helps me do effectively serialization of the dag","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2413s"},{"start_seconds":2422,"text":"right right you see these two things are separate like the structure of the data is independent from like the serialization format i happen to be","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2422s"},{"start_seconds":2430,"text":"using yes though i think it brings in an interesting point that in a very simple model right so like at the very beginning we started","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2430s"},{"start_seconds":2438,"text":"with a model where there's a file and there's a hash of it right and so in that module i think it was implied uh that the hash is predefined uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2438s"},{"start_seconds":2446,"text":"in that case if there is a particular file that file has only one address right but once we have multiple hashes multiple ways to chunk it up suddenly the same","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2446s"},{"start_seconds":2455,"text":"war and peace can can exist in in multiple like if i'm uploading war and peace the probability that it will","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2455s"},{"start_seconds":2461,"text":"that the hash will be the same as the same yeah uploaded by someone else is very little now right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2461s"},{"start_seconds":2468,"text":"right so so so totally so this is this is like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2468s"},{"start_seconds":2475,"text":"this is a really i even if we don't really have that much time i think we should spend a few minutes on this because it's like really interesting","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2475s"},{"start_seconds":2483,"text":"and and like there's there's some like some some nuance here which is like absolutely so","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2483s"},{"start_seconds":2492,"text":"in in sort of in the abstract sense right these two things that are chunked up differently","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2492s"},{"start_seconds":2500,"text":"are not the same right like if mike if my machine has like you know two megabytes of ram and the and one version of this had things in one megabyte blocks and","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2500s"},{"start_seconds":2508,"text":"the other had five megabyte blocks like i could reproduce one thing like one of these i'd be like i'm having problems with this one dag but not the other one","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2508s"},{"start_seconds":2516,"text":"right so these things are not the same but yet they are right like the thing the thing that most people care about when describing this","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2516s"},{"start_seconds":2524,"text":"is basically what happens when i concatenate all the leaf nodes is what does that look like right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2524s"},{"start_seconds":2531,"text":"um and that and there's multiple right you can have uh you know there is in the sense there there's a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2531s"},{"start_seconds":2539,"text":"there's a one hat like hash goes to data exactly once","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2539s"},{"start_seconds":2549,"text":"you know one this is a one-to-one mapping but data can go to many different","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2549s"},{"start_seconds":2559,"text":"like hashes or cids right if we're talking about the set like the maybe this is the backwards way to describe it","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2559s"},{"start_seconds":2567,"text":"right but but yeah that's that's totally right um in figuring out whether","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2567s"},{"start_seconds":2574,"text":"when right it's the other way around it's it's like it depends on how you think about it i guess whether data means like the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2574s"},{"start_seconds":2581,"text":"data i've received over the wire or data means the concatenation of the leaf nodes that you care about i would as i was writing it i was like i don't even know what the right way is to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2581s"},{"start_seconds":2589,"text":"describe this which is like a little it's a little tricky um but the question is like does this matter","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2589s"},{"start_seconds":2596,"text":"um and i think there are certain cases where it does matter but those in those cases are we'll call them like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2596s"},{"start_seconds":2604,"text":"uh i don't want to say legacy because legacy seems to imply like you know we are the future they are the past but i'll say like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2604s"},{"start_seconds":2612,"text":"interoperability with existing systems right um what would be nice to have is if i","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2612s"},{"start_seconds":2620,"text":"could go if i go to the ubuntu website and i try and download the and i see the iso and they post the sha-256 of the iso","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2620s"},{"start_seconds":2628,"text":"if i could search the ipfs network to say hey i'm looking for the shot i'm looking for this like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2628s"},{"start_seconds":2636,"text":"shot 256 anyone know where it is um but someone could be storing it using the shot three someone like there's any","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2636s"},{"start_seconds":2645,"text":"number of things that could change here even without the dag itself right even just the hash function referencing it could change um and so it would be nice if i could do","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2645s"},{"start_seconds":2653,"text":"that um and and we can talk about that in a sec but for a lot of cases","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2653s"},{"start_seconds":2663,"text":"right no one really wants to interact with us with a hash right like no one really wants to look at these long strings of numbers","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2663s"},{"start_seconds":2670,"text":"and letters right these can occur behind the scenes so","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2670s"},{"start_seconds":2678,"text":"i don't go to most urls by typing them in the address bar right that's that's this is not what happens uh certainly anything","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2678s"},{"start_seconds":2687,"text":"beyond like the root of the website you know you know google.com etc i don't type into the web browser","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2687s"},{"start_seconds":2694,"text":"i like navigate to it through some other linking and so the linking is is precise like if","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2694s"},{"start_seconds":2704,"text":"i if i upload some data if i add some data set and then people start using that data set they're going to they're not going to be re-adding it to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2704s"},{"start_seconds":2712,"text":"they're not going to be re-adding it and chunking it themselves they're going to be using my graph they're going to be downloading the graph from me and then re-sharing that","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2712s"},{"start_seconds":2720,"text":"one right in a sense this is like how the web this is like how the web is built and so it's one of these things that like it it","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2720s"},{"start_seconds":2729,"text":"hurts but it doesn't hurt that bad um and there are i think schemes that you could use to get around this","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2729s"},{"start_seconds":2737,"text":"um you know you could have schemes some of which could be like kind of really you know you could go","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2737s"},{"start_seconds":2746,"text":"like you know full trust and just or you know not full trust but you know high trust and be like okay i'm just going to ask you to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2746s"},{"start_seconds":2753,"text":"send me the shot 256 of the whole file if you have it you'll keep a separate index that has it for for files in particular and then","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2753s"},{"start_seconds":2762,"text":"like you could send it to me and i have to download the whole thing before i can check it i could do that but it's a little but that would be a little unfortunate right we'd lose some of these other properties","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2762s"},{"start_seconds":2770,"text":"uh you could take advantage of the fact that things like sha-1 and shot 2 are streaming hashes and i could download the file backwards","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2770s"},{"start_seconds":2778,"text":"so that i could keep confirming that the i could download it from the end going back so that i could keep confirming","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2778s"},{"start_seconds":2785,"text":"that that the blocks i'm downloading can hash into the final results um i could use some sort of you know","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2785s"},{"start_seconds":2795,"text":"zero knowledge proof to to prove that two dags are equivalent under some rep under some set of properties that i care","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2795s"},{"start_seconds":2803,"text":"about uh i think those are interesting but they're not the they're not it's it's not brutal that","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2803s"},{"start_seconds":2810,"text":"they're not there um okay so with some time remaining because","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2810s"},{"start_seconds":2818,"text":"got we got a bunch of other stuff to cover here um all right so we've talked basically about uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2818s"},{"start_seconds":2828,"text":"the why we care about content addressing and then what the content address data structures look like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2828s"},{"start_seconds":2835,"text":"uh there is also like how we how we transport things and how we actually um how we transport","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2835s"},{"start_seconds":2844,"text":"data around um what and you know so exchanging data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2844s"},{"start_seconds":2853,"text":"exchanging data and uh finding who has it","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2853s"},{"start_seconds":2861,"text":"right uh who has who has the data um ipfs is built on the p2p","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2861s"},{"start_seconds":2870,"text":"uh um which is a peer-to-peer networking stack or using it for","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2870s"},{"start_seconds":2878,"text":"for all sorts of stuff right the idea is to make it sort of pluggable to what you need do you need tcp or websockets or quick you know or you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2878s"},{"start_seconds":2886,"text":"do you want tls or noise like all sorts of you know what kind of multiplexer do you want it has like all these components sort of","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2886s"},{"start_seconds":2893,"text":"in there for you and it also has some higher level peer-to-peer components like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2893s"},{"start_seconds":2900,"text":"um like you know a pub sub network uh and a distributed hash table","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2900s"},{"start_seconds":2908,"text":"um so we're gonna for finding who has the data uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2908s"},{"start_seconds":2917,"text":"all right first we'll talk about how we exchange data so we have two protocols for exchanging data uh the one which is mostly in","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2917s"},{"start_seconds":2925,"text":"which is mostly in use now is called bitswap and the other one is called graphsync um not going to talk too much about","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2925s"},{"start_seconds":2934,"text":"graphsync because it's not what is commonly used with ipfs right now although is it's sort of it is it is","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2934s"},{"start_seconds":2941,"text":"newer and very useful for for reasons that will sort of become obvious as we look at bitswap um and bitswa","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2941s"},{"start_seconds":2949,"text":"uh and this finding who has the data is mostly the dht although also a little bit of bit slop","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2949s"},{"start_seconds":2960,"text":"uh which we'll we'll come to um so bit swap works like this uh we have alice","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2960s"},{"start_seconds":2969,"text":"and we have bob and we have some graph um or we have we have a set of people we","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2969s"},{"start_seconds":2976,"text":"have whatever we have bob we have charlie we have dan uh and we have some graph","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2976s"},{"start_seconds":2985,"text":"you know uh you know a b c d e f g","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2985s"},{"start_seconds":2993,"text":"um and alice wants to download the entire graph rooted at a right so the root note of this thing is","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=2993s"},{"start_seconds":3001,"text":"a so she'll send she'll ask for she'll ask for a she'll get it back","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3001s"},{"start_seconds":3011,"text":"she'll ask for b and c she'll get back and then she'll ask for you know d e f g","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3011s"},{"start_seconds":3020,"text":"and she'll get them back and these can be divided up between","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3020s"},{"start_seconds":3026,"text":"all of the uh all of the peers right the more peers there are the more i can send off the blocks right so","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3026s"},{"start_seconds":3033,"text":"obviously for the first round a i can't chunk that one up i don't have enough information i don't know what else i need uh b and c i can send to different people","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3033s"},{"start_seconds":3042,"text":"the efg i can send to three this whole thing is sort of very right pretty simple and straightforward right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3042s"},{"start_seconds":3048,"text":"and allows us given this primitive we can then start to build on some of the more smarts right um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3048s"},{"start_seconds":3057,"text":"let me send let me send requests to people who respond faster and send me data quicker and uh you know let me penalize people who","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3057s"},{"start_seconds":3065,"text":"are who are problematic or our connections drop um you know uh things like that","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3065s"},{"start_seconds":3074,"text":"of course the unfortunate thing here is like obviously in the case where bob has the data is the only person who","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3074s"},{"start_seconds":3082,"text":"has the data right if bob is the only person who has the data um or your data looks like it looks like a chain instead of","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3082s"},{"start_seconds":3090,"text":"like uh if your data is deep instead of wide right if it looks like this you're going to be really sad because this whole process is latency","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3090s"},{"start_seconds":3099,"text":"bound right latency because like in order to know","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3099s"},{"start_seconds":3106,"text":"what the next block is to ask for i have to have already gotten the previous block so i can look at what its children are","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3106s"},{"start_seconds":3113,"text":"um and so and so graph sync which is uh i guess in many ways the successor","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3113s"},{"start_seconds":3121,"text":"um you can you know alice can send some","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3121s"},{"start_seconds":3130,"text":"like selector along with a so she could say like you know a you know plus everything underneath","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3130s"},{"start_seconds":3138,"text":"for example right or you could ask for subsets of a you know i want a b and everything underneath and i want to ask someone else for ac and everything underneath so you can","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3138s"},{"start_seconds":3146,"text":"you can do more of that um this will probably make its way into ipvs you know in in the future but it requires you might","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3146s"},{"start_seconds":3155,"text":"imagine this is a little more complicated to do all of this like uh we call it session management all of the smarts around who do i ask for data when right how do","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3155s"},{"start_seconds":3164,"text":"i divide up all the tasks um all right so this is this is transfer protocols","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3164s"},{"start_seconds":3170,"text":"um bitswap is just like uh uh so so bitswap is effective i'm","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3170s"},{"start_seconds":3178,"text":"traversing the graph locally i'm just sending requests and receiving responses yeah so so bitswap is basically it swaps","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3178s"},{"start_seconds":3185,"text":"bits it knows nothing about the structure of graphs there's a thing that lives above it that knows what graphs are and so it tries to traverse the graph","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3185s"},{"start_seconds":3192,"text":"and it's like i want a bit swap can you or do i have a locally no bit swap can you get me a sure and then it says okay well now i","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3192s"},{"start_seconds":3201,"text":"need b bit swap do you have b and and so on um and again it uses the sessions to keep track of like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3201s"},{"start_seconds":3210,"text":"the various smarts of who and when it should ask for stuff um but obviously graph sync you could","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3210s"},{"start_seconds":3217,"text":"you can ask for if you can if you are a graph aware you can make more intelligent decisions around how to download the whole graph","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3217s"},{"start_seconds":3226,"text":"right right like downloading downloading a blockchain using bitswap would like not be nearly as fun as using graphsync","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3226s"},{"start_seconds":3234,"text":"because you know the latency boundness of this long chain would be could be could be pretty brutal right","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3234s"},{"start_seconds":3243,"text":"um yeah now now the question is uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3243s"},{"start_seconds":3250,"text":"finding data so there are there are two ways in which we find data um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3250s"},{"start_seconds":3258,"text":"no all right two ways in which we find data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3258s"},{"start_seconds":3266,"text":"uh one is uh we just ask everyone we're","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3266s"},{"start_seconds":3275,"text":"connected to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3275s"},{"start_seconds":3282,"text":"we are connected to right i mean those are those are like pretty cheap right it's like one you know just like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3282s"},{"start_seconds":3290,"text":"send a packet do you have stuff and they're like yes or no um and if yes they get added to the session","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3290s"},{"start_seconds":3299,"text":"and then we don't have to broadcast to everyone anymore right because you've already found the limited subset of people who actually have some stake have some interest in this data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3299s"},{"start_seconds":3308,"text":"uh and the second is we ask the dht","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3308s"},{"start_seconds":3313,"text":"um you know ask the dht and say who has this right the dhc does not have the data we","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3313s"},{"start_seconds":3321,"text":"asked the dht who has the data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3321s"},{"start_seconds":3326,"text":"right so this is really instead of finding data it's finding who has data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3326s"},{"start_seconds":3334,"text":"um so uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3334s"},{"start_seconds":3341,"text":"running a little long time so i we can go into some of this a little bit uh here is the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3341s"},{"start_seconds":3348,"text":"crash course to cademya which is the uh dh which is sort of the the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3348s"},{"start_seconds":3356,"text":"the structure of the dht that we use we do some things a little bit differently but that's like the the main the main underlying component is it can never","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3356s"},{"start_seconds":3364,"text":"is based on kedamalia um okay uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3364s"},{"start_seconds":3372,"text":"all right this is um the main underlying idea is that you have all these peers","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3372s"},{"start_seconds":3380,"text":"that are sort of server nodes in your dht and you've ordered them right based on","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3380s"},{"start_seconds":3388,"text":"some you know these are ordered uh based on some metric below below the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3388s"},{"start_seconds":3395,"text":"screen again yeah sorry these are ordered based on some some metric um right these are going to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3395s"},{"start_seconds":3404,"text":"be ordered by the sha-256","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3404s"},{"start_seconds":3411,"text":"of the peer ids of each node you know and a peer id","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3411s"},{"start_seconds":3419,"text":"is basically the is basically","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3419s"},{"start_seconds":3426,"text":"it's basically a public key yeah it may be a hash of a public key also if the public key is too big rsa keys we use hashes of public keys elliptic curve","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3426s"},{"start_seconds":3435,"text":"keys we just use the public keys because they're small enough um but they all have to be in the same space no matter what key type they are","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3435s"},{"start_seconds":3442,"text":"so we shot 256 all of them we ordered by sha256 you know as integer","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3442s"},{"start_seconds":3449,"text":"um interpreted as integers okay and now they're all ordered um and what we're going to do is basically treat this thing like a skip","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3449s"},{"start_seconds":3457,"text":"list right so in a skip list you are connected to people you know one two four eight away","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3457s"},{"start_seconds":3465,"text":"right and so you can get to any you know you can get to like the 70th element you can do all of those in like logarithmic","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3465s"},{"start_seconds":3471,"text":"number of hops um but you can't just use a skip list because computers go on and off the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3471s"},{"start_seconds":3479,"text":"network all the time right so like imagine you were trying to build a skip list and chunks of your memory started disappearing and new parts started popping in at like random","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3479s"},{"start_seconds":3487,"text":"intervals like that would not be happy so instead um what we do is every peer has","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3487s"},{"start_seconds":3496,"text":"a routing table um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3496s"},{"start_seconds":3509,"text":"every pier has a routing table uh and","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3509s"},{"start_seconds":3516,"text":"ah sorry um has a routing table","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3516s"},{"start_seconds":3524,"text":"where","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3524s"},{"start_seconds":3528,"text":"you're basically tracking the peers that are closest to you in that are closest to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3528s"},{"start_seconds":3536,"text":"you in reality we're actually using the the xor metric so um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3536s"},{"start_seconds":3544,"text":"what it'll be is i'll say you know i want to keep track of the peers","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3544s"},{"start_seconds":3551,"text":"where you know say uh zero i want to keep track of the peers where","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3551s"},{"start_seconds":3559,"text":"uh you know me you know xor you know","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3559s"},{"start_seconds":3566,"text":"pierre um is like where we have zero we have zero bits in common","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3566s"},{"start_seconds":3574,"text":"right where we have our common prefix of bits is zero um","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3574s"},{"start_seconds":3582,"text":"so yes the metric is what happens underneath i guess this is really um you know","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3582s"},{"start_seconds":3591,"text":"common prefix length","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3591s"},{"start_seconds":3596,"text":"is is zero and then one and i thought you know to 256. although realistically i mean you really only","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3596s"},{"start_seconds":3604,"text":"need like 15 of these things or 20 of these things right because the the closer you go the odds are that somebody is going to be that close to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3604s"},{"start_seconds":3612,"text":"you is zero right nobody's gonna have all their bits in common with you yep right it's like breaking a hash function","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3612s"},{"start_seconds":3620,"text":"um and and i'll keep maybe you know we'll i'll keep some number of these guys maybe i'll keep 20 maximum in","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3620s"},{"start_seconds":3628,"text":"each bucket and so that will allow me to sort of jump around","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3628s"},{"start_seconds":3634,"text":"the routing table because when i everyone always knows the people closest to them everyone knows always the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3634s"},{"start_seconds":3642,"text":"the 20 people closest to them because those guys those those will be down here you know in this end and those tables will be sparse i'm","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3642s"},{"start_seconds":3650,"text":"not going to run out of slots for them uh and so i can always get traverse closer and closer because i ask this node","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3650s"},{"start_seconds":3660,"text":"hey do you know anyone closer to the the target i'm looking for and they say yeah i know this guy and you ask them do you know anyone closer and so on until","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3660s"},{"start_seconds":3668,"text":"you find the target um and and in order to sort of i guess the the generic form of this is that","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3668s"},{"start_seconds":3676,"text":"when we do a when we do a put slash get","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3676s"},{"start_seconds":3683,"text":"uh we will take we'll call like the sort of the key","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3683s"},{"start_seconds":3690,"text":"and then we'll we'll sort of we'll shot 256 it to put it into the same space and then we'll say","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3690s"},{"start_seconds":3698,"text":"the rule is that you know the some numbers say the 20 closest","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3698s"},{"start_seconds":3705,"text":"people 20 closest peers","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3705s"},{"start_seconds":3712,"text":"to the key store it they store they they store that record","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3712s"},{"start_seconds":3720,"text":"um and in the case of ipfs these records are what we call provider records so there's not just one record","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3720s"},{"start_seconds":3728,"text":"they're they're additive right because you know alice says i have the data and bob says i have the data and charlie says i have the data and they all end up in the same in the","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3728s"},{"start_seconds":3737,"text":"same place they're additive on the same key","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3737s"},{"start_seconds":3744,"text":"is that that sound good makes sense okay that makes sense okay and this is and this is basically","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3744s"},{"start_seconds":3750,"text":"kid emilia um has logarithmic lookup time logarithms are are good um they","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3750s"},{"start_seconds":3759,"text":"they make this like really small the number of hops you need to find data should be like three at most","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3759s"},{"start_seconds":3768,"text":"right um as these networks get large and maybe get up to like four i mean like as these networks start to get like really large they go up but","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3768s"},{"start_seconds":3775,"text":"the constant number of hops is like not that big and there are there there if you get if you start to get really concerned as","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3775s"},{"start_seconds":3783,"text":"some of the late with some of the latency of these things you're like well this one kilobyte text file i don't want to wait or web page i don't want to wait for three hops","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3783s"},{"start_seconds":3791,"text":"three hops is a whole bunch of hundreds of milliseconds there are other schemes you could use uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3791s"},{"start_seconds":3798,"text":"to to speed things up right the other schemes like coral which uh which focus on like how do i create latency ring","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3798s"},{"start_seconds":3806,"text":"based dhts so that i sort of create smaller groups before i go up to the larger groups so that i can find the data closest to me more efficiently right so","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3806s"},{"start_seconds":3814,"text":"there's other sorts of schemes here but reminder it does not matter what","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3814s"},{"start_seconds":3822,"text":"this routing system right the dht is like not intrinsic it's like it's like a thing that ipfs","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3822s"},{"start_seconds":3830,"text":"has to help you find stuff right but you can find content however you want to you can plug in some external mechanism uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3830s"},{"start_seconds":3839,"text":"if you're using a building ipf as a library it's like pretty easy as long as it fulfills this contract of give i give you a cid and you give me people who","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3839s"},{"start_seconds":3847,"text":"have the cid you're good to go and even if you're using it as a binary if you if you have some ideas as to who","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3847s"},{"start_seconds":3855,"text":"might have it you could connect to those peers and then request the data","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3855s"},{"start_seconds":3862,"text":"via your external system and this whole stuff with bitswap asking everyone will guarantee that you get it too","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3862s"},{"start_seconds":3869,"text":"um so the you know and there are people like there are people who are building on top of ipfs in various ways","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3869s"},{"start_seconds":3877,"text":"who um you know we've improved the dht a lot in the last year in particular and there's still there's still some really","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3877s"},{"start_seconds":3884,"text":"like you know nice big big chunks of work that uh we've got ahead um but for the people who've had like more specialized these cases where they","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3884s"},{"start_seconds":3892,"text":"they need to find other ways to do this they've been able to build other routing systems or they are they're in the process of doing so because","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3892s"},{"start_seconds":3900,"text":"why not ipfs is this flat space right everybody's hashes they live in the shared space right if there's a c","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3900s"},{"start_seconds":3908,"text":"id of data stored in swarm all right because swarm can use cids to refer to data and we have a swarm codec which we do","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3908s"},{"start_seconds":3916,"text":"which there is right then realistically i i could find i could look for swarm data in ikfs and if","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3916s"},{"start_seconds":3924,"text":"anyone who who was publishing it and you know if i knew where to go look for them i could set up a routing system that way","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3924s"},{"start_seconds":3932,"text":"if they were providing and saying i have this data and telling everyone in the public dht that they have it i could find the swarm data in ipfs and that would just work","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3932s"},{"start_seconds":3941,"text":"so i would say like the key like i guess the the the key takeaway for me at least with this stuff is like content addressing","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3941s"},{"start_seconds":3950,"text":"is really nice because i i do not have to care where the data comes from not just from like a which ip address","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3950s"},{"start_seconds":3958,"text":"has a perspective but even from a system perspective right there there are there are blockchains that incentivize data storage","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3958s"},{"start_seconds":3965,"text":"and it doesn't matter which one of those you would use right it doesn't really matter if i'm storing the data on amazon it doesn't","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3965s"},{"start_seconds":3973,"text":"matter if i'm running my own server um it i can just it doesn't i don't have to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3973s"},{"start_seconds":3982,"text":"depend on sort of the success of other systems i don't have to be tied so deeply to them i can diversify in the same way i don't have to be tied to the success of a","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3982s"},{"start_seconds":3990,"text":"particular sterilization format by pld i can diversify same with all of these things the idea is to the stack has been designed to try and make it as easy to","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3990s"},{"start_seconds":3999,"text":"swap out things as possible so that we don't have to guess in advance what's going to become successful we can","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=3999s"},{"start_seconds":4007,"text":"adapt to the ecosystem and how it is evolving um and and i think that's that is something","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4007s"},{"start_seconds":4015,"text":"i would i would recommend other projects to to look at because i that's been i think very helpful for us um it also","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4015s"},{"start_seconds":4022,"text":"helps with our engagement with with other people in the web3 ecosystem is like we just we just want to work with them because we want to make sure that if they have good ideas that like","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4022s"},{"start_seconds":4032,"text":"people who exactly fast can make use of their good ideas yep um right this sort of like collaborative open source","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4032s"},{"start_seconds":4040,"text":"thing where like you don't need to use everything you just need to use the parts you need um and i think we are uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4040s"},{"start_seconds":4048,"text":"we are we are out of time but uh we can talk about naming another time because mutability is a whole other can of worms yeah yeah i think we covered a lot of","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4048s"},{"start_seconds":4057,"text":"stuff today yeah yeah thanks a lot for giving us the uh the overview","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4057s"},{"start_seconds":4065,"text":"and yeah that was very exciting all right thank you happy uh","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4065s"},{"start_seconds":4069,"text":"happy to be here","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4069s"},{"start_seconds":4077,"text":"you","url":"https://www.youtube.com/watch?v=J-drqD2UebM&t=4077s"}],"youtube_url":"https://youtube.com/watch?v=J-drqD2UebM","url":"https://near-pedia.xyz/videos/J-drqD2UebM"}