Whiteboard Series with NEAR | Ep: 30 David Vorick from Sia

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:03

   hi everyone this is Alex and we have David from SIA today and David will present us the technology and we will will go through it and so to start can you introduce yourself and I walked through the what's powering SIA the technology sure yes my name is David work I co-founded taya with Luc champagne in 2014 I've been in the Bitcoin space / like observing studying tracking Bitcoin

2. 0:36

   since 2011 to spend a long time in an exciting journey for me SIA is a decentralized cloud storage platform we really took inspiration from Bitcoin and the idea that you can create a banking infrastructure or a financial infrastructure with no trusted third party and so we asked the question could this be applied to something else and in the case of SIA where we're applying that to you can create a cloud storage platform that has no trusted third party

3. 1:08

   and so our goal is to be able to make it for you to put data into the cloud without having to give up control to like say in Amazon or an Apple or Google we want we want you to be able to use the cloud without having to trust anyone else and and SIA launched in 2015 we've been building and enhancing it ever since and I do think we've achieved the goal at this point of decentralized cloud storage so I would say that it's here today it works and you can try it

4. 1:39

   out cool so let's let's dive into the tech ok so the first thing to know is that size a proof-of-work blockchain I don't know what I'm gonna write at this exact moment but it is a consensus system very similar to Bitcoin we have one special edition called the file contract so rather than use a like whole smart contracting platform we thought the complexity was high we felt to make the cyanine work we really

5. 2:13

   needed one decentralized primitive or one like smart contract and so we just made that like built that right into the chain so here's something called the file contract and what a file contract is an agreement between a renter and a host and spell them out go each put money I'll just draw a dollar sign into a file contract and of course it's not US dollars it is cyclones so what this

6. 2:50

   does is this file contract has a size and a merkel route in it and so the size is how much data is in there the merkel route is the merkel route of the data so we take we take the data and we merkel eyes it we put it into a Merkle tree we take the route and we put that here and so the size for this guy would be 4 and then and those are bytes or like how big is it yeah so each each leaf is a 64 byte 64 byte set of data

7. 3:25

   that you hatch and then so each each node or branch is 32 bytes of data and then of course the route is 32 bytes so you have 64 bytes into a 32 byte leaf then you combine all the Leafs into a root so oh and then and then there's a time we call it a start time and an end time will come come back to time in a moment so what the money is is money from the renter this is getting paid to the host so as long as the host is honest and delivers on the file storage

8. 4:06

   promise the host will get this money and then yeah this money from the host is actually like its collateral so it's a promise to the renter if I don't deliver on my promise to store the data I will forfeit this money and so if the host succeeds all the money goes to the host if the host fails all the money destroyed and oh they don't go to their anthers Briggs no they get destroyed and this is a game theory optimization what we don't want is if the money goes back

9. 4:38

   to the renter and let's say the renter has some pretty big storage bills you know hundreds of thousands of dollars owed to hosts now the renter has a incentive to make the hosts fail the renter can get substantial revenue or substantial cash back by seeing the host fail so we want to make sure that there's never any incentive to attack the host nobody wins when the host loses oh you're saying if renters aw yes for

10. 5:00

    example yeah like someone who hosts hosts yeah that's it and so we don't we don't want anyone to be able to make money by harming hosts and so that's why the money gets destroyed there's no revenue to be had by you make a little bit of revenue right because total supply reduces what is it yeah it's just not as much I see yeah it's much more indirect and that would require a much much bigger attack to be profitable

11. 5:30

    okay so we'll dive into the start time and end time so this is this is how how do we know that the host is storing the data so the host has this merkel root that micro root and size that describe the data that's being stored and a start time and end time so the start time might be let's say 10,000 blocks in the future and then time might be ten thousand two hundred eighty eight blocks in the future some quick context sia

12. 5:56

    blocks are ten minutes just like Bitcoin so two hundred eighty eight blocks is two days so the start time is when the host has the opportunity to provide a storage proof and the end time is when the network assumes that the host was unable to provide the storage proof so we don't want the host to provide a storage proof earlier because then that doesn't the host hasn't stored the data

13. 6:22

    for for enough time hasn't fulfilled their commitment but we want to make sure they have you know they more than one block because it can be hard to get a you know a transaction into a single block so we give them give them some time to get the proof onto the blockchain so go ahead and erase this the way the proof it self is structured do they periodically submit proofs just one just one okay we'll come back to why that is go ahead

14. 6:53

    and make eight so we have a Merkle tree and let's say it's block so let's say the start height sh th is ten ten thousand and then tide is ten thousand to 88 what's going to happen is at height 999 9999 we're gonna take the hash of that block so we're going to get you know equals you know ABCD we're going to take the hash of this h of ABCD and we're going to turn that into a random number RNG we're going to use the random number to select one of these and

15. 7:49

    random so let's say this one gets chosen so now the host has to provide a proof on this node that he has this data in particular so this if you remember corresponds to 64 bytes so the host is going to provide this 64 bytes not the hash of them but what the actual bytes themselves mm-hmm and then also going to provide see what would that be that movies the Merkle proof that the 64

16. 8:18

    bytes belongs in the merkel root in the contract and so this is a probabilistic proof if the host can present this proof then we accept it as though the host proved they have all the data if the host cannot present this proof then we assume the host doesn't have any data at all even if they have other the host knows no has one one chance to prove they have the state and that happens okay so because they using a hash of the previous block then the proof will be submitted like at this block or shortly

17. 8:43

    after or it has to be sent me that at this height exactly it has to be submitted in this range oh oh that's a range yes the proof against or fic yeah and so if if we get to 10,000 288 and no proof has shown up yet then the contract will destroy all the money mm-hmm if we do get a proof in this range though obviously because we use this block to figure out what what the challenge is the hosts unable to provide a proof until this height mm-hmm cuz the host

18. 9:15

    doesn't know where the challenge is going to be this is a secure random number but why so so let's say host also happens to be one of the miners can they not grind well I guess it's expensive it's a good question yes so and that's that's precisely the argument so the hash of a block is AC is a secure random number in the sense that it's expensive to try again and each time if you don't like the random number you get you can roll again and it's gonna cost you

19. 9:47

    whatever the block reward is so if the block reward is say $400 it's gonna cost you $400 to try and get her in another random number now if you're the network's busy if you're a host with say 200 storage proofs and you have a ran a number that that's selects on all 200 what are the chances that you're gonna roll another random number if you're trying to cheat what are the chances that your next random number is going to

20. 10:11

    be good across all 200 contracts so if you do the the math out basically the upper bound on how safe it is like if you make a you know 200 million dollar contract on the sign network today which I think does several hundred dollars per block is how much it costs to do a block from estée network today that's probably a bad idea but if you have contracts even that are you know in the thousands of dollars probabilistically it should be fine it's more expensive to

21. 10:41

    manipulate the random number even if you assume a 100 percent minor than it is to just store all the data mm-hmm cheating is very expensive so the other the other trick that we use is the collateral so we can we can prove that as long as the host is putting up more than one X collateral the expected value of cheating is lower than the expected value of or the yeah the expected penalty of cheating is

22. 11:11

    lower than the expected costs of just continuing to store the data and so one like subtle corner of the network is actually when the price drops because the reward is tracked in SIA coin so if the cycloid price goes way down the host reward goes way down so if you have let's say like a collateral that's 2x the revenue we can say like you know 1000 signs of revenue mm sighs coins of collateral and the price drops to minus 50 percent the price falls in half the

23. 11:43

    host reward the host reward has halved and the host collateral has halved this ratio is still safe if you were doing like 1000 and and 1000 this ratio may not be safe be because now what now that it's fallen in half the host can go to someone else and get 2000 signs for the same amount of storage and so you want your collateral multiple to you want to take into account network volatility so this is one things where SIA tries to

24. 12:15

    avoid volatility in its token price if we could have a stable coin I think that would be best of all at this point we don't have one but the more volatile the network is the higher you need the collateral to be just because the host can go and get a better deal by reselling the hard drive Montee of the collateral is 2x and the price goes down three times then jeyonce then it's then it's not safe and that actually is a function of how old the

25. 12:39

    contract is so if the contract is like has 3% transpired then probably not safe but if the contract is 99% transpired the host only has 2 or even like 50% transpired the the host only has to hold the data for half the time of the contract if they go to someone else they're going to have to spend you know say double or yeah that someone else is going to have to compensate for more

26. 13:09

    time and so here so this start start hash that's the time I start criteria this is the time at which the ran does not need data anymore right yes the data stored up until the start height at which point the host once provides the cache and so the renter should assume that as soon as as soon as this height hits the host is going to make the proof and delete the data and in reality the host is going to wait a little bit in case there's a reorg or something so the host does wait to submit the proof but the

27. 13:40

    renter has no control over that the host doesn't have to wait depending on the hosts model of the network but not now the host has two two kinds of expenses right they store data but they also serve data yes so what makes them serve data yeah so that is purely revenue based thing so every time that you we've tried to structure the sign at work so that every time you spend resources on the side net work you pay for them so upload bandwidth download bandwidth storage cost contract

28. 14:11

    fees write this the host has to get a storage proof onto the blockchain that's a resource that the host that's a cost that the host incurs when it enters into a file contract so the renter actually pre pays for that so for bandwidth renters pay for the hosts every time they download and then data availability is actually one of the one of the bigger challenges of the sign at work the only so one one thing we can't do is a decentralized trustless proof of data

29. 14:42

    delivery so if I'm a third party like any anyone looking at the blockchain anyone trying to verify the blockchain as a third party so if you and I if you're a host I'm a renter and I say to the network he's not sending me the data but then every time I try and like this is the data he's not sending me and you just proved it you you prove the network that you have it there's no way for me to prove that you are not sending the

30. 15:06

    data you can always you can always withhold until the last second and then give it to me or I could just like falsely claim that you're not sending me the data we can't resolve who's who's being honest in a disagreement like that so we have to use other means then we can't we can't slash people for not making data available because we can't prove that they are actually not making data available we can only prove whether

31. 15:31

    or not they're actually storing it that's what we do instead is we fall back on redundancy so we store this is its own thing so I think remove on from file contracts for now we will come back to state channels so our file contracts are actually state channels this is a scalability optimization and a super important one that that is like fundamental it's making scienc callable or Oh redundancy okay so every single we use something

32. 16:03

    called reed-solomon code it's a que and redundancy scheme so I'll take a piece of data I'll split it up into K pieces on the sign at work today it's a ton of 30 I'm gonna go ahead and draw two of six just because it's less things to draw so we end up with do it this way so we end up with six pieces of data out of these six pieces of data two of them and it doesn't matter which two are all I need to reconstruct the original data so

33. 16:49

    we fragment the file out in a two of six scheme so we can pick any value we want for K and n up two on most libraries this is not a like 255 255 that's correct 0 to 255 and that's just because most libraries don't don't support bigger values so we pick 10 of 30 on the side network the redundancy overhead this is called a maximum distance separable code is n divided by K so on the sign out

34. 17:28

    work that would be 30 divided by 10 equals 3 X 3 X 4 done in C so it's it's basically the same overhead is doing a 1 of 3 to do a 10 of 30 but the difference is instead of needing in a 1 of 3 you're hosed if you get three rare events three three hosts are not responsive in a 10 of 30 you don't get a failure until you have 21 rare events or 21 failures and so probabilistically one is exponentially better than the other as

35. 18:02

    you increase the total number of pieces for a fixed redundancy your robustness goes way up but for a particular cost so that's it there's nothing else right in the network then for a particular host it is it has been official not to serve files right no because if you are not serving files you are missing out on revenue because you get paid to serve files what that I see okay so let's so let's say this so

36. 18:29

    there's the renter in the host right Surrender says please serve me a file yep here's my money how but I'm not giving you the file yeah so so why would they not get the revenue because this where this where payment channels come in so the way our payment channels work we're going to go ahead and accept this is solved so so far it feels to me that at least one of them should be able at some point to to trick the other right so either so between you

37. 18:57

    and I want to let's say you are enter I'm a host right yeah this is the moment to an eye when you actually receive the file and there is a moment when you release the payment to me yeah there is no way that is atomic right that's one of those must happen before the other so actually there is a way to make that atomic I'm not gonna go into it because it's not how we do it so we we do something called packetized payments so

38. 19:22

    what the renter and I'll just go ahead and use the numbers typically so the renter is going to send payment the first step is the renter sense payment whether it's a small payment yeah for four megabytes are you doing inter ledger yes although we predate inter ledger so we've had this live since before the inner layer protocol was devised this this came out in I think August 2015 on the site network so then

39. 19:48

    we trust the host for four megabytes worth payment then the host sends four megabytes and then we do it again and so if we're downloading a one gigabyte file we're gonna run this circle a bunch of times if at any point the hosts it takes our payment and doesn't give us data now I know it's a bad host yeah and because we're talking to 30 different hosts one this gives us super super high parallelism so even if each

40. 20:12

    host is pretty slow as an aggregate the throughput is super high and since since we've been doing 10 of 30 which is also many years the bottleneck for speed has always been how fast your computer is and how fast our code is not how fast the networks are but also if I remember I can read Solomon library's work right they so I give if this is our 6 6 shares right yeah then each each byte is encoded separately right so don't really need to have two out of six full shares for each particular byte we need any two

41. 20:47

    shares yeah so the way we do the way the sign-out works structured you do need a full full 12 to 6 but for each chunk so we reach UNK it okay so actually the same thing I guess yeah just the chunk is not one byte that yes right the chunks not one bite it is 200 and I think it's 64 bytes so I think our chunks are 64 bytes at the at the reed-solomon level at the network level are chunks so we take what we do with a large file let's say we have a 4 gigabyte file we're gonna take this 4

42. 21:31

    gigabyte file we're going to split it into basically so the sector size on the network is 4 megabytes so we're gonna split it into sector size times min redundancy in redundancy for just 3 X no that's min pieces I'm in or those mini pieces which is 10 we need 10 pieces to recover so that would be a foot cables 10 4 times 10 equals 40 megabytes so we're gonna turn this into 100 chunks of

43. 22:08

    40 megabytes each mm-hmm becomes 120 Meg's that's correct operator cooled it and so we call this the logical data we erase your code that into 120 megabytes of erasure coded data and then each and this is of course sliced into four megabytes each each slice gets sent to a different spot on the side net work and so each of this is a separate file contract yes okay within one chunk all right so now all that will go to the

44. 22:41

    payment channels state channel as if there was no state channels each of those would result in on chain transaction yes I see but because there are state channels really what you do is before we get into the technical details you're gonna pick 50 hosts so I'll just draw five 10 10 I don't know if that's time 2 4 6 8 10 okay and so we're gonna have you put the renter up here the renter is going to

45. 23:15

    form a contract file contract with each of the 10 hosts now it looks like a spider happy Halloween and when we upload a file each of these are state channels we're going to use the state channel operation we're going to update each of the file contracts to now include the new data so your average file upload results in 0 on chain transactions that's assuming that the rent are already has state channels open to the loan right yes so part of the part of the setup process for saya

46. 23:49

    decided processes like 1 download download client or get it from whatever source you have to get side daemon to sync cane sync chain 3 form contracts so this will open state channels and yeah this is this phase and then for ready to go so so opening a state channels is an unchanged transaction right yes so we need to open we need to we need to create 50 own chain transactions yes to

47. 24:23

    begin with yeah yeah and then once we have them we can use them to upload essentially an unlimited amount of data as far as far as the protocol is concerned as you saw the the file contract itself is a constant size it just has a size and a Merkel root that scales infinitely the storage proof is logarithmic it's a Merkle tree proof so that doesn't scale infinitely but log a a logarithm in a finite universe is constant right

48. 24:55

    so by the time you're up to proof sizes of I think around three kilobytes you've exceeded it isn't linear in the finest universal so constant alright you know that's that's fair but if if you boil if you say storage proofs are you know always no more than four kilobytes mhm that's bigger than all the data in the universe so and whereas linear you're gonna get much less yeah that's the

49. 25:26

    ledger yeah so so the chain can handle effectively an infinite amount of data per file contract and and the largest proof size you'll ever practically see is 4 kilobytes but the saiga client can't there's a lot of metadata it has to manage this is actually something that we improved progressively so like when we first launched sia we actually didn't have the state channels up so the

50. 25:49

    whole entire side network was good for about I think 10,000 files and the largest acceptable file was like 50 megabytes this was like you know 2015 and then later in 2015 also in 2015 we got it up to you know like 10 million files for the whole sign at work and then you know maybe each file could be up to I think 200 megabytes today we're doing much better so each each node a single cyano today is effectively good

51. 26:24

    up to around 50,000 files in a single node and then around 20 terabytes in a secure file opera note print per node or these there's no limit on the file size anymore right if it's a single file at 20 terabytes you probably still good mmm and then the 20 terabytes most that limit comes from the repair process we're still ironing that out we actually have we have an internal fix that we're gonna push out we don't

52. 26:49

    know if this it bumps this number up or how much it bumps this number it's early but these this is what you can do today and in theory with like practical engineering as we keep working on it we think that in the end this number will be several million and this number will be several petabytes per node no like no problem yep so one question I had was so let's say so as a as a host I still get the

53. 27:28

    reward even if so even though I'm losing revenue from not serving a file right I'm still making some money because we store from the storage right yes so someone might contemplate the idea of having like a huge node which is not connected to the network or like to fast Network yeah right because if I have 20 terabytes of data I in principle that is a quite a bit right so I might choose not to have a network yeah well I guess

54. 27:53

    yeah so but rather my question is is there anything today in the see a network which would like for a particular enter band yes like some heuristics to ban host switch yes so we we have a module that we call a host DB mm-hmm as the name suggests it is a database of hosts and the renter so each renter has their own host DB the blockchain when a host wants to announce the network it puts into the blockchain

55. 28:18

    an announcement I'm a host here's my IP address ask me about my wares and and so the renter will ping the host get prices availability you know whatever a bunch of statistics and then that renter keeps in its database just a big list of how good each host is and so if it's using as it starts to use house it gets better data if it noticed that a host is slower or is withholding data or otherwise unavailable it down

56. 28:49

    ranks it in the host DB and so if you are having trouble Italy host will let you host something that is currently hosted so let's say I put this file for like 10 month and like 1 month in I realize that one of the hosts is is so so yeah automatically hosted it will yes so that's called the repair process so let's say that the renter decides a particular host is bad whether it's because it's offline it increased prices it you know there are there are a number

57. 29:19

    of triggers that could cause the renter to decide a hostess bad basically what it does is it will when it wants to do a repair it will download the file so if we're if we're doing a ton of 30 scheme it'll find 10 pieces download them reconstruct the original 30 and then whatever hosts need replacing it will find new hosts and replace those it will only do that once 25% of the redundancy is missing so 25% is missing in so the

58. 29:53

    redundancy in our case is 20 pieces right we call them parity pieces fun of you parity pieces I spelled that that's ok times 0.25 equals 5 pieces mm-hmm so once a particular not going five percent of all D ok all the pieces just just 25% of the parity so once five hosts are bad on a particular file it will go ahead download that file replace those five pieces whichever whichever five are the bad ones it'll replace them upload them to new hosts and so the the

59. 30:27

    renter is constantly doing this we call it churn just constantly doing this repair churn and obviously that's can be a scalability limitation if it's happening a lot but for them for the renter that doesn't mean that the renter can also stop the file contract the renter didn't stop uploading to the host but the host the renter already put money into it I see so this like that that would result in a money loss rental rate like four they already paid for

60. 30:58

    storage which they not using for yeah for some someone on a storage yeah that's unfortunate yes because I presume oh but I see so the so let's say I'm running a host machine yeah right and oh yeah so that's that's what is not very nice idea and the host machine I'm a host motion crushed and I'm in Hawaii yes I'm not in a particular hurt to recover it I need to recover it the first time my next file contract expires right No

61. 31:29

    so what's gonna happen is if you so the renters have certain expectations we can get in into those later but I'll me I'll make a thing for it actually I'll just go ahead and run him out so our expectations the big one is 95 percent uptime uptime per month so if you're offline for more than 5% of a single month which is about 1 day so you can you can afford to lose about one day of downtime in a row the renter will start to penalize you and will start to turn

62. 32:02

    you and so if a renter is active is constantly uploading new data it's constantly repairing if you're a host and you have downtime all the renters who are using you are starting to down rank you all right what time is ready so so I will start losing revenue what that meant is that for me to lose collateral yes it's it's like it's not particularly depressing I need to recover my you start my note before file contracts

63. 32:27

    expire yes because that's a way bigger loss for me right Michael at you yep then yeah yeah and generally speaking if you're a busy host I would expect you to have file contracts that turn up constantly but as we said that that window is about two days right so you still like it's intentional that you're not in a super big hurry because we have we have 29 other hosts that we can depend on and that's actually one of the reasons the network is so cheap right who's like Amazon their data centers they have full-time engineers

64. 32:59

    that are on site 24/7 awake alert trained technicians they know how to operate the datacenter they have multiple ISPs multiple power companies our hosts it's fine if you go on vacation right and then the thing crashes and you have to call your friend and get it to reboot it or whatever if there's some like if you're slower and have less uptime then Amazon datacenter this is seen as perfectly acceptable because you're not

65. 33:24

    a single point of failure well while you're down for you know 24 hours we have 29 other people right but presumably we had behind the scenes on Amazon there's also quite a bit of redundancy right oh yeah of course of course so so for them for them it's crucial that the the proxy between you and Amazon is always life yes what is behind the scenes that that could be very redundant right that's correct but in practice keeping that proxy has exerted high uptime yeah it's very

66. 33:56

    expensive once we don't have so one other thing where I cannot see anything bad happening yet but because you thought more about that you probably know if there's something happening there sure when I send those thing is right some of those entities could be the same entity right yes could each because this is a racial code that could could they cheat in some way especially if it happens that many of them yeah

67. 34:16

    this gets into encryption so the process is we not only I can recover they cannot recover we split the file yeah and then we encrypt it I'll just do e and then I'll put it into I'll make it a negative puff after encryption so even if all of these are the same host mm-hmm and they each have one piece and they know that the pieces go together they can't put them together because the encryption is applied after the erasure coding yep so you cannot the redundancy

68. 34:56

    the renter has perfect confidence that the redundancy can't be cheated mm-hmm this comes with a trade-off and this is actually like this is this is the trade-off that SIA makes that file coin does not make that resulted in file coin doing like requiring you know tons of open research problems so this this is fairly easy but it means that the renter or someone that the renter trusts with the encryption key is the only one able

69. 35:20

    to do repair so if the renter is like on vacation and you know three hosts disappear or let's say you know half half the hosts disappear so now the renter is in trouble she needs to repair as fast as possible if the ranch is on vacation the network can't repair on behind the renter because the network doesn't know what the encryption keys are so either the renter themselves or someone that the renter has trusted the

70. 35:45

    encryption keys - has to be the one to do the repairs whereas file coin has all these like deep research questions that are open how do we make it so that we can repair it without someone being able to cheat and well if you believe in magic you can use SGX right that's yes that's one option SGX and they have a bunch of crypto protocols require trusted setup so similar similar assumptions to have SGX

71. 36:12

    in my opinion regardless we don't like that we do up here like verifiable route we make the trade off the renter has to be around or someone they trust has to be around to do file repairs that's just a limitation of the sign at work if our coin figures out proof of replication one day maybe we can move to it answer in practice in CN Network SIA met work other participants who like other

72. 36:37

    let's call them watchtowers but I guess they not real watch towers like other entities whom you give the key and then they will do it unless you feel practice today no there exists we haven't needed them although maybe some like maybe someday there'll be you will see that pop up Model T in practice how frequently do you observe or first of all can you observe someone the cow doing churn from us we observe when we

73. 37:03

    so we run our own notes writes we can observe when we have to deal with churn and of course we talked to our community so lots of people run large nodes and actually right now again we already have the fix for this in the lab we're trying to push it out we were trying to push it out this week didn't quite make it maybe next week but right now Turin is actually fairly high on the side network

74. 37:24

    and this is due to a scoring error um what Mekhi is like how high high would be like maybe twenty five percent per month I saw after after the scoring fix I think will be closer to I predict it will be closer to five percent per month which is still you know if you have a hundred terabytes of data that means you're repairing five terabytes a month that's still fairly substantial but we'll see so churn is definitely like from a

75. 38:01

    scalability perspective that's probably the biggest bottleneck is how much repair bandwidth do you need per terabyte you put on the network every month and this big focus right now is barely 10 out of 32 card kulit or can I configure it like oh it's fully configurable okay so if I want to if I want to go to Hawaii I will just say that to five out of a hundred sure and okay so and like for probably for several month I'm fine yeah I'm

76. 38:23

    adding more for storage though and in practice data that's on maintained for a full month at 10 of 30 is completely fine yeah you don't lose you don't lose 21 in a single month normally it would take three three to six months one other thing on the sign at work file contracts are typically they typically lasts around three months and so if you're gone for more than three months all your

77. 38:48

    file contracts expire host to lead all the data you need to renew them you need to let the hosts know you're still interested you need to repay to keep the file contracts alive but I can renew the contract without your applauding file right that's correct yep yep okay so so I guess is there anything else interesting or gives you cover almost everything there's lots of interesting things

78. 39:12

    so one thing we didn't cover is how the state channels work what are these are they significantly different from state channels not really used we the one thing we have is an update number a lot of state channels do have this a lot of a lot don't but it's basically like so the file the file contract has Merkle root size and height start height and then it also has this like revision

79. 39:35

    number which is a un 64 so if you want a file contract that cannot be revised for whatever reason you just set it to you in 64 max that contract can't be revised and practice I think anyone does this because I you you also there's also a renter renter private key renter public key sorry and a host host public key so you need signatures from both to update anything yeah so that's what if I said to reveal them to into max that means I can close it without yes judge period well I don't

80. 40:09

    know if it's implement it but that would that would enable it right so actually all you have to do so there's also a so there's a success success balance so these are I see success Pal success actually success outputs so the renter gets some money back the host gets the money back and it's just how much right and then there's fail where the renter gets the money back the host gets them

81. 40:32

    get some money back and so and then and then the void that has to add up to that and then there's an input there's a payout payout up here these two have to add up to the payout these three have to add up to the payout so this is the void so you have renter host void enter host and so in the fail these two are significantly smaller because we destroy money in a success usually it's mostly host weighted but it

82. 40:58

    depends on how much you use the contract if you open a contract and then upload it almost no data the host never put in any collateral the renter never put in any revenue and so both of them basically just get out what they put in and so if you want to close out a contract early what you can do is you change the start height and then tight to just be now right and then you create the success outputs and fail outputs to

83. 41:20

    be equal to each other and then it'll just fail immediately write the file contract will fail immediately and the host will get will get paid awesome so the I as the renter can just close the contract and pay to the host yes I see and so that that that's how you do that on the protocol level that will not have any challenge period or or rather even if it does nobody will challenge it

84. 41:43

    right yeah I see but what I get no money in this case all the money go to host oh they still something it depends on what they're entering host agree because on this channel right if I have the channel with the host it could be that they have more money in the channel that I need to pay for anything we have right now open with them right yes so yeah so like what what could happen is you put a hundred

85. 42:06

    signs into a channel right and then initially let's say the we close the contract super early so what's happened is the renter paid you know Tennessee to the hosts so well let me break this out a little more so we have a file contract renter in mentor in will call one hundred side coins hosting will call 210 points these are recommended parameters and then what's going to happen is if we've all if the renter has only stored a little

86. 42:39

    bit you might have success look like this where it's renter gets back the renter gets back ninety nine side coins the host gets back will that be to one side coins go inside claims and then fail would be renter and two ninety-nine side coins host gets back that would be one ninety eight SIA coins and then void is going to collect the rest for just three side points so what is this number 198 come from so the host was originally at two side two hundred side coins the renter

87. 43:17

    the renter paid one side coin to store some amount of data oh those collateral and the host put in two signs of collateral hmm and so the void gets three in the in the failed case but if we want to do an early close what we would do is we would you know X out this fail or actually had okay if we ignore this X we would we would X this one out and this one out and this would go to 201 and this would go to zero I

88. 43:40

    see and so such contract can be closed without challenge period exactly okay you you'd reduce the challenge period to just immediately and it'll fail immediately but the failure condition is the same as the success condition the host doesn't care they get all the all the same money if it had been its success yep I see cool so you also mention that it is cheaper than Amazon so in practice what is that show yeah comparison of prices oversee overseas let's say st and then we can talk about

89. 44:13

    snapshot backups which I think are super cool and we should definitely talk about and then we can talk about streaming which is also pretty cool but okay so prices let me see you I have signal in this room I just pull up the live feed of our prices and I do just already have the paid roping because as soon as it loads storage what would they also know AWS prices yes upload bandwidth so a storage bandwidth

90. 44:51

    on the sign at work today our storage cost is $1 and 20 cents per terabyte per month upload bandwidth costs 16 cents 0.16 per terabyte download bandwidth DL BW is whatever 83 cents 0.8 3 cents per terabyte on Amazon this is I believe $21 upload bandwidth I think is free on Amazon I don't I don't think they charge you to upload and for download this is about $90 so we have between a 10 and 20x improvement on storage costs and between a 50 and 100 X improvement on

91. 45:35

    bandwidth costs actually right now we're over 100x improvement unbanned with cost so prices are super competitive on this high network and doesn't sound bad at all not bad at all but it does surprise me a little bit because on Amazon the storage itself is heavily optimized right yes so is it the case that all the hosts also some professional so some of them are professionals and I think the end game of the sign at work kind of like Bitcoin mining there will all be professionals at some point right now

92. 46:07

    the total amount of storage on the site network is about 675 terabytes as of writing this video or as it right now so you multiply this by this you see it's not that much money that's getting paid per month but as this goes up I think we'll see increasingly more and more professionals come in they'll continue to push the prices down but the real the real cost advantage I think for us comes from a significant relaxed relaxing of

93. 46:37

    trust requirements a significant reduction of barrier to entry so if you have a you know host on a fast internet connection you can just buy some hard drives and and offer them up to network and and then like there's no the up time requirements ninety-five percent is more than you can do on a laptop but it's pretty easy from just any you know you don't you don't need a crazy ISP or a crazy power supply

94. 47:03

    agreement cool okay so a snapshot snapshots okay where to start I'll start with we won't dive into the technology initially we'll just talk about the end result so I can do is I can upload a bunch of files the SCI Network will just call my FS right my NFS and this could be you know some some folder tree 141 the final one file to file 3 etc and so I could have you know a ton of files a ton of folders and so I'll just create a

95. 47:48

    snapshot of these folders so snapshot and then I can say save my snapshot and so when I create a snapshot it will essentially freeze the metadata make a copy of the metadata upload the metadata to the sign at work and then create a bunch of pointers such that in the future if I lose my machine my house whatever and I have to start fresh I can grab a new computer download SIA put my wallet seat in and this snapshot will

96. 48:22

    show up and all the files listed in the snapshot will be recoverable and I need to know nothing more than my wallet seat to get all of my data back and this I think is the feature that really makes SIA like a viable proper backup solution so this is something that's live today and now I can dive into kind of how that works it's actually fairly complicated this this was one and this a feature we added earlier this year so

97. 48:55

    it took us almost five years to get saya to a maturity level where I could where could do this okay so step one is we have a blockchain full of information this is the blockchain blockchain and in the blockchain are all of our file contracts so we'll just call this and actually they're probably they're probably all in one in one period of time so I like right here I might have my slice of 50 file contracts that belong to me so when we create the file

98. 49:30

    contracts all transactions in SIA have this so IFC and then so here's the here's the file contract they also have something called arbitrary data so inside the arbitrary data what we do is we put a color color which is really just a hash of our seed appended with some extra this is a little bit of a simplification but but let's say the host pub key the post pub key and so if I have my seed I can see in the file contract with the host pub key is I know what the color would be of this contract

99. 50:11

    if it was mine and importantly because it depends on my seed which is a secret nobody else can make a file contract that I'll mistakenly think has my color they don't know what data they need so basically thanks to and again it's a little more complicated than this because a little bit safer than just using your seed in the raw but but I think one thing we give them define is the problem right oh yes why was it hard because because I lost all my state channels right you lost everything so you have no information at all except

100. 50:42

     the blockchain which is public so we can download the SIA client which is public we can download the blockchain which is public and we have our seed but we don't have an estate channel and we don't have anything else besides our seed and so from our seed we have to derive everything else and so this is why the color is important we know our seed we know the host pub key has to send the file contract so as we're scanning through we're gonna find I think a million file contracts in the blockchain that's how many are

101. 51:06

     the blockchain today and so a million times you're gonna check and you're gonna say oh no this is not mine this is not mine this is mine not mine but 50 or more depending on if there's been turn but some or if you've done renewals or whatever some of the contracts you're gonna find you're gonna be like yes this is mine you're gonna be able to pull out your file contract also in the blockchain are a bunch of host

102. 51:27

     announcements right and so what that is a public key to IP address so I just we're gonna query them for the for the latest state that's correct but this is risky right yeah that we've lost our data so yes we we can identify our final contracts we know from the public key inside the blockchain there's also this public key to IP address mapping we know what IP addresses to ping something that we do in our protocol is that every time

103. 52:01

     we connect to a host no matter what we open the connection by pretending we've lost our data I see so we're trying to connect we're trying to trick the host into taking advantage of us when I think we're getting a little bit we would immediately smash them exactly we we immediately know that they're cheating we know not to use them anymore but so the one time that we do we we have no

104. 52:24

     ability to double check if they're cheating they don't know because our behavior is identical it's all the other situations as well one third disk wittingly fine right so what you can actually do is you can give we don't do this yet thank you this is something we would like to like to add but what you can do is you can put the entire state you put every state channel on every host so every time you connect to a host you can ask them for all your state okay and this actually creates an interest it

105. 52:54

     was sent to you wouldn't put this twenty is almost that's yeah then you can get everything back and hosts are all in competition with each other right so the other people that I'm using are this guys competition this guy you know the honest host actually kind of has an incentive to have his dishonest competitors go out of business so he actually wants to give me he may not want to give me my channel state for him

106. 53:19

     but he wants to give me my channel state for everybody else so I can penalize everybody else and so there's this like nice little second layer incentive working in our favor but this this particular feature of always requesting as a pillows data yeah was it added as part of this change with snapshots or was it also needed before for something so it wasn't needed before we we've been planning this for a long

107. 53:42

     time so we've had that behavior in the network for a long time but we did we did at it specifically because we knew we would need it down the road yep interesting inside so now now we have our file contracts back we have our state channels back we still need some way to fetch the metadata and so basically the first a file contract is okay this is kind of interesting architectural stuff is your mat again how's the file contract orchestrated or

108. 54:15

     managed so we have a file contract today file contracts are broken into two sections the first section is a like a fixed fixed data section fixed section and then the second section is a random access section so the fixed section is where we store a pointer to when we created that snapshot we actually uploaded it as a file so somewhere in this random access section there's a there's a file that has you know snapshot snapshot and then that snapshot of course ot that snapshot of course

109. 55:04

     will point to all the other files that we have so here we have a pointer and so we'll just look at this is index 0 so in index 0 we have a pointer that says this thing this root is a snapshot so most of our files are stored by their merkel root the host keeps the table on its side that goes from merkel root to data and so if I ask the host for a particular merkel root it can give me the data I know that it gave me the right data because just hash it up I see if the Merkel

110. 55:35

     routes match and then what that means is that when we want to do something like deletion what we can do because so if this if this is a million nodes long which on for some filed contracts that that's four terabytes a million nodes is four terabytes if it's a million nodes long we don't want to have to recompute the merkel route across four terabytes of data that's way too much hashing so

111. 56:00

     what we do instead is we will swap this guy with the tail and then we only have to do logarithmic number of operations instead of sliding everything over we do a swap we have every cashed log number of updates we get a new merkel root nice and easy we have the two cashed for a fourth terabyte file that's pretty big it's not too bad and we don't cache every layer just the top once we cache yeah so so the big one is we cache every four megabyte so we we cache every four

112. 56:31

     megabytes and then I think there's a second layer seven above that so whatever four megabytes x to the 7 is hmm I should I dunno 512 megabytes M hmm we save another you have another hash so that this allows us to do things efficiently and then something we're rolling out in the next three months or so is garbage collection so if you want to sew the delete operation I explained

113. 57:08

     we don't actually have right now this will eventually be broken up into three sections you'll have your fixed section it's not a one question I have is why why is this not the very first file and I'm coming in why why is that just now why snaps are not the very first file that's right that they have like why would it not be just at the beginning so that they don't have to look it up yeah because then we can repair the snapshot

114. 57:33

     we can make changes to it I say also we can make a bunch of them so if we have like actually I think I think the correct answer is so we can have a bunch of them if we have 50 different snapshots it becomes a pain to try and front load 50 things and keep them ordered and like and then what you want to prune them or delete them figure out how to do that when they're all in fixed if you like the snapshots to be random

115. 57:57

     access so if we want to shuffle things around mm-hmm it's free fix the fixed section everything's referenced by index so it means whatever is there can't move and we have no flexibility to touch it if you delete something in the middle of a fixed section you can't reclaim that data you can only wait until someone wants new fixed data no there whereas in the random section we just pop things around a bunch because we don't nothing points to it by index everything points by Merkle root

116. 58:22

     and so that's why it's snapshots go out in the random section because it's easier so this is a final Quinto between me and the particular host right yes you and one particular house are snapshots oh and that's the other thing so every host has the fixed section this is replicated if you have 50 hosts your fixed section is replicated across 50 hosts but your snapshot can be erasure

117. 58:49

     coded and so the snapshot which can be quite a bit larger you know the the fixed section the pointer to the snapshot is you know either 32 bytes or if you've erased your code that the snapshot maybe it's 32 times 50 mm-hmm bytes it's yeah 32 times 50 pointers but the snapshot itself we can erase your code that hmm we don't have to we don't have to fix 50 extra complicated so yeah a

118. 59:12

     couple couple of good reasons to keep the SAP shot in random land mm-hmm cool yeah that's that's interesting so the last thing is that we're streaming streaming video streaming see yeah this is really just tip of the iceberg for engineering engineering advances on the side platform but one thing that we're really excited to be able to support is video streaming and fuse which is file

119. 59:43

     system and user space what this means is that you can mount SIA as a drive on your computer as though something like VLC or Plex Media Center will recognize the folder that is actually a size folder as a normal file system folder and so we're working on an integration with Plex called size stream that allows you to put your media library onto the sign at work and then stream from it to

120. 1:00:18

     any of your any of your Plex endpoints and and view your content so that's that's like one interesting use case that we're we're rolling out design network cool and so the idea so streaming works because effectively like doing I'm already steaming my file from the from the host right yeah so I guess but it is the Razr coded oh but it's arrays are coded in chunks so I like I recover one chunk yeah I play it they recover the second chunk yep and then

121. 1:00:50

     because so the the actual erasure coding bits are I think divisible down to 64 bytes mm-hmm I don't I don't think we have a specific vocab word for the 64 byte mm-hmm segments but they but so if you want just the first like say you know two megabytes of a 40 megabyte chunk uh-huh you can download just the first two bytes you don't have to download the whole 40 mega voice oh that makes it a responsive streaming platform hmm awesome yeah cool so we've covered everything

122. 1:01:21

     I wish this way more awesome yeah okay so that was fun thanks a lot for coming and talking about SIA solutely thanks for the pronounce it awesome thanks a lot everybody until next time