{"id":"NfRbWVcoJ-4","title":"Whiteboard Series with NEAR | Ep: 54 Aptos Labs w/Illia Polosukhin, Avery Ching, Sasha Spiegelman","description":"This Whiteboard Series episode features\nIllia Polosukhin, Co-Founder, NEAR Protocol\nAvery Ching, Co-Founder, CTO, Aptos Labs \nSasha Spiegelman, Head of Research, Aptos Labs \n\nAptos leverages decentralized technology for applications like remittances and payments, utilizing the Move programming language and a Diem blockchain designed for scalability and mass adoption. The platform includes features like data dissemination, consensus, parallel execution, storage, and proof generation. \n\nThis overview examines the protocol’s implementation in Move, focusing on anchor ordering, crash management, and the benefits of a leaderless system. It also discusses parallel execution trade-offs in consensus protocols and introduces BlockSTM, a software transactional memory system that improves concurrency and reduces conflicts by integrating STM, OCC, and multi-version data structures.\n\nTakeaways:\n• Aptos aims to leverage decentralized technology for remittance and payments\n• Their infrastructure includes a move language \n• The architecture consists of data dissemination, consensus, parallel execution, storage, and proof generation\n• Parallel execution in consensus protocols aims to reduce conflicts and maximize concurrency.\n• Static parallelism requires the user to specify read and write sets, while dynamic parallelism is agnostic to the specific data accessed.\n\nChapters:\n00:00 Introduction to Aptos and Their Goals\n19:16 The Leaderless Consensus Protocol\n29:18 Ensuring Safety through Quorum Intersection\n37:10 Static vs Dynamic Parallelism: Trade-offs and Benefits\n\nJoin NEAR's community:\nTwitter:   / nearprotocol  \nBlog: https://near.org/blog/\nWebsite: https://near.org/\nReddit:   / nearprotocol  \nDiscord: https://near.chat/\nMedium:   / nearprotocol  \nGitHub: https://github.com/near  https://github.com/nearprotocol\nDev Docs: https://docs.near.org/\nCreate a wallet: https://wallet.near.org/create\nLearn to Build On NEAR: https://www.near.university/\n#chainabstraction #NEAR #nearprotocol","channel_name":"NEAR Protocol","published_at":"2024-04-16T00:00:00.000Z","thumbnail_url":"https://img.youtube.com/vi/NfRbWVcoJ-4/hqdefault.jpg","transcript_text":"hey everyone this is Ilia joining from Bay Area palalo it's amazing to be in the office of abas so we have Avery and Sasha from abas guys introduce yourself and let's dig great I'll just keep it short I'm Avery uh co-founder and CTO here at abas Sasha head of research with awesome so let's dive in so let's maybe explain what you're trying to achieve as aptus and then uh go from there and start decomposing the system sure so just a little bit of history about apas uh 2018 uh you know meta started kind of launching an initiative uh called Libra and then DM over time uh I was very fortunate to work with folks like Sasha and a bunch of other people on our team on this project from from almost day one um and the idea was how do we kind of support use case of billions of people uh you know at meta and beyond that beyond the leverage decentralized technology to do things like remids and payments uh and support like the largest use cases in the world uh and so from that kind of point on we thought about from a first principal standpoint what do we want out of a decentralized infrastructure and that started off with the idea of a blockchain like you know thinking about the way that we scale systems that meted before in the past scaling you know to massive amounts of transaction throughput uh having maintaining great latency and then from a language perspective how do you kind of think about smart cont languages from from the beginning again you know if you think about the general purpose languages they're they're built to support very different things high performance fast iteration fast development times fix bugs in prod uh roll forward that kind of thing whereas smart contract development uh harder to fix bugs that are al already in the network also dealing with the idea that these bugs may cause massive asset loss of course is a is a huge problem and so with that we kind of have the idea of a move language coming forward uh the idea of the DM blockchain going forward and then building these in tandem the same way kind of think about you know Hardware software codesign being to think about the developer platform and kind of the infrastructure behind it you know working in tandem towards this ultimate goal of of mass skill adoption all right so for later ones I usually like to go from bottom to the top so consensus kind of the design of the state and transaction processing and then VM and then kind of polish on VM around like gas transaction sure we can do that yeah I thought maybe it also be useful to think about like the way our architecture set up be an easy way to to start off with so the first thing we have in the way we think about transaction processing is a data dissin dissemination phase and so you have clients that are going to be basically pushing transactions uh into validators and so we have the notion of a Valor set um many validators here that all have um fully fully connected uh one more here and clients can kind of send these transactions to each any one of these validators mhm and in order to make sure we leverage effective use of this bandwidth um we leverage something called like a narwal which Sasha actually worked on um that technique and it's also called corm stor our current model and so the idea is that by having each of these validators um kind of producing their own blocks and then being able to pass those blocks off to each other uh we can leverage the full bandwidth of the Valor set itself and so if you think about like traditionally in many older systems you have this idea of a leader this protocol leader kind of push transactions to other uh values in the network you have to deal with kind of either a limitation of bandwidth and or a latency delay and so for instance if each of these nodes has like n bandwidth you know and I can you know have a leader that's pushing out to these other two nodes I have to divide this bandwidth out going into two and I can push these transactions at a rate of n/ two uh and this then slows the weight the rate at which I can kind of uh get my consensus complet over time so instead what you have here is everyone is going to be broadcasting these transactions that they receive in blocks to each other kind of all toall communication but they and they're leveraging that n over two bandwidth and now you're also receiving en bandwidth which means you kind of get a full utilization of your network pipe which is great the next phase is what we call consensus and so for consensus we've actually iterated a lot over this design over time we started off with ideas of Hot Stuff uh this is kind of my first project I worked on when I was at uh meta um and I think we quickly realized there were things that were missing like there was no pacem think ER there was idea of you just having exponential back off until you kind of achieve this partial synchrony model uh but that in practice that works really really poorly uh and so we started to make changes to it like adding things like leader reputation uh where you could kind of uh support the you know the idea of how the leaders done in the past have they voted have they proposed correctly successfully in the past and then also um things around an active pacemaker where we kind of um proactively if you see like the leaders Miss around you can kind of start broadcasting to each other and trying to figure out the next lead ought to be and advance that cycle forward faster um that kind of led into um a new line of work which you know Nar one Tusk is one thing and then bullshark protocol which is dag base protocol and then finally unto SCH um but the idea here is that we have our consensus protocol here that that doesn't agree it doesn't pass off transactions as a part of that protocol that transactions H propagation happening in the first stage of data dissemination this is where a lot of the bandwidth goes because if you think about again the system every Valor needs to receive every transaction eventually and every executed output um and this is the bulk of the traffic and that's where we maximize our throughput in the system then we have a consensus protocol and as part of consensus we also kind of have this interesting like Oracle Services built into it and so we have uh time stamps like uh that are built in and we also have um uh R onchain Randomness which is really interesting which I probably won to dig into too much because a littleit deep but the time stamp process is like you know as a part of the consensus protocol you only vote when certain times are passed times are part of the the voting protocol itself and that means that every single block has this time stamp associated with it and typically inas you see something like 200 to 250 millisecond blocks and every time you know you have this very accurate clock that kind of goes forward with that process after this we have uh parel execution sorry about that bad handwriting and so at this point in the in the process we have agreed on a set of transactions that will be um kind of uper execution and so you kind of Imagine these set of blocks like you know block zero and then block one block two full of all full of transactions so you know TX tx0 tx1 tx2 tx3 so on so forth and this is where I think a lot of the magic happens all kind of Sasha R into but we leverage um block STM as a technique for dynamic kind of parallelism we can kind of Val value all the transactions we can we currently evalue block by block but you can imagine looking at a window blocks and the idea here is that you have this set of transactions you're going to execute and we want to make sure we fill up all the resources in the in the in the computer so all the CPU cores all the memory we have all the ssds you know so on so forth uh we want to leverage all those resources to maximize and and execute as much as we can in parallel the next phase is going to be on the um batch storage so once you kind of have done all this execution you need to write to to in our case SST um in many cases I think that's the case uh and so what happens is you know we try to bulk up our rights into larger uh larger operations this allows us to maximize the throughput and you know reduce the number of seeks that are necessary we also start to explore um how do we take our Rox DB instances and scale them out to multiple instances so you have you know a couple of these in the back end and this allows you actually get much much higher through but we call this kind of storage charting it is an internal sharing technique it does not require anything like anything user visible um just gives us a lot more more throughput on kind of a critical path of our system and the Final Phase is kind of the the proof generation so at this point we kind of generate the um the proof on the state state root hash and then we broadcast this to other validators in the network and so at this point once you have 2+1 uh stake in the system uh that has a Pro on it you kind of know that this is the state that's agreed upon you can use this for State synchronization you can use this for light client proofs any kind of things like that uh so I think what's also important about the think about this design is that each of these phases can be executed All In Parallel the dissemination is consistently going on consensus is going on execution storage and improv Generation all happening at the same time and so this means we kind of Leverage all the ma machine resources pretty efficiently and also the also we can you know if you think about we can expand this this offer stack to work on the set of machines over time as well um because every phase is kind of inherently pretty parallel so any questions thoughts yeah a lot so yeah start di through this yeah let's start digging in okay so data dissemination right so you have a bunch of validators uh they're interconnected yes um so let's say I'm validator like one I received uh a transaction uh what what am I like I'm just sending it out to everybody else like what what are the guarantees that I'm looking for yep so the first thing you're going to do is you're going to basically broadcast to other validators and network and first you wait a period of time a small period of time you batch it up and the batching is meant to be more more of like a efficiency win right you can process transaction by transaction but that's that's not very efficient for the system right so by batching you have like a signature now on that batch we call this the proof availability I'll write this down Prevail ability and this kind of when you sign this you're saying I'm going to store I'm going to store this data I'm not going to forget about it right uh and that way when I start going to consensus I'm sending this proof and I'm agreeing on these proofs and the important thing about these proofs means that I don't actually even need uh the the transactions to vote on the transactions I'm agree on on I can fetch those just prior to the pr execution phase okay so yeah so so pretty much there like small batches of transactions that as you said you wait for like smaller amount of time like very short period of time yeah kind of customizable depending on each validator yeah then you have a signature I send it to everybody else do they like if they have the same transaction and they shared it what like what what is a ah so you're actually what happens with duplicate transactions duplicate transactions as well as like do it propagated I mean you you're saying everybody's interconnected what if like some links are severed yeah so if some links are SE so if some links are severed like as long as one when a Valor broadcast this information to others they're getting the proof availability they're get two of plus one stake signatures mhm so they collecting like for for any transactions they pretty much for for batch for batch for batch for a batch of transactions you collect this proof availability on it okay so so I I created my batch I send it to everybody else and everybody else then signs as well this batch and sends it to everybody else no they sign it and since you're broadcasting that since the the creator of the batch is responsible for making sure everyone gets it okay they don't need they don't need to broadcast anybody else necessarily what happens is that he this this valer is kind of responsible for making sure it's disseminated yeah if they don't if they don't if whatever reason they don't receive it before execution they'll have to fetch it and they can fetch it when you have this proof of avability you know exactly who has it you know you can go get it from but but how does this get aggregated like by the whoever disseminated it yes okay so you send it and they reply with their signatures that they received it okay so you just get you collect receipts okay yes and then broadcast that information and that information now is that that information which is the proability is used in the consensus protocol okay so so now pretty much when you move into contest protocol you have uh like batch like batch validator one for example batch validator 2 with a bunch of signatures right yeah they're aggregated but yes yes yeah well aggregate signature yeah is like BLS yes okay all right so you have this batches mhm okay and then uh and then so now consensus is what actually orders them yes and I think it's less of an ordering which is important here what is is you're kind of agreeing on that set of transactions and the reason why I called that out specifically is that when it comes time to like we've agreed on these blocks basically and we've agreed that these will be sent to execution the nice thing is that I kind of at runtime here as long as the valers agree on it they can do all kinds of interesting Transformations on that data sure they can reorder it uh to maybe throw it MV type attx if you want to you can also try to extend that window if you want to achieve more more concurrency in the way that you execute those transactions you can even optimize if you want to do it around gas as well like if you want to think about the way that gas markets work in prioritizing certain transactions we can also add the optimization there so it's kind of like a later late stage runtime um optimization problem around uh parallelism gas fees as well as any other kind of um objectives your optimization functions is targeted for okay but I mean you want everybody to to execute in the same order right yeah they to have the same process for so you mean like your protocol can like decide how to execute them exactly so over time the protocol can evolve depending on the kind of the needs and desires of the network okay um which is which is really nice yeah okay um so how how does this batches transform into this so I have these got these essentially we have these like proof avilability kind of U thing so you have like this U maybe like a hash or something like that and you have this you know the signatures right mhm and you kind of have a bunch of these yeah and these are then um the you know in consensus someone's proposing that this is the the thing we agree onh and once you get the agreement on this at this point in time you should have probably received this batch if Val there is like broadcasting the stuff to everybody and if not then in said before you execute you'll go and fetch that data and so now you end up with the block which has all these transactions kind of matching to each of these this is like the the this is like the batch of actual transactions now TX zero tx1 all that so and so you kind of make sure all that information is present in the execution there before you execute okay and so so this is like this is still a leader leader leader based leader based consensus and so so somebody like rotate between validators who is actually proposing the blog yes they like look at whatever they received the uh proof of availability order them and uh uh kind of do they then the leader execute like is this happens on the leader pretty much or this is all happening everyone's happening so yeah let me show you let me draw the exact like so didin there's kind of a you know everyone's all valers are talking to each other essentially here in consensus they're all talking to each other as well uh they're just but here you're talking about like the proof availabilities essentially MH uh and then here this is this is no networking unless okay there's fetching if you need if you're missing blocks missing uh uh blocks batches of transactions you fetch them so so here leader leader says okay these are the batches that then block and then sends it to everybody else these are the no they send the the only proof availabilities well Pro sorry Pro availabilities and now everybody can execute in parallel exactly yeah and then this is then this is going to happen locally on each machine and this is going to be generally locally and then broadcast to everyone so they can agree on the 2+ one uh sorry yeah two 2+ one um signatures stake yeah so in this case the only thing you leader needs is pretty much say these are the proof availabilities included here's my signature everybody executes it uh and then there magic that happens in the I mean we're skipping some magic in the consens yeah I mean I I would like to zoom in into that more but yeah we can do it okay then everybody executes it in parallel and parallel executes it yes and then I mean storage yeah Pro generation and then they sent out again the kind of this is the certificate kind of the state Roots yeah State Roots sign by the individual validator and then you the two plus one stake on this two plus one stes on this and so everybody just broadcast everybody else the like hey I you know very small amount of data it's just a broadcast all right and that you can use again for like client proofs and all the fun stuff so so here you have like very low kind of latency around creating the batches how frequently does the leader create kind of a block this is pretty much uh just consistently ongoing but like what's uh like it need so like I said basically what we have is like the block time is around 200 to 250 milliseconds today got it yeah so that's every time there's a block created that includes kind of all of the consensus now of course like this is the the happy path but but in most cases I think in in many blockchains we see the happy path yeah and then so this can like take arbitrary I mean dep depends on the execution yeah on execution do you have a limit on like how how do you limit the yeah so we so each each block has a kind of a gas limit individually um and there are a couple different limits here with respect to IO um to gas and storage as well and the idea is you don't want to have any block kind of blocking any other too long um but at the same time like I said the interesting thing is you think about like looking at a window that's beyond a Block M you can start to think about like maybe it doesn't matter as much to have those limits um it's just uh for now we start with the limits and see how it goes yeah okay then badge storage Generations so so this is like in a way validators are continuously processing right they actually they actually don't need this per they don't need that yeah they don't need that technically so this is a stage that we could actually remove in theory but uh we like this just because it allows us to actually support the state sync yeah well for like clients and like those those are nice things and we don't do this this is actually done on a block basis and you could change the kind of you kind of customize how frequently want to do the proof generation then think about like for like client it means you have to do some extra execution which is kind of annoying or find other like longer proofs to to satisfy the client demands yeah yeah okay okay makes sense um I mean would love to zoom in into consensus sure into magic but uh we can do into consensus as well yeah and then we can dive in more into execution and move as well and um yeah let's do that actually maybe for consensus Sasha do you want to yeah sure yeah I'll let you I'll just get into that he's been working on a lot of recens protocols for a long time you can probably talk about uh where we at and where we're going to this is a tough eraser it Roe too much here I think the black color was col yeah that's a workout you get it's the only workout we're getting these days man all right for Sasha all right so consensus yeah yeah let's talk about the new consensus let's do it yeah so we're going to switch into a do based consensus um basically like every said the bottleneck in the lader based consensus is the lader MH and if you think about it you have one leader that needs to disseminate the data to all the validators then the entire performance of the consensus determined by the ability of the leader to push data M right so want to want to eliminate that and this what you talked about data dissemination then then this is basically what we're doing we have instead of the data everybody are doing it in parallel this is validator one of the validators validator one is going to transmit the data to everybody collect the signatures back and then here he has Pro of avilability and then this is going to go into consent just just on a summarize whatever we said this this happening in parallel all validator yeah so um let me erase this thing okay but we have like a few if few so now we have like if you remember data dissemination then consensus we can consensus a blackbox we can use any consensus with this uh metadata already and we already we can scale any consensus now the question becomes like how far we can scale it and what's the latency mhm So currently we have this data dissemination it's one and a half round trips which we need to add to the consensus whatever consensus we're using right so there is so what we want to do we want to want to merge them and also even if the metadata now is small the ler can still be a button leg if there are a lot of proofs that still we can see that the is a button leg so we moving to a du based consensus basically what it means that there's no lader it's liter lless so what we're going to do is the following there is like two separated processes first we want to build a dug after we have a dug we will order the dog so this is how the dog going to look like I'm going to use four validators because this the minimum we need in order to do 3 f plus one so um each this these are the validators validator one validator two validator three validator four and the Doug is going to be constructed in in a round based manner these are going to be the rounds so in every round um I'm going to let's say this is the beginning Genesis then I'm going to produce a new round and you know and I can contribute a new node and this node needs to link to n minus F which is 2 f+ one in our case three uh notes from the previous round this is how I this is how we we collectively Advance the the dog well building the dog that's all we are doing like forget about consensus for a second we're building the duck each of these node is going to contain this proof of availability that we said before I'm proposing and also I'm going to link this so it doesn't have to be the same links right it can be slightly different and this is how this is how it's going to the dog is going to go mhm and um I I can move on to the next round once I receive 2 f+ one um nodes in this round because I can already link to them and I can I can broadcast my next my next node and this is um a local view of one one of the validators because the processes continue ASN completely as synchronously no timeouts nothing you just wait for this 2 plus1 link to you propose a new a new a new node you move forward so each validators might see slightly different view of the Doug M um but the important property here is that because we are we are we're not just sending the nodes we actually using something that's called consistent broadcast it's like weaker form of reliable broadcast which guarantees that if we both see this node it's going to be the same node like there is no equivocation uh this validator even even though it might be B bantine and it's trying to you cannot cuz the way it's going to work is that I'm going to send you the node like each of these node is actually hide one and a half round trips each of this round I'm going to send you a note to everybody they're going to sign send it back to me I'm going to aggregate the signatures and now I have a certificate and then I'm going to and then I'm going to send you this node again with a certificate and then you're going to add it to the Doug so I cannot equate and I cannot produce certificate to different two different nodes in the same in the same round so the property here is that we have two validators they might see slightly different view of the dog but whatever they see is the same and then Al and it means that if we both see this this node it's going to be the same and the entire causal history is going to be the same because the links are the same and like we can go back and everything is the same so this is what we we're doing we keep doing it there's no timeout we just keep building it and now the question is uh okay so we build it that's nice and this is completely symmetric protocol there's no leader everything perfectly loow balanced that's a very nice property the thing now the the real magic here is that why I think people are like like these protocols now is that once I do that I don't do anything else in order to have the consens now you know the consensus is two three on 2/3 and two3 no that's you that that's actually challenging you need to go and order all of this right because um we might see different slightly different uh different um different dos but how can we order it I mean I can I can do 1 2 three 4 5 6 7 you may decide a different order right so there is um I don't know if you want to go to the specific details yeah you want to go yeah let's go okay cool cool okay let's go yeah so good so we're going to get get I'm going to explain bullshark now and that's just then sha is actually an improvement on top of it but let's start with bullshark so we have these rounds right so one this round is going to be an enor round and this is going to be a vult round and so on and so forth this is going to be an enor round and this is going to be a vote round so by enor ah before that I want to say that we're we're going to look each validator is going to look on this Doug and and interet in inet what's the world there and like look at the like look at it as a consensus protocol this is going to be the proposals and the links are going to be the votes like mentally this is how the validator is going to see it so before running the protocol there is some a prior agreement on the en course there is some mapping you can think of around Robin or whatever and we can later expand it to whatever he said we have this lead reputation to use the history in order to choose the future enors based on the strong validators but okay let's say there is some mapping so let's say that this here going to be an end call this validator this validator is going to be an end call and in the future we have we have we have other en course so we all agree on this who is going to be the en course this is easy we just some some function right the terministic function now the now the goal is to go and we want to order the entire thing but we're going to go it in steps first we're going to Al the ankles we're going to basically decide which ankle I want to alel and which ankle I can skip mhm and even though we see slightly different views of the dog we want to come up to the same conclusion this is for safety right so and this is the first step and the the next step is is Trivial because then what we're going to do we're going to go older by enle by enle what we old and we're just going to Old the causal history by some deterministic rule again remember we said that all the causal histories are the same if you apply the same deterministic rule you're going to end up in the same with the same total order so really the challenge is that to decide which enors to order and which to skip we even know which enors are are the candidates we just need to decide which and we not skipping so here is the logic this is this is pretty simple so we're going to go let's say we have a dug okay now we want to aler it and I have some local local view of the duck so I'm going to see and uh uh okay how I going to do it so the example will be nice um yeah okay let's we have I'm going to go enor by enor and do the following I I will try first to directly Commit This enor the commit rule is that if I have F plus1 votes for the enor a vote is just a link from the next um round so in this case we have one two three even four right we have four votes we can all do this Anor m that's okay so we decide to aler this enle now imagine a different scenario and then and then after we do that we move to the next one and try to aler this now imagine a different scenario at which we don't have this so we don't have this link mhm and uh this validator actually doesn't vote this validator doesn't vote for the en call so the en call in this case um f+ one is actually two so this one also does so okay this en code has only one vote mm we need f plus plus one like I said before so we need two in this example and only one so we cannot aler it but the thing the tricky thing is that um we don't have this node yet because we will it asynchronously maybe somebody else already have it right so in some other cases I'm going to draw it in red this is like your this is how you see it right and you actually see this right so you decide to Al the it and I decide to skip it that's bad right that's what we want to avoid so here's what we do we rely on quum intersection here because uh we need f plus one in order to directly commit the enor and remember I told you that we build the dog with 2 f+ one we link every every node links to 2 f+ one from the previous round right so in and and nobody can lie it's also important important property is that you cannot we all see the same links so in if if I directly committed uh I mean you right you have these two things if you directly committed this Anor it means that every future um node here will have a puff to this right this is like a simple quum intersection f+1 to f+1 we have at least one in the intersection so we're going to leverage that and let's say I don't see this right but what I do see is this enor and I'm going to I'm going to aler it because I have all this this is my continuous of the dog so I do all the it so now it's very simple all I need to do is to go back and see if there is a puth between this ankle to this ankle if there is a puth see here there is a pu mhm that means that it could be the case that somebody you in this example actually directly committed this enchor so I need to go and order this anchor before this anchor mhm this is and this is how we maintain the safety I also go now if there is not if there is no PA there are the other logic if there is no puff say this doesn't exist then nobody nobody could nobody could no nobody could directly commit it again this this is just the opposite logic of the previous argument just the the other the other direction so no so it's it's safe to skip it nobody directly committed it so um and we can prove it by induction the reason it's safe to skip it because we're all going to order this and we're all going to agree that there is no puff because we all agree on the caal history and then we're all going to skip it so so that's basically that's basically it that's basically the entire protocol very super simple and from a system point of view all you need to build is the dog which is actually not that simple nothing is simple when are actually very complicated yeah nothing is simple while doing it right now a lot of a lot of like cases and performance building this in 19 that's yeah probably we building hasra what what we building well we we we had hasra we we we were calling transaction flow it was very similar to this um there's a lot of caveats around like when people was holding this or like retrieving it back like do do you I mean we can let let's go step by step y yeah so let's dive in um I guess my my first question is so for for actually making ordering of this anchor right like you you need to get to where you're starting to order the anchor yes how long can it be that you kind of in this halfway state right where like the next anchor you cannot order yet because you haven't seen this and so you kind of keep keep having this yeah good good point yeah that's actually an issue with with bullshark with like the vanilla bullshark protocol because uh I mean if if the validators keep start crashing you don't want them to become their NS to become anchors because then it's actually can go arbitrary um also if you think about um the flp impossibility result the the theory says that it can go indefinitely but I mean nobody everybody are implementing partial synchronous protocols that's kind of true for everybody no no matter what you do there is a low bound if you have a deterministic protocol there is a scenario very unlikely in which you you don't have liveness right but so that's not an issue but from practical point of view if the if the no if the want to start crashing you can go far uh into the future of the dog so as ever said before we have this little reputation that we that we use so um and this is already an improvement that we did in show so we added one of the contribution there was to add a little reputation to the dog so what we do is that once we older an anle we can look at the causal history and look we can see okay these validators didn't participate they didn't contribute nodes there's High likely that they are crashed or like very slow they're struggling now they're far behind so let's exclude them from the possible sets of validators that can contribute enors so we end up with validators contributing enor and Only The Strongest validators Will contribute enors and then it will reduce the scenario that we're describing in which you you will need to skip and go ahead in the duck got it got it so now another now to compare to a leader based protocols what is like nice property of that is that in leader based protocols the leader can also be faulty right but there it's unavoidable that you have you have to use these timeouts right so if the leader is faulty you and typical timeout is around second one one and a half second in which you just do nothing you just you you gave the leader a chance for one and a half second just to realize that he actually crashed nothing happened and then you view change and continue so this is like a one and a half second penalty the good thing about this is that you don't have this penalty because the dog you keep building right network speed mhm and uh yeah you might pay some latency price because it wouldn't hand this enor but this Anor is already here you can continue whereas in little base you just just do nothing for one and a half second and then you then you try again yeah no of course leaderless consensus has huge benefits so uh what happens when so let's say I have a uh so let's say I'm this V V4 yeah so I sent out this with some confirmation but I don't I actually didn't publish this uh so I didn't publish the round two but I sent out approval so what happens to the you point to something that you didn't publish right this is what you're saying so we will ignore you we will like we will u i mean we will process the node but we will not add it to the Doug unless you give us this we will fetch it from you wait from you to give it to us but if you don't then we just not add it to the DG and we will wait to receive 2 f+ one different nodes and we consider this byzantin behavior and because there is only F Byzantine to f+1 Are Not bantin So eventually we will get to F plus1 nodes that behave correctly and then we can continue M yeah so pretty much just like I mean obviously uh so I'm I'm sending this to to some other validators they haven't seen this they'll go fetch it yes right from me I'm let's say maybe giving to some not giving to others uh do they like broad rebroadcast like intermediate notes to each other or uh they that's they don't that's an implementation details we could but we now we don't do it what we do is what we actually do is that this um well I'm going to give you I I point to a certificate yeah and the certificate is 2 f+1 signatures MH um so I can go and fetch from any of this I don't I don't need to fetch specifically for this this is for load balancing I can and fetch for any of the um signatures any of the signing validators right sorry I didn't know yes so like like I said a node a node here before remember I said yeah yeah you collect the signature I collect the signature and then here I already send you something with the signature so you can add to the dog and when I point it back I actually point it back to the certificate like to a pretty much aggregated signature yeah yeah to aggregated but it has the bit the bit Vector I know I know who signed it so I I could just go and fetch from this validator for but I don't want everybody to go and fetch from validator for so I'm want to load balancing it so I can use the information in the v Vector of the signature because whever in the signature they have should should have should have it right should have should have the block because they signed this they signed it right so uh so I will go and like load balance like in yep got it f plus one the two f plus one have it yeah yeah for sure yeah under the buting assumption at least F plus1 I'm going to have it here all right and so uh and so you're progressing pretty much to next round after you have uh 2f+ one yes published uh so I guess like latency wise like let's say I publish this and then I'm waiting to see everybody else's like meanwhile it can be just ordering transactions so that I can I just thinking what what's the delay here looks like what's what's the delay yeah so for for an enor to older an enor you need this round and this round yeah and to all the um to all this node for example is going to be all the as part of this en Co history so this is additional round yeah and this one is will actually require more so because this will need this round this now this now and this one this will come for but in short we get rid of this we can we can talk about this iation as well so in sh we actually make sure that we have an anchor in every round this is the one of our latest optimization and the idea is actually very simple just took us some time to realize it but let's say we believe that this protocol works okay safe and everything works and like like I told you we have this that we we divide it into two phases first is to agree on the end course and then we agree on everything as part of of agreeing on the enor in particular we agree what is the first Anor we need to order right in this example it was this but does necessarily has to be we might skip him and agree this right but if we believe bullsh is correct that's in particular also correct we all agree which anchor we need to order first so in sh what we said we just realized it that it doesn't matter how we went in the future what we did in the dog we can always deterministically go back and reinterpret the dog again so what we do is that think of it think about it we start like I said before an instance of bullshark and we come to a conclusion that this is the first anchor we need to all do right now if we have to run the bullart protocol the vanilla bullart protocol we would skip this round and try to hold this enle right and so on however we we realize that we don't need to do it so now we we all agree this enle we need to order we go order it caal history like before and now we just decide okay let's kill this bullshark instance we don't care about about it anymore we can just deterministically and we start over from this from this from this round and do exactly the same we're going to change the mapping and say okay this was the ankle now this is the ankle right mhm and now let's start a CLE clean instance of bullsh starting starting from here mhm and and and again and again and again we're going to continue with the same trick and then we're having we're having an anchor in every round MH um yes and that's uh that's one and then another one is the lead reputation like I said before because now once we order this enle we can not only that we are moving the ankles we can only change the mapping based on the history yeah so your assignment function pretty much changes based on history yep and with this observation together we can get rid of timeouts completely because like I said before this is a terministic protocol to fight the flp we need to use timeouts so one of one of places we use timeouts in bullshark is that we wait like I told you before that we we build the Doug in network speed this wasn't entirely true in order to get liveness you need to wait for 2 f+ one but you also want to get a chance to get the enor m so you're going to wait for uh either you wait for and and and timeout or anchor right this is this is what you do so now with this leader reputation we actually don't need this because um if the anchors are fast like one of the fast validators fast or fast validators everybody going to vote for them good if they crashed um we are going to if they crashed or slow the little reputation mechanism will make we we just filter them out in the future rounds so we end up with the strongest validated contributors and call and we just get rid of the time outs we just keep we just keep going um of course for the flp just to be you know theoretically on the safe site so nobody can argue we don't have liveness right so what we can do is completely unrealistic but say like some configurable number of anchors we cannot H order like we skip 10 anchors then we can say okay let's fall back to use timeouts and then we go back to the same theoretical liveness guarantees we had before where in practice we will never skip the en course this is like you a very powerful adversary yeah to pull this off right so um well we pretty much need split Network and somebody yeah yep manipulating dags on both sides yeah I personally really like it because uh I come from Fury and there FP you take it very seriously and you cannot do the terministic protocols without timeouts so yes we we cannot but we can like very corner it into some very very theoretical place right so in practice we will never use timeouts MH so yeah par like if you remember from the beginning execution then par sorry consensus par execution so in par execution the goal is really to execute the the given transaction as fast as possible and we can we can use all like threads in the system right and if we need a scalable algorithm to make take use of threads and in order to be able to do it we need to the goal is to basically reduce conflicts mhm my so that that's the goal reduce conflict that you want to execute in parallel you need to reduce the conflict so there are like a few ways to do it one is like super trivial which is like a um excess vity that's like meaning that if you have if you group together some resources mhm Resource Group and you group them together and you have like Resource One resource two resource three and this is like you do it because it's friendly for your storage you want to store them together get them together and then you have like one transaction that try to um to touch R1 and another transaction try to touch R2 if the if you if the granularity of the ex is entire Resource Group that's going to be a conflict right so you always need to you always need you always need to try to make them do made the they made the ex is as lity as possible so if I touch L1 and you touch L2 we should be able to go in parallel so this is this is trivial stuff I think you just want to the more important thing I I want to talk about is there is like this debate now about static versus Dynamic um parallelism MH so I want to maybe touch about this a little bit so we have static versus Dynamic okay okay and by Static I mean that U the user needs to specify the read and right set mhm like completely yep and dynamic is completely agnostic agnostic yeah meaning that the user doesn't need to give anything just provide you the transactions well this here that's to use need to give you light set so yeah so this is this is less convenient for sure this is maybe a little bit inconvenient but I want to uh set yeah I want to I want to from a performance point of view I want to touch upon it and why why we decided to do Dynamic and I want to go and maybe give an example of a one use case which is an older book let's say I want to implement a concurrent older book older book on chain so for an older book you have basically this price and you have the uh this is the amount so so and say this is 10 so I want to I want to sell this amount in 10 this amount and 11 you know and the higher the price the more I want to sell and the buyer wants to buy a little bit in nine more than eight seven like this is how old the book looks and then here we can have a match right this is what we do so how would you represent something like this um on chain this is like this is a market let's say Bitcoin and ethereum so so you have a you you need some sort of tree concurrent tree um so you will have some something like this mhm uh this is how you represent the sellers and you will have also the representation of the buyers uh okay now you want if you want to place a Noel modify a Mel cancel a Noel you need to go to the tree and basically find your place and add a node right or delete a node remove a node so if you think about um static parallelism like uh Solana s this this these blockchains you need to statically declare everything you're going to touch now the thing is that you don't know you don't know which path you which path you're going to go because you don't know the state of the tree so there is nothing better you can do with static paraly than to lock everything you just lock everything right and if you lock everything then basically there's no concurrency if me and you are going to going to place a we're going to place a a bid yeah we loog everything we look everything we're going to go sequentially where in Dynamic paral and say I want to add these and you want to add this we this is why it call dynamic because we dynamically on the Fly realize what we are actually going to touch so you are touching this area I'm touching this area there's no no reason why we cannot go go in in parallel so it gives you like Dynamic parallelism gives you much more much more concurrency we can place a lot of and think of this a big tree right so we can we can place a lot of bids concurrently cancel modify same on the other side and um this is like a strong argument why why we choose dynamic because this is one use case but we believe like more more use case most complex use cases will come in which static will just the GL the Locking grity is going to be bad where dynamically you can just figure out a better complexity I mean one of the limitations here is that also like you actually need to declare specific pieces of data that you're touching which becomes a lot especially in this case and it expires quickly right by the time you prepared transaction you sent it and somebody already matched three right and like now your transaction is valid now now transaction about that's that's that's the Solana right and they all actually no matter you actually need to limit right because you specified yeah they have 16 pieces of account like account data that they can tou you they can increase it but somewhere they will need to put a limit right so if the tree is bigger then you basically cannot you need to hack some stuff in know they have a crank for that yeah y yeah so dynamically just no issue right you just go and figure out on the flight mhm so how how do you do that yeah good good next question yeah uh all right so let's me me eras this so yeah block STM is like something we're really proud of um actually there are a few other blockchains that currently already either implementing it or already deployed it so it's uh which for us is like a big researchers for us is a big deal you know somebody is using your somebody's using your check yeah this is this is more than any best for us you know so so okay so let's talk about block STM we're going to for blackm we're going to take advantage of a few Concepts um there is the STM as the name concept OC concept and multi multiversion data structure okay so what is the STM um and and these all of these were like heavily studied in the literature it just uh not so much in practice unfortunately but with this specific use case for blockchain we figured out a very practical way to to and I will explain in a second what exactly led to this to combine everything together into a system that actually performing very well so STM is a sof stands for software transactional memory um just think of like a layer of software that abstracts away all the concurrency you you want to do so it's like makes programming much easier you don't need to deal with locks you're just going to say this this piece of code I want to execute conai like automically send it to the library the library will do the magic and execute it which is perfect fit for blockchains because what we have is actually transactions that we want to execute automically now this OC is optimistic on currency control for stms we have two technique we have pessimistic on currency control and optimistic on currency control and this is BAS this most of the time gives better performance and the general structure is that you want to go optimistic and you want to say oh I'm going to execute the transaction and you split into you you you you can execute transaction then you can validate the transaction so you execute all the transactions and then you you go after the fact and some later point and you validate if validation succeeds basically what you what you most of the time you are doing is that you are checking that whatever you read during execution the values that they're still they they haven't changed so whatever decision you made back then they're still relevant you can commit so then you you execute optimistically then you validate if validate validation pass you can commit otherwise you need to abort retry re execute so this is optimistic concurrency control and multivalue uh multiversion data structure is just also like simple concept like say you have a memor some memory location and you have basically what you have is a mapping uh between the version and let's say you have like a you have a you have you have a list of transactions that you want to execute so now instead of having one memory location for transaction the transaction keep overwriting M you basically remember all the all the values that this this transaction wanted to write to this memory location so let's say one wres 100 transaction two is going to write 200 and transaction four is going to write 400 so you have all these information why is it useful because for us uh this is this is like okay why why do we actually manage to make all these things work very efficiently is that we realize that we have this fixed order that we need to follow mhm right the later we realize that we can like do whatever we want with the order but eventually we just thought okay consensus gives us an order we need to follow it right and it has to be not only the result has to be deterministic across other validators it's actually need to respect the order so it looks like a limitation you limit your system you probably should get like wor performance but turns out that um it it actually give us a better performance and the reason is because we can follow the older uh explaining for different point of view so first with this multiversion data structure we can completely eliminate right right conflicts this is kind of know technique nothing like we invented here is that think about these are these are the rights everything is executed in parallel these are the rights and now transaction three comes and want to read right so the way it works is that the transaction will Traverse uh all the versions and will take the higher version that is smaller than the transaction number so it will take this value this this is going to be the read because because we're going to respect this order of execution transaction three needs to go after two so so three is going to is going to read the value that to wrote right so right right conflict is not an issue and yeah remember the goal is to reduce conflict so no right right conflicts now the question is um read right conflicts right how can we how can we how can we minimize the problem minimize the number of aborts number of executions so what we're going to do is actually Al also simp idea so let's for example in this example say that transaction number um two we executed it uh we try to execute right optimistically but then we figure out it needs to be aborted because transaction one conflict with some transaction one right so what we are going to do is what and we're going to go back to this multiversion data structure now uh we can be much smarter we remember I told you we have no no idea at the beginning what are the read right sets right but now we have much more information that we can use because we already know we executed transaction two one time we already know we already have a good estimation where transaction two wants to write particular it right to this memory location so what we're going to do we going to delete this value and we keep the entry and instead we are going to have some buttom here okay the buttom will represent the fact that there was high likely transaction two is going to right here currently it's going to be re executed it's AB bolted it's in the process of being re executed but it's high likely it's going to right here so when transaction 3 now comes and read it sees that there is an entry for transaction two the transaction should take this value but this is the bottom here MH which means that there is no point for transaction to go and execute because it's going to be a bolted right so instead we're going to just suspend transaction 3 and and transaction three is going to um J's going and we were going to wake it up whenever two finishes and then transaction three can go so we because we have this fixed order and we respect the order we are able to reduce most of the aborts and most of the execution there is really nothing better you can do if you think about it transaction three depends on transaction two right so transaction 3 has to go after transaction two we couldn't execute them in parallel and now we we executed transaction three until as much as we could but now we need to suspend wait for two and then we can continue this is um this is like one of the call contribution of block and why can scale so you pretty much just so you like you run pretty much all of this VMS in parallel and then yes when you see it reading something that like transaction two have updated you just suspend it until until until it's ready until the value is ready and then we can we can go again with transaction three maybe in the future we will need to suspend it again but we will we we keep doing progress yeah yeah so another thing that we another way we leverage this order is how we actually dispatch the tasks right we have this transaction a lot of threads which which one is is doing what and the trick is always withm is to minimize the overhead if you pay a lot of overhead in order to synchronize the thread it's not going to scale it's not going to work so what we what we end up doing is that we we maintain like roughly speaking we maintain two indexes uh validation index and uh execution index mhm so execution is the next transaction we the index we're going to point to the next transaction we need to execute and validate the next transaction validation the next transaction need to validate and because we respect the older we also need to validate in all of these things so basically the all all the contention is going to be on these two indexes when a trade comes wants to check the task going to read both of them see which one is the lower and then going to perform a Fetch aned and then just go and take okay I need to go and execute this transaction and then and then the pointer and then the index will Point here so the next next red will going to come and take this transaction so very very minimum overhead for all this uh uh collaborative schedule for like the transactions execution but I guess I mean there's a there's a big Ram overhead of having this like transaction suspended right I mean not big but like per transaction there's some amount of ram that you allocate for running VM and everything yes and so as you kind of spinning up more and then going back to revalidate so yeah that actually was an issue the the thing was that um at the beginning um move doesn't support um there's no USNC Concept in move so we cannot just suspend inside move itself we had to Abol the first version of block STM we had to Abol the transaction and re execute from scratch everything and it was kind of annoying because algorithmically we didn't need to do it we already reached some point we can continue from there so now what we are currently doing is that we're implementing our own H tread garage tread pool in in rust inside inside rust and we're going to use it so we have all these abilities that we need to block STM suspend the thread wake up the thread so um then it will give us another performance gain I hope got it um okay and then so just for my understanding uh so let's say you start let's say you have four threads you start executing all this four in parallel uh let's say there is like a indeed I mean all of them actually write uh and then T2 reads from T1 so they'll write into some address but then T2 actually specifically reads from 1 so they will all execute t334 will complete with their own but You2 will suspend because it it reads whatever T1 wrote out it will uh then wait until two un finishes Y and then you'll like unsuspend it or restart itend it yeah you restart it yeah and then it will write out but then actually that value will be replaced by D4 and you just kind of Def when you save the final State you just take whatever the final values are for that final step yeah that's exactly what going to happen and the cool thing about it is that if you don't have any conflict yeah it's you just execute right nothing you don't you don't need to pre execute you don't need nothing you just execute but if you have conflicts you already have pretty good estimation of what's going on and for the future you can be very efficient okay yep yeah let's yeah I mean we're kind of D um all right cool yeah this is this is really cool um I think this was a really educational hopefully so uh thanks Sasha and Avery for joining and uh yeah join more episodes of whiteboard series thank you","transcript_synced_at":"2026-11-03T06:02:56.516Z","segments":[{"start_seconds":4,"text":"hey everyone this is Ilia joining from Bay Area palalo it's amazing to be in the office of abas so we have Avery and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=4s"},{"start_seconds":12,"text":"Sasha from abas guys introduce yourself and let's dig great I'll just keep it short I'm Avery uh co-founder and CTO","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=12s"},{"start_seconds":20,"text":"here at abas Sasha head of research with awesome so let's dive in so let's","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=20s"},{"start_seconds":28,"text":"maybe explain what you're trying to achieve as aptus and then uh go from there and start decomposing the system sure so just a little bit of history","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=28s"},{"start_seconds":37,"text":"about apas uh 2018 uh you know meta started kind of launching an initiative uh called Libra and then DM over time uh","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=37s"},{"start_seconds":45,"text":"I was very fortunate to work with folks like Sasha and a bunch of other people on our team on this project from from almost day one um and the idea was how","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=45s"},{"start_seconds":53,"text":"do we kind of support use case of billions of people uh you know at meta and beyond that beyond the leverage decentralized technology to do things","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=53s"},{"start_seconds":61,"text":"like remids and payments uh and support like the largest use cases in the world uh and so from that kind of point on we thought about from a first principal","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=61s"},{"start_seconds":70,"text":"standpoint what do we want out of a decentralized infrastructure and that started off with the idea of a blockchain like you know thinking about the way that we scale systems that meted","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=70s"},{"start_seconds":78,"text":"before in the past scaling you know to massive amounts of transaction throughput uh having maintaining great latency and then from a language perspective how do you kind of think","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=78s"},{"start_seconds":86,"text":"about smart cont languages from from the beginning again you know if you think about the general purpose languages they're they're built to support very different things high performance fast","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=86s"},{"start_seconds":93,"text":"iteration fast development times fix bugs in prod uh roll forward that kind of thing whereas smart contract development uh harder to fix bugs that","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=93s"},{"start_seconds":102,"text":"are al already in the network also dealing with the idea that these bugs may cause massive asset loss of course is a is a huge problem and so with that","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=102s"},{"start_seconds":110,"text":"we kind of have the idea of a move language coming forward uh the idea of the DM blockchain going forward and then building these in tandem the same way kind of think about you know Hardware","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=110s"},{"start_seconds":118,"text":"software codesign being to think about the developer platform and kind of the infrastructure behind it you know working in tandem towards this ultimate goal of of mass skill adoption all right","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=118s"},{"start_seconds":127,"text":"so for later ones I usually like to go from bottom to the top so consensus kind of the design of the state and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=127s"},{"start_seconds":136,"text":"transaction processing and then VM and then kind of polish on VM around like gas transaction sure we can do that yeah","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=136s"},{"start_seconds":144,"text":"I thought maybe it also be useful to think about like the way our architecture set up be an easy way to to start off with so the first thing we have in the way we think about","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=144s"},{"start_seconds":150,"text":"transaction processing is a data dissin dissemination phase and so you have clients that are","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=150s"},{"start_seconds":160,"text":"going to be basically pushing transactions uh into validators and so we have the notion of a Valor set um","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=160s"},{"start_seconds":167,"text":"many validators here that all have um fully fully connected uh one more","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=167s"},{"start_seconds":176,"text":"here and clients can kind of send these transactions to each any one of these validators mhm and in order to make sure we leverage effective use of this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=176s"},{"start_seconds":184,"text":"bandwidth um we leverage something called like a narwal which Sasha actually worked on um that technique and it's also called corm stor our current","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=184s"},{"start_seconds":192,"text":"model and so the idea is that by having each of these validators um kind of producing their own blocks and then being able to pass those blocks off to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=192s"},{"start_seconds":199,"text":"each other uh we can leverage the full bandwidth of the Valor set itself and so if you think about like traditionally in many older systems you have this idea of","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=199s"},{"start_seconds":207,"text":"a leader this protocol leader kind of push transactions to other uh values in the network you have to deal with kind of either a limitation of bandwidth and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=207s"},{"start_seconds":216,"text":"or a latency delay and so for instance if each of these nodes has like n bandwidth you know and I can you know have a leader that's pushing out to these other two nodes I have to divide","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=216s"},{"start_seconds":224,"text":"this bandwidth out going into two and I can push these transactions at a rate of n/ two uh and this then slows the weight the rate at which I can kind of uh get","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=224s"},{"start_seconds":232,"text":"my consensus complet over time so instead what you have here is everyone is going to be broadcasting these transactions that they receive in","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=232s"},{"start_seconds":241,"text":"blocks to each other kind of all toall communication but they and they're leveraging that n over two bandwidth and now you're also receiving en bandwidth","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=241s"},{"start_seconds":248,"text":"which means you kind of get a full utilization of your network pipe which is great the next phase is what we call","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=248s"},{"start_seconds":256,"text":"consensus and so for consensus we've actually iterated a lot over this design over time we started off with ideas of Hot Stuff uh this is kind of my first","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=256s"},{"start_seconds":263,"text":"project I worked on when I was at uh meta um and I think we quickly realized there were things that were missing like there was no pacem think ER there was idea of you just having exponential back","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=263s"},{"start_seconds":272,"text":"off until you kind of achieve this partial synchrony model uh but that in practice that works really really poorly uh and so we started to make changes to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=272s"},{"start_seconds":279,"text":"it like adding things like leader reputation uh where you could kind of uh support the you know the idea of how the leaders done in the past have they voted","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=279s"},{"start_seconds":288,"text":"have they proposed correctly successfully in the past and then also um things around an active pacemaker where we kind of um proactively if you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=288s"},{"start_seconds":296,"text":"see like the leaders Miss around you can kind of start broadcasting to each other and trying to figure out the next lead ought to be and advance that cycle forward faster um that kind of led into","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=296s"},{"start_seconds":303,"text":"um a new line of work which you know Nar one Tusk is one thing and then bullshark protocol which is dag base protocol and then finally unto SCH um but the idea","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=303s"},{"start_seconds":311,"text":"here is that we have our consensus protocol here that that doesn't agree it doesn't pass off transactions as a part of that protocol that transactions H","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=311s"},{"start_seconds":319,"text":"propagation happening in the first stage of data dissemination this is where a lot of the bandwidth goes because if you think about again the system every Valor needs to receive every transaction","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=319s"},{"start_seconds":327,"text":"eventually and every executed output um and this is the bulk of the traffic and that's where we maximize our throughput in the system then we have a consensus","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=327s"},{"start_seconds":335,"text":"protocol and as part of consensus we also kind of have this interesting like Oracle Services built into it and so we have uh time stamps like uh","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=335s"},{"start_seconds":343,"text":"that are built in and we also have um uh R onchain Randomness which is really interesting which I probably won to dig","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=343s"},{"start_seconds":351,"text":"into too much because a littleit deep but the time stamp process is like you know as a part of the consensus protocol you only vote when certain times are passed times are part of the the voting","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=351s"},{"start_seconds":359,"text":"protocol itself and that means that every single block has this time stamp associated with it and typically inas you see something like 200 to 250","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=359s"},{"start_seconds":368,"text":"millisecond blocks and every time you know you have this very accurate clock that kind of goes forward with that process after this we have uh parel","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=368s"},{"start_seconds":381,"text":"execution sorry about that bad handwriting and so at this point in the in the process we have agreed on a set of transactions that will be um kind of","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=381s"},{"start_seconds":390,"text":"uper execution and so you kind of Imagine these set of blocks like you know block zero and then block one block","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=390s"},{"start_seconds":398,"text":"two full of all full of transactions so you know TX tx0 tx1 tx2 tx3 so on so","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=398s"},{"start_seconds":406,"text":"forth and this is where I think a lot of the magic happens all kind of Sasha R into but we leverage um block","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=406s"},{"start_seconds":414,"text":"STM as a technique for dynamic kind of parallelism we can kind of Val value all the transactions we can we currently evalue block by block but you can","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=414s"},{"start_seconds":423,"text":"imagine looking at a window blocks and the idea here is that you have this set of transactions you're going to execute and we want to make sure we fill up all the resources in the in the in the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=423s"},{"start_seconds":431,"text":"computer so all the CPU cores all the memory we have all the ssds you know so on so forth uh we want to leverage all those resources to maximize and and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=431s"},{"start_seconds":439,"text":"execute as much as we can in parallel the next phase is going to be on the um batch storage","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=439s"},{"start_seconds":449,"text":"so once you kind of have done all this execution you need to write to to in our case SST um in many cases I think that's the case uh and so what happens is you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=449s"},{"start_seconds":457,"text":"know we try to bulk up our rights into larger uh larger operations this allows us to maximize the throughput and you know reduce the number of seeks that are","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=457s"},{"start_seconds":464,"text":"necessary we also start to explore um how do we take our Rox DB instances and scale them out to multiple","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=464s"},{"start_seconds":472,"text":"instances so you have you know a couple of these in the back end and this allows you actually get much much higher through but we call this kind of storage charting it is an internal sharing","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=472s"},{"start_seconds":480,"text":"technique it does not require anything like anything user visible um just gives us a lot more more throughput on kind of a critical path of our system and the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=480s"},{"start_seconds":489,"text":"Final Phase is kind of the the proof generation so at this point we kind of","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=489s"},{"start_seconds":496,"text":"generate the um the proof on the state state root hash and then we broadcast this to other validators in the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=496s"},{"start_seconds":503,"text":"network and so at this point once you have 2+1 uh stake in the system uh that has a Pro on it you kind of know that","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=503s"},{"start_seconds":510,"text":"this is the state that's agreed upon you can use this for State synchronization you can use this for light client proofs any kind of things like that uh so I think what's also important about the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=510s"},{"start_seconds":519,"text":"think about this design is that each of these phases can be executed All In Parallel the dissemination is consistently going on consensus is going on execution storage and improv","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=519s"},{"start_seconds":527,"text":"Generation all happening at the same time and so this means we kind of Leverage all the ma machine resources pretty efficiently and also the also we can you know if you think about we can","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=527s"},{"start_seconds":536,"text":"expand this this offer stack to work on the set of machines over time as well um because every phase is kind of","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=536s"},{"start_seconds":543,"text":"inherently pretty parallel so any questions thoughts yeah a lot so yeah start di through this yeah let's start digging in okay so data dissemination","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=543s"},{"start_seconds":551,"text":"right so you have a bunch of validators uh they're interconnected yes um so let's say I'm validator like one I","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=551s"},{"start_seconds":560,"text":"received uh a transaction uh what what am I like I'm just sending it out to everybody else like what what are the guarantees that","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=560s"},{"start_seconds":568,"text":"I'm looking for yep so the first thing you're going to do is you're going to basically broadcast to other validators and network and first you wait a period of time a small period of time you batch","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=568s"},{"start_seconds":576,"text":"it up and the batching is meant to be more more of like a efficiency win right you can process transaction by transaction but that's that's not very efficient for the system right so by","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=576s"},{"start_seconds":584,"text":"batching you have like a signature now on that batch we call this the proof availability I'll write this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=584s"},{"start_seconds":593,"text":"down Prevail ability and this kind of when you sign this you're saying I'm going to store I'm going to store this data I'm not going to forget about it right uh and that way when I start going","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=593s"},{"start_seconds":602,"text":"to consensus I'm sending this proof and I'm agreeing on these proofs and the important thing about these proofs means that I don't actually even need uh the the transactions to vote on the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=602s"},{"start_seconds":611,"text":"transactions I'm agree on on I can fetch those just prior to the pr execution phase okay so yeah so so pretty much","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=611s"},{"start_seconds":619,"text":"there like small batches of transactions that as you said you wait for like smaller amount of time like very short period of time yeah kind of customizable","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=619s"},{"start_seconds":627,"text":"depending on each validator yeah then you have a signature I send it to everybody else do they like if they have","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=627s"},{"start_seconds":634,"text":"the same transaction and they shared it what like what what is a ah so you're actually what happens with duplicate transactions duplicate transactions as","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=634s"},{"start_seconds":641,"text":"well as like do it propagated I mean you you're saying everybody's interconnected what if like some links are severed yeah","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=641s"},{"start_seconds":649,"text":"so if some links are SE so if some links are severed like as long as one when a Valor broadcast this information to others they're getting the proof","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=649s"},{"start_seconds":656,"text":"availability they're get two of plus one stake signatures mhm so they collecting like for for any","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=656s"},{"start_seconds":663,"text":"transactions they pretty much for for batch for batch for batch for a batch of transactions you collect this proof availability on it okay so so I I","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=663s"},{"start_seconds":671,"text":"created my batch I send it to everybody else and everybody else then signs as well this batch and sends it to everybody else no they sign it and since","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=671s"},{"start_seconds":679,"text":"you're broadcasting that since the the creator of the batch is responsible for making sure everyone gets it okay they don't need they don't need to broadcast anybody else necessarily what happens is","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=679s"},{"start_seconds":686,"text":"that he this this valer is kind of responsible for making sure it's disseminated yeah if they don't if they don't if whatever reason they don't receive it before execution they'll have","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=686s"},{"start_seconds":694,"text":"to fetch it and they can fetch it when you have this proof of avability you know exactly who has it you know you can go get it from but but how does this get","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=694s"},{"start_seconds":702,"text":"aggregated like by the whoever disseminated it yes okay so you send it and they reply with their signatures that they received it okay so you just","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=702s"},{"start_seconds":710,"text":"get you collect receipts okay yes and then broadcast that information and that information now is that that information which is the proability is used in the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=710s"},{"start_seconds":718,"text":"consensus protocol okay so so now pretty much when you move into contest protocol you have uh like","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=718s"},{"start_seconds":727,"text":"batch like batch validator one for example batch validator 2 with a bunch of signatures right yeah they're","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=727s"},{"start_seconds":735,"text":"aggregated but yes yes yeah well aggregate signature yeah is like BLS yes","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=735s"},{"start_seconds":741,"text":"okay all right so you have this batches mhm okay and then","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=741s"},{"start_seconds":752,"text":"uh and then so now consensus is what actually orders them yes and I think it's less of an ordering which is important here what is is you're kind of","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=752s"},{"start_seconds":760,"text":"agreeing on that set of transactions and the reason why I called that out specifically is that when it comes time to like we've agreed on these blocks basically and we've agreed that these","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=760s"},{"start_seconds":768,"text":"will be sent to execution the nice thing is that I kind of at runtime here as long as the valers agree on it they can do all kinds of interesting Transformations on that data","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=768s"},{"start_seconds":776,"text":"sure they can reorder it uh to maybe throw it MV type attx if you want to you can also try to extend that window if you want to achieve more more","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=776s"},{"start_seconds":784,"text":"concurrency in the way that you execute those transactions you can even optimize if you want to do it around gas as well like if you want to think about the way that gas markets work in prioritizing","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=784s"},{"start_seconds":791,"text":"certain transactions we can also add the optimization there so it's kind of like a later late stage runtime um optimization problem around uh","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=791s"},{"start_seconds":799,"text":"parallelism gas fees as well as any other kind of um objectives your optimization functions is targeted","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=799s"},{"start_seconds":806,"text":"for okay but I mean you want everybody to to execute in the same order right yeah they to have the same process for","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=806s"},{"start_seconds":814,"text":"so you mean like your protocol can like decide how to execute them exactly so over time the protocol can evolve depending on the kind of the needs and desires of the network okay um which is","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=814s"},{"start_seconds":823,"text":"which is really nice yeah okay um so how how does this batches transform into this so I have","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=823s"},{"start_seconds":831,"text":"these got these essentially we have these like proof avilability kind of U thing so you have like this U maybe like a hash or something like that and you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=831s"},{"start_seconds":838,"text":"have this you know the signatures right mhm and you kind of have a bunch of these yeah and these are then um the you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=838s"},{"start_seconds":846,"text":"know in consensus someone's proposing that this is the the thing we agree onh and once you get the agreement on this at this point in time you should have","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=846s"},{"start_seconds":853,"text":"probably received this batch if Val there is like broadcasting the stuff to everybody and if not then in said before you execute you'll go and fetch that data and so now you end up with the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=853s"},{"start_seconds":860,"text":"block which has all these transactions kind of matching to each of these this is like the the this is like the batch of actual transactions now TX zero tx1","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=860s"},{"start_seconds":870,"text":"all that so and so you kind of make sure all that information is present in the execution there before you execute okay and so so this is like this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=870s"},{"start_seconds":878,"text":"is still a leader leader leader based leader based consensus and so so somebody like rotate between validators who is actually proposing the blog yes","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=878s"},{"start_seconds":887,"text":"they like look at whatever they received the uh proof of availability order","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=887s"},{"start_seconds":894,"text":"them and uh uh kind of do they then the leader execute","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=894s"},{"start_seconds":902,"text":"like is this happens on the leader pretty much or this is all happening everyone's happening so yeah let me show you let me draw the exact like so didin","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=902s"},{"start_seconds":909,"text":"there's kind of a you know everyone's all valers are talking to each other essentially here in consensus they're all talking to each other as well uh","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=909s"},{"start_seconds":917,"text":"they're just but here you're talking about like the proof availabilities essentially MH uh and then here this is this is no networking unless okay","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=917s"},{"start_seconds":924,"text":"there's fetching if you need if you're missing blocks missing uh uh blocks batches of transactions you fetch them","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=924s"},{"start_seconds":932,"text":"so so here leader leader says okay these are the batches that then block and then sends it to everybody else these are the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=932s"},{"start_seconds":939,"text":"no they send the the only proof availabilities well Pro sorry Pro availabilities and now everybody can execute in parallel exactly yeah and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=939s"},{"start_seconds":947,"text":"then this is then this is going to happen locally on each machine and this is going to be generally locally and then broadcast to everyone so they can agree on the 2+","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=947s"},{"start_seconds":955,"text":"one uh sorry yeah two 2+ one um signatures stake","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=955s"},{"start_seconds":964,"text":"yeah so in this case the only thing you leader needs is pretty much say these are the proof availabilities included","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=964s"},{"start_seconds":971,"text":"here's my signature everybody executes it uh and then there magic that happens in the I mean we're skipping some magic in the consens yeah I mean I I would","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=971s"},{"start_seconds":979,"text":"like to zoom in into that more but yeah we can do it okay then everybody executes it in parallel and parallel","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=979s"},{"start_seconds":986,"text":"executes it yes and then I mean storage yeah Pro generation and then they sent out again the kind of this is the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=986s"},{"start_seconds":995,"text":"certificate kind of the state Roots yeah State Roots sign by the individual validator and then you the two plus one stake on this two plus one stes on this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=995s"},{"start_seconds":1004,"text":"and so everybody just broadcast everybody else the like hey I you know very small amount of data it's just a broadcast all","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1004s"},{"start_seconds":1013,"text":"right and that you can use again for like client proofs and all the fun stuff so so here you have like very low","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1013s"},{"start_seconds":1020,"text":"kind of latency around creating the batches how frequently does the leader create kind of a block this is pretty","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1020s"},{"start_seconds":1029,"text":"much uh just consistently ongoing but like what's uh like it need so like I said basically what we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1029s"},{"start_seconds":1037,"text":"have is like the block time is around 200 to 250 milliseconds today got it yeah so that's every time there's a block created that includes kind of all","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1037s"},{"start_seconds":1045,"text":"of the consensus now of course like this is the the happy path but but in most cases I think in in many blockchains we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1045s"},{"start_seconds":1051,"text":"see the happy path yeah and then so this can like take arbitrary I mean dep","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1051s"},{"start_seconds":1059,"text":"depends on the execution yeah on execution do you have a limit on like how how do you limit the yeah so we so each each block has a kind of a gas limit individually um and there are a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1059s"},{"start_seconds":1067,"text":"couple different limits here with respect to IO um to gas and storage as well and the idea is you don't want to have any block kind","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1067s"},{"start_seconds":1075,"text":"of blocking any other too long um but at the same time like I said the interesting thing is you think about like looking at a window that's beyond a Block M you can start to think about","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1075s"},{"start_seconds":1083,"text":"like maybe it doesn't matter as much to have those limits um it's just uh for now we start with the limits and see how it goes yeah okay then badge storage","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1083s"},{"start_seconds":1092,"text":"Generations so so this is like in a way validators are continuously","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1092s"},{"start_seconds":1099,"text":"processing right they actually they actually don't need this per they don't need that yeah they don't need that technically so this is a stage that we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1099s"},{"start_seconds":1106,"text":"could actually remove in theory but uh we like this just because it allows us to actually support the state sync yeah well for like clients and like those","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1106s"},{"start_seconds":1114,"text":"those are nice things and we don't do this this is actually done on a block basis and you could change the kind of you kind of customize how frequently want to do the proof generation then","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1114s"},{"start_seconds":1122,"text":"think about like for like client it means you have to do some extra execution which is kind of annoying or find other like longer proofs to to satisfy the client demands yeah yeah","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1122s"},{"start_seconds":1131,"text":"okay okay makes sense um I mean would love to zoom in into consensus sure into magic but uh we can","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1131s"},{"start_seconds":1141,"text":"do into consensus as well yeah and then we can dive in more into execution and move as well","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1141s"},{"start_seconds":1148,"text":"and um yeah let's do that actually maybe for consensus Sasha do you want to yeah sure yeah I'll let you I'll just get","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1148s"},{"start_seconds":1155,"text":"into that he's been working on a lot of recens protocols for a long time you can probably talk about uh","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1155s"},{"start_seconds":1163,"text":"where we at and where we're going to this is a tough eraser it Roe too much here I think the black","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1163s"},{"start_seconds":1170,"text":"color was col yeah that's a workout you get it's the only workout we're getting","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1170s"},{"start_seconds":1178,"text":"these days man all right for Sasha all right so consensus yeah yeah let's talk about the new consensus let's do it yeah","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1178s"},{"start_seconds":1187,"text":"so we're going to switch into a do based consensus um basically like every said the bottleneck in the lader based consensus is the lader MH and if you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1187s"},{"start_seconds":1196,"text":"think about it you have one leader that needs to disseminate the data to all the validators then the entire performance of the consensus determined by the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1196s"},{"start_seconds":1203,"text":"ability of the leader to push data M right so want to want to eliminate that and this what you talked about data dissemination then then this is","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1203s"},{"start_seconds":1212,"text":"basically what we're doing we have instead of the data everybody are doing it in parallel this is validator one of the validators validator one is going to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1212s"},{"start_seconds":1220,"text":"transmit the data to everybody collect the signatures back and then here he has Pro of avilability and then this is going to go into consent just just on a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1220s"},{"start_seconds":1228,"text":"summarize whatever we said this this happening in parallel all validator yeah so um let me erase this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1228s"},{"start_seconds":1238,"text":"thing okay but we have like a few if few so now we have like if you remember data dissemination then consensus we can","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1238s"},{"start_seconds":1245,"text":"consensus a blackbox we can use any consensus with this uh metadata already and we already we can scale any consensus now the question becomes like","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1245s"},{"start_seconds":1253,"text":"how far we can scale it and what's the latency mhm So currently we have this data dissemination it's one and a half round trips which we need to add to the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1253s"},{"start_seconds":1261,"text":"consensus whatever consensus we're using right so there is so what we want to do we want to want to merge them and also even if the metadata now is small the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1261s"},{"start_seconds":1270,"text":"ler can still be a button leg if there are a lot of proofs that still we can see that the is a button leg so we moving to a du based consensus basically what it means that there's no lader it's","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1270s"},{"start_seconds":1279,"text":"liter lless so what we're going to do is the following there is like two separated processes first we want to build a dug","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1279s"},{"start_seconds":1287,"text":"after we have a dug we will order the dog so this is how the dog going to look like I'm going to use four validators because this the minimum we need in","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1287s"},{"start_seconds":1295,"text":"order to do 3 f plus one so um each this these are the validators validator one validator two validator three validator","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1295s"},{"start_seconds":1304,"text":"four and the Doug is going to be constructed in in a round based manner these are going to be the rounds so in every round um I'm going to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1304s"},{"start_seconds":1314,"text":"let's say this is the beginning Genesis then I'm going to produce a new round and you know and I can contribute a new node and this node needs to link to n","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1314s"},{"start_seconds":1323,"text":"minus F which is 2 f+ one in our case three uh notes from the previous round this is how I this is how we we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1323s"},{"start_seconds":1331,"text":"collectively Advance the the dog well building the dog that's all we are doing like forget about consensus for a second we're building the duck each of these node is going to contain this proof of","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1331s"},{"start_seconds":1339,"text":"availability that we said before I'm proposing and also I'm going to link this so it doesn't have to be the same","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1339s"},{"start_seconds":1347,"text":"links right it can be slightly different and this is how this is how it's going to the dog is going to go mhm and um I I","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1347s"},{"start_seconds":1356,"text":"can move on to the next round once I receive 2 f+ one um nodes in this round because I can already link to them and I","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1356s"},{"start_seconds":1365,"text":"can I can broadcast my next my next node and this is um a local view of one one of the validators because the processes","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1365s"},{"start_seconds":1373,"text":"continue ASN completely as synchronously no timeouts nothing you just wait for this 2 plus1 link to you propose a new a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1373s"},{"start_seconds":1380,"text":"new a new node you move forward so each validators might see slightly different view of the Doug M um but the important","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1380s"},{"start_seconds":1389,"text":"property here is that because we are we are we're not just sending the nodes we actually using something that's called consistent broadcast it's like weaker","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1389s"},{"start_seconds":1397,"text":"form of reliable broadcast which guarantees that if we both see this node it's going to be the same node like there is no equivocation uh this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1397s"},{"start_seconds":1405,"text":"validator even even though it might be B bantine and it's trying to you cannot cuz the way it's going to work is that I'm going to send you the node like each","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1405s"},{"start_seconds":1413,"text":"of these node is actually hide one and a half round trips each of this round I'm going to send you a note to everybody they're going to sign send it back to me I'm going to aggregate the signatures","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1413s"},{"start_seconds":1422,"text":"and now I have a certificate and then I'm going to and then I'm going to send you this node again with a certificate and then you're going to add it to the Doug so I cannot equate and I cannot","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1422s"},{"start_seconds":1430,"text":"produce certificate to different two different nodes in the same in the same round so the property here is that we have two validators they might see","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1430s"},{"start_seconds":1438,"text":"slightly different view of the dog but whatever they see is the same and then Al and it means that if we both see this this node it's going to be the same and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1438s"},{"start_seconds":1447,"text":"the entire causal history is going to be the same because the links are the same and like we can go back and everything is the same so this is what we we're","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1447s"},{"start_seconds":1456,"text":"doing we keep doing it there's no timeout we just keep building it and now the question is uh okay so we build it that's nice and this is completely","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1456s"},{"start_seconds":1465,"text":"symmetric protocol there's no leader everything perfectly loow balanced that's a very nice property the thing now the the real magic here is that why","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1465s"},{"start_seconds":1473,"text":"I think people are like like these protocols now is that once I do that I don't do anything else in order to have the consens now you know the consensus","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1473s"},{"start_seconds":1480,"text":"is two three on 2/3 and two3 no that's you that that's actually challenging you need to go and order all of this right because","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1480s"},{"start_seconds":1489,"text":"um we might see different slightly different uh different um different dos but how can we order it I mean I can I","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1489s"},{"start_seconds":1496,"text":"can do 1 2 three 4 5 6 7 you may decide a different order right so there is um I don't know if you want to go to the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1496s"},{"start_seconds":1504,"text":"specific details yeah you want to go yeah let's go okay cool cool okay let's go yeah so good so we're going to get","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1504s"},{"start_seconds":1513,"text":"get I'm going to explain bullshark now and that's just then sha is actually an improvement on top of it but let's start with bullshark so we have these rounds","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1513s"},{"start_seconds":1522,"text":"right so one this round is going to be an enor round and this is going to be a vult round and so on and so forth this is","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1522s"},{"start_seconds":1530,"text":"going to be an enor round and this is going to be a vote round so by enor ah before that I want to say that we're we're going to look each validator is","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1530s"},{"start_seconds":1537,"text":"going to look on this Doug and and interet in inet what's the world there and like look at the like look at it as","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1537s"},{"start_seconds":1545,"text":"a consensus protocol this is going to be the proposals and the links are going to be the votes like mentally this is how the validator is going to see it so","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1545s"},{"start_seconds":1553,"text":"before running the protocol there is some a prior agreement on the en course there is some mapping you can think of around Robin or whatever and we can","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1553s"},{"start_seconds":1562,"text":"later expand it to whatever he said we have this lead reputation to use the history in order to choose the future enors based on the strong validators but","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1562s"},{"start_seconds":1570,"text":"okay let's say there is some mapping so let's say that this here going to be an end call this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1570s"},{"start_seconds":1578,"text":"validator this validator is going to be an end call and in the future we have we have we have other en course so we all agree on this who is going to be the en","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1578s"},{"start_seconds":1585,"text":"course this is easy we just some some function right the terministic function now the now the goal is to go and we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1585s"},{"start_seconds":1593,"text":"want to order the entire thing but we're going to go it in steps first we're going to Al the ankles we're going to basically decide which ankle I want to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1593s"},{"start_seconds":1600,"text":"alel and which ankle I can skip mhm and even though we see slightly different views of the dog we want to come up to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1600s"},{"start_seconds":1608,"text":"the same conclusion this is for safety right so and this is the first step and the the next step is is Trivial because then what we're going to do we're going","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1608s"},{"start_seconds":1615,"text":"to go older by enle by enle what we old and we're just going to Old the causal history by some deterministic rule again","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1615s"},{"start_seconds":1623,"text":"remember we said that all the causal histories are the same if you apply the same deterministic rule you're going to end up in the same with the same total order so really the challenge is that to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1623s"},{"start_seconds":1631,"text":"decide which enors to order and which to skip we even know which enors are are the candidates we just need to decide","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1631s"},{"start_seconds":1638,"text":"which and we not skipping so here is the logic this is this is pretty simple so we're going to go let's say we have a dug okay now we want to aler it and I","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1638s"},{"start_seconds":1647,"text":"have some local local view of the duck so I'm going to see and uh uh okay how I going to do it so the example will be","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1647s"},{"start_seconds":1655,"text":"nice um yeah okay let's we have I'm going to go enor by enor and do the following I I will try first to directly","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1655s"},{"start_seconds":1663,"text":"Commit This enor the commit rule is that if I have F plus1 votes for the enor a vote is just a link from the next um","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1663s"},{"start_seconds":1672,"text":"round so in this case we have one two three even four right we have four votes we can all do this Anor m that's okay so we decide to aler this enle now imagine","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1672s"},{"start_seconds":1681,"text":"a different scenario and then and then after we do that we move to the next one and try to aler this now imagine a different scenario at which we don't","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1681s"},{"start_seconds":1688,"text":"have this so we don't have this link mhm and uh this validator actually doesn't","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1688s"},{"start_seconds":1695,"text":"vote this validator doesn't vote for the en call so the en call in this case um f+ one is actually two so this one also","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1695s"},{"start_seconds":1704,"text":"does so okay this en code has only one vote mm we need f plus plus one like I said before so we need two in this example and only one so we cannot aler","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1704s"},{"start_seconds":1711,"text":"it but the thing the tricky thing is that um we don't have this node yet because we will it asynchronously maybe somebody else already have it right so","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1711s"},{"start_seconds":1720,"text":"in some other cases I'm going to draw it in red this is like your this is how you see it right and you actually see this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1720s"},{"start_seconds":1727,"text":"right so you decide to Al the it and I decide to skip it that's bad right that's what we want to avoid so here's what we do we rely on quum intersection","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1727s"},{"start_seconds":1736,"text":"here because uh we need f plus one in order to directly commit the enor and remember I told you that we build the dog with 2 f+ one we link every every","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1736s"},{"start_seconds":1744,"text":"node links to 2 f+ one from the previous round right so in and and nobody can lie it's also important important property","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1744s"},{"start_seconds":1751,"text":"is that you cannot we all see the same links so in if if I directly committed uh I mean you right you have these two","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1751s"},{"start_seconds":1760,"text":"things if you directly committed this Anor it means that every future um node here will have a puff to this right this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1760s"},{"start_seconds":1768,"text":"is like a simple quum intersection f+1 to f+1 we have at least one in the intersection so we're going to leverage that and let's say I don't see this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1768s"},{"start_seconds":1777,"text":"right but what I do see is this enor and I'm going to I'm going to aler it because I have","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1777s"},{"start_seconds":1785,"text":"all this this is my continuous of the dog so I do all the it so now it's very simple all I need to do is to go back and see if there is a puth between this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1785s"},{"start_seconds":1794,"text":"ankle to this ankle if there is a puth see here there is a pu mhm that means that it could be the case that somebody","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1794s"},{"start_seconds":1802,"text":"you in this example actually directly committed this enchor so I need to go and order this anchor before this anchor mhm this is and this is how we maintain","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1802s"},{"start_seconds":1810,"text":"the safety I also go now if there is not if there is no PA there are the other logic if there is no puff say this doesn't exist then nobody nobody could","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1810s"},{"start_seconds":1819,"text":"nobody could no nobody could directly commit it again this this is just the opposite logic of the previous argument just the the other the other direction","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1819s"},{"start_seconds":1828,"text":"so no so it's it's safe to skip it nobody directly committed it so um and we can prove it by induction the reason","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1828s"},{"start_seconds":1836,"text":"it's safe to skip it because we're all going to order this and we're all going to agree that there is no puff because we all agree on the caal history and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1836s"},{"start_seconds":1844,"text":"then we're all going to skip it so so that's basically that's basically it that's basically the entire protocol very super simple and from a system","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1844s"},{"start_seconds":1853,"text":"point of view all you need to build is the dog which is actually not that simple nothing is simple when are actually very complicated yeah","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1853s"},{"start_seconds":1861,"text":"nothing is simple while doing it right now a lot of a lot of like cases and performance building this in 19 that's","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1861s"},{"start_seconds":1868,"text":"yeah probably we building hasra what what we building well we we we had hasra we we we were calling transaction flow","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1868s"},{"start_seconds":1875,"text":"it was very similar to this um there's a lot of caveats around like when people was holding this or like retrieving it","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1875s"},{"start_seconds":1884,"text":"back like do do you I mean we can let let's go step by step y yeah so let's dive in um I guess my my first question","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1884s"},{"start_seconds":1891,"text":"is so for for actually making ordering of this anchor right like you you need to get to where you're starting to order","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1891s"},{"start_seconds":1898,"text":"the anchor yes how long can it be that you kind of in this halfway state right where like the next anchor you cannot","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1898s"},{"start_seconds":1907,"text":"order yet because you haven't seen this and so you kind of keep keep having this yeah good good point yeah that's actually an issue with with bullshark","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1907s"},{"start_seconds":1915,"text":"with like the vanilla bullshark protocol because uh I mean if if the validators keep start crashing you don't want them to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1915s"},{"start_seconds":1923,"text":"become their NS to become anchors because then it's actually can go arbitrary um also if you think about um","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1923s"},{"start_seconds":1931,"text":"the flp impossibility result the the theory says that it can go indefinitely but I mean nobody everybody are","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1931s"},{"start_seconds":1938,"text":"implementing partial synchronous protocols that's kind of true for everybody no no matter what you do there is a low bound if you have a deterministic protocol there is a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1938s"},{"start_seconds":1946,"text":"scenario very unlikely in which you you don't have liveness right but so that's not an issue but from practical point of view if the if the no if the want to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1946s"},{"start_seconds":1954,"text":"start crashing you can go far uh into the future of the dog so as ever said before we have this little reputation","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1954s"},{"start_seconds":1961,"text":"that we that we use so um and this is already an improvement that we did in show so we added one of the contribution there was to add a little reputation to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1961s"},{"start_seconds":1970,"text":"the dog so what we do is that once we older an anle we can look at the causal history and look we can see okay these","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1970s"},{"start_seconds":1978,"text":"validators didn't participate they didn't contribute nodes there's High likely that they are crashed or like very slow they're struggling now they're far behind so let's exclude them from","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1978s"},{"start_seconds":1987,"text":"the possible sets of validators that can contribute enors so we end up with validators contributing enor and Only","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1987s"},{"start_seconds":1995,"text":"The Strongest validators Will contribute enors and then it will reduce the scenario that we're describing in which you you will need to skip and go ahead","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=1995s"},{"start_seconds":2003,"text":"in the duck got it got it so now another now to compare to a leader based protocols what is like nice property of","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2003s"},{"start_seconds":2010,"text":"that is that in leader based protocols the leader can also be faulty right but there it's unavoidable that you have you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2010s"},{"start_seconds":2017,"text":"have to use these timeouts right so if the leader is faulty you and typical timeout is around second one one and a half second in which you just do nothing","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2017s"},{"start_seconds":2026,"text":"you just you you gave the leader a chance for one and a half second just to realize that he actually crashed nothing happened and then you view change and continue so this is like a one and a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2026s"},{"start_seconds":2033,"text":"half second penalty the good thing about this is that you don't have this penalty because the dog you keep building right network speed mhm and uh yeah you might","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2033s"},{"start_seconds":2042,"text":"pay some latency price because it wouldn't hand this enor but this Anor is already here you can continue whereas in little base you just just do nothing for","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2042s"},{"start_seconds":2049,"text":"one and a half second and then you then you try again yeah no of course leaderless consensus has huge benefits so uh what","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2049s"},{"start_seconds":2058,"text":"happens when so let's say I have a uh so let's say I'm this V V4 yeah so I sent","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2058s"},{"start_seconds":2066,"text":"out this with some confirmation but I don't I actually didn't publish this uh","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2066s"},{"start_seconds":2073,"text":"so I didn't publish the round two but I sent out approval so what happens to the you point to something that you didn't publish right this is what you're saying","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2073s"},{"start_seconds":2082,"text":"so we will ignore you we will like we will u i mean we will process the node but we will not add it to the Doug unless you give us this we will fetch it","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2082s"},{"start_seconds":2090,"text":"from you wait from you to give it to us but if you don't then we just not add it to the DG and we will wait to receive 2 f+ one","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2090s"},{"start_seconds":2098,"text":"different nodes and we consider this byzantin behavior and because there is only F Byzantine to f+1 Are Not bantin","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2098s"},{"start_seconds":2106,"text":"So eventually we will get to F plus1 nodes that behave correctly and then we can continue M","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2106s"},{"start_seconds":2114,"text":"yeah so pretty much just like I mean obviously uh so I'm I'm sending this to to some other validators they haven't","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2114s"},{"start_seconds":2123,"text":"seen this they'll go fetch it yes right from me I'm let's say maybe giving to some not giving to others uh do they","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2123s"},{"start_seconds":2131,"text":"like broad rebroadcast like intermediate notes to each other or uh they that's they don't that's an implementation details we could but we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2131s"},{"start_seconds":2140,"text":"now we don't do it what we do is what we actually do is that this um well I'm going to give you I I point to a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2140s"},{"start_seconds":2147,"text":"certificate yeah and the certificate is 2 f+1 signatures MH um so I can go and fetch from any of this I don't I don't need to fetch specifically for this this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2147s"},{"start_seconds":2156,"text":"is for load balancing I can and fetch for any of the um signatures any of the signing validators right sorry I didn't know yes","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2156s"},{"start_seconds":2165,"text":"so like like I said a node a node here before remember I said yeah yeah you collect the signature I collect the signature and then here I already send you something with the signature so you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2165s"},{"start_seconds":2173,"text":"can add to the dog and when I point it back I actually point it back to the certificate like to a pretty much aggregated signature yeah yeah to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2173s"},{"start_seconds":2181,"text":"aggregated but it has the bit the bit Vector I know I know who signed it so I I could just go and fetch from this validator for but I don't want everybody","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2181s"},{"start_seconds":2190,"text":"to go and fetch from validator for so I'm want to load balancing it so I can use the information in the v Vector of the signature because whever in the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2190s"},{"start_seconds":2197,"text":"signature they have should should have should have it right should have should have the block because they signed this they signed it right so uh so I will go","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2197s"},{"start_seconds":2205,"text":"and like load balance like in yep got it f plus one the two f plus one have it yeah yeah for sure yeah under the buting","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2205s"},{"start_seconds":2214,"text":"assumption at least F plus1 I'm going to have it here all right and so uh and so you're progressing pretty much to next","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2214s"},{"start_seconds":2219,"text":"round after you have uh 2f+ one yes published uh","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2219s"},{"start_seconds":2230,"text":"so I guess like latency wise like let's say I publish this and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2230s"},{"start_seconds":2238,"text":"then I'm waiting to see everybody else's like meanwhile it can be just","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2238s"},{"start_seconds":2244,"text":"ordering transactions so that I can I just thinking what what's the delay here looks like what's what's the delay yeah","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2244s"},{"start_seconds":2252,"text":"so for for an enor to older an enor you need this round and this round yeah and to all the um to all this node for","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2252s"},{"start_seconds":2260,"text":"example is going to be all the as part of this en Co history so this is additional round yeah and this one is will actually require more so because","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2260s"},{"start_seconds":2270,"text":"this will need this round this now this now and this one this will come for but in short we get rid of this we can we can talk about this iation as well so in","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2270s"},{"start_seconds":2279,"text":"sh we actually make sure that we have an anchor in every round this is the one of our latest optimization and the idea is actually very simple just took us some","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2279s"},{"start_seconds":2287,"text":"time to realize it but let's say we believe that this protocol works okay safe and everything works and like like I told you we have this that we we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2287s"},{"start_seconds":2296,"text":"divide it into two phases first is to agree on the end course and then we agree on everything as part of of agreeing on the enor in particular we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2296s"},{"start_seconds":2304,"text":"agree what is the first Anor we need to order right in this example it was this but does necessarily has to be we might skip him and agree this right but if we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2304s"},{"start_seconds":2313,"text":"believe bullsh is correct that's in particular also correct we all agree which anchor we need to order first so in sh what we said we just realized it","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2313s"},{"start_seconds":2321,"text":"that it doesn't matter how we went in the future what we did in the dog we can always deterministically go back and reinterpret the dog again so what we do","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2321s"},{"start_seconds":2329,"text":"is that think of it think about it we start like I said before an instance of bullshark and we come to a conclusion that this is the first anchor we need to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2329s"},{"start_seconds":2336,"text":"all do right now if we have to run the bullart protocol the vanilla bullart protocol we would skip this round and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2336s"},{"start_seconds":2343,"text":"try to hold this enle right and so on however we we realize that we don't need to do it so now we we all agree this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2343s"},{"start_seconds":2350,"text":"enle we need to order we go order it caal history like before and now we just decide okay let's kill this bullshark instance we don't care about about it","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2350s"},{"start_seconds":2357,"text":"anymore we can just deterministically and we start over from this from this from this round and do exactly the same we're going to change the mapping and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2357s"},{"start_seconds":2365,"text":"say okay this was the ankle now this is the ankle right mhm and now let's start a CLE clean instance of bullsh starting","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2365s"},{"start_seconds":2373,"text":"starting from here mhm and and and again and again and again we're going to continue with the same trick and then we're having we're having an anchor in","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2373s"},{"start_seconds":2381,"text":"every round MH um yes and that's uh that's one and then","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2381s"},{"start_seconds":2389,"text":"another one is the lead reputation like I said before because now once we order this enle we can not only that we are moving the ankles we can only change the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2389s"},{"start_seconds":2397,"text":"mapping based on the history yeah so your assignment function pretty much changes based on history yep and with this observation together we can get rid","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2397s"},{"start_seconds":2406,"text":"of timeouts completely because like I said before this is a terministic protocol to fight the flp we need to use timeouts so one of one of places we use","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2406s"},{"start_seconds":2414,"text":"timeouts in bullshark is that we wait like I told you before that we we build the Doug in network speed this wasn't","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2414s"},{"start_seconds":2421,"text":"entirely true in order to get liveness you need to wait for 2 f+ one but you also want to get a chance to get the enor m so you're","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2421s"},{"start_seconds":2430,"text":"going to wait for uh either you wait for and and and timeout or anchor right this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2430s"},{"start_seconds":2437,"text":"is this is what you do so now with this leader reputation we actually don't need this because um if the anchors are fast","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2437s"},{"start_seconds":2446,"text":"like one of the fast validators fast or fast validators everybody going to vote for them good if they crashed um we are","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2446s"},{"start_seconds":2453,"text":"going to if they crashed or slow the little reputation mechanism will make we we just filter them out in the future rounds so we end up with the strongest","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2453s"},{"start_seconds":2462,"text":"validated contributors and call and we just get rid of the time outs we just keep we just keep going um of course for the flp just to be you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2462s"},{"start_seconds":2472,"text":"know theoretically on the safe site so nobody can argue we don't have liveness right so what we can do is completely unrealistic but say like some","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2472s"},{"start_seconds":2480,"text":"configurable number of anchors we cannot H order like we skip 10 anchors then we can say okay let's fall back to use","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2480s"},{"start_seconds":2487,"text":"timeouts and then we go back to the same theoretical liveness guarantees we had before where in practice we will never skip the en course this is like you a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2487s"},{"start_seconds":2496,"text":"very powerful adversary yeah to pull this off right so um well we pretty much need split Network and somebody yeah yep","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2496s"},{"start_seconds":2505,"text":"manipulating dags on both sides yeah I personally really like it because uh I come from Fury and there FP you take it","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2505s"},{"start_seconds":2512,"text":"very seriously and you cannot do the terministic protocols without timeouts so yes we we cannot but we can like very corner it into some very very","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2512s"},{"start_seconds":2521,"text":"theoretical place right so in practice we will never use timeouts MH so yeah par like if you remember from","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2521s"},{"start_seconds":2529,"text":"the beginning execution then par sorry consensus par execution so in par execution the goal is really to execute","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2529s"},{"start_seconds":2537,"text":"the the given transaction as fast as possible and we can we can use all like threads in the system right and if we need a scalable algorithm to make take","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2537s"},{"start_seconds":2546,"text":"use of threads and in order to be able to do it we need to the goal is to basically reduce conflicts","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2546s"},{"start_seconds":2553,"text":"mhm my so that that's the goal reduce conflict that you want to execute in parallel you need to reduce the conflict","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2553s"},{"start_seconds":2561,"text":"so there are like a few ways to do it one is like super trivial which is like a um excess","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2561s"},{"start_seconds":2573,"text":"vity that's like meaning that if you have if you group together some resources mhm Resource Group and you group them","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2573s"},{"start_seconds":2582,"text":"together and you have like Resource One resource two resource three and this is like you do it because it's friendly for your storage you want to store them together get them together and then you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2582s"},{"start_seconds":2591,"text":"have like one transaction that try to um to touch R1 and another transaction try to touch R2 if the if you if the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2591s"},{"start_seconds":2599,"text":"granularity of the ex is entire Resource Group that's going to be a conflict right so you always need to you always need you always need to try to make them","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2599s"},{"start_seconds":2607,"text":"do made the they made the ex is as lity as possible so if I touch L1 and you touch L2 we should be able to go in parallel so this is this is trivial","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2607s"},{"start_seconds":2615,"text":"stuff I think you just want to the more important thing I I want to talk about is there is like this debate now about","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2615s"},{"start_seconds":2623,"text":"static versus Dynamic um parallelism MH so I want to maybe touch about this a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2623s"},{"start_seconds":2630,"text":"little bit so we have static versus Dynamic okay okay and by Static I mean","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2630s"},{"start_seconds":2638,"text":"that U the user needs to specify the read and right set mhm like completely yep and dynamic is completely","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2638s"},{"start_seconds":2648,"text":"agnostic agnostic yeah meaning that the user doesn't need to give anything just provide you the transactions well this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2648s"},{"start_seconds":2655,"text":"here that's to use need to give you light set so yeah so this is this is less convenient for sure this is maybe a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2655s"},{"start_seconds":2663,"text":"little bit inconvenient but I want to uh set yeah I want to I want to from a performance point of view I want to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2663s"},{"start_seconds":2671,"text":"touch upon it and why why we decided to do Dynamic and I want to go and maybe give an example of a one use case which","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2671s"},{"start_seconds":2680,"text":"is an older book let's say I want to implement a concurrent older book older book on chain so for an older book you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2680s"},{"start_seconds":2689,"text":"have basically this price and you have the uh this is the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2689s"},{"start_seconds":2696,"text":"amount so so and say this is 10 so I want to I want to sell this amount in 10","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2696s"},{"start_seconds":2703,"text":"this amount and 11 you know and the higher the price the more I want to sell and the buyer wants to buy a little bit in nine more than eight seven like this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2703s"},{"start_seconds":2711,"text":"is how old the book looks and then here we can have a match right this is what we do so how would you represent something like this um on chain this is","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2711s"},{"start_seconds":2720,"text":"like this is a market let's say Bitcoin and ethereum so so you have a you you need some sort","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2720s"},{"start_seconds":2727,"text":"of tree concurrent tree um so you will have some something like this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2727s"},{"start_seconds":2734,"text":"mhm uh this is how you represent the sellers and you will have also the representation of the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2734s"},{"start_seconds":2744,"text":"buyers uh okay now you want if you want to place a Noel modify a Mel cancel a Noel you need to go to the tree and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2744s"},{"start_seconds":2751,"text":"basically find your place and add a node right or delete a node remove a node so if you think about um static parallelism","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2751s"},{"start_seconds":2761,"text":"like uh Solana s this this these blockchains you need to statically declare everything you're going to touch now the thing is that you don't know you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2761s"},{"start_seconds":2769,"text":"don't know which path you which path you're going to go because you don't know the state of the tree so there is nothing better you can do with static paraly than to lock everything you just","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2769s"},{"start_seconds":2777,"text":"lock everything right and if you lock everything then basically there's no concurrency if me and you are going to going to place a we're going to place a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2777s"},{"start_seconds":2785,"text":"a bid yeah we loog everything we look everything we're going to go sequentially where in Dynamic paral and say I want to add these and you want to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2785s"},{"start_seconds":2792,"text":"add this we this is why it call dynamic because we dynamically on the Fly realize what we are actually going to touch so you are touching this area I'm","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2792s"},{"start_seconds":2800,"text":"touching this area there's no no reason why we cannot go go in in parallel so it gives you like Dynamic parallelism gives","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2800s"},{"start_seconds":2807,"text":"you much more much more concurrency we can place a lot of and think of this a big tree right so we can we can place a lot of bids concurrently cancel modify","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2807s"},{"start_seconds":2816,"text":"same on the other side and um this is like a strong argument why why we choose dynamic because this is one use case but","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2816s"},{"start_seconds":2823,"text":"we believe like more more use case most complex use cases will come in which static will just the GL the Locking grity is going to be bad where","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2823s"},{"start_seconds":2831,"text":"dynamically you can just figure out a better complexity I mean one of the limitations here is that also like you actually need to declare specific pieces of data that you're touching which","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2831s"},{"start_seconds":2840,"text":"becomes a lot especially in this case and it expires quickly right by the time you prepared transaction you sent it and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2840s"},{"start_seconds":2849,"text":"somebody already matched three right and like now your transaction is valid now now transaction about that's that's that's the Solana right and they all","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2849s"},{"start_seconds":2856,"text":"actually no matter you actually need to limit right because you specified yeah they have 16 pieces of account like account data that they can tou you they","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2856s"},{"start_seconds":2864,"text":"can increase it but somewhere they will need to put a limit right so if the tree is bigger then you basically cannot you need to hack some stuff in know they","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2864s"},{"start_seconds":2872,"text":"have a crank for that yeah y yeah so dynamically just no issue right you just go and figure out on the flight mhm so how how do you do that yeah good good","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2872s"},{"start_seconds":2881,"text":"next question yeah uh all right so let's me me eras this so yeah block STM is","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2881s"},{"start_seconds":2887,"text":"like something we're really proud of um actually there are a few other blockchains that currently already","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2887s"},{"start_seconds":2895,"text":"either implementing it or already deployed it so it's uh which for us is like a big researchers for us is a big deal you know somebody is using your","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2895s"},{"start_seconds":2903,"text":"somebody's using your check yeah this is this is more than any best for us you know so so okay so let's talk about block","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2903s"},{"start_seconds":2915,"text":"STM we're going to for blackm we're going to take advantage of a few Concepts um there is the STM as the name","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2915s"},{"start_seconds":2923,"text":"concept OC concept and multi multiversion data structure okay so what is the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2923s"},{"start_seconds":2930,"text":"STM um and and these all of these were like heavily studied in the literature it just uh not so much in practice","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2930s"},{"start_seconds":2938,"text":"unfortunately but with this specific use case for blockchain we figured out a very practical way to to and I will explain in a second what exactly led to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2938s"},{"start_seconds":2947,"text":"this to combine everything together into a system that actually performing very well so STM is a sof stands for software","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2947s"},{"start_seconds":2953,"text":"transactional memory um just think of like a layer of software that abstracts away all the concurrency you you want to do so it's like makes programming much","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2953s"},{"start_seconds":2963,"text":"easier you don't need to deal with locks you're just going to say this this piece of code I want to execute conai like automically send it to the library the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2963s"},{"start_seconds":2971,"text":"library will do the magic and execute it which is perfect fit for blockchains because what we have is actually transactions that we want to execute","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2971s"},{"start_seconds":2978,"text":"automically now this OC is optimistic on currency control for stms we have two technique we have pessimistic on currency control and optimistic on","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2978s"},{"start_seconds":2985,"text":"currency control and this is BAS this most of the time gives better performance and the general structure is that you want to go optimistic and you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2985s"},{"start_seconds":2993,"text":"want to say oh I'm going to execute the transaction and you split into you you you you can execute transaction then you can validate the transaction so you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=2993s"},{"start_seconds":3000,"text":"execute all the transactions and then you you go after the fact and some later point and you validate if validation succeeds basically what you what you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3000s"},{"start_seconds":3009,"text":"most of the time you are doing is that you are checking that whatever you read during execution the values that they're still they they haven't changed so","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3009s"},{"start_seconds":3017,"text":"whatever decision you made back then they're still relevant you can commit so then you you execute optimistically then you validate if validate validation pass","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3017s"},{"start_seconds":3025,"text":"you can commit otherwise you need to abort retry re execute so this is optimistic concurrency control and multivalue uh multiversion data","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3025s"},{"start_seconds":3034,"text":"structure is just also like simple concept like say you have a memor some memory location and you have basically","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3034s"},{"start_seconds":3041,"text":"what you have is a mapping uh between the version and let's say you have like a you have a you have you have a list of","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3041s"},{"start_seconds":3049,"text":"transactions that you want to execute so now instead of having one memory location for transaction the transaction keep overwriting M you basically","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3049s"},{"start_seconds":3057,"text":"remember all the all the values that this this transaction wanted to write to this memory location so let's say one wres","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3057s"},{"start_seconds":3064,"text":"100 transaction two is going to write 200 and transaction four is going to write 400 so you have all these information","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3064s"},{"start_seconds":3073,"text":"why is it useful because for us uh this is this is like okay why why do we actually manage to make all these things","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3073s"},{"start_seconds":3081,"text":"work very efficiently is that we realize that we have this fixed order that we need to follow mhm right the later we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3081s"},{"start_seconds":3089,"text":"realize that we can like do whatever we want with the order but eventually we just thought okay consensus gives us an order we need to follow it right and it","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3089s"},{"start_seconds":3096,"text":"has to be not only the result has to be deterministic across other validators it's actually need to respect the order so it looks like a limitation you limit","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3096s"},{"start_seconds":3105,"text":"your system you probably should get like wor performance but turns out that um it it actually give us a better performance","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3105s"},{"start_seconds":3112,"text":"and the reason is because we can follow the older uh explaining for different point of view so first with this multiversion data structure we can","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3112s"},{"start_seconds":3121,"text":"completely eliminate right right conflicts this is kind of know technique nothing like we invented here is that think about these are these are the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3121s"},{"start_seconds":3129,"text":"rights everything is executed in parallel these are the rights and now transaction three comes and want to read right so the way it works is that the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3129s"},{"start_seconds":3138,"text":"transaction will Traverse uh all the versions and will take the higher version that is smaller than the transaction number so it will take this","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3138s"},{"start_seconds":3146,"text":"value this this is going to be the read because because we're going to respect this order of execution transaction three needs to go after two so so three is going to is going to read the value","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3146s"},{"start_seconds":3154,"text":"that to wrote right so right right conflict is not an issue and yeah remember the goal is to reduce conflict","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3154s"},{"start_seconds":3161,"text":"so no right right conflicts now the question is um read right conflicts right how can we how can we how can we","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3161s"},{"start_seconds":3169,"text":"minimize the problem minimize the number of aborts number of executions so what we're going to do is actually Al also","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3169s"},{"start_seconds":3176,"text":"simp idea so let's for example in this example say that transaction number um","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3176s"},{"start_seconds":3183,"text":"two we executed it uh we try to execute right optimistically but then we figure out it needs to be aborted because","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3183s"},{"start_seconds":3190,"text":"transaction one conflict with some transaction one right so what we are going to do is what and we're going to go back to this multiversion data","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3190s"},{"start_seconds":3197,"text":"structure now uh we can be much smarter we remember I told you we have no no idea at the beginning what are the read","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3197s"},{"start_seconds":3204,"text":"right sets right but now we have much more information that we can use because we already know we executed transaction two one time we already know we already","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3204s"},{"start_seconds":3212,"text":"have a good estimation where transaction two wants to write particular it right to this memory location so what we're going to do we going to delete this value and we keep the entry and instead","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3212s"},{"start_seconds":3221,"text":"we are going to have some buttom here okay the buttom will represent the fact that there was high likely transaction two is going to right here currently","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3221s"},{"start_seconds":3230,"text":"it's going to be re executed it's AB bolted it's in the process of being re executed but it's high likely it's going to right here so when transaction 3 now","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3230s"},{"start_seconds":3237,"text":"comes and read it sees that there is an entry for transaction two the transaction should take this value but this is the bottom here MH which means","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3237s"},{"start_seconds":3244,"text":"that there is no point for transaction to go and execute because it's going to be a bolted right so instead we're going to just suspend transaction 3 and and","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3244s"},{"start_seconds":3253,"text":"transaction three is going to um J's going and we were going to wake it up whenever two finishes and then transaction three can go so we because","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3253s"},{"start_seconds":3261,"text":"we have this fixed order and we respect the order we are able to reduce most of the aborts and most of the execution there is really nothing better you can","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3261s"},{"start_seconds":3270,"text":"do if you think about it transaction three depends on transaction two right so transaction 3 has to go after transaction two we couldn't execute them","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3270s"},{"start_seconds":3277,"text":"in parallel and now we we executed transaction three until as much as we could but now we need to suspend wait for two and then we can continue this is","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3277s"},{"start_seconds":3287,"text":"um this is like one of the call contribution of block and why can scale so you pretty much just so you like you run pretty much all of this VMS in","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3287s"},{"start_seconds":3295,"text":"parallel and then yes when you see it reading something that like transaction two have updated you just suspend it until until until it's ready until the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3295s"},{"start_seconds":3303,"text":"value is ready and then we can we can go again with transaction three maybe in the future we will need to suspend it again but we will we we keep doing","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3303s"},{"start_seconds":3311,"text":"progress yeah yeah so another thing that we another way we leverage this order is how we actually dispatch the tasks right","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3311s"},{"start_seconds":3319,"text":"we have this transaction a lot of threads which which one is is doing what and the trick is always withm is to minimize the overhead if you pay a lot","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3319s"},{"start_seconds":3328,"text":"of overhead in order to synchronize the thread it's not going to scale it's not going to work so what we what we end up doing is that we we maintain like","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3328s"},{"start_seconds":3336,"text":"roughly speaking we maintain two indexes uh validation index and uh execution index mhm so execution is the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3336s"},{"start_seconds":3345,"text":"next transaction we the index we're going to point to the next transaction we need to execute and validate the next transaction validation the next transaction need to validate and because","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3345s"},{"start_seconds":3354,"text":"we respect the older we also need to validate in all of these things so basically the all all the contention is going to be on these two indexes when a","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3354s"},{"start_seconds":3361,"text":"trade comes wants to check the task going to read both of them see which one is the lower and then going to perform a Fetch aned and then just go and take","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3361s"},{"start_seconds":3369,"text":"okay I need to go and execute this transaction and then and then the pointer and then the index will Point here so the next next red will going to","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3369s"},{"start_seconds":3376,"text":"come and take this transaction so very very minimum overhead for all this uh uh collaborative schedule for like the","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3376s"},{"start_seconds":3384,"text":"transactions execution but I guess I mean there's a there's a big Ram overhead of having this like","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3384s"},{"start_seconds":3392,"text":"transaction suspended right I mean not big but like per transaction there's some amount of ram that you allocate for","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3392s"},{"start_seconds":3400,"text":"running VM and everything yes and so as you kind of spinning up more and then going back to revalidate so yeah that","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3400s"},{"start_seconds":3407,"text":"actually was an issue the the thing was that um at the beginning um move doesn't support um there's no USNC Concept in","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3407s"},{"start_seconds":3416,"text":"move so we cannot just suspend inside move itself we had to Abol the first version of block STM we had to Abol the transaction and re execute from scratch","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3416s"},{"start_seconds":3425,"text":"everything and it was kind of annoying because algorithmically we didn't need to do it we already reached some point we can continue from there so now what","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3425s"},{"start_seconds":3432,"text":"we are currently doing is that we're implementing our own H tread garage tread pool in in rust inside inside rust and we're going to use it so we have all","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3432s"},{"start_seconds":3441,"text":"these abilities that we need to block STM suspend the thread wake up the thread so um then it will give us another performance gain I","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3441s"},{"start_seconds":3448,"text":"hope got it um okay and then so just for my understanding uh so let's say you start","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3448s"},{"start_seconds":3457,"text":"let's say you have four threads you start executing all this four in parallel uh let's say there is like a indeed I mean all of them actually","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3457s"},{"start_seconds":3467,"text":"write uh and then T2 reads from T1 so they'll write into some address but then","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3467s"},{"start_seconds":3474,"text":"T2 actually specifically reads from 1 so they will all execute t334 will complete","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3474s"},{"start_seconds":3480,"text":"with their own but You2 will suspend because it it reads whatever T1 wrote out it","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3480s"},{"start_seconds":3490,"text":"will uh then wait until two un finishes Y and then you'll like unsuspend it or restart itend it yeah you restart it","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3490s"},{"start_seconds":3499,"text":"yeah and then it will write out but then actually that value will be replaced by D4 and you just kind of Def when you save the final State you just take","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3499s"},{"start_seconds":3508,"text":"whatever the final values are for that final step yeah that's exactly what going to happen and the cool thing about it is that if you don't have any","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3508s"},{"start_seconds":3515,"text":"conflict yeah it's you just execute right nothing you don't you don't need to pre execute you don't need nothing you just execute but if you have","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3515s"},{"start_seconds":3522,"text":"conflicts you already have pretty good estimation of what's going on and for the future you can be very efficient okay yep yeah let's yeah I","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3522s"},{"start_seconds":3531,"text":"mean we're kind of D um all right cool yeah this is this is really cool um I think this was a really educational","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3531s"},{"start_seconds":3539,"text":"hopefully so uh thanks Sasha and Avery for joining and uh yeah join more","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3539s"},{"start_seconds":3546,"text":"episodes of whiteboard series thank you","url":"https://www.youtube.com/watch?v=NfRbWVcoJ-4&t=3546s"}],"youtube_url":"https://youtube.com/watch?v=NfRbWVcoJ-4","url":"https://near-pedia.xyz/videos/NfRbWVcoJ-4"}