Nightshade 3.0: The Next Evolution of NEAR Protocol - Bowen Wang

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:09

   Hello, hello, hello. Uh, hey everyone. Thanks for being here. I'm Bowen and today I want to talk about Nightshade 3.0, the next evolution of Near Protocol. So before we talk about the future, I want to take a look back at um um the last five plus years of amazing development u of the protocol since we launched mainet in October 2020. Um we've maintained 100% uptime which is a rare achievement uh in the blockchain space and not only that uh we've made great progress in terms of scaling the blockchain improving the throughput uh

2. 0:44

   and we did this redesign of um the protocol based on stress validation and launched the niche 2.0 no design in August 2024 uh made the blockchain fully sharded uh both transaction processing and state are fully sharded and not only we actually uh designed this uh highly scalable architecture implemented it and we also um proven the scalability uh in practice uh we now have nine shards on mainet and in December last year we've run this uh test on the 70 shard network

3. 1:18

   uh and proved 1 million transaction per second in this public benchmark that anyone can actually run and it's also the improvement is not just about um scalability it's also about the speed uh we've actually decreased uh block time by half last year and now we have 600 millcond block time 1.2 2 seconds time to finality and overall I would say in the past um eight plus years since we uh started working on we've built this um fast scalable and very reliable blockchain.

4. 1:50

   So all of that sounds great. Uh the question is what challenge do we still face today? Uh so I would say there are a couple. One is asynchronous execution um that comes with a starting design. So this is how we allow the blockchain to scale um and to have this uh ability to linearly scale with number of shards. Uh but it also comes with some problems. Uh one of the problem is that uh even

5. 2:15

   though um the the blocks are very fast you know produce every 600 millisecond some transaction actually take several blocks uh to be executed. Uh so the reason is that the transaction would first execute it on the shard of the sender then uh generate a receipt uh go and send to the uh shard of the receiver gets executed there and if you're calling a a function then uh that may actually generate uh more receipts uh that go to some other shards to be

6. 2:41

   executed uh which would be the case uh if you have a complex D5 transaction. Um so that's a problem for users and uh it's actually a bigger problem for developers who need to write those smart contracts uh especially when it involves uh this multicontract interaction. The main problem is that you need to handle uh the cases when the cross contract call fails and actually you need to attach to this call back to make sure the state is restored in a correct way.

7. 3:05

   Uh making sure there's enough gas attached to it uh so that the callback doesn't fail and and so on so forth. Uh so that's actually a sticky point uh in nearest developer experience. So another uh problem is uh how we manage storage. Uh so we introduced this uh storage staking mechanism uh when we launched mainet uh in 2020. The idea is that uh in order to occupy some uh state on the blockchain you actually need to

8. 3:30

   lock down your tokens uh to do that and there's a fixed conversion rate between the uh number of tokens you lock down versus number of uh storage amount of bytes you can get in the state. Uh so all of that was fine except that uh when actually uh gets used more and more uh so problem emerge. Uh so basically for uh both developers and users this is not a great experience. So developers need to think about uh how to actually charge for storage on the smart contract. For

9. 4:00

   example, when a user does smart function call, let's say they write some more bytes to the state, then they actually need to get charged more because um they actually increase the storage staking requirement for the contract. And if you don't do that, then obviously people can abuse this and so the contract can like run out of storage and and so on. Um and for users it's also u annoying because uh if you actually do it correctly then

10. 4:26

    uh for some contract call you may need to attach a variable amount depending on how much state you plan to write to the contract. Uh so that's going to be very annoying. So what happened in practice is that many public smart contract actually give up on managing the storage properly. they either just say like they will subsidize uh the storage management uh the storage staking cost or because it's some defy contract they already

11. 4:49

    have a lot of near uh in some like u pool or some something else that's um they already have enough near to pay for the storage so they don't actually worry about managing the storage properly. Uh so we actually introduced a partial escape hatch in uh 2023 called zero balance account. Uh the idea is that uh uh back then the cost of uh creating near account is actually expensive because in order for the account to even

12. 5:14

    exist you need to have enough um near locked in the in the state uh for for the storage and uh that actually makes it actually very high cost to acquire new users. So the idea of the balance account is that uh if the state of the uh account is under 770 bytes then you don't actually need to maintain any balance on the account. Um essentially the um creation cost of the account is paid for uh as part of the transaction fee uh when the account was created.

13. 5:47

    Okay. So here I talk about some of the problems we face today and uh I want to give uh a bit high level overview of a number of different initiatives we're working on to make the protocol better. Uh so I would say each of them requires own uh presentation to do a deep dive and here um I'm mostly giving a high level overview of how uh those initiatives are and um some of the thinking behind. So the first one is

14. 6:13

    called spice uh which is stand for um separation of consensus and execution. Uh this is idea that the consensus should not get blocked on execution. Uh this would allow consensus to run faster uh blocks to be produced faster because um unlike today um the the bad won't need to actually uh apply all the transaction and receipt uh do the state transition update in order to produce uh

15. 6:36

    the next block. So they would not only lead to faster block time but also generally uh better utilization of the hardware resources and a result uh higher throughput. So it also uh makes the execution model much more flexible because the execution does not need to actually update every single block. It can update asynchronously every several blocks. Uh so the new state root um uh don't necessarily need to get posted

16. 7:00

    every block. And this uh has some very interesting properties. For example, it allows for transactions that are more long running than today. So you don't necessarily need to get limited by uh the gas limit of single block. It can actually for run for like longer period of time. And also it allows for time to generate uh zero knowledge proofs which would address one of the scaling bottlenecks which is the network bandwidth usage on transmitting state

17. 7:24

    witness. It also make the uh state witness much cheaper to verify uh on the stateless file side. So here is an illustration of how roughly it would look like. There's this consensus chain uh that just keeps running forward uh producing blocks uh packaging transactions and there is this execution chain that um keeps catching up. It executes transactions and receives state update and every once in a while it'll post the state update uh on chain.

18. 7:55

    So we talked about the problem with storage management. Uh so the idea to address it is that uh we want to decouple the actual state storage from execution. So we [snorts] want to have this concept of um stateless executors and have them separate from the state storage nodes which are can be very heavy and can store the state of many shards at a time. Uh so the the observation here is that uh if we don't actually need to uh replicate the state storage on every single chunk producer

19. 8:25

    node uh for the shard then the state will become even cheaper uh than what we have today. Um which is already good with uh the sharding and and so on. Um so so the idea is that uh if we can do that then um we can we can further reduce the cost of storage and actually replace the storage staking uh with gas payment uh for the state storage.

20. 8:47

    Essentially you just pay for uh writing to this to the state as uh part of the transaction fee uh similar to what we did for the zero balance account and you don't need to worry about uh storage sticking which is a major pain point for both developers as well as users.

21. 9:04

    And uh I also talk about uh the um problem with asynchron asynchronous execution. Uh so I want to talk about this um optional atomic transaction that we want to introduce. Uh so basically that the idea is that uh today transaction would specify uh who the receiver is and um based on that uh it actually gets executed on the shard of the receiver. But you can think of it as actually um it's it's specifying that it's going to touch uh this key that

22. 9:33

    represents uh this piece of account um a piece of state in in um uh in the blockchain. And the idea is that the transaction should should have the ability to specify the list of state keys you're going to touch uh during the execution. And today's transaction is just a special case of that. Uh so the idea is that the executor would actually pull the state specified from the uh state storage nodes or already have them

23. 9:56

    cached locally if it's popular contract and then can execute the transaction synchronously. Uh and and here the the actual spice design is key to this because it actually allows for longer running transaction where it takes longer to pull the state if you touch a um not so popular contract or account and for the popular ones they the executor should already have them cached locally.

24. 10:20

    So this is roughly how the flow would look like. you have this transaction, it specify the list of keys that's going to touch and then uh the executor when they execute the transaction they would actually pull the state from potentially uh different uh state nodes and then does the execution uh in atomic fashion.

25. 10:37

    Obviously there are still a lot of unresolved problem here. For example, um how do you resolve uh the cases when multiple transaction need to touch the same state uh and how do you minimize failure of transactions in in those cases. Uh so yeah, a lot of work ahead of us.

26. 10:52

    So so far we talked about uh decoupling consensus from execution uh decoupling execution from uh state storage introduction of uh atomic transactions on the year and also removal of store staking. I think altogether they already give rise to a much faster more scalable blockchain with much better developer experience. Uh but there's another thing I want to talk about which is uh privacy. Uh so as more financial

27. 11:17

    activities move on ching I think the right to privacy is in becoming increasingly more important uh so businesses do not want their payment information and payment history to be publicly visible uh and individuals do not want their assets or transaction to be seen or analyzed publicly. Uh but introducing privacy is definitely not an easy task uh because you want um the blockchain to have both private state and public state. Um and um this is

28. 11:44

    because uh for a certain smart contract having the transparency uh having the state being public uh having the audit auditability of the state uh is important property on its own. So you want both to exist. Uh obviously there's still a lot of work uh to be done on actually properly design how the private state will look like. Um but our current thinking is as follows. uh the validator will run inside the trust execution environments uh and using the

29. 12:09

    confidentiality that comes with the trust execution environment to provide uh privacy for the state. Uh there will be uh the accounts and contract that actually um live inside the trust execution environment uh with some metadata being public and then you have composibility between the private contracts and composibility between uh the public contracts uh but the public contract will not be able to call the private contracts. uh and then the users

30. 12:33

    will have uh what's called like a viewing key. This is an instrument that allows them to view um the the state of their own on the general private state of the blockchain. So roughly it will look like uh as follows. The onchain state roughly split into the private uh private state and public state. So they can interact between each other uh but the public contracts cannot call the private contracts uh to avoid leaking data. And then the users would have this thing

31. 12:59

    called viewing key. it can can view the uh it account on on the private state or uh certain data on the private contract in the private state. For example, uh some balance on a funable token contract can only view its own balance and cannot view other people's balances and and so on.

32. 13:18

    To summarize, um I talk about uh quite a few exciting features that we plan to build uh that together shape shade 3.0 uh which is our vision for the future. It's going to uh to become a much faster and more scalable blockchain compared to today. Uh it's going to be much easier to develop than today and we also have uh built-in privacy and actually Alex will talk about tomorrow how this uh privacy part will play important role uh

33. 13:44

    in near intents and uh that's it for my talk. Thank you for coming.