[Protocol technical on-boarding series] Testing in Nearcore (March 3rd, 2023)

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:02

   hello hi so the not team is coming to you learn about this it's not for you guys it's definitely not for you Nikolai you've G to be super bored with this I have no idea about naok I'm not going to talk a lot about naok we're killing it it's in the title it's a releases CI and N yeah it's just a question okay yeah I was thinking a lot about what I want to talk about and then I started writing it down and it took me an hour to write write all the things I

2. 0:42

   want to cover and I thought well I should maybe just cover the basics I wanted to talk about how to write integration test like what basic integration test we have what basic P test we have because it's so super useful and it's super scary to do on your own but is it scary it's a nice python script okay um okay maybe we should wait for another minute and then we shall start what did you write any rust integration test like tests in

3. 1:32

   integration tests directory of near core yeah I think one uh why not a long time ago was it scary for the first time user or no it wasn't too bad I me like copy pasted another test and yeah changed it to do what I do usual stuff okay okay so let's start hi uh I will cover some testing with doing new core uh as part of continuous integration as part of uh making sure that we are

4. 2:18

   releasing uh good enough binaries and we are releasing good enough versions of our code and uh it's going to cover some technology and mainly I'm going to focus on the basics and in the end I'm going to tell you what you should know about testing in general after this presentation so let's first think about the pass from zero to main net like from developing of your code to this code being run in main net on all of the notes so first you write your code and

5. 2:55

   test for a code and you make sure your test pass uh in whatever order doesn't matter uh you you would want to call Cargo format because it is part of the test your Bill Kite will not be happy if your code is not formatted properly if it's not an intuitive part of your pipeline to call this command please include it in your workflow but still uh you commit you push you already know this part you experien it it is easy

6. 3:26

   after that you may have noticed that uh something called build kite creates uh a test run for your commit build kite generally configured to create uh a run for every new latest commit in the branch and if uh the previous run hasn't finished and you pushed the new commit uh the previous run will be dropped and the new run will be started for the new latest commit uh this build kite run is uh

7. 3:58

   running testing Comm commands build kite is generally a tool that we use to run uh any commands on some cloud machines we use it for two things to run tests and to release binaries release is a tricky word we use it like in three different minutes throughout the whole testing releasing process here I mean releasing binaries as in uh compiling binaries and uploading them to S3 so build kite uh in our uh infrastructure does these two things it

8. 4:37

   uh tests are called and it publishes binaries so build guide will create a test run for your commit after that you can see that probably your tests are green you open your PR then you go through iterations of improvements for every new commit build kite will again create a new test run you approve you merge and you stop caring about your Cod you move on to the next thing but after

9. 5:02

   that testing doesn't stop what happens after merch uh at night probably by some time zone uh head of Master will be tested on naok with more tests uh naok is another testing system it is created by us and it is maintained by us and we are trying to kill it for that specific reason on N do we run more expensive tests tests that require more time to complete and also fing which we will not cover

10. 5:35

    here uh next day after your code was merged to master uh the head of Master is deployed to our Canary nodes uh it is a specific kind of nodes that we use for testing and it's uh that code is also deployed to betet specific Network that we use for testing I will cover them uh in detail uh later and uh somewhere next months after your code was pushed to master it will end up in the release branch that will be cut from

11. 6:10

    masterhead so from that point your code is on the PA to be definitely released what happens between cutting the release branch and releasing on mainnet so cutting the release branch is still action kind of visible to everybody but ignored by everybody it is only meaningful to us uh we are cutting the release Branch to kind of make sure that we are set uh the commit that we want to release and make it made this

12. 6:41

    decision sort of final no new features no new commits from Master will automatically end up in the release we can uh cherry pick them into that Branch but uh it is a manual process when you need to contact no team if you want your feature to uh as soon as possible end up on the main net after after we cut the branch after we settled on the commit that we want to release we make sure that uh this commit is actually good we do some testing we

13. 7:14

    do a testing we mcknet so we are creating uh a mcknet and running uh some special codes there to make sure that uh uh our new binary will be okay and uh we after that after a week of waiting uh the week because we want to make sure that that code is also was tested on canies on vonet uh and manually on mcknet uh we are releasing it to test it then comes iterations of improvements because no release is perfect there are always some new

14. 7:53

    patches but for every new release we're creating a specific mnet test some after 3 weeks on testnet I am not sure on this date it is always sliding but somewhere after 3 weeks on testnet we release the branch uh to main net and that's it your Cod is now in mainnet so the whole process from cing to releasing on mainnet can take somewhere from a week in like very major security patch cases uh or a few days in major major security

15. 8:27

    patch cases to like months in case you just miss the release you have to wait a month for test net release and then kind the months for many it release okay now I dropped a lot of words that you may not understand let's go through them first word is build guite what is build guite first of all build guite uh test pipeline uh is kind of separate SC uh to what we use build kite regularly for mostly we use build guide like by the number of scripts we mostly use build guide for uh publishing

16. 9:06

    binaries but we also have build kite test Pipeline and you can find specification of that pipeline in uh near core uh in this pass so uh there are clearly defined steps of testing uh mainly you should be interested in cargo tests so R tests we are running inexpensive tests so tests that weren't marked expensive and tests that weren't marked ignored uh and we are running them with uh both stable and nightly build and to

17. 9:45

    speed things up we are running them in batches so we're separating integration tests like a specific directory integration tests and all the other tests uh and we're also running some tests that are written in Python uh the three last lines are python tests backwards compatible upgradable and DB migration they're basically making sure that uh newest uh binary is compatible with uh latest release that we can up

18. 10:17

    move from latest release to this commit also there some Sage checks for code you probably noticed that's why carg format uh build cite test pipeline is C automatically for every latest commit in the branch so to trigger it you just have to push your code and it is super easy you have to wait 10 minutes and you will have results it is longer to do all of this testing on your machine so just push your

19. 10:51

    code but build cut is also used for other things oh yeah troubleshooting uh usually you want to see this green stuff but sometimes it's not that uh first thing you have to do is like sanity check that you're looking at the right page uh I want to explain some things about uh this page so you can see uh here at the top uh you see the number of builds but you can see also the name of the branch from which this build was pulled and uh a specific

20. 11:24

    uh commit you can click on it and go to the page and the name of the will be the name of the commit so make sure that it is actually like correct commit that you want to test uh then uh you can also hit retry like rebuild the whole thing from scratch I don't know why you want to do that maybe the major build guite problems like we experiencing now and they just fix them and you can now click rebuild to uh make sure that uh your

21. 11:58

    test is actually tested properly but in other cases you probably see that only one of those steps is failing because of w tests uh usually you can see that uh here you can see that I've retried this test you will see uh somewhere in this place a button to retry if you don't see it please log in if you log in and don't see it contact somebody from a team some someone messed up this permissions and uh how to make sure that

22. 12:33

    it is actually a f test and not just your C or introducing someb well you can expand commands uh jump to the end and then see uh specific tests that were failed so here kind of in this sign color uh you see uh the name of the test that failed I know that this test has nothing to do with my code just from looking at the name so I hit retry and as it turns out the tast test was just

23. 13:04

    faking uh in this case you should also notify someone in zulip or not if you but probably should notify somebody in zulip uh that you found another fake test uh and Link this build kite built if after ret try you still find yourself with uh the same test failing you always can rerun this specific test uh so just don't forget to get in the guts of the commands uh it is a tricky thing so you

24. 13:40

    can always look at the name of the test uh in GitHub and take look at the blame who wrote this test but that person might be already not working in Pagoda and usually the tests that a flake are very old and so you just notify General and and maybe if the test is persistently fakey or nobody's responding being the whole cor team or not team just be bold with it um people usually are glad to fix for

25. 14:16

    test because it's easy uh usually okay uh if you ever merged Golden Master and also hovered over that uh green check mark near your commit you may have noticed that there's a bunch of checks performed for your commit a lot more than just for your commit in PR like nine checks compared to two it's because most of those checks are not checks those those are builds and specifically build guite builds so you can see the suffix release like New

26. 14:58

    York nightly releasee near core per release near core release uh these are jobs that are triggered on any Master commit and they're building uh binary and releasing it to A3 publishing it to3 they're also doing other stuff but I don't know who uses that other stuff and you may find binaries useful so I'm only going to talk about them so on Master it is happening out automatically uh the pipeline sort of looks like this it has four combined steps like three combined steps fetch

27. 15:37

    pipeline doesn't count uh and after all of those steps are completed we are updating the latest version on S fre so how S3 what is S3 S3 you can think about is basically uh a file manager that is on the cloud and open to the public uh we have a nice three structures there and your build will end up in a uh subdirectory Linux subdirectory Branch uh your branch name subdirectory your commit so all of the commits related to

28. 16:14

    master will end up in Linux master all commits related to specific release will end up in Linux and number of that release because that's how we name branches for releases and for that specific release we will update the latest commits that we pushed to always kind of know what is the freshest binary for that specific branch that is why you can also call this pipeline for your uh special branches that are not master uh you can manually in the

29. 16:49

    interface create a new build give it arbitrary name uh choose a commit uh that you want to build from and uh choose a range that you want a build from this is basically creates a publicly available build on S3 uh it is useful in case you want to do some big testing like with mcknet you will need to uh make uh binary downloadable by those nodes or if you want in any case in any way publish a

30. 17:20

    binary this is the easiest way to do this or if you are lazy and don't want to wait for 30 minutes for build from zero on your new machine I don't know yes if you're building on uh as we were advised on your fork I believe build kite is not available how do you publish can you still use build kite to publish uh a binary oh no I don't think that this works for for Forex so it is kind of PR privilege for people who are in

31. 17:59

    near organization and it may go away it is still a useful tool for uh near release so and we use it for near releases so releases are still branches in the new core uh repository and for releases it's as likely it is going to stay that way that we are using build kite to build and release binaries so if you ever want to check out a release that is not the latest but older uh you

32. 18:31

    can search through builds in build K or you can search directly through as3 directory like uh Linux and uh version of the release that you want to check out the binary is wiely there uh again if you are to ways to build and you should be it takes too much time uh okay so if you are confused where your binary ends up uh you go to Binary release step you expand running commands you jump to the end and you see the

33. 19:08

    whole wall of files being uploaded to three multiple times to multiple path just for the fun of it for us to easily use it uh but at the top you will see that uh for 1301 release uh the near D was uploaded to the pass like new core Linux 13.1 new and for like latest Master you will be uh finding new G in New Linux Master new easier just explaining some very basic stuff here okay uh also you can look what update latest version does and at the bottom

34. 19:56

    you will see that it again updates latest file in the uh deer very specific to your branch so if you are developing in near core branches not in Forks uh don't be afraid to build your specific binaries it will not mess anything up it will be just your little secret your easy way to use your binary in on other machines okay let's move to n n is something made by us uh is something maintained by Us and something that we are soon going to kill

35. 20:37

    in order of in uh in order to make our lives easier but also in order to use build kitee more because one instrument is better than two uh N is a system uh is an orchestrating system so basically we have uh a lot of machine machines not a lot a lot but some machines in Google Cloud that we use specifically to run tests and fing tests uh and we call this in N uh n runs test specified in uh this

36. 21:19

    file and they are for some reason split into three categories and not in sendbox spy test and expensive press tests uh sandbox tests uh for purposes of this meeting does not differ from pie tests they are still pie tests that spin up local net and test some stuff so you should know about two types of tests that we run on Native first type of tests is Pyon tests that usually just create a local net environment usually

37. 21:58

    with four machines and do any kind of testing that you want to do with uh a network some tests use different binaries on different uh nodes some tests upgrade database some test uh kill one of the machines to make sure that that machine is Ban uh that node is banned as validat in the next Epoch all kinds of stuff and expensive rust tests are run also in naok because we cannot afford to run them per commit it's way too long so they run

38. 22:38

    nightly uh as a user you may see uh some context around any failed test uh for rust tests I didn't include screenshot because it's trival it's just uh two outputs error and regular output uh for carg test command but for python tests it is more trickier context so python tests again usually spin up a local net so we need not only uh an output of specific uh python file like python run we also need logs from those machines and they

39. 23:25

    can be find found in test zero test one and so on and also in order to debug this test we probably would need uh a home directory of those nodes and they can be found in node States those directories include everything note keys validator keys if everything configuration file anything saved in that directory during the run so if you run as command and just uh saved output there it will will be saved

40. 23:58

    in the context and you can look at later when the test fails and also add data uh DB directory uh it is very useful if you want to kind of not recreate the test but recreate the failure uh locally but uh there are no Silver Bullet here because uh how you should approach is heavily depends on the test you should just know that you can access the full state of every note in your local net

41. 24:30

    cluster uh after the test failed if it didn't fail I think we save only outputs and we do not save uh home uh uh states of of every note as a developer uh you can use it to make sure that your new big feature is not breaking anything that is not tested in build guite or make sure that your test is actually correct so in first case you would want to schedule a run with regular nightly tests that are

42. 25:10

    defined in near core you can do this by just calling uh special python script n.p and specifying your branch and commit which you want to test uh if you want to check out your new test or make sure that some test is no longer flaky like I usually do uh you can run your uh test uh even several types so uh you will call pretty much the same command as for regular nly testing but with different specification file and that specification file can

43. 25:50

    include a first number of times you want to run a test it can Amit that number if it's one but if you want to run several times you can include like 1,000 and uh expensive means that it's rust test P test means that it's py test for expensive it's kind of uh a weird notation you have to specify the package of new core in which the test lies you have to specify the name of the binary

44. 26:21

    uh which you want to produce I assume it can be an arbitrary name you have to specify the test and anything that you want to add to a carg test command for example features nightly and for p tests uh you just need to specify like uh imagine that you will call that would call that test and specify any argument that you want to pass so you can use n in your development it may be useful if your if your feature is very big and it can touch on several different things and

45. 27:02

    you are not sure that uh build kite tests are really covering uh your feature and it can be useful if you are writing some tricky test and want to make sure that the test will not break uh later for no reason okay candary notes candary notes in terms of testing come after nck so if you remember the timeline build kite is during your development n is uh night after and canies come like after that continuously at least for a week before

46. 27:41

    we actually release your code to testet Kary is a note in our production Network that runs Uh custom code a fresh code so it is a note uh that works with real data real traffic real network but the code can be arbitrary uh we are maintaining several Canary nodes in testet and Main net uh in test net we are maintaining four of type archival RPC RPC but with binary with extra assertions and a validator note also is

47. 28:19

    a canary in testet uh and in main that we do not run a validator node we do not own validator and mainnet it's a principal thing I don't know why but we just don't so we have only three convers mainnet and you can create your own Canary it is very easy there are instructions they will be linked uh in the end and uh you can create arbitrary setup if you wish to know that your feature uh doesn't break communication with the network that this note will not get arbitrary banned

48. 28:57

    isolated it will not get uh out of think with others it will not go into some Fork if you are touching in at any Behavior Uh that's going to affect block hash or like Network layer maybe uh you would want to spin up at some point can R note and make sure that it works uh betet uh okay uh for gener note uh you shouldn't run it for weeks like we do it is uh our protocols that we want to test code for at least a week before releasing it it's extra precaution but

49. 29:40

    uh for you it can be whatever your test goes for if you know that your code either will not break or will break on the first block then run it for several blocks if you have like uh something concerning Epoch boundaries or protocol versions uh run for at least one to Epoch usually the trickiest cases are Epoch switches so start some somewhere before and EPO switches and end somewhere after and it should be okay like the general rule rule of time so it can be just few hours but it's not like

50. 30:21

    uh few hours of you monitoring heavily locks you set up the note and in few hours you return and make sure that it didn't crash okay I have another question is there way to simulate EPO change faster than waiting for half a day in test in may not no because it is not controlled by you but in next topic yes okay so Bonet and mcknet betet is uh a network of 10 notes all of which run the latest code and mcknet is kind of

51. 31:04

    arbitrary way of defining any network uh that runs custom codes that is isolated of our production networks that runs on a custom number of nodes so everything is custom uh even your traffic is defined by you there are no limits besides your imagination and you're willing to tediously code every case of transaction you want to uh send to your network and at what intervals you want to do this uh we have some scripts that do it automatically but they may not

52. 31:43

    cover your specific cases mcknet we spin up on demand Bonet is running continuously and is updating every day and Bonet as you may have noticed is a type of mock so in mnet environment you can control everything including Genesis file and Genesis file controls the lens of an ebook so you can create arbitrary network with arbitrary number of charts even uh and uh arbitrary settings configurations uh to stress test your

53. 32:20

    code and again it may be a lot of work but you probably can do anything okay now to touch on the release as I said we use three different uh meanings for the word release release as Act of making a release is an act of assigning specific commit a GitHub tag so release as an object is just a reference to a commit in GitHub that's it uh cutting the release is an act of creating a branch from master so it's an

54. 33:02

    act of creating a point in time uh after which we say okay we do not automatically accept any new changes and Publishing uh publishing the release is creating attack to that latest commit in that selected branch and also notifying people making a publishing some binaries uh whatever else is required by infrastructure but the minimum required action is to create a tack to some branch and release binaries can be found in uh this path in

55. 33:46

    S3 uh as I previously explained uh releasing a binary is just creating a branch uh a subdirectory in uh S3 with the name of the branch and name of the release branch is version of the release so for example right now we are on 132.0 and that's it there is nothing else to talk about the R so let's cover what you should know uh generally you should know that in terms of complexity coverage and

56. 34:23

    flakiness it all goes like this unit tests then rust integration tests then P tests that spins local net then can re note and mcknet they are kind of different cases you cannot really compare them as I said Canary node is using real data in real traffic it just using some other code and mcknet creates uh arbitrary conditions uh which we can fully control uh and in terms of uh complex

57. 34:56

    lexity it is easier to write and execute unit test than it is to write and execute uh rest integration test and so on it is extremely difficult to write a good mcknet test uh that is why you shouldn't know how to do that if you ever find yourself in need of mcknet test you should immediately contact note team and uh dump all of the information that you know all of the requirements and we will assist you however we can for example we have uh several mcknet

58. 35:31

    tests already uh created so we have mcknet test for release uh as I mentioned that checks uh upgradability basically we are creating a network uh that uh operates on the latest release and then uh we gradually update nodes in batches to the new used version of release and making sure that nothing breaks also we have several version of transactions that we can send to that mcknet test so we can send fully synthetic transactions that cover basic

59. 36:13

    transaction types or we can somehow mirror uh main net or test net real traffic and it takes a lot a lot of time to set this up uh because of computational complexity I suppose I I don't really know but it takes a lot of time like a day to set it up uh but it is very useful in case you're introducing like a major change and if you're doing some risky protocol upgrades you should definitely consider doing mcknet tests before your feature

60. 36:46

    goes to the release stages okay uh so what you should know in terms of Technology build is a quick way to run basic tests quicker than running them yourself uh n can be used for expensive tests or repeated test runs so either to make sure that your big feature is okay or to make sure that your big test is okay uh the binary you're looking for is probably in A3 and if it's not it is easy to put it there and you can run your own Canary

61. 37:27

    note so you can test your code on real data it is easy and it is fast you also can run your own separate Network that is not local net that is more sophisticated that does whatever you want automatically it's a matter of code it's a matter of effort and uh we are not given any specific instructions on that as than basic instruction how to run mnet test for the release because there are no Silver Bullet because it's such a powerful hard and uh V

62. 38:05

    to so final table if you want a build kite test just push your code if you want a build guite binary build either merge it into master or manually trigger the build from build gu interface if you want to n test run push your code and then schedule run there is a documentation I will send this presentation into protocol team uh slack uh for local net test uh try to copy something already written in pest directory anything that uses the function start cluster is probably a

63. 38:44

    good template for what you want to do for Canary notes uh there is a documentation called go on Main net to create your own main net or test net kry note if you find that something breaks that something doesn't work contact not team we will H we will be happy to help and for M net test just contact note team uh and we will help you get rid of that idea okay I have a lot of time

64. 39:15

    left and I can talk about a lot of stuff but it's better if you have questions I do have one question M um you mentioned the release mcknet tests M that we we run a mcknet with a few nodes and then test that we can upgrade from one version to the Target version do we run this continuously or it's something that we have to run manually we have to run it manually we run it whenever we want to make a release uh we run it manually because

65. 39:49

    first of all it is expensive and uh just having a we're creating a network of 100 nodes I think or 200 so a lot of notes and just having those instances in Google cloud and some of them with pretty big databases is expensive okay oh yeah betet is less expensive type of test because uh we don't have real traffic there all of the traffic is synthetic and databases are smaller the small number of accounts

66. 40:30

    like way smaller I think around 200 accounts total and we just endlessly send the same types of transactions there and Bonet uh is only 10 notes with a small state so we run this continuously uh betet is always operational and we are updating codes there every day up dating the code is not the hard part it's fully automated I didn't cover that tool here because you probably don't need it but uh we have a tool to update notes in

67. 41:13

    MTH um I just wanted to clarify some points so uh I want to talk more about uh local net tests inite tests and mcknet is definitely an interesting topic and canri uh so mcknet uh if you want to at some point create a very specific conditions for your test so kind of stress test your code you should be thinking about mcknet uh you should be thinking about uh creating uh a lot of sharts or a lot of uh transactions or very short epox uh

68. 41:58

    we can manipulate all of that and stress test your code on those assumptions with Moet uh why can't you do that with other tests so you cannot do that with r test because R test is definitely not uh suitable for uh a lot of validators a lot of clients with different types of databases where you can control everything it is mostly suitable for kind of like one client tests maybe two validators

69. 42:34

    tests um local net is a good example of creating again local net is a type of mock net is uh it is a separate network with arbitrary code uh with arbitrary configurations but local net is fully run on one machine and that means that it has limits to how many nodes we can run simultaneously so you cannot create local net with 100 nodes you will have to do some Google Cloud magic for that uh some distributed system and we already have like mcknet setup so you

70. 43:15

    would want to use mcknet setup to create uh big nodes so on local net you cannot test 100 charts but in Google Cloud with a specific mnet you can can test 100 charts it will be expensive though and uh if you want uh real data on your local net even if you want just four nodes it is hard to do on your machine to create four databases uh maybe uh it will be hard because of space maybe it will be hard because of speed

71. 43:54

    but it still will be a bad and inconsistent test and we don't want you and it is hard to spin up so again if you want big state in any sense if you want a lot of nodes if you want a lot of uh uh space for database you want to think about mcknet rather than local net test but usually you don't want that usually you want to test some simple behavior that is uh easily reproducible in small Network works so for example

72. 44:27

    what happens if one node upgrades what happens during database migration what happens if we kill uh 33% of validators it is easily reproducible with small networks and it is even more prominent with smaller networks uh problems of local net is that those tests uh local net and mck net that those tests are freaking local net tests are mostly flaky because uh we want to run them fast and we want

73. 45:02

    to uh make them as small as possible uh that means that any arbitrary mistake in synchronization any arbitrary mistake in uh delay in starting up one note messes up the whole tests so those test can be incredibly and stable uh but I think it was fixed uh like a months ago to more stable version now we all better still you have to think about synchronization and you have to think about uh not as human

74. 45:35

    uh too much for mnet tests uh you need to be aware that there is a lot that can go wrong with a note during like 24hour span and it may not be related to your code so if every node crashes sure that is probably your feature but if one note crashes it may be something that is completely unrelated to any of your developments I have a question about the Smet test where do I see the log and what's happening to the mcknet to my

75. 46:17

    mnet so mcknet test is just spinning up a bunch of instances in Google Cloud and uh communicating with them by sending transactions and running uh scripts on them with well with various Technologies but just running scripts on them will work in this context so every note is a separate instance in Google quote in a uh separate project and uh you will be able to access uh any specific node and

76. 46:59

    just look at the logs as you would for any other node that you're running so it is as close to real life example as possible it is just a separate Network so it is close in the way that we run it it is close in the way that we execute it and monitor it uh but it has different configurations for the network itself okay okay okay I should probably stop blobbing about mcknet uh if uh I want to you to

77. 47:38

    remember anything from this is that existence of the stable and uh it is mostly uh aimed at people who are only starting working at Pagoda and working in near core if you are more advanced you may be only interested in existence of mcknet test and their ability to mirror real life traffic and if you ever think about stress testing uh your code on a lot of machines and making sure that productivity like BRS per second Do not

78. 48:18

    drop uh mcknet is the way to go okay if there are no more questions I think we can take yeah I I have a question so is there a way to let's say um like uh for example if I want to shut down a node on the mock net and then send send some commands to that node and then restart can I do it on the mcknet uh yes you can run arbitrary commands uh as user of the notes on mcknet to again it's it's it's just code is just uh I don't know how specifically it is

79. 49:01

    implemented but Google cloud has capabilities to run arbitrary commands uh from uh users that you authenticated uh on the machine just from your machine so you can write like Google Cloud execute and like whatever you want to execute like uh stop near D and it will stop near G and yeah yeah yeah yeah sure uh so you can do whatever you want uh in part of uh update Running uh code in

80. 49:44

    part of configuring and uh also as sending transaction to whatever note you want and however you want it is complete creative freedom you should think about it as uh just another Network where you control everything from the start okay any more questions no okay then thank you for watching like And subscribe and yeah see you around see you awesome thank

81. 50:54

    you