Block Zero Stage | Utilizing NEAR Blockchain Data for dApps and Analytics

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   next up uh we are going to be hearing from pavl kudinov from the pagota team on utilizing near blockchain data for daps and analytics the mic can you hear me guys all right good good to see you all thanks for coming uh I'm uh very excited to talk about uh data um I'm working at pagod I'm the engineering manager at the data platform team and we're looking after uh different tools for analytics and uh for developers to build uh reach user interfaces um I'm going to be

2. 0:51

   talking about indexers today um so there are there is a little bit of confusion when people talk about IND indexing what what is indexing it's actually a lot of different things uh so as you know blockchain is a stream of transactions actually stream of blocks and every block has a bunch of transactions uh but it's very hard to fetch some uh transactions in the past or make some

3. 1:17

   aggregate queries on these transactions uh to make these queries you actually need to do indexing but the catch is uh you need to do that differently depending on the use case you uh are solving and uh I found through uh almost two years working in the space that there are two major use cases uh usually uh and one is analytics let's say you're a product manager or a blockchain analyst and uh you wants to see the behavior of users of certain uh smart contracts or

4. 1:49

   certain apps um and you want to see this in time uh and you maybe you want to find some insights so this is what we call analytical use case and the other use case is probably more important to uh attract more more users is to build reach user interfaces um for that you need a different type of indexer which is uh real time and which gives you uh all information needed to show on the user interface so let's talk a little bit about analytics

5. 2:19

   first and uh we recently released a public data set on Google Cloud uh this is supported by Pagoda and uh this is uh updated near real time and this is solving this analytical use case anybody can come in uh and build SQL queries to get different insights uh we store uh and provide the full block uh data uh all the transactional data receipts execution outcomes uh and even parts

6. 2:49

   near social DB uh data so you can see all the content and query all the content generated on new.org and other gateways um one of the simplest uh use cases is to actually understand the uh the number of users that interact with uh your smart contracts right so uh with this this is a simple query that does that I I'll just quickly show you how you could use this query uh if you go to docs.

7. 3:22

   new.org and search for big query you will see uh a guide on how to use this and here you can just click click on the uh view the public data set this is my account on Google Cloud um click view data set uh and you can just take this query that I had on the slide and I'll walk you through this quickly oh sorry paste it here and execute so uh what it does is we're looking at all the receipt actions so these are all the

8. 4:04

   receipt actions we are then joining these receipt actions with uh the transaction that this receipt is originated from and if you analyzed near blockchain data you know how sometimes complex that is because receipts are executed in uh different blocks so when you're looking at receipt you don't necessarily know which transaction it was originated from but for that we created this table receip a region transaction so we're joining on receipt ID and then finally we're joining this receipt uh this this data set with uh

9. 4:37

   transactions table and to get the uh by transaction hash to get the signer so this is what we're interested in the signer who signed the regional transaction that produced a receipt that interacted with in this case it's talking. SWAT uh contract and I'm only interested in function calls and I'm only interested in anything that happened uh this month and so as you can see um we have uh roughly 80 to 100,000 different signers daily uh

10. 5:10

    signing transactions that interact with token. s contract if you replace this with um any other contract that you uh manage or contracts you can use obviously things like like uh statements Etc um you can uh build different insights um this is available uh for anybody to use uh it is not completely free uh it is free from pagota side we're not charging for this but you're charging uh the Google cloud is charging for compute every time you run the query you need to

11. 5:47

    pay for the service on Google Cloud you can copy this data daily to your data warehouse you can take the portion of this data and we have some uh big companies already integrating it to their etls ETL pipelines um we are soon going to be adding more data to that and we want to hear your feedback uh what else do you want to see there uh we're also looking at the patterns of uh data usage uh but the things that are coming are fungible

12. 6:18

    tokens nfts and all the different events so basically if you want to look at the events pars events for your smart contracts uh you'll just be able to make simple SQL queries uh directly on b b query all right so let's talk about reach user interfaces um basically what is that it's allowing users to find information on your apps and I see designers smiling because designers always have a lot of things a lot of ideas to uh uh to show

13. 6:53

    to to make uh the the use of interfaces uh easier right but how to get the data there and how to get this data um in the right shape and form and also make it up to date right so that's the real question um what are the common UI patterns that we have so first of all we want to show user specific information when we have a logged in user we want to show um assets and information that they generated or that is relevant to them uh we want to aggregate across multiple

14. 7:26

    smart contracts so for example if you use defi application there might be multiple defi pools or lending pools uh and you want to aggregate across all these smart contracts and as new smart contracts are deployed and created you want to automatically track these smart contracts too uh you want to aggregate over time so for example you want to show weekly information or information

15. 7:50

    for the past few days uh or monthly uh and you might want to aggregate across users so if there uh if a user has m multiple accounts and they want to see information across all these accounts you want to show this information together you want to allow users to filter information so let's say there is a history of transaction and they might want to filter by certain parameters so filtering should be quick which means you need indexes on this columns that

16. 8:16

    you're uh uh indexing uh filtering on uh of course you might want to implement pation because there might be too much information and you want don't want to download all of it which means you need efficient ordering um and of course you want this data to be near real time which means that when your users make the transaction uh and your user inter your user interface reads from uh from your database you want it to be up to

17. 8:42

    dat uh ideally instantaneously right so how to achieve all of that well actually you need a database for reads um as you can see we have a very happy user who uses near blockchain uh a dab that is built on near blockchain um it makes rights to near protocol directly which means sending transactions signing and sending transactions through read RPC and um the blogs are then streamed

18. 9:12

    um and then how to get them to the user interface well you actually need to look at every block and write something that we call a projection uh and then this projection is a piece of code that looks at every block extracts information that is needed and relevant for your dab and puts it into the database which we call the read database and this is fairly traditional uh web 2 so to speak

19. 9:35

    approach when you have a database specifically designed to serve queries that are needed for certain user interfaces uh and you can build what we call materialized views there and then this user inter the user interface the user when they want to see something they actually query from this database and not from the protocol itself because making queries the protocol will result in hundreds or maybe thousands of RPC requests and making user interface very

20. 10:06

    slow um so how to do that um well first of all uh we more than a year ago maybe more than a half one and a half years ago we introduced near Lake that allows you to stream blocks directly to your server you don't have to uh run a node running a node is very expensive you need to maintain this node you need to upgrade it so if you don't want to do that you can use near Lake it's a storage of all the blocks uh in Json

21. 10:36

    format um it is uh always up to date it's less than three blocks away from the tip of the network usually uh it is roughly 25 times cheaper than running a node it can stream 300 sometimes up to 600 blocks per second um and currently we have more than 50 indexers live on Main net um you receiver pays so this is AWS S3 receiver pays policy which means we pay for storage for maintaining it but whoever is running the indexer will need to pay for every querry for every

22. 11:13

    download of this Json uh block and as I said it's 25 times cheaper than running your own node we have uh indexers life in JavaScript rust uh python go and even Ruby we officially support JavaScript and rust and majority of indexers are actually written in JavaScript um so how will the architecture look like with this approach well we have near Lake which is an awss 3 um we then create here nap1 141 indexer or near balances indexer

23. 11:50

    these are two Standalone applications that just read uh blocks perceive blocks as they show up in near Lake uh process them this is our projection process them extract needed information put in pogress SQL and then HTTP server specifically written for this use case is uh serving this data as users requested um we have this running as an enhanced API on pag. uh these two GitHub repositories have the source code for it if you need tokens history balances uh

24. 12:27

    for near tokens for fun ible tokens for non-fungible tokens this architecture is uh implemented in these two repositories and on pagoto doco there is enhanced API that you can see live for yourself this was one of the first implementations of production indexers that we had um but then as as we were building more and more indexers we found that actually first of all this is the pattern that keeps repeating it's almost always the same pattern and and if you

25. 12:59

    run multiple indexers you need to look after the architecture uh the infrastructure sorry of these uh indexers and you need to make sure that they're not failing they're not lagging behind and the the API is always live so we find that the this is one of the best practices to uh have the database optimized for the use case and to have graphql uh ideally automatically generated after your schema to make sure you do fewer queries to to the

26. 13:30

    API and then we thought well uh developers want to build apps faster they normally don't have database engineers in the team uh they don't they normally don't have time to build a proper architecture right away because they need to show traction to their investors so they want something quickly um so we uh started working on uh query API it's a fully managed serverless indexers uh and Stor storage and graphql end

27. 14:01

    points this allows you to create an indexer without maintaining any infrastructure this is what means serverless uh we do have servers of course but you don't have to worry about them at all um we are autoprovisioning custom indexers for you uh this is not a specific indexer for let's say tokens this is the indexer that you need for your specific set of smart contct contct s so without query apis you had to build smart contracts the data from Smart

28. 14:34

    contracts comes to near Lake that I described before automatically then you need to create a database schema you need to write a JavaScript or rust code to uh process every block then you need to run a database instance and to manage this database instance uh you need to run an HTTP API server you actually need a load balancer because if you have a spike of transactions you want to scale this and you want to serve this traffic

29. 15:01

    um and ultimately you need probably to do some client side Transformations if you built rest interface as people usually do uh when they build um these apis so with query API all you need to do besides your smart contracts of course is to write think about the SQL schema and the JavaScript code that reads every block and transforms data from this block into the schema that you just created all of the other components are provisioned automatically by the

30. 15:34

    service um so this is a snip I I want to show you a live demo but this is a screenshot of uh a ddl schema for the new.org main page social feed it is powered by query API today and uh this is the a couple of tables actually three tables for posts comments and likes um fair Fairly standard hress schema so you just create this write and query API after that you write JavaScript indexer logic to process blocks as they appear and then after that you have

31. 16:15

    automatically provisioned graphql API to access these data so what is the workflow how do you work with query API what is what is the process um so first first of all you go to designs and once you have the designs the user interface that you need to implement as let's say as a front-end engineer you look uh at all the screens and you think of these screens and think of uh what are the queries need to be made to the back end to receive this

32. 16:45

    information and you want to actually have fewer queries on every load of the screen right so you need to design database after the schema that is needed to show uh to implement mement the the figma Prototype right so you create this SQL schema and then you find few block Heights uh that block numbers right uh that you want to test on um and you'll write JavaScript code to inspect those blocks extract the information you can

33. 17:16

    even debug this code right in the in the browser I'll show you in a couple of minutes um then you store the code and you watch the logs the code and the schema is stored on CH on chain on your blockchain um this is the the storage layer for query API um and then after a few minutes the uh uh infrastructure will be provisioned for you and you will see the logs and you will have the database and the API uh obviously you might face some bugs and certain things won't work right away so you'll go back

34. 17:49

    and fix those bugs following your own logs um and then after that once everything's working you can go to graphical playground and generate queries and and Export them so when you're creating an SQL schema um you need to try to design it for absurds which means you might want to reindex your data you want might want to rerun your indexer from scratch and your scheme should support that as well

35. 18:15

    as the code um to make graphql work well and efficiently uh you need to create foreign Keys um you shouldn't forget about indexers indexes to ensure that the data there there are no full scans uh when when you're querying the data um and sometimes you might want to show some aggregated data let's say Group by date or group by week uh so for that you can use views and graph K will automatically generate a special query for these views um this will reduce the number the

36. 18:46

    amount of code you need to write in JavaScript so when you're writing um the index or logic code in JavaScript uh we're providing a near Lake Primitives Library which allows you to extract data from the Block easier with fewer lines of code um you still get access to the whole block you can extract the receipt args uh arguments outcomes uh receipt outcomes uh events everything that is

37. 19:12

    stored in the block and then you can use a context. DB method to access the database write data to the database that we created and you can also write logs there so once everything's working uh you can go and look at graphql queries we have a graphical playground um you can click through and just you know select the fields that you need uh you can try the queries right there debug the query uh and then from there you can directly export uh code to be used in

38. 19:45

    the boss component you can even change a query uh keyword to subscription keyword and graphql and you will receive websocket connection uh with uh with that subscription automatically uh let's try to look at the demo how this works so first of all this is new.org main page uh how can you find query API you go to Applications develop uh find near query API here you can see different indexers

39. 20:16

    that uh people build there is one specific that I want to show you this indexer is currently running you can can see the the status um it shows the logs that the user created um you can see the current block height you can see the status that it's running let's look at the code um this is written directly on new.org as a widget stored on chain um the editor experience is powered by a separate uh web instance because we wanted to produce as much uh you know as

40. 20:57

    reach experience for editing as possible um and here um you can see the code that was written to extract the posts the comments and the likes for the main social feed it all starts somewhere here when we look at the blog we look at all the actions we go through all the operations we extract function calls and set methods um and you need to know this information you need to know what the query uh you need to know your

41. 21:28

    own smart contract um but this is a key value storage on social. near so we need to look at all the set methods and then we're basically looking for posts and likes and uh here if we found found a post we run handle post creation or we handle we run handle comment creation or we handle a like and here these are the functions let's say handle post creation we assemble the post data and we just simply context. db. posts. insert the

42. 22:02

    post data um this is the schema that I showed you before we have the post table we have the comments table the cool thing is we generate Types on the fly so that it's easier for you to um right so let's say you can type context. DB do uh comments and then you can say insert and you can uh add the the object that you need names of the fields are uh the same as in your uh schema that makes uh makes access to the database

43. 22:39

    easier um so to I just switch to the uh actual indexer that is running uh in production right now that powers the main feed it's um deployed under data platform. near um I wanted to quickly show the uh bugging experience so let's say there is a block height where there is a post we can actually uh take the post uh I'm choosing a block but let's say this block in reality when you're editing an

44. 23:18

    indexer code you need to use Explorer and look at the transactions of uh into your smart contract and you need to find the blocks where these transactions um generate receipts to your smart contract and this is where you want to debug your indexer but here it's just handy for me here on the stage to find this block I go to indexer and I can switch out switch on the debug mode and paste this uh block height here I can press

45. 23:50

    Plus open the console the the browser console and just hit play here and so so what will happen is uh we will fetch the block and we will execute this function right away in the browser and everything that you're outputting these are the the logs that the developer uh output in this function we will show it here in the console and hopefully this will make it easier for you to uh troubleshoot uh the the indexing on every single block that you

46. 24:23

    have um we don't touch the database in this debug mode um so you you cannot select and you cannot insert but uh you can simulate the processing for every block very easily um now let's look at the graphical interface so here if we switch off the debug mode uh here I can create a query for the indexer that I just created we have a bunch of indexer a bunch of tables here we name them after

47. 24:55

    the Creator on the indexer the data platform near and then social feed is the name of the indexer and then the table uh that was created in the schema uh follows after that so let's say I want to query the posts I want to find five last posts last means I need to order by block height descending and I want to take the account ID I want to take the content and maybe I want to look at the

48. 25:29

    post likes uh aggregate which means I want to know the the number of likes that were under this post so I just assemble this query I can hit play and I'm right away getting the index data from here with the latest posts now what if I want to build a user interface with that uh if you want to use it in the components uh on new.org you can hit uh share uh icon here uh which opens the

49. 26:00

    code exporter and right away here you have the code that you can directly copy paste into uh sendbox which I will open here so you can copy paste it here and if you look at the preview we're just rendering the Json object but you can put anything here so for example a do account account ID will show us the account names that created the last posts and after that you can implant your user interface um we have uh I have also the

50. 26:42

    example of the widget activity feed uh that is a subscription hopefully somebody com uh sends uh uh Let me refresh maybe but while I'm talking maybe somebody will uh edit the widget and this will this user interface will automatically update because uh it maintains a websocket connection to query API uh and and and shows um live stream of all the edits to widgets uh but it didn't happen uh it doesn't happen very often but you can

51. 27:16

    check the code and uh here uh you can look at the details of this component and if you want you can Fork it there is a bunch of Styles as always um it looks at near query API at pagot pag. and here we have um a bunch of code that needed that is needed to maintain a web socet connection and update the user interface all right that's pretty much it for uh for the demo and my uh presentation today uh query API is available in a beta uh mode where where

52. 27:58

    uh basically we need to add you to Whit list to use it because we don't want to uh we want to maintain the service for everybody and want to make sure that everything is stable uh we'll just have a short conversation uh with you about your use case if this is one of the smaller smart contracts and you don't need to index the whole blockchain you're just building a dab uh we will give access to you and uh work with you

53. 28:23

    to uh bring your indexer to production uh you can use this Q code to fill up the form um and uh we have uh two minutes if you have any questions I can answer them now or you can find me afterwards right I see people scanning the code that's good so uh see you see you around yeah so you don't want the query API you want to build query API but the API itself you want it to be uh private you

54. 29:21

    want right uh well query API uh everything is going to be public with the current setup uh query API is open source so you can find it on under near account in GitHub and you can run a copy of query API yourself uh it's a bunch of Docker containers you run them um and uh this will make a private uh API for you all right it's great to be on time thank you so much