by Unity Nodes
Data

[Announcement] Automated Indexer Deployment from PrimeLab

3 posts649 views2 likesLast activity Apr 2022

Open on Forum
  1. WillHOP 2

    Automated Deployment of Indexer

    PrimeLab is excited to launch our Automated Indexer Deployment Utility for NEAR Protocol. Quickly deploy your own PRODUCTION READY Indexer 82% FASTER!

    Whether you’re a new dApp or a seasoned publisher, you have a need for reliable & verifiable blockchain data. Running an Indexer node enables you to maintain a reliable & verifiable “Source of Truth” for blockchain data.

    Current Public Solutions

    The near-indexer-for-explorer repository available in the NEAR GitHub org provides you with a basic Indexer project that can be deployed to start your own indexer. It assumes that you have experience and knowledge of Postgres Databases, Cloud Architecture, Docker, and a high-level understanding of using Cargo.

    Our Improved Solution

    We’re providing a Terraform based solution that enables you to deploy the near-indexer-for-explorer project AND all of the infrastructure, databases, security group settings, backups, etc. needed to ensure your project is up and running quickly.

    Automatic Resource Deployment

    Screen Shot 2022-03-29 at 2.08.00 PM
    Screen Shot 2022-03-29 at 2.08.00 PM2234×1532 374 KB

    Inputs

    When deploying an Indexer with our proposed solution, you’re able to build and deploy using the default parameters, or adjust the input parameters to fit your infrastructure or use-case.

    Name Description Type Default Required
    account The account number associated with the environment. number null no
    ecs_asg_desired_capacity EC2 Capacity Providers ASG - Min Number of Instances number 1 no
    ecs_asg_max_size EC2 Capacity Providers ASG - Max Number of Instances number 1 no
    ecs_asg_min_size EC2 Capacity Providers ASG - Min Number of Instances number 1 no
    ecs_instance_type Instance Type for ECS Cluster string “c5.2xlarge” no
    efs_filesystems A map of EFS filesystems to create using the EFS module. map {} no
    environment Name of Environment for deployment. string null no
    indexer_archive_health_check_block_id Block ID To Check from for the Healthcheck number 0 no
    indexer_archive_node_cpu ECS Container - Indexer Node CPU number 4096 no
    indexer_archive_node_ram ECS Container - Indexer Node RAM number 8192 no
    indexer_aurora_autoscaling_enabled n/a bool false no
    indexer_aurora_autoscaling_max_capacity n/a number 1 no
    indexer_aurora_autoscaling_min_capacity n/a number 0 no
    indexer_aurora_engine n/a string “aurora-postgresql” no
    indexer_aurora_engine_version n/a string “13.4” no
    indexer_aurora_instance_class n/a string “db.r6g.large” no
    indexer_aurora_instances n/a any { “one”: {}} no
    indexer_aurora_monitoring_interval n/a number 60 no
    indexer_chain_id n/a string “localnet” no
    indexer_command_concurrency The number of outbound connections to the blockchain number 1 no
    indexer_data_archive_url n/a string “” no
    indexer_docker_tag n/a string null no
    indexer_generate_config n/a string “” no
    indexer_is_archive Weither to set the indexer node in archive mode bool true no
    indexer_near_dirpath n/a string “/indexer/near/” no
    indexer_run_migration n/a string “” no
    indexer_set_archive n/a string “” no
    indexer_sync_delta n/a string “” no
    indexer_sync_from_command n/a string “” no
    redshift_automated_snapshot_retention_period How long will we retain backups number 0 no
    redshift_cluster_database_name The name of the database to create string “” no
    redshift_cluster_master_username Master username string “etl” no
    redshift_cluster_node_type Node Type of Redshift cluster string “” no
    redshift_cluster_number_of_nodes Number of nodes in the cluster (values greater than 1 will trigger ‘cluster_type’ of ‘multi-node’) number 1 no
    region Name of region for deployment. string “us-east-1” no

    Build/Deployment Process

    Terraform automatically provisions and deploys all required infrastructure, databases, and packages needed to run your Indexer.

    Screen Shot 2022-03-29 at 2.23.02 PM
    Screen Shot 2022-03-29 at 2.23.02 PM1920×1543 240 KB

    Outputs

    After deploying your core infrastructure, Indexer, and the required databases, you’re provided with all of the resources created during the deployment.

    Screen Shot 2022-03-29 at 2.09.04 PM
    Screen Shot 2022-03-29 at 2.09.04 PM2906×2130 458 KB

    Name Description
    ecs_cluster_name The name of the ECS cluster
    multi_region_primary_aurora_kms_key_arn Multi-Region primary key for indexer-aurora-primary Cluster
    primary_aurora_global_cluster_engine Name of the database engine to be used for this DB cluster.
    primary_aurora_global_cluster_engine_version Engine version of the Aurora global database.
    primary_aurora_global_cluster_id RDS Global Cluster identifier
    primary_aurora_reader_endpoint A read-only endpoint for the cluster, automatically load-balanced across replicas
    primary_aurora_security_group_id The security group ID of the cluster
    primary_aurora_writer_endpoint Writer endpoint for the cluster
    redshift_cluster_arn The Redshift cluster ARN
    redshift_cluster_automated_snapshot_retention_period The backup retention period
    redshift_cluster_availability_zone The availability zone of the Cluster
    redshift_cluster_database_name The name of the default database in the Cluster
    redshift_cluster_encrypted Whether the data in the cluster is encrypted
    redshift_cluster_endpoint The connection endpoint
    redshift_cluster_hostname The hostname of the Redshift cluster
    redshift_cluster_id The Redshift cluster ID
    redshift_cluster_identifier The Redshift cluster identifier
    redshift_cluster_node_type The type of nodes in the cluster
    redshift_cluster_nodes Cluster nodes in the Redshift cluster
    redshift_cluster_parameter_group_name The name of the parameter group to be associated with this cluster
    redshift_cluster_port The port the cluster responds on
    redshift_cluster_preferred_maintenance_window The backup window
    redshift_cluster_public_key The public key for the cluster
    redshift_cluster_revision_number The specific revision number of the database in the cluster
    redshift_cluster_security_groups The security groups associated with the cluster
    redshift_cluster_subnet_group_name The name of a cluster subnet group to be associated with this cluster
    redshift_cluster_type The Redshift cluster type
    redshift_cluster_version The version of Redshift engine software
    redshift_cluster_vpc_security_group_ids The VPC security group ids associated with the cluster
    redshift_parameter_group_id The ID of Redshift parameter group created by this module
    redshift_subnet_group_id The ID of Redshift subnet group created by this module
    secondary_aurora_reader_endpoint A read-only endpoint for the cluster, automatically load-balanced across replicas
    secondary_aurora_security_group_id The security group ID of the cluster

    That’s it! :partying_face:

    You’re now running a Production-Ready Indexer deployed and syncing with the NEAR MainNet!

    Release Roadmap:

    1. RDS Restore Points & Compute Restore Points
      1. Requester-Pays RDS backups enabling you to skip the long initial sync process.
    2. Indexer APIs
      1. Basic REST/GraphQL APIs that enable any development team to quickly query the indexer data without creating complex indices or queries.
  2. erik.near

    Thanks for open sourcing this! I hope it makes it easier for people to build indexers of their own! It may be best categorized under the development tag to get visibility from the devs who would use it.

  3. WillH

    Awesome, thanks for the feedback! I’ve included the development tag :slight_smile:

Back to Discussions