Writing an Anchor program indexer for Solana using Helius webhooks, Google Cloud functions, and Firestore — Part I, Backend.

“solana beach, digital art” dalle-2 prompt output

When I first started to learn how to develop for the Solana ecosystem it wasn’t quite made clear to me how applications were supposed to read relevant onchain data. The most obvious and naive approach is to use a Solana network RPC to get data for the client whenever the frontend needs to hydrate a particular account state or balance. This approach comes with data consistency problems and doesn’t work on a large-scale dApp that needs to load in <100 milliseconds. When you want your web3 application to feel like a web2 application one comes to the conclusion that it is important to listen to state changes (transactions) and store relevant state into a database so it can be queried cheaper (RPC calls are expensive).

For the Solana Sandstorm hackathon several teams came together to build a realtime onchain drawing dApp similar to r/place. With Solana’s 400ms latency drawing with others in real-time is possible! But if hundreds or thousands of users are all making RPC calls for the same onchain data, you should probably consider moving your client RPC calls to your backend and when relevant onchain state is changed, broadcast it to your users.

This is where Helius webhooks, Google Cloud functions, and Firestore come in. Helius webhooks act as a “geyser”-esq transaction feed that streams newly processed transactions (related to a selected address) to a target HTTP server — in this case a Google Cloud function that accepts HTTPs requests and processes the transactions and stores onchain data into Firestore (a non-SQL database). Firestore is great for broadcasting state changes to clients who are listening, meaning you can get near real-time state changes sent to all users currently on your site with little effort.

bonkboard.gg

Setting Up Google Cloud

First you’ll need to go to Google Firebase console. Firebase is a subset of Google Cloud services console.firebase.google.com. Follow the setup wizard for a new project. The project name isn’t relevant but it does act as an project identity.

You may be prompted to enable Google Analytics but you don’t need the service for this tutorial.

After your project is created, you need to enable firestore for your project.

Navigate to create database in the firebase console

Select start in production mode and click Next.

Select the region you’d like your firestore database to be. nam5 is the best region for US based projects.

Setting up your local environment

Clone the following GitHub repository to your local environment. The google cloud function is written in typescript and has npm scripts which make deploying easy.

You’ll need to setup up gcloud CLI (command-line interface) to be able to deploy your indexer to Google Cloud.

Once installed, run:

$ gcloud auth login

You should be re-directed to your browser to sign in to your google account to get an authentication token from Google.

Before you run this next command, you’ll need to find your project id. To do this go to https://console.cloud.google.com/ and click the project dropdown.

Project dropdown
Project display name (left) and project id (right)

Use the project id from the project popup and run the following command.

$ gcloud config set project <project-id>

Run the following code to install relevant packages.

$ cd ./functions/webhook-indexer
$ yarn install

Understanding the indexer code

Inside src/index.ts is an exported function called handleWebhookIndexer. It’s two arguments are of Request and a Response type, types imported from the @google-cloud/functions-framework package. This handler function is for our cloud function. It will receive webhook requests from Helius in the form of HTTPs POST requests. Inside these HTTP requests is a body which contains an array of transaction responses. A transaction response is an object which contains transaction metadata, blocktime, slot, and the transaction itself.

The transaction response object sent from the Helius webhook is identical to the web3.js library web3.TransactionResponse type except all properties typed as web3.PublicKey are actually typed as string (because JSON has no concept of web3.PublicKey). TransactionResponseJson type does some clever typing to transform the web3.PublicKey fields inside web3.TransactionResponse into string.

Handler function used as the entry point to the Google Cloud function.

Next, we iterate through the transaction response array. We store signature, slot, and blocktime in variables as we will use them later. We check if the transaction failed (txResponse.meta.err is not null), if there was an error we definitely don’t want to process a failed transaction so we skip it by continuing.

Setting up variables, filtering, and logging

Moving on, each Solana transaction can have many instructions in it. We want to iterate through each instruction in the transaction. If the instruction’s program id (the entry program mentioned by the instruction) doesn’t match the bonkboard program id — we do not process that instruction. In our case, the program id (the global const named BB_PROGRAM_ID) is the bonkboard’s program address.

programIdIndex is an index to the transaction accountKeys array.

Bonkboard is an anchor program. We can use its anchor IDL to decode raw instruction data. BB_CODER is of type BorshCoder from the @project-seurm/anchor package which takes an IDL (bonkboard’s IDL).

Decoding instruction so we can process the one we want.
A borsh coder object used to encode/decode accounts and instructions for the bonkboard anchor program

Anchor IDLs define the instructions, accounts, and structs for a particular anchor program. The following code snippet is an excerpt from the bonkboard anchor IDL which is located in src/IDL.ts. Instructions are how we communicate with Solana programs so we want to process them to know what happened in the transaction. The instruction we want to process in our indexer is named draw, it has several accounts, the accounts that are relevant for processing are payer and boardAccount (I’ll explain why later). The instruction’s argument pixels is of the vector<Pixel>. The anchor IDL struct Pixel is defined lower down in the IDL file.

{
name: 'draw',
accounts: [
{
name: 'payer',
isMut: false,
isSigner: true,
},
{
name: 'payerTokenAccount',
isMut: true,
isSigner: false,
},
{
name: 'boardAccount',
isMut: true,
isSigner: false,
},
{
name: 'feeAccount',
isMut: false,
isSigner: false,
},
{
name: 'feeDestination',
isMut: true,
isSigner: false,
},
{
name: 'boardDataAccount',
isMut: true,
isSigner: false,
},
{
name: 'tokenProgram',
isMut: false,
isSigner: false,
},
],
args: [
{
name: 'pixels',
type: {
vec: {
defined: 'Pixel',
},
},
},
],
},

The Pixel struct is defined as having coord and color fields. Which are also structs defined in the IDL. The coord struct is defined as having fields x and y which are both u16 types. And color struct is defined as having the fields r, g, and b — which are all of type u8 (0–255 integers).

{
name: 'Pixel',
type: {
kind: 'struct',
fields: [
{
name: 'coord',
type: {
defined: 'Coord',
},
},
{
name: 'color',
type: {
defined: 'Color',
},
},
],
},
},
{
name: 'Coord',
type: {
kind: 'struct',
fields: [
{
name: 'x',
type: 'u16',
},
{
name: 'y',
type: 'u16',
},
],
},
},
{
name: 'Color',
type: {
kind: 'struct',
fields: [
{
name: 'r',
type: 'u8',
},
{
name: 'g',
type: 'u8',
},
{
name: 'b',
type: 'u8',
},
],
},
},

Now that we understand the anchor IDL for bonkboard a little better (it’s okay if you didn’t quite get it all!!!) lets go back to the code.

We need to use the BorshCoder we talked about earlier to decode the instruction name and instruction data. If the decoding fails, we skip and log the failure. Next, we look to see if the decoded instruction’s name is draw, since thats the instruction we want process in the transaction.

getAccountKey, is a helper function I wrote which gets the public key of the account referenced in an anchor IDL instruction by the name given to the account. If we want the user who paid for the transaction we pass in payer string since that is the name of the account in the anchor IDL instruction for draw. We do the same with the boardAccount which is the account that stores the board’s state.

We want to store the most up-to-date state of the drawing board in firestore. A hacky way to do that is to keep fetching the account info for the board account until we get a version of the board which has a context slot that is greater or equal to the transaction’s slot. We keep looping and waiting 250ms until we get the latest context slot. We store the base58 encoding of the data instead the boardAccount for later (it’s going into firestore).

This next part is the fun stuff! We finally get to store our transaction data into firestore. First, we get a firestore instance using our firebase-admin application which is initialized in src/firebase.ts. Then we define firestore document references for the user, global stats, activity, and board. For the user firestore document we want to store user statistics, for the stats/mainnet-beta document we want to store global stats about bonkboard in it. For the activity document, we use the signature as the document id and store information about which pixels were drawn in that transaction. For the board/mainnet-beta document we are planning to store the board’s state inside the document.

Now that we have the decoded data and the firestore document references. It’s just a matter of updating the firestore document with the decoded data. We do this in a Promise.all so that all the firestore writes can be done all at once, asynchronously.

For the activities document, we store relevant transaction info and the pixel coordinates / colors into an array.

In the firestore webapp, this document looks like:

Next, for the user document — we increment the amount of pixels placed, the bonk burned, and the amount of bonk paid (variables we created earlier). firestore.FieldValue.increment is a field value for firestore which increments whatever value firestore currently has in its database.

What the user document looks like in the firestore webapp

Lastly, for the globals stats document — we increment the burned/paid bonk and pixels placed but these values will be global (all users). For the board document, we store the base58 encoded board state which we looped to find earlier.

What the global stats document looks like in the firestore webapp
What the global board state document looks like in the firestore webapp

Deploying the Cloud Function and Connecting Helius

Now that you understand the code (more or less) you’re ready to deploy the Google Cloud function!

$ cd ./functions/webhook-indexer
$ yarn run deploy

You should see a success message in your cli after it is deployed.

Next, go to the Google Cloud console and navigate to your project’s Cloud Functions. You should see the handleWebhookIndexer cloud function, click the name to navigate to the function details page.

Navigate to the TRIGGER tab of the function and copy the trigger URL. This is the URL that Helius will send HTTP POST payloads to.

Go to https://dev.helius.xyz, setup a free account (comes with one webhook), go to the Webhooks page, and click New Webhook.

To create the webhook we need. Choose the raw webhook type, insert the trigger URL from our Google Cloud function. The account address we need to listen to is our bonkboard program who’s address is bbggT3MZKdJ2cgHQpfSZJFvKHrvAm3NHSqxHq2zoe7A. After filling out the form, click Create Webhook.

Testing the indexer

Head to bonkboard.gg and draw a pixel on the board and submit the drawing.

Back on the functions details page, in the LOGS tab you should see the function execute, in the code we log the pixel position and color — so we should see that in the cloud function logs.

Now in the Google Cloud console navigate to Firestore. You should see several collections which are populated with documents.

The pixel you drew should be inside the activity collection. The document’s id is the signature transaction of the pixel. This signature is the same as the signature you plug into solana transaction explorers. Check out the documents in the other collections!

Wrap-up

Congratulations you’ve got a program indexer up and running! This is a very important piece of infrastructure for any Solana-based dApp. This badboy will process any transactions that include the bonkboard draw instruction! Note: this tutorial’s HTTPs cloud function doesn’t authenticate the requests it receives, meaning if anyone got your function’s URL, they can send malicious transaction payloads — for more about webhook security read here.

In the next tutorial we will show you how to synchronize the firestore data with the bonkboard frontend, stay tuned!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store