Decentralizing Blockchain Data with Indexers: Interview with SubQuery’s James Bayly

Blockchains are sometimes referred to as decentralized but very slow databases that sacrifice much of the performance of modern computing. Although it is a bit cheeky, it is true that an average bank’s database would be able to handle many times more transactions than Bitcoin (BTC-USD) or other leading blockchains.

But there is a reason why blockchains are “slow”. The benefits of decentralization are hard to overstate, and in any case, technology is advancing at a rapid pace to increase the capacity of blockchain systems.

As the data throughput of blockchain systems increases, we will increasingly need solutions to efficiently read and organize this data. This is where the ecosystem of blockchain indexers comes in: services that create an easy-to-query interface to access the data stored in blockchains, including historical transactions.

That’s why we sat down with James Bayly, COO of blockchain indexing project SubQuery, to talk about the space as it is right now and how Bayly sees it evolving over time. The interview focused on how data indexing is made tamper-proof and reliable, who uses it, and how the project plans to outperform the competition with the SubQuery Network.

Hi James, nice to have you here! Let’s start from a general overview. What is data indexing in a blockchain setting and why is it necessary? Why can’t we just get this data directly from the blockchain nodes?

JB: One of the critical components of every Web3 decentralized application (dApp) is the indexer. Simply put, an indexer is software that collects and organizes data stored on the blockchain network into a more efficient and accessible medium, allowing developers to query that data quickly and efficiently. They are important because they allow dApps to work quickly and support the load of millions of users.

The way blockchain data is stored on a network makes it difficult and time-consuming for developers to retrieve this data. A simple example is reading the last 10 transactions made by the current user – this is an extremely difficult request for blockchain nodes, as they have to scan each individual block in sequence to search for the data.

Indexers like SubQuery can quickly retrieve data from a blockchain network and store it in a format that is easy to work with, such as searchable and sortable fields that allow developers to quickly find the information they need.

What are the challenges in making indexed blockchain data available to users? One would imagine that ensuring that the data is correct and unmanipulated is one of the key requirements, right? What are some ways to decentralize the indexing ecosystem?

JB: Indeed, the indexer is often the last remaining centralized service critical to the performance of a dApp. A decentralized indexer can help ensure that the data is accurate, objective and accessible to everyone and [that] your dApp is more performant and reliable.

In SubQuery’s case, we solve that by facilitating a network where any participant can join and run decentralized indexing tasks for the network in a trustless but verifiable way. Since your decentralized data is not controlled by a single entity or group, you can verify the accuracy of the data in the indexing and ensure that there is no manipulation or bias.

What about other providers in the area? How does SubQuery compare to some competitors like The Graph, Covalent, Subsquid and others?

JB: Some indexers, such as Covalent and Unmarshal, are general-purpose indexers for standard data sets, such as transaction lists and blocks. The problem is that they don’t have any flexibility – you can’t add more data you need to make the dApp more feature-rich or intuitive.

Some other providers, like SubSquid, are centralized, meaning you’re dependent on a server run by their team, and you can’t easily verify the accuracy of the data in the indexer.

SubQuery is most similar to Graph, but provides some major improvements, including the ability to make external API calls, import external libraries, and protect against DoS attacks. Additionally, we have no plans to discontinue our simplified managed service.

Both SubQuery and The Graph are designed to index data quickly, but analysis shows that SubQuery is 1.85 times faster for common projects over The Graph. With faster sync times, developers can iterate faster and deliver features to market faster.

Who uses indexed data today? What about some interesting use cases for using indexed data in the future?

JB: Every Web3 application, website, business intelligence tool or extension needs indexed data today. This is such a critical aspect of application development that most developers need to leverage fast and efficient indexing to build a competitive advantage.

We have customers building wallets, running DeFi exchanges, monitoring blockchains for events, managing marketplaces for NFTs, and even running AI workloads across chain data. We also look beyond Web3 to the latest innovations in Big Data to bring their benefits and changes into our industry.

A potentially important use for data indexing is to serve dedicated decentralized data storage protocols such as Filecoin or IPFS. Do you see demand for such a combination now or in the future?

JB: Just as data indexing is a major component of a Web3 dApp, another is decentralized storage. For example, they can be used to host resources needed for the front-end interface of the dApp, allowing users to interact with the application in a completely decentralized way.

We work closely with some decentralized storage providers such as Crust and IPFS and also use both solutions to a large extent in our decentralized data infrastructure. Many of our customers want to use a decentralized data indexer to populate and store key data in decentralized data storage – and we help them do just that.

What is SubQuery Network? You recently announced your launch on Polygon (MATIC-USD). What is the long-term vision for it?

JB: The SubQuery Network indexes and serves data to the global community in an incentivized and verifiable way. We’re building the most open, high-performance, reliable, and scalable data service for dApp developers, and we’re launching on Polygon.

We chose Polygon for several reasons. Firstly, performance; Our plans for the SubQuery Network are huge, so we need a network that scales to serve billions of data API calls to millions of recipients every day. We wanted to find a hub with bridges and integrations to all other chains that we support, so that our customers can easily access the network.

Finally, the size and commitment of the community means that there are a number of developers in Polygon who will benefit from the growth of SubQuery in the future.

We begin with the Kepler Network, the pre-mainnet of the SubQuery Network, which enables users to gradually launch and test the functions of the SubQuery Network.

Kepler runs on the same smart contracts that our mainnet will; The main difference is that certain features are slowly activated and brought online in a sustainable way. When ready, Kepler will transform into the SubQuery Network. We expect this to happen mostly seamlessly, as the contracts won’t change much, and kSQT can be burned for SQT.

From this point on, the future of decentralized data indexing will be live.

Mediation