bft blockchain consensus consistency cryptocurrency cryptography db distributed-systems go

17 KiB

Raw Blame History

RFC 012: Event Indexing Revisited

Changelog

11-Feb-2022: Add terminological notes.
10-Feb-2022: Updated from review feedback.
07-Feb-2022: Initial draft (@creachadair)

Abstract

A Tendermint node allows ABCI events associated with block and transaction processing to be "indexed" into persistent storage. The original Tendermint implementation provided a fixed, built-in proprietary indexer for such events.

In response to user requests to customize indexing, ADR 065 introduced an "event sink" interface that allows developers (at least in theory) to plug in alternative index storage.

Although ADR-065 was a good first step toward customization, its implementation model does not satisfy all the user requirements. Moreover, this approach leaves some existing technical issues with indexing unsolved.

This RFC documents these concerns, and discusses some potential approaches to solving them. This RFC does not propose a specific technical decision. It is meant to unify and focus some of the disparate discussions of the topic.

Background

We begin with some important terminological context. The term "event" in Tendermint can be confusing, as the same word is used for multiple related but distinct concepts:

ABCI Events refer to the key-value metadata attached to blocks and transactions by the application. These values are represented by the ABCI Event protobuf message type.
Consensus Events refer to the data published by the Tendermint node to its pubsub bus in response to various consensus state transitions and other important activities, such as round updates, votes, transaction delivery, and block completion.

This confusion is compounded because some "consensus event" values also have "ABCI event" metadata attached to them. Notably, block and transaction items typically have ABCI metadata assigned by the application.

Indexers and RPC clients subscribed to the pubsub bus receive consensus events, but they identify which ones to care about using query expressions that match against the ABCI events associated with them.

In the discussion that follows, we will use the term event item to refer to a datum published to or received from the pubsub bus, and ABCI event or event metadata to refer to the key/value annotations.

Indexing in this context means recording the association between certain ABCI metadata and the blocks or transactions they're attached to. The ABCI metadata typically carry application-specific details like sender and recipient addresses, catgory tags, and so forth, that are not part of consensus but are used by UI tools to find and display transactions of interest.

The consensus node records the blocks and transactions as part of its block store, but does not persist the application metadata. Metadata persistence is the task of the indexer, which can be (optionally) enabled by the node operator.

History

The original indexer built in to Tendermint stored index data in an embedded tm-db database with a proprietary key layout. In ADR 065, we noted that this implementation has both performance and scaling problems under load. Moreover, the only practical way to query the index data is via the query filter language used for event subscription. Issue #1161 appears to be a motivational context for that ADR.

To mitigate both of these concerns, we introduced the EventSink interface, combining the original transaction and block indexer interfaces along with some service plumbing. Using this interface, a developer can plug in an indexer that uses a more efficient storage engine, and provides a more expressive query language. As a proof-of-concept, we built a PostgreSQL event sink that exports data to a PostgreSQL database.

Although this approach addressed some of the immediate concerns, there are several issues for custom indexing that have not been fully addressed. Here we will discuss them in more detail.

For further context, including links to user reports and related work, see also the Pluggable custom event indexing tracking issue issue.

Issue 1: Tight Coupling

The EventSink interface supports multiple implementations, but plugging in implementations still requires tight integration with the node. In particular:

Any custom indexer must either be written in Go and compiled in to the Tendermint binary, or the developer must write a Go shim to communicate with the implementation and build that into the Tendermint binary.
This means to support a custom indexer, it either has to be integrated into the Tendermint core repository, or every installation that uses that indexer must fetch or build a patched version of Tendermint.

The problem with integrating indexers into Tendermint Core is that every user of Tendermint Core takes a dependency on all supported indexers, including those they never use. Even if the unused code is disabled with build tags, users have to remember to do this or potentially be exposed to security issues that may arise in any of the custom indexers. This is a risk for Tendermint, which is a trust-critical component of all applications built on it.

The problem with not integrating indexers into Tendermint Core is that any developer who wants to use a particular indexer must now fetch or build a patched version of the core code that includes the custom indexer. Besides being inconvenient, this makes it harder for users to upgrade their node, since they need to either re-apply their patches directly or wait for an intermediary to do it for them.

Even for developers who have written their applications in Go and link with the consensus node directly (e.g., using the Cosmos SDK), these issues add a potentially significant complication to the build process.

Issue 2: Legacy Compatibility

The EventSink interface retains several limitations of the original proprietary indexer. These include:

The indexer has no control over which event items are reported. Only the exact block and transaction events that were reported to the original indexer are reported to a custom indexer.
The interface requires the implementation to define methods for the legacy search and query API. This requirement comes from the integation with the event subscription RPC API, but actually supporting these methods is not trivial.

At present, only the original KV indexer implements the query methods. Even the proof-of-concept PostgreSQL implementation simply reports errors for all calls to these methods.

Even for a plugin written in Go, implementing these methods "correctly" would require parsing and translating the custom query language over whatever storage platform the indexer uses.

For a plugin not written in Go, even beyond the cost of integration the developer would have to re-implement the entire query language.

Issue 3: Indexing Delays Consensus

Within the node, indexing hooks in to the same internal pubsub dispatcher that is used to export event items to the event subscription RPC API. In contrast with RPC subscribers, however, indexing is a "privileged" subscriber: If an RPC subscriber is "too slow", the node may terminate the subscription and disconnect the client. That means that RPC subscribers may lose (miss) event items. The indexer, however, is "unbuffered", and the publisher will never drop or disconnect from it. If the indexer is slow, the publisher will block until it returns, to ensure that no event items are lost.

In practice, this means that the performance of the indexer has a direct effect on the performance of the consensus node: If the indexer is slow or stalls, it will slow or halt the progress of consensus. Users have already reported this problem even with the built-in indexer (see, for example, #7247). Extending this concern to arbitrary user-defined custom indexers gives that risk a much larger surface area.

Discussion

It is not possible to simultaneously guarantee that publishing event items will not delay consensus, and also that all event items of interest are always completely indexed.

Therefore, our choice is between eliminating delay (and minimizing loss) or eliminating loss (and minimizing delay). Currently, we take the second approach, which has led to user complaints about consensus delays due to indexing and subscription overhead.

If we agree that consensus performance supersedes index completeness, our design choices are to constrain the likelihood and frequency of missing event items.
If we decide that consensus performance is more important than index completeness, our option is to minimize overhead on the event delivery path and document that indexer plugins constrain the rate of consensus.

Since we have user reports requesting both properties, we have to choose one or the other. Since the primary job of the consensus engine is to correctly, robustly, reliablly, and efficiently replicate application state across the network, I believe the correct choice is to favor consensus performance.

An important consideration for this decision is that a node does not index application metadata separately: If indexing is disabled, there is no built-in mechanism to go back and replay or reconstruct the data that an indexer would have stored. The node does store the blockchain itself (i.e., the blocks and their transactions), so potentially some use cases currently handled by the indexer could be handled by the node. For example, allowing clients to ask whether a given transaction ID has been committed to a block could in principle be done without an indexer, since it does not depend on application metadata.

Inevitably, a question will arise whether we could implement both strategies and toggle between them with a flag. That would be a worst-case scenario, requiring us to maintain the complexity of two very-different operational concerns. If our goal is that Tendermint should be as simple, efficient, and trustworthy as posible, there is not a strong case for making these options configurable: We should pick a side and commit to it.

Design Principles

Although there is no unique "best" solution to the issues described above, there are some specific principles that a solution should include:

A custom indexer should not require integration into Tendermint core. A developer or node operator can create, build, deploy, and use a custom indexer with a stock build of the Tendermint consensus node.
Custom indexers cannot stall consensus. An indexer that is slow or stalls cannot slow down or prevent core consensus from making progress.

The plugin interface must give node operators control over the tolerances for acceptable indexer performance, and the means to detect when indexers are falling outside those tolerances, but indexer failures should "fail safe" with respect to consensus (even if that means the indexer may miss some data, in sufficiently-extreme circumstances).
Custom indexers control which event items they index. A custom indexer is not limited to only the current transaction and block events, but can observe any event item published by the node.
Custom indexing is forward-compatible. Adding new event item types or metadata to the consensus node should not require existing custom indexers to be rebuilt or modified, unless they want to take advantage of the new data.
Indexers are responsible for answering queries. An indexer plugin is not required to support the legacy query filter language, nor to be compatible with the legacy RPC endpoints for accessing them. Any APIs for clients to query a custom index are the responsibility of the indexer, not the node.

Open Questions

Given the constraints outlined above, there are important design questions we must answer to guide any specific changes:

What is an acceptable probability that, given sufficiently extreme operational issues, an indexer might miss some number of events?

There are two parts to this question: One is what constitutes an extreme operational problem, the other is how likely we are to miss some number of events items.
- If the consensus is that no event item must ever be missed, no matter how bad the operational circumstances, then we must accept that indexing can slow or halt consensus arbitrarily. It is impossible to guarantee complete index coverage without potentially unbounded delays.
- Otherwise, how much data can we afford to lose and how often? For example, if we can ensure no event item will be lost unless the indexer halts for at least five minutes, is that acceptable? What probabilities and time ranges are reasonable for real production environments?
What level of operational overhead is acceptable to impose on node operators to support indexing?

Are node operators willing to configure and run custom indexers as sidecar type processes alongside a node? How much indexer setup above and beyond the work of setting up the underlying node in isolation is tractable in production networks?

The answer to this question also informs the question of whether we should keep an "in-process" indexing option, and to what extent that option needs to satisfy the suggested design principles.

Relatedly, to what extent do we need to be concerned about the cost of encoding and sending event items to an external process (e.g., as JSON blobs or protobuf wire messages)? Given that the node already encodes event items as JSON for subscription purposes, the overhead would be negligible for the node itself, but the indexer would have to decode to process the results.
What (if any) query APIs does the consensus node need to export, independent of the indexer implementation?

One typical example is whether the node should be able to answer queries like "is this transaction ID in a block?" Currently, a node cannot answer this query unless it runs the built-in KV indexer. Does the node need to continue to support that query even for nodes that disable the KV indexer, or which use a custom indexer?

Informal Design Intent

The design principles described above implicate several components of the Tendermint node, beyond just the indexer. In the context of ADR 075, we are re-working the RPC event subscription API to improve some of the UX issues discussed above for RPC clients. It is our expectation that a solution for pluggable custom indexing will take advantage of some of the same work.

On that basis, the design approach I am considering for custom indexing looks something like this (subject to refinement):

A custom indexer runs as a separate process from the node.
The indexer subscribes to event items via the ADR 075 events API.

This means indexers would receive event payloads as JSON rather than protobuf, but since we already have to support JSON encoding for the RPC interface anyway, that should not increase complexity for the node.
The existing PostgreSQL indexer gets reworked to have this form, and no longer built as part of the Tendermint core binary.

We can retain the code in the core repository as a proof-of-concept, or perhaps create a separate repository with contributed indexers and move it there.
(Possibly) Deprecate and remove the legacy KV indexer, or disable it by default. If we decide to remove it, we can also remove the legacy RPC endpoints for querying the KV indexer.

If we plan to do this, we should also investigate providing a way for clients to query whether a given transaction ID has landed in a block. That serves a common need, and currently only works if the KV indexer is enabled, but could be addressed more simply using the other data a node already has stored, without having to answer more general queries.

17 KiB Raw Blame History