diff --git a/docs/app-dev/app-development.md b/docs/app-dev/app-development.md index 6c2ae9aba..3aaebb230 100644 --- a/docs/app-dev/app-development.md +++ b/docs/app-dev/app-development.md @@ -1,5 +1,10 @@ # Application Development Guide +## XXX + +This page is undergoing deprecation. All content is being moved to the new [home +of the ABCI specification](../spec/abci/README.md). + ## ABCI Design The purpose of ABCI is to provide a clean interface between state diff --git a/docs/spec/abci/README.md b/docs/spec/abci/README.md new file mode 100644 index 000000000..c0956db6f --- /dev/null +++ b/docs/spec/abci/README.md @@ -0,0 +1,19 @@ +# ABCI + +ABCI is the interface between Tendermint (a state-machine replication engine) +and an application (the actual state machine). It consists of a set of +*methods*, where each method has a corresponding `Request` and `Response` +message type. Tendermint calls the ABCI methods on the ABCI application by sending the `Request*` +messages and receiving the `Response*` messages in return. + +All message types are defined in a [protobuf file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto). +This allows Tendermint to run applications written in any programming language. + +This specification is split as follows: + +- [Methods and Types](abci.md) - complete details on all ABCI methods and + message types +- [Applications](apps.md) - how to manage ABCI application state and other + details about building ABCI applications +- [Client and Server](client-server.md) - for those looking to implement their + own ABCI application servers diff --git a/docs/spec/abci/abci.md b/docs/spec/abci/abci.md index d2638f6c8..a1ca42f08 100644 --- a/docs/spec/abci/abci.md +++ b/docs/spec/abci/abci.md @@ -1,13 +1,7 @@ -# Application Blockchain Interface (ABCI) +# ABCI Methods and Types ## Overview -ABCI is the interface between Tendermint (a state-machine replication engine) -and an application (the actual state machine). It consists of a set of -*methods*, where each method has a corresponding `Request` and `Response` type. -Tendermint calls the methods on the ABCI application by sending the `Request*` -messages and receiving the `Response*` messages in return. - The ABCI message types are defined in a [protobuf file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto). @@ -26,7 +20,6 @@ The `Info Connection` is for initialization and for queries from the user. Additionally, there is a `Flush` method that is called on every connection, and an `Echo` method that is just for debugging. - ## Errors Some methods (`Echo, Info, InitChain, BeginBlock, EndBlock, Commit`), diff --git a/docs/spec/abci/apps.md b/docs/spec/abci/apps.md new file mode 100644 index 000000000..fcba47355 --- /dev/null +++ b/docs/spec/abci/apps.md @@ -0,0 +1,207 @@ +# ABCI Applications + +Please ensure you've first read the spec for [ABCI Methods and Types](abci.md) + +Here we cover the following components of ABCI applications: + +- [State](#State) - the interplay between ABCI connections and application state + and the differences between `CheckTx` and `DeliverTx`. +- [Validator Set Updates](#Validator-Set-Updates) - how validator sets are + changed during `InitChain` and `EndBlock` +- [Query](#Query) - standards for using the `Query` method +- [Crash Recovery](#Crash-Recovery) - handshake protocol to synchronize + Tendermint and the application on startup. + +## State + +Since Tendermint maintains multiple concurrent ABCI connections, it is typical +for an application to maintain a distinct state for each, and for the states to +be sycnronized during `Commit`. + +### Commit + +Before `Commit` is called, Tendermint locks and flushes the mempool so that no new messages will +be received on the mempool connection. This provides an opportunity to safely update all three +states to the latest committed state at once. + +When `Commit` completes, it unlocks the mempool. + +Note that it is not possible to send transactions to Tendermint during `Commit` - if your app +tries to send a `/broadcast_tx` to Tendermint during Commit, it will deadlock. + +### Consensus Connection + +The Consensus Connection should maintain a `DeliverTxState` - +the working state for block execution. It should be updated by the calls to +`BeginBlock`, `DeliverTx`, and `EndBlock` during block execution and committed to +disk as the "latest committed state" during `Commit`. + +Updates made to the DeliverTxState by each method call must be readable by each subsequent method - +ie. the updates are linearizeable. + +### Mempool Connection + +The Mempool Connection should maintain a `CheckTxState` - +to process pending transactions in the mempool that have +not yet been committed. It should be initialized to the latest committed state +at the end of every `Commit`. Note it may be updated concurrently with the +DeliverTxState. + +Before calling `Commit`, Tendermint will lock and flush the mempool, +ensuring that all existing CheckTx are responded to and no new ones can +begin. + +After `Commit`, CheckTx is run again on all transactions that remain in the +node's local mempool after filtering those included in the block. To prevent the +mempool from rechecking all transactions every time a block is committed, set +the configuration option `mempool.recheck=false`. + +Finally, the mempool will unlock and new transactions can be processed through CheckTx again. + +Note that CheckTx doesn't have to check everything that affects transaction validity; the +expensive things can be skipped. In fact, CheckTx doesn't have to check +anything; it might say that any transaction is a valid transaction. +Unlike DeliverTx, CheckTx is just there as +a sort of weak filter to keep invalid transactions out of the blockchain. It's +weak, because a Byzantine node doesn't care about CheckTx; it can propose a +block full of invalid transactions if it wants. + +### Info Connection + +The Mempool Connection should maintain a `QueryState` for answering queries from the user, +and for initialization when Tendermint first starts up. +It should always contain the latest committed state associated with the +latest commited block. + +QueryState should be set to the latest `DeliverTxState` at the end of every `Commit`, +ie. after the full block has been processed and the state committed to disk. +Otherwise it should never be modified. + +## Validator Updates + +### EndBlock + +Updates to the Tendermint validator set can be made by returning +`ValidatorUpdate` objects in the `ResponseEndBlock`: + +``` +message ValidatorUpdate { + PubKey pub_key + int64 power +} + +message PubKey { + string type + bytes data +} +``` + +The `pub_key` currently supports only one type: + +- `type = "ed25519" and`data = ` + +The `power` is the new voting power for the validator, with the +following rules: + +- power must be non-negative +- if power is 0, the validator must already exist, and will be removed from the + validator set +- if power is non-0: + - if the validator does not already exist, it will be added to the validator + set with the given power + - if the validator does already exist, its power will be adjusted to the given power + +### InitChain + +ResponseInitChain has the option to return a list of validators. +If the list is not empty, Tendermint will adopt it for the validator set. +This way the application can determine the initial validator set for the +blockchain. + +ResponseInitChain also includes ConsensusParams, but these are presently +ignored. + +## Query + +Query is a generic message type with lots of flexibility to enable diverse sets +of queries from applications. Tendermint has no requirements from the Query +message for normal operation - that is, the ABCI app developer need not implement Query functionality if they do not wish too. +That said, Tendermint makes a number of queries to support some optional +features. These are: + +### Peer Filtering + +When Tendermint connects to a peer, it sends two queries to the ABCI application +using the following paths, with no additional data: + +- `/p2p/filter/addr/`, where `` denote the IP address and + the port of the connection +- `p2p/filter/id/`, where `` is the peer node ID (ie. the + pubkey.Address() for the peer's PubKey) + +If either of these queries return a non-zero ABCI code, Tendermint will refuse +to connect to the peer. + + +## Crash Recovery + +On startup, Tendermint calls the `Info` method on the Info Connection to get the latest +committed state of the app. The app MUST return information consistent with the +last block it succesfully completed Commit for. + +If the app succesfully committed block H but not H+1, then `last_block_height = H` and `last_block_app_hash = `. If the app +failed during the Commit of block H, then `last_block_height = H-1` and +`last_block_app_hash = `. + +We now distinguish three heights, and describe how Tendermint syncs itself with +the app. + +``` +storeBlockHeight = height of the last block Tendermint saw a commit for +stateBlockHeight = height of the last block for which Tendermint completed all + block processing and saved all ABCI results to disk +appBlockHeight = height of the last block for which ABCI app succesfully + completely Commit +``` + +Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight` +Note also we never call Commit on an ABCI app twice for the same height. + +The procedure is as follows. + +First, some simeple start conditions: + +If `appBlockHeight == 0`, then call InitChain. + +If `storeBlockHeight == 0`, we're done. + +Now, some sanity checks: + +If `storeBlockHeight < appBlockHeight`, error +If `storeBlockHeight < stateBlockHeight`, panic +If `storeBlockHeight > stateBlockHeight+1`, panic + +Now, the meat: + +If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`, +replay all blocks in full from `appBlockHeight` to `storeBlockHeight`. +This happens if we completed processing the block, but the app forgot its height. + +If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done +This happens if we crashed at an opportune spot. + +If `storeBlockHeight == stateBlockHeight+1` +This happens if we started processing the block but didn't finish. + + If `appBlockHeight < stateBlockHeight` + replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`, + and replay the block at `storeBlockHeight` using the WAL. + This happens if the app forgot the last block it committed. + + If `appBlockHeight == stateBlockHeight`, + replay the last block (storeBlockHeight) in full. + This happens if we crashed before the app finished Commit + + If appBlockHeight == storeBlockHeight { + update the state using the saved ABCI responses but dont run the block against the real app. + This happens if we crashed after the app finished Commit but before Tendermint saved the state. diff --git a/docs/spec/abci/client-server.md b/docs/spec/abci/client-server.md new file mode 100644 index 000000000..1923b5e08 --- /dev/null +++ b/docs/spec/abci/client-server.md @@ -0,0 +1,104 @@ +# ABCI Client and Server + +This section is for those looking to implement their own ABCI Server, perhaps in +a new programming language. + +You are expected to have read [ABCI Methods and Types](abci.md) and [ABCI +Applications](apps.md). + +See additional details in the [ABCI +readme](https://github.com/tendermint/tendermint/blob/develop/abci/README.md)(TODO: deduplicate +those details). + +## Message Protocol + +The message protocol consists of pairs of requests and responses defined in the +[protobuf file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto). + +Some messages have no fields, while others may include byte-arrays, strings, integers, +or custom protobuf types. + +For more details on protobuf, see the [documentation](https://developers.google.com/protocol-buffers/docs/overview). + +For each request, a server should respond with the corresponding +response, where the order of requests is preserved in the order of +responses. + +## Server + +To use ABCI in your programming language of choice, there must be a ABCI +server in that language. Tendermint supports two kinds of implementation +of the server: + +- Asynchronous, raw socket server (Tendermint Socket Protocol, also + known as TSP or Teaspoon) +- GRPC + +Both can be tested using the `abci-cli` by setting the `--abci` flag +appropriately (ie. to `socket` or `grpc`). + +See examples, in various stages of maintenance, in +[Go](https://github.com/tendermint/tendermint/tree/develop/abci/server), +[JavaScript](https://github.com/tendermint/js-abci), +[Python](https://github.com/tendermint/tendermint/tree/develop/abci/example/python3/abci), +[C++](https://github.com/mdyring/cpp-tmsp), and +[Java](https://github.com/jTendermint/jabci). + +### GRPC + +If GRPC is available in your language, this is the easiest approach, +though it will have significant performance overhead. + +To get started with GRPC, copy in the [protobuf +file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto) +and compile it using the GRPC plugin for your language. For instance, +for golang, the command is `protoc --go_out=plugins=grpc:. types.proto`. +See the [grpc documentation for more details](http://www.grpc.io/docs/). +`protoc` will autogenerate all the necessary code for ABCI client and +server in your language, including whatever interface your application +must satisfy to be used by the ABCI server for handling requests. + +### TSP + +If GRPC is not available in your language, or you require higher +performance, or otherwise enjoy programming, you may implement your own +ABCI server using the Tendermint Socket Protocol, known affectionately +as Teaspoon. The first step is still to auto-generate the relevant data +types and codec in your language using `protoc`. Messages coming over +the socket are proto3 encoded, but additionally length-prefixed to +facilitate use as a streaming protocol. proto3 doesn't have an +official length-prefix standard, so we use our own. The first byte in +the prefix represents the length of the Big Endian encoded length. The +remaining bytes in the prefix are the Big Endian encoded length. + +For example, if the proto3 encoded ABCI message is 0xDEADBEEF (4 +bytes), the length-prefixed message is 0x0104DEADBEEF. If the proto3 +encoded ABCI message is 65535 bytes long, the length-prefixed message +would be like 0x02FFFF.... + +Note this prefixing does not apply for grpc. + +An ABCI server must also be able to support multiple connections, as +Tendermint uses three connections. + + +### Async vs Sync + +The main ABCI server (ie. non-GRPC) provides ordered asynchronous messages. +This is useful for DeliverTx and CheckTx, since it allows Tendermint to forward +transactions to the app before it's finished processing previous ones. + +Thus, DeliverTx and CheckTx messages are sent asycnhronously, while all other +messages are sent synchronously. + +## Client + +There are currently two use-cases for an ABCI client. One is a testing +tool, as in the `abci-cli`, which allows ABCI requests to be sent via +command line. The other is a consensus engine, such as Tendermint Core, +which makes requests to the application every time a new transaction is +received or a block is committed. + +It is unlikely that you will need to implement a client. For details of +our client, see +[here](https://github.com/tendermint/tendermint/tree/develop/abci/client). diff --git a/docs/spec/software/abci.md b/docs/spec/software/abci.md index aacdb43df..44d48f170 100644 --- a/docs/spec/software/abci.md +++ b/docs/spec/software/abci.md @@ -1,185 +1,3 @@ # Application Blockchain Interface (ABCI) -ABCI is the interface between Tendermint (a state-machine replication engine) -and an application (the actual state machine). - -The ABCI message types are defined in a [protobuf -file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto). - -For full details on the ABCI message types and protocol, see the [ABCI -specification](https://github.com/tendermint/tendermint/blob/develop/docs/app-dev/abci-spec.md). -Be sure to read the specification if you're trying to build an ABCI app! - -For additional details on server implementation, see the [ABCI -readme](https://github.com/tendermint/tendermint/blob/develop/abci/README.md). - -Here we provide some more details around the use of ABCI by Tendermint and -clarify common "gotchas". - -## ABCI connections - -Tendermint opens 3 ABCI connections to the app: one for Consensus, one for -Mempool, one for Queries. - -## Async vs Sync - -The main ABCI server (ie. non-GRPC) provides ordered asynchronous messages. -This is useful for DeliverTx and CheckTx, since it allows Tendermint to forward -transactions to the app before it's finished processing previous ones. - -Thus, DeliverTx and CheckTx messages are sent asycnhronously, while all other -messages are sent synchronously. - -## CheckTx and Commit - -It is typical to hold three distinct states in an ABCI app: CheckTxState, DeliverTxState, -QueryState. The QueryState contains the latest committed state for a block. -The CheckTxState and DeliverTxState may be updated concurrently with one another. -Before Commit is called, Tendermint locks and flushes the mempool so that no new changes will happen -to CheckTxState. When Commit completes, it unlocks the mempool. - -Thus, during Commit, it is safe to reset the QueryState and the CheckTxState to the latest DeliverTxState -(ie. the new state from executing all the txs in the block). - -Note, however, that it is not possible to send transactions to Tendermint during Commit - if your app -tries to send a `/broadcast_tx` to Tendermint during Commit, it will deadlock. - -## EndBlock Validator Updates - -Updates to the Tendermint validator set can be made by returning `Validator` -objects in the `ResponseBeginBlock`: - -``` -message Validator { - PubKey pub_key - int64 power -} - -message PubKey { - string type - bytes data -} -``` - -The `pub_key` currently supports two types: - -- `type = "ed25519" and`data = ` -- `type = "secp256k1" and `data = <33-byte OpenSSL compressed public key>` - -If the address is provided, it must match the address of the pubkey, as -specified [here](/docs/spec/blockchain/encoding.md#Addresses) - -(Note: In the v0.19 series, the `pub_key` is the [Amino encoded public -key](/docs/spec/blockchain/encoding.md#public-key-cryptography). -For Ed25519 pubkeys, the Amino prefix is always "1624DE6220". For example, the 32-byte Ed25519 pubkey -`76852933A4686A721442E931A8415F62F5F1AEDF4910F1F252FB393F74C40C85` would be -Amino encoded as -`1624DE622076852933A4686A721442E931A8415F62F5F1AEDF4910F1F252FB393F74C40C85`) - -(Note: In old versions of Tendermint (pre-v0.19.0), the pubkey is just prefixed with a -single type byte, so for ED25519 we'd have `pub_key = 0x1 | pub`) - -The `power` is the new voting power for the validator, with the -following rules: - -- power must be non-negative -- if power is 0, the validator must already exist, and will be removed from the - validator set -- if power is non-0: - - if the validator does not already exist, it will be added to the validator - set with the given power - - if the validator does already exist, its power will be adjusted to the given power - -## InitChain Validator Updates - -ResponseInitChain has the option to return a list of validators. -If the list is not empty, Tendermint will adopt it for the validator set. -This way the application can determine the initial validator set for the -blockchain. - -ResponseInitChain also includes ConsensusParams, but these are presently -ignored. - -## Query - -Query is a generic message type with lots of flexibility to enable diverse sets -of queries from applications. Tendermint has no requirements from the Query -message for normal operation - that is, the ABCI app developer need not implement Query functionality if they do not wish too. -That said, Tendermint makes a number of queries to support some optional -features. These are: - -### Peer Filtering - -When Tendermint connects to a peer, it sends two queries to the ABCI application -using the following paths, with no additional data: - -- `/p2p/filter/addr/`, where `` denote the IP address and - the port of the connection -- `p2p/filter/id/`, where `` is the peer node ID (ie. the - pubkey.Address() for the peer's PubKey) - -If either of these queries return a non-zero ABCI code, Tendermint will refuse -to connect to the peer. - -## Info and the Handshake/Replay - -On startup, Tendermint calls Info on the Query connection to get the latest -committed state of the app. The app MUST return information consistent with the -last block it succesfully completed Commit for. - -If the app succesfully committed block H but not H+1, then `last_block_height = H` and `last_block_app_hash = `. If the app -failed during the Commit of block H, then `last_block_height = H-1` and -`last_block_app_hash = `. - -We now distinguish three heights, and describe how Tendermint syncs itself with -the app. - -``` -storeBlockHeight = height of the last block Tendermint saw a commit for -stateBlockHeight = height of the last block for which Tendermint completed all - block processing and saved all ABCI results to disk -appBlockHeight = height of the last block for which ABCI app succesfully - completely Commit -``` - -Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight` -Note also we never call Commit on an ABCI app twice for the same height. - -The procedure is as follows. - -First, some simeple start conditions: - -If `appBlockHeight == 0`, then call InitChain. - -If `storeBlockHeight == 0`, we're done. - -Now, some sanity checks: - -If `storeBlockHeight < appBlockHeight`, error -If `storeBlockHeight < stateBlockHeight`, panic -If `storeBlockHeight > stateBlockHeight+1`, panic - -Now, the meat: - -If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`, -replay all blocks in full from `appBlockHeight` to `storeBlockHeight`. -This happens if we completed processing the block, but the app forgot its height. - -If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done -This happens if we crashed at an opportune spot. - -If `storeBlockHeight == stateBlockHeight+1` -This happens if we started processing the block but didn't finish. - - If `appBlockHeight < stateBlockHeight` - replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`, - and replay the block at `storeBlockHeight` using the WAL. - This happens if the app forgot the last block it committed. - - If `appBlockHeight == stateBlockHeight`, - replay the last block (storeBlockHeight) in full. - This happens if we crashed before the app finished Commit - - If appBlockHeight == storeBlockHeight { - update the state using the saved ABCI responses but dont run the block against the real app. - This happens if we crashed after the app finished Commit but before Tendermint saved the state. +This page has [moved](../spec/abci/apps.md).