* move spec/software/abci.md to spec/abci/apps.md and improve it * move some of app-dev/app-development.md to spec/abci/client-server.mdpull/2343/head
@ -0,0 +1,19 @@ | |||||
# ABCI | |||||
ABCI is the interface between Tendermint (a state-machine replication engine) | |||||
and an application (the actual state machine). It consists of a set of | |||||
*methods*, where each method has a corresponding `Request` and `Response` | |||||
message type. Tendermint calls the ABCI methods on the ABCI application by sending the `Request*` | |||||
messages and receiving the `Response*` messages in return. | |||||
All message types are defined in a [protobuf file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto). | |||||
This allows Tendermint to run applications written in any programming language. | |||||
This specification is split as follows: | |||||
- [Methods and Types](abci.md) - complete details on all ABCI methods and | |||||
message types | |||||
- [Applications](apps.md) - how to manage ABCI application state and other | |||||
details about building ABCI applications | |||||
- [Client and Server](client-server.md) - for those looking to implement their | |||||
own ABCI application servers |
@ -0,0 +1,207 @@ | |||||
# ABCI Applications | |||||
Please ensure you've first read the spec for [ABCI Methods and Types](abci.md) | |||||
Here we cover the following components of ABCI applications: | |||||
- [State](#State) - the interplay between ABCI connections and application state | |||||
and the differences between `CheckTx` and `DeliverTx`. | |||||
- [Validator Set Updates](#Validator-Set-Updates) - how validator sets are | |||||
changed during `InitChain` and `EndBlock` | |||||
- [Query](#Query) - standards for using the `Query` method | |||||
- [Crash Recovery](#Crash-Recovery) - handshake protocol to synchronize | |||||
Tendermint and the application on startup. | |||||
## State | |||||
Since Tendermint maintains multiple concurrent ABCI connections, it is typical | |||||
for an application to maintain a distinct state for each, and for the states to | |||||
be sycnronized during `Commit`. | |||||
### Commit | |||||
Before `Commit` is called, Tendermint locks and flushes the mempool so that no new messages will | |||||
be received on the mempool connection. This provides an opportunity to safely update all three | |||||
states to the latest committed state at once. | |||||
When `Commit` completes, it unlocks the mempool. | |||||
Note that it is not possible to send transactions to Tendermint during `Commit` - if your app | |||||
tries to send a `/broadcast_tx` to Tendermint during Commit, it will deadlock. | |||||
### Consensus Connection | |||||
The Consensus Connection should maintain a `DeliverTxState` - | |||||
the working state for block execution. It should be updated by the calls to | |||||
`BeginBlock`, `DeliverTx`, and `EndBlock` during block execution and committed to | |||||
disk as the "latest committed state" during `Commit`. | |||||
Updates made to the DeliverTxState by each method call must be readable by each subsequent method - | |||||
ie. the updates are linearizeable. | |||||
### Mempool Connection | |||||
The Mempool Connection should maintain a `CheckTxState` - | |||||
to process pending transactions in the mempool that have | |||||
not yet been committed. It should be initialized to the latest committed state | |||||
at the end of every `Commit`. Note it may be updated concurrently with the | |||||
DeliverTxState. | |||||
Before calling `Commit`, Tendermint will lock and flush the mempool, | |||||
ensuring that all existing CheckTx are responded to and no new ones can | |||||
begin. | |||||
After `Commit`, CheckTx is run again on all transactions that remain in the | |||||
node's local mempool after filtering those included in the block. To prevent the | |||||
mempool from rechecking all transactions every time a block is committed, set | |||||
the configuration option `mempool.recheck=false`. | |||||
Finally, the mempool will unlock and new transactions can be processed through CheckTx again. | |||||
Note that CheckTx doesn't have to check everything that affects transaction validity; the | |||||
expensive things can be skipped. In fact, CheckTx doesn't have to check | |||||
anything; it might say that any transaction is a valid transaction. | |||||
Unlike DeliverTx, CheckTx is just there as | |||||
a sort of weak filter to keep invalid transactions out of the blockchain. It's | |||||
weak, because a Byzantine node doesn't care about CheckTx; it can propose a | |||||
block full of invalid transactions if it wants. | |||||
### Info Connection | |||||
The Mempool Connection should maintain a `QueryState` for answering queries from the user, | |||||
and for initialization when Tendermint first starts up. | |||||
It should always contain the latest committed state associated with the | |||||
latest commited block. | |||||
QueryState should be set to the latest `DeliverTxState` at the end of every `Commit`, | |||||
ie. after the full block has been processed and the state committed to disk. | |||||
Otherwise it should never be modified. | |||||
## Validator Updates | |||||
### EndBlock | |||||
Updates to the Tendermint validator set can be made by returning | |||||
`ValidatorUpdate` objects in the `ResponseEndBlock`: | |||||
``` | |||||
message ValidatorUpdate { | |||||
PubKey pub_key | |||||
int64 power | |||||
} | |||||
message PubKey { | |||||
string type | |||||
bytes data | |||||
} | |||||
``` | |||||
The `pub_key` currently supports only one type: | |||||
- `type = "ed25519" and`data = <raw 32-byte public key>` | |||||
The `power` is the new voting power for the validator, with the | |||||
following rules: | |||||
- power must be non-negative | |||||
- if power is 0, the validator must already exist, and will be removed from the | |||||
validator set | |||||
- if power is non-0: | |||||
- if the validator does not already exist, it will be added to the validator | |||||
set with the given power | |||||
- if the validator does already exist, its power will be adjusted to the given power | |||||
### InitChain | |||||
ResponseInitChain has the option to return a list of validators. | |||||
If the list is not empty, Tendermint will adopt it for the validator set. | |||||
This way the application can determine the initial validator set for the | |||||
blockchain. | |||||
ResponseInitChain also includes ConsensusParams, but these are presently | |||||
ignored. | |||||
## Query | |||||
Query is a generic message type with lots of flexibility to enable diverse sets | |||||
of queries from applications. Tendermint has no requirements from the Query | |||||
message for normal operation - that is, the ABCI app developer need not implement Query functionality if they do not wish too. | |||||
That said, Tendermint makes a number of queries to support some optional | |||||
features. These are: | |||||
### Peer Filtering | |||||
When Tendermint connects to a peer, it sends two queries to the ABCI application | |||||
using the following paths, with no additional data: | |||||
- `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and | |||||
the port of the connection | |||||
- `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the | |||||
pubkey.Address() for the peer's PubKey) | |||||
If either of these queries return a non-zero ABCI code, Tendermint will refuse | |||||
to connect to the peer. | |||||
## Crash Recovery | |||||
On startup, Tendermint calls the `Info` method on the Info Connection to get the latest | |||||
committed state of the app. The app MUST return information consistent with the | |||||
last block it succesfully completed Commit for. | |||||
If the app succesfully committed block H but not H+1, then `last_block_height = H` and `last_block_app_hash = <hash returned by Commit for block H>`. If the app | |||||
failed during the Commit of block H, then `last_block_height = H-1` and | |||||
`last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>`. | |||||
We now distinguish three heights, and describe how Tendermint syncs itself with | |||||
the app. | |||||
``` | |||||
storeBlockHeight = height of the last block Tendermint saw a commit for | |||||
stateBlockHeight = height of the last block for which Tendermint completed all | |||||
block processing and saved all ABCI results to disk | |||||
appBlockHeight = height of the last block for which ABCI app succesfully | |||||
completely Commit | |||||
``` | |||||
Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight` | |||||
Note also we never call Commit on an ABCI app twice for the same height. | |||||
The procedure is as follows. | |||||
First, some simeple start conditions: | |||||
If `appBlockHeight == 0`, then call InitChain. | |||||
If `storeBlockHeight == 0`, we're done. | |||||
Now, some sanity checks: | |||||
If `storeBlockHeight < appBlockHeight`, error | |||||
If `storeBlockHeight < stateBlockHeight`, panic | |||||
If `storeBlockHeight > stateBlockHeight+1`, panic | |||||
Now, the meat: | |||||
If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`, | |||||
replay all blocks in full from `appBlockHeight` to `storeBlockHeight`. | |||||
This happens if we completed processing the block, but the app forgot its height. | |||||
If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done | |||||
This happens if we crashed at an opportune spot. | |||||
If `storeBlockHeight == stateBlockHeight+1` | |||||
This happens if we started processing the block but didn't finish. | |||||
If `appBlockHeight < stateBlockHeight` | |||||
replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`, | |||||
and replay the block at `storeBlockHeight` using the WAL. | |||||
This happens if the app forgot the last block it committed. | |||||
If `appBlockHeight == stateBlockHeight`, | |||||
replay the last block (storeBlockHeight) in full. | |||||
This happens if we crashed before the app finished Commit | |||||
If appBlockHeight == storeBlockHeight { | |||||
update the state using the saved ABCI responses but dont run the block against the real app. | |||||
This happens if we crashed after the app finished Commit but before Tendermint saved the state. |
@ -0,0 +1,104 @@ | |||||
# ABCI Client and Server | |||||
This section is for those looking to implement their own ABCI Server, perhaps in | |||||
a new programming language. | |||||
You are expected to have read [ABCI Methods and Types](abci.md) and [ABCI | |||||
Applications](apps.md). | |||||
See additional details in the [ABCI | |||||
readme](https://github.com/tendermint/tendermint/blob/develop/abci/README.md)(TODO: deduplicate | |||||
those details). | |||||
## Message Protocol | |||||
The message protocol consists of pairs of requests and responses defined in the | |||||
[protobuf file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto). | |||||
Some messages have no fields, while others may include byte-arrays, strings, integers, | |||||
or custom protobuf types. | |||||
For more details on protobuf, see the [documentation](https://developers.google.com/protocol-buffers/docs/overview). | |||||
For each request, a server should respond with the corresponding | |||||
response, where the order of requests is preserved in the order of | |||||
responses. | |||||
## Server | |||||
To use ABCI in your programming language of choice, there must be a ABCI | |||||
server in that language. Tendermint supports two kinds of implementation | |||||
of the server: | |||||
- Asynchronous, raw socket server (Tendermint Socket Protocol, also | |||||
known as TSP or Teaspoon) | |||||
- GRPC | |||||
Both can be tested using the `abci-cli` by setting the `--abci` flag | |||||
appropriately (ie. to `socket` or `grpc`). | |||||
See examples, in various stages of maintenance, in | |||||
[Go](https://github.com/tendermint/tendermint/tree/develop/abci/server), | |||||
[JavaScript](https://github.com/tendermint/js-abci), | |||||
[Python](https://github.com/tendermint/tendermint/tree/develop/abci/example/python3/abci), | |||||
[C++](https://github.com/mdyring/cpp-tmsp), and | |||||
[Java](https://github.com/jTendermint/jabci). | |||||
### GRPC | |||||
If GRPC is available in your language, this is the easiest approach, | |||||
though it will have significant performance overhead. | |||||
To get started with GRPC, copy in the [protobuf | |||||
file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto) | |||||
and compile it using the GRPC plugin for your language. For instance, | |||||
for golang, the command is `protoc --go_out=plugins=grpc:. types.proto`. | |||||
See the [grpc documentation for more details](http://www.grpc.io/docs/). | |||||
`protoc` will autogenerate all the necessary code for ABCI client and | |||||
server in your language, including whatever interface your application | |||||
must satisfy to be used by the ABCI server for handling requests. | |||||
### TSP | |||||
If GRPC is not available in your language, or you require higher | |||||
performance, or otherwise enjoy programming, you may implement your own | |||||
ABCI server using the Tendermint Socket Protocol, known affectionately | |||||
as Teaspoon. The first step is still to auto-generate the relevant data | |||||
types and codec in your language using `protoc`. Messages coming over | |||||
the socket are proto3 encoded, but additionally length-prefixed to | |||||
facilitate use as a streaming protocol. proto3 doesn't have an | |||||
official length-prefix standard, so we use our own. The first byte in | |||||
the prefix represents the length of the Big Endian encoded length. The | |||||
remaining bytes in the prefix are the Big Endian encoded length. | |||||
For example, if the proto3 encoded ABCI message is 0xDEADBEEF (4 | |||||
bytes), the length-prefixed message is 0x0104DEADBEEF. If the proto3 | |||||
encoded ABCI message is 65535 bytes long, the length-prefixed message | |||||
would be like 0x02FFFF.... | |||||
Note this prefixing does not apply for grpc. | |||||
An ABCI server must also be able to support multiple connections, as | |||||
Tendermint uses three connections. | |||||
### Async vs Sync | |||||
The main ABCI server (ie. non-GRPC) provides ordered asynchronous messages. | |||||
This is useful for DeliverTx and CheckTx, since it allows Tendermint to forward | |||||
transactions to the app before it's finished processing previous ones. | |||||
Thus, DeliverTx and CheckTx messages are sent asycnhronously, while all other | |||||
messages are sent synchronously. | |||||
## Client | |||||
There are currently two use-cases for an ABCI client. One is a testing | |||||
tool, as in the `abci-cli`, which allows ABCI requests to be sent via | |||||
command line. The other is a consensus engine, such as Tendermint Core, | |||||
which makes requests to the application every time a new transaction is | |||||
received or a block is committed. | |||||
It is unlikely that you will need to implement a client. For details of | |||||
our client, see | |||||
[here](https://github.com/tendermint/tendermint/tree/develop/abci/client). |
@ -1,185 +1,3 @@ | |||||
# Application Blockchain Interface (ABCI) | # Application Blockchain Interface (ABCI) | ||||
ABCI is the interface between Tendermint (a state-machine replication engine) | |||||
and an application (the actual state machine). | |||||
The ABCI message types are defined in a [protobuf | |||||
file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto). | |||||
For full details on the ABCI message types and protocol, see the [ABCI | |||||
specification](https://github.com/tendermint/tendermint/blob/develop/docs/app-dev/abci-spec.md). | |||||
Be sure to read the specification if you're trying to build an ABCI app! | |||||
For additional details on server implementation, see the [ABCI | |||||
readme](https://github.com/tendermint/tendermint/blob/develop/abci/README.md). | |||||
Here we provide some more details around the use of ABCI by Tendermint and | |||||
clarify common "gotchas". | |||||
## ABCI connections | |||||
Tendermint opens 3 ABCI connections to the app: one for Consensus, one for | |||||
Mempool, one for Queries. | |||||
## Async vs Sync | |||||
The main ABCI server (ie. non-GRPC) provides ordered asynchronous messages. | |||||
This is useful for DeliverTx and CheckTx, since it allows Tendermint to forward | |||||
transactions to the app before it's finished processing previous ones. | |||||
Thus, DeliverTx and CheckTx messages are sent asycnhronously, while all other | |||||
messages are sent synchronously. | |||||
## CheckTx and Commit | |||||
It is typical to hold three distinct states in an ABCI app: CheckTxState, DeliverTxState, | |||||
QueryState. The QueryState contains the latest committed state for a block. | |||||
The CheckTxState and DeliverTxState may be updated concurrently with one another. | |||||
Before Commit is called, Tendermint locks and flushes the mempool so that no new changes will happen | |||||
to CheckTxState. When Commit completes, it unlocks the mempool. | |||||
Thus, during Commit, it is safe to reset the QueryState and the CheckTxState to the latest DeliverTxState | |||||
(ie. the new state from executing all the txs in the block). | |||||
Note, however, that it is not possible to send transactions to Tendermint during Commit - if your app | |||||
tries to send a `/broadcast_tx` to Tendermint during Commit, it will deadlock. | |||||
## EndBlock Validator Updates | |||||
Updates to the Tendermint validator set can be made by returning `Validator` | |||||
objects in the `ResponseBeginBlock`: | |||||
``` | |||||
message Validator { | |||||
PubKey pub_key | |||||
int64 power | |||||
} | |||||
message PubKey { | |||||
string type | |||||
bytes data | |||||
} | |||||
``` | |||||
The `pub_key` currently supports two types: | |||||
- `type = "ed25519" and`data = <raw 32-byte public key>` | |||||
- `type = "secp256k1" and `data = <33-byte OpenSSL compressed public key>` | |||||
If the address is provided, it must match the address of the pubkey, as | |||||
specified [here](/docs/spec/blockchain/encoding.md#Addresses) | |||||
(Note: In the v0.19 series, the `pub_key` is the [Amino encoded public | |||||
key](/docs/spec/blockchain/encoding.md#public-key-cryptography). | |||||
For Ed25519 pubkeys, the Amino prefix is always "1624DE6220". For example, the 32-byte Ed25519 pubkey | |||||
`76852933A4686A721442E931A8415F62F5F1AEDF4910F1F252FB393F74C40C85` would be | |||||
Amino encoded as | |||||
`1624DE622076852933A4686A721442E931A8415F62F5F1AEDF4910F1F252FB393F74C40C85`) | |||||
(Note: In old versions of Tendermint (pre-v0.19.0), the pubkey is just prefixed with a | |||||
single type byte, so for ED25519 we'd have `pub_key = 0x1 | pub`) | |||||
The `power` is the new voting power for the validator, with the | |||||
following rules: | |||||
- power must be non-negative | |||||
- if power is 0, the validator must already exist, and will be removed from the | |||||
validator set | |||||
- if power is non-0: | |||||
- if the validator does not already exist, it will be added to the validator | |||||
set with the given power | |||||
- if the validator does already exist, its power will be adjusted to the given power | |||||
## InitChain Validator Updates | |||||
ResponseInitChain has the option to return a list of validators. | |||||
If the list is not empty, Tendermint will adopt it for the validator set. | |||||
This way the application can determine the initial validator set for the | |||||
blockchain. | |||||
ResponseInitChain also includes ConsensusParams, but these are presently | |||||
ignored. | |||||
## Query | |||||
Query is a generic message type with lots of flexibility to enable diverse sets | |||||
of queries from applications. Tendermint has no requirements from the Query | |||||
message for normal operation - that is, the ABCI app developer need not implement Query functionality if they do not wish too. | |||||
That said, Tendermint makes a number of queries to support some optional | |||||
features. These are: | |||||
### Peer Filtering | |||||
When Tendermint connects to a peer, it sends two queries to the ABCI application | |||||
using the following paths, with no additional data: | |||||
- `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and | |||||
the port of the connection | |||||
- `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the | |||||
pubkey.Address() for the peer's PubKey) | |||||
If either of these queries return a non-zero ABCI code, Tendermint will refuse | |||||
to connect to the peer. | |||||
## Info and the Handshake/Replay | |||||
On startup, Tendermint calls Info on the Query connection to get the latest | |||||
committed state of the app. The app MUST return information consistent with the | |||||
last block it succesfully completed Commit for. | |||||
If the app succesfully committed block H but not H+1, then `last_block_height = H` and `last_block_app_hash = <hash returned by Commit for block H>`. If the app | |||||
failed during the Commit of block H, then `last_block_height = H-1` and | |||||
`last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>`. | |||||
We now distinguish three heights, and describe how Tendermint syncs itself with | |||||
the app. | |||||
``` | |||||
storeBlockHeight = height of the last block Tendermint saw a commit for | |||||
stateBlockHeight = height of the last block for which Tendermint completed all | |||||
block processing and saved all ABCI results to disk | |||||
appBlockHeight = height of the last block for which ABCI app succesfully | |||||
completely Commit | |||||
``` | |||||
Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight` | |||||
Note also we never call Commit on an ABCI app twice for the same height. | |||||
The procedure is as follows. | |||||
First, some simeple start conditions: | |||||
If `appBlockHeight == 0`, then call InitChain. | |||||
If `storeBlockHeight == 0`, we're done. | |||||
Now, some sanity checks: | |||||
If `storeBlockHeight < appBlockHeight`, error | |||||
If `storeBlockHeight < stateBlockHeight`, panic | |||||
If `storeBlockHeight > stateBlockHeight+1`, panic | |||||
Now, the meat: | |||||
If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`, | |||||
replay all blocks in full from `appBlockHeight` to `storeBlockHeight`. | |||||
This happens if we completed processing the block, but the app forgot its height. | |||||
If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done | |||||
This happens if we crashed at an opportune spot. | |||||
If `storeBlockHeight == stateBlockHeight+1` | |||||
This happens if we started processing the block but didn't finish. | |||||
If `appBlockHeight < stateBlockHeight` | |||||
replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`, | |||||
and replay the block at `storeBlockHeight` using the WAL. | |||||
This happens if the app forgot the last block it committed. | |||||
If `appBlockHeight == stateBlockHeight`, | |||||
replay the last block (storeBlockHeight) in full. | |||||
This happens if we crashed before the app finished Commit | |||||
If appBlockHeight == storeBlockHeight { | |||||
update the state using the saved ABCI responses but dont run the block against the real app. | |||||
This happens if we crashed after the app finished Commit but before Tendermint saved the state. | |||||
This page has [moved](../spec/abci/apps.md). |