zolfa
/
tendermint
1
0
Fork 0
Code Issues 0 Pull Requests 0 Projects 0 Releases 221 Wiki Activity
Browse Source

docs: refactor ABCI docs 

* move spec/software/abci.md to spec/abci/apps.md and improve it
* move some of app-dev/app-development.md to spec/abci/client-server.md
pull/2343/head
Ethan Buchman 6 years ago
parent
3fd54c5df5
commit
1144e72c61
6 changed files with 337 additions and 191 deletions
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. +5
    -0
      docs/app-dev/app-development.md
  2. +19
    -0
      docs/spec/abci/README.md
  3. +1
    -8
      docs/spec/abci/abci.md
  4. +207
    -0
      docs/spec/abci/apps.md
  5. +104
    -0
      docs/spec/abci/client-server.md
  6. +1
    -183
      docs/spec/software/abci.md

+ 5
- 0
docs/app-dev/app-development.md View File

@ -1,5 +1,10 @@
# Application Development Guide
## XXX
This page is undergoing deprecation. All content is being moved to the new [home
of the ABCI specification](../spec/abci/README.md).
## ABCI Design
The purpose of ABCI is to provide a clean interface between state


+ 19
- 0
docs/spec/abci/README.md View File

@ -0,0 +1,19 @@
# ABCI
ABCI is the interface between Tendermint (a state-machine replication engine)
and an application (the actual state machine). It consists of a set of
*methods*, where each method has a corresponding `Request` and `Response`
message type. Tendermint calls the ABCI methods on the ABCI application by sending the `Request*`
messages and receiving the `Response*` messages in return.
All message types are defined in a [protobuf file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto).
This allows Tendermint to run applications written in any programming language.
This specification is split as follows:
- [Methods and Types](abci.md) - complete details on all ABCI methods and
message types
- [Applications](apps.md) - how to manage ABCI application state and other
details about building ABCI applications
- [Client and Server](client-server.md) - for those looking to implement their
own ABCI application servers

+ 1
- 8
docs/spec/abci/abci.md View File

@ -1,13 +1,7 @@
# Application Blockchain Interface (ABCI)
# ABCI Methods and Types
## Overview
ABCI is the interface between Tendermint (a state-machine replication engine)
and an application (the actual state machine). It consists of a set of
*methods*, where each method has a corresponding `Request` and `Response` type.
Tendermint calls the methods on the ABCI application by sending the `Request*`
messages and receiving the `Response*` messages in return.
The ABCI message types are defined in a [protobuf
file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto).
@ -26,7 +20,6 @@ The `Info Connection` is for initialization and for queries from the user.
Additionally, there is a `Flush` method that is called on every connection,
and an `Echo` method that is just for debugging.
## Errors
Some methods (`Echo, Info, InitChain, BeginBlock, EndBlock, Commit`),


+ 207
- 0
docs/spec/abci/apps.md View File

@ -0,0 +1,207 @@
# ABCI Applications
Please ensure you've first read the spec for [ABCI Methods and Types](abci.md)
Here we cover the following components of ABCI applications:
- [State](#State) - the interplay between ABCI connections and application state
and the differences between `CheckTx` and `DeliverTx`.
- [Validator Set Updates](#Validator-Set-Updates) - how validator sets are
changed during `InitChain` and `EndBlock`
- [Query](#Query) - standards for using the `Query` method
- [Crash Recovery](#Crash-Recovery) - handshake protocol to synchronize
Tendermint and the application on startup.
## State
Since Tendermint maintains multiple concurrent ABCI connections, it is typical
for an application to maintain a distinct state for each, and for the states to
be sycnronized during `Commit`.
### Commit
Before `Commit` is called, Tendermint locks and flushes the mempool so that no new messages will
be received on the mempool connection. This provides an opportunity to safely update all three
states to the latest committed state at once.
When `Commit` completes, it unlocks the mempool.
Note that it is not possible to send transactions to Tendermint during `Commit` - if your app
tries to send a `/broadcast_tx` to Tendermint during Commit, it will deadlock.
### Consensus Connection
The Consensus Connection should maintain a `DeliverTxState` -
the working state for block execution. It should be updated by the calls to
`BeginBlock`, `DeliverTx`, and `EndBlock` during block execution and committed to
disk as the "latest committed state" during `Commit`.
Updates made to the DeliverTxState by each method call must be readable by each subsequent method -
ie. the updates are linearizeable.
### Mempool Connection
The Mempool Connection should maintain a `CheckTxState` -
to process pending transactions in the mempool that have
not yet been committed. It should be initialized to the latest committed state
at the end of every `Commit`. Note it may be updated concurrently with the
DeliverTxState.
Before calling `Commit`, Tendermint will lock and flush the mempool,
ensuring that all existing CheckTx are responded to and no new ones can
begin.
After `Commit`, CheckTx is run again on all transactions that remain in the
node's local mempool after filtering those included in the block. To prevent the
mempool from rechecking all transactions every time a block is committed, set
the configuration option `mempool.recheck=false`.
Finally, the mempool will unlock and new transactions can be processed through CheckTx again.
Note that CheckTx doesn't have to check everything that affects transaction validity; the
expensive things can be skipped. In fact, CheckTx doesn't have to check
anything; it might say that any transaction is a valid transaction.
Unlike DeliverTx, CheckTx is just there as
a sort of weak filter to keep invalid transactions out of the blockchain. It's
weak, because a Byzantine node doesn't care about CheckTx; it can propose a
block full of invalid transactions if it wants.
### Info Connection
The Mempool Connection should maintain a `QueryState` for answering queries from the user,
and for initialization when Tendermint first starts up.
It should always contain the latest committed state associated with the
latest commited block.
QueryState should be set to the latest `DeliverTxState` at the end of every `Commit`,
ie. after the full block has been processed and the state committed to disk.
Otherwise it should never be modified.
## Validator Updates
### EndBlock
Updates to the Tendermint validator set can be made by returning
`ValidatorUpdate` objects in the `ResponseEndBlock`:
```
message ValidatorUpdate {
PubKey pub_key
int64 power
}
message PubKey {
string type
bytes data
}
```
The `pub_key` currently supports only one type:
- `type = "ed25519" and`data = <raw 32-byte public key>`
The `power` is the new voting power for the validator, with the
following rules:
- power must be non-negative
- if power is 0, the validator must already exist, and will be removed from the
validator set
- if power is non-0:
- if the validator does not already exist, it will be added to the validator
set with the given power
- if the validator does already exist, its power will be adjusted to the given power
### InitChain
ResponseInitChain has the option to return a list of validators.
If the list is not empty, Tendermint will adopt it for the validator set.
This way the application can determine the initial validator set for the
blockchain.
ResponseInitChain also includes ConsensusParams, but these are presently
ignored.
## Query
Query is a generic message type with lots of flexibility to enable diverse sets
of queries from applications. Tendermint has no requirements from the Query
message for normal operation - that is, the ABCI app developer need not implement Query functionality if they do not wish too.
That said, Tendermint makes a number of queries to support some optional
features. These are:
### Peer Filtering
When Tendermint connects to a peer, it sends two queries to the ABCI application
using the following paths, with no additional data:
- `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and
the port of the connection
- `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the
pubkey.Address() for the peer's PubKey)
If either of these queries return a non-zero ABCI code, Tendermint will refuse
to connect to the peer.
## Crash Recovery
On startup, Tendermint calls the `Info` method on the Info Connection to get the latest
committed state of the app. The app MUST return information consistent with the
last block it succesfully completed Commit for.
If the app succesfully committed block H but not H+1, then `last_block_height = H` and `last_block_app_hash = <hash returned by Commit for block H>`. If the app
failed during the Commit of block H, then `last_block_height = H-1` and
`last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>`.
We now distinguish three heights, and describe how Tendermint syncs itself with
the app.
```
storeBlockHeight = height of the last block Tendermint saw a commit for
stateBlockHeight = height of the last block for which Tendermint completed all
block processing and saved all ABCI results to disk
appBlockHeight = height of the last block for which ABCI app succesfully
completely Commit
```
Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight`
Note also we never call Commit on an ABCI app twice for the same height.
The procedure is as follows.
First, some simeple start conditions:
If `appBlockHeight == 0`, then call InitChain.
If `storeBlockHeight == 0`, we're done.
Now, some sanity checks:
If `storeBlockHeight < appBlockHeight`, error
If `storeBlockHeight < stateBlockHeight`, panic
If `storeBlockHeight > stateBlockHeight+1`, panic
Now, the meat:
If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`,
replay all blocks in full from `appBlockHeight` to `storeBlockHeight`.
This happens if we completed processing the block, but the app forgot its height.
If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done
This happens if we crashed at an opportune spot.
If `storeBlockHeight == stateBlockHeight+1`
This happens if we started processing the block but didn't finish.
If `appBlockHeight < stateBlockHeight`
replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`,
and replay the block at `storeBlockHeight` using the WAL.
This happens if the app forgot the last block it committed.
If `appBlockHeight == stateBlockHeight`,
replay the last block (storeBlockHeight) in full.
This happens if we crashed before the app finished Commit
If appBlockHeight == storeBlockHeight {
update the state using the saved ABCI responses but dont run the block against the real app.
This happens if we crashed after the app finished Commit but before Tendermint saved the state.

+ 104
- 0
docs/spec/abci/client-server.md View File

@ -0,0 +1,104 @@
# ABCI Client and Server
This section is for those looking to implement their own ABCI Server, perhaps in
a new programming language.
You are expected to have read [ABCI Methods and Types](abci.md) and [ABCI
Applications](apps.md).
See additional details in the [ABCI
readme](https://github.com/tendermint/tendermint/blob/develop/abci/README.md)(TODO: deduplicate
those details).
## Message Protocol
The message protocol consists of pairs of requests and responses defined in the
[protobuf file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto).
Some messages have no fields, while others may include byte-arrays, strings, integers,
or custom protobuf types.
For more details on protobuf, see the [documentation](https://developers.google.com/protocol-buffers/docs/overview).
For each request, a server should respond with the corresponding
response, where the order of requests is preserved in the order of
responses.
## Server
To use ABCI in your programming language of choice, there must be a ABCI
server in that language. Tendermint supports two kinds of implementation
of the server:
- Asynchronous, raw socket server (Tendermint Socket Protocol, also
known as TSP or Teaspoon)
- GRPC
Both can be tested using the `abci-cli` by setting the `--abci` flag
appropriately (ie. to `socket` or `grpc`).
See examples, in various stages of maintenance, in
[Go](https://github.com/tendermint/tendermint/tree/develop/abci/server),
[JavaScript](https://github.com/tendermint/js-abci),
[Python](https://github.com/tendermint/tendermint/tree/develop/abci/example/python3/abci),
[C++](https://github.com/mdyring/cpp-tmsp), and
[Java](https://github.com/jTendermint/jabci).
### GRPC
If GRPC is available in your language, this is the easiest approach,
though it will have significant performance overhead.
To get started with GRPC, copy in the [protobuf
file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto)
and compile it using the GRPC plugin for your language. For instance,
for golang, the command is `protoc --go_out=plugins=grpc:. types.proto`.
See the [grpc documentation for more details](http://www.grpc.io/docs/).
`protoc` will autogenerate all the necessary code for ABCI client and
server in your language, including whatever interface your application
must satisfy to be used by the ABCI server for handling requests.
### TSP
If GRPC is not available in your language, or you require higher
performance, or otherwise enjoy programming, you may implement your own
ABCI server using the Tendermint Socket Protocol, known affectionately
as Teaspoon. The first step is still to auto-generate the relevant data
types and codec in your language using `protoc`. Messages coming over
the socket are proto3 encoded, but additionally length-prefixed to
facilitate use as a streaming protocol. proto3 doesn't have an
official length-prefix standard, so we use our own. The first byte in
the prefix represents the length of the Big Endian encoded length. The
remaining bytes in the prefix are the Big Endian encoded length.
For example, if the proto3 encoded ABCI message is 0xDEADBEEF (4
bytes), the length-prefixed message is 0x0104DEADBEEF. If the proto3
encoded ABCI message is 65535 bytes long, the length-prefixed message
would be like 0x02FFFF....
Note this prefixing does not apply for grpc.
An ABCI server must also be able to support multiple connections, as
Tendermint uses three connections.
### Async vs Sync
The main ABCI server (ie. non-GRPC) provides ordered asynchronous messages.
This is useful for DeliverTx and CheckTx, since it allows Tendermint to forward
transactions to the app before it's finished processing previous ones.
Thus, DeliverTx and CheckTx messages are sent asycnhronously, while all other
messages are sent synchronously.
## Client
There are currently two use-cases for an ABCI client. One is a testing
tool, as in the `abci-cli`, which allows ABCI requests to be sent via
command line. The other is a consensus engine, such as Tendermint Core,
which makes requests to the application every time a new transaction is
received or a block is committed.
It is unlikely that you will need to implement a client. For details of
our client, see
[here](https://github.com/tendermint/tendermint/tree/develop/abci/client).

+ 1
- 183
docs/spec/software/abci.md View File

@ -1,185 +1,3 @@
# Application Blockchain Interface (ABCI)
ABCI is the interface between Tendermint (a state-machine replication engine)
and an application (the actual state machine).
The ABCI message types are defined in a [protobuf
file](https://github.com/tendermint/tendermint/blob/develop/abci/types/types.proto).
For full details on the ABCI message types and protocol, see the [ABCI
specification](https://github.com/tendermint/tendermint/blob/develop/docs/app-dev/abci-spec.md).
Be sure to read the specification if you're trying to build an ABCI app!
For additional details on server implementation, see the [ABCI
readme](https://github.com/tendermint/tendermint/blob/develop/abci/README.md).
Here we provide some more details around the use of ABCI by Tendermint and
clarify common "gotchas".
## ABCI connections
Tendermint opens 3 ABCI connections to the app: one for Consensus, one for
Mempool, one for Queries.
## Async vs Sync
The main ABCI server (ie. non-GRPC) provides ordered asynchronous messages.
This is useful for DeliverTx and CheckTx, since it allows Tendermint to forward
transactions to the app before it's finished processing previous ones.
Thus, DeliverTx and CheckTx messages are sent asycnhronously, while all other
messages are sent synchronously.
## CheckTx and Commit
It is typical to hold three distinct states in an ABCI app: CheckTxState, DeliverTxState,
QueryState. The QueryState contains the latest committed state for a block.
The CheckTxState and DeliverTxState may be updated concurrently with one another.
Before Commit is called, Tendermint locks and flushes the mempool so that no new changes will happen
to CheckTxState. When Commit completes, it unlocks the mempool.
Thus, during Commit, it is safe to reset the QueryState and the CheckTxState to the latest DeliverTxState
(ie. the new state from executing all the txs in the block).
Note, however, that it is not possible to send transactions to Tendermint during Commit - if your app
tries to send a `/broadcast_tx` to Tendermint during Commit, it will deadlock.
## EndBlock Validator Updates
Updates to the Tendermint validator set can be made by returning `Validator`
objects in the `ResponseBeginBlock`:
```
message Validator {
PubKey pub_key
int64 power
}
message PubKey {
string type
bytes data
}
```
The `pub_key` currently supports two types:
- `type = "ed25519" and`data = <raw 32-byte public key>`
- `type = "secp256k1" and `data = <33-byte OpenSSL compressed public key>`
If the address is provided, it must match the address of the pubkey, as
specified [here](/docs/spec/blockchain/encoding.md#Addresses)
(Note: In the v0.19 series, the `pub_key` is the [Amino encoded public
key](/docs/spec/blockchain/encoding.md#public-key-cryptography).
For Ed25519 pubkeys, the Amino prefix is always "1624DE6220". For example, the 32-byte Ed25519 pubkey
`76852933A4686A721442E931A8415F62F5F1AEDF4910F1F252FB393F74C40C85` would be
Amino encoded as
`1624DE622076852933A4686A721442E931A8415F62F5F1AEDF4910F1F252FB393F74C40C85`)
(Note: In old versions of Tendermint (pre-v0.19.0), the pubkey is just prefixed with a
single type byte, so for ED25519 we'd have `pub_key = 0x1 | pub`)
The `power` is the new voting power for the validator, with the
following rules:
- power must be non-negative
- if power is 0, the validator must already exist, and will be removed from the
validator set
- if power is non-0:
- if the validator does not already exist, it will be added to the validator
set with the given power
- if the validator does already exist, its power will be adjusted to the given power
## InitChain Validator Updates
ResponseInitChain has the option to return a list of validators.
If the list is not empty, Tendermint will adopt it for the validator set.
This way the application can determine the initial validator set for the
blockchain.
ResponseInitChain also includes ConsensusParams, but these are presently
ignored.
## Query
Query is a generic message type with lots of flexibility to enable diverse sets
of queries from applications. Tendermint has no requirements from the Query
message for normal operation - that is, the ABCI app developer need not implement Query functionality if they do not wish too.
That said, Tendermint makes a number of queries to support some optional
features. These are:
### Peer Filtering
When Tendermint connects to a peer, it sends two queries to the ABCI application
using the following paths, with no additional data:
- `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and
the port of the connection
- `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the
pubkey.Address() for the peer's PubKey)
If either of these queries return a non-zero ABCI code, Tendermint will refuse
to connect to the peer.
## Info and the Handshake/Replay
On startup, Tendermint calls Info on the Query connection to get the latest
committed state of the app. The app MUST return information consistent with the
last block it succesfully completed Commit for.
If the app succesfully committed block H but not H+1, then `last_block_height = H` and `last_block_app_hash = <hash returned by Commit for block H>`. If the app
failed during the Commit of block H, then `last_block_height = H-1` and
`last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>`.
We now distinguish three heights, and describe how Tendermint syncs itself with
the app.
```
storeBlockHeight = height of the last block Tendermint saw a commit for
stateBlockHeight = height of the last block for which Tendermint completed all
block processing and saved all ABCI results to disk
appBlockHeight = height of the last block for which ABCI app succesfully
completely Commit
```
Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight`
Note also we never call Commit on an ABCI app twice for the same height.
The procedure is as follows.
First, some simeple start conditions:
If `appBlockHeight == 0`, then call InitChain.
If `storeBlockHeight == 0`, we're done.
Now, some sanity checks:
If `storeBlockHeight < appBlockHeight`, error
If `storeBlockHeight < stateBlockHeight`, panic
If `storeBlockHeight > stateBlockHeight+1`, panic
Now, the meat:
If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`,
replay all blocks in full from `appBlockHeight` to `storeBlockHeight`.
This happens if we completed processing the block, but the app forgot its height.
If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done
This happens if we crashed at an opportune spot.
If `storeBlockHeight == stateBlockHeight+1`
This happens if we started processing the block but didn't finish.
If `appBlockHeight < stateBlockHeight`
replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`,
and replay the block at `storeBlockHeight` using the WAL.
This happens if the app forgot the last block it committed.
If `appBlockHeight == stateBlockHeight`,
replay the last block (storeBlockHeight) in full.
This happens if we crashed before the app finished Commit
If appBlockHeight == storeBlockHeight {
update the state using the saved ABCI responses but dont run the block against the real app.
This happens if we crashed after the app finished Commit but before Tendermint saved the state.
This page has [moved](../spec/abci/apps.md).

Write Preview
Loading…
Cancel
Save