diff --git a/spec/README.md b/spec/README.md index ba881643e..00c112f79 100644 --- a/spec/README.md +++ b/spec/README.md @@ -3,3 +3,7 @@ This folder houses the spec of Tendermint the Protocol. **Note: We are currently working on expanding the spec and will slowly be migrating it from Tendermint the repo** + +### Table of Contents + +- [ABCI Spec](./abci/README.md) diff --git a/spec/abci/README.md b/spec/abci/README.md new file mode 100644 index 000000000..56d5e8aaf --- /dev/null +++ b/spec/abci/README.md @@ -0,0 +1,19 @@ +# Overview + +ABCI is the interface between Tendermint (a state-machine replication engine) +and your application (the actual state machine). It consists of a set of +_methods_, where each method has a corresponding `Request` and `Response` +message type. Tendermint calls the ABCI methods on the ABCI application by sending the `Request*` +messages and receiving the `Response*` messages in return. + +All message types are defined in a [protobuf file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto). +This allows Tendermint to run applications written in any programming language. + +This specification is split as follows: + +- [Methods and Types](./abci.md) - complete details on all ABCI methods and + message types +- [Applications](./apps.md) - how to manage ABCI application state and other + details about building ABCI applications +- [Client and Server](./client-server.md) - for those looking to implement their + own ABCI application servers diff --git a/spec/abci/abci.md b/spec/abci/abci.md new file mode 100644 index 000000000..275c4dcd8 --- /dev/null +++ b/spec/abci/abci.md @@ -0,0 +1,532 @@ +# Methods and Types + +## Overview + +The ABCI message types are defined in a [protobuf +file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto). + +ABCI methods are split across 3 separate ABCI _connections_: + +- `Consensus Connection`: `InitChain, BeginBlock, DeliverTx, EndBlock, Commit` +- `Mempool Connection`: `CheckTx` +- `Info Connection`: `Info, SetOption, Query` + +The `Consensus Connection` is driven by a consensus protocol and is responsible +for block execution. +The `Mempool Connection` is for validating new transactions, before they're +shared or included in a block. +The `Info Connection` is for initialization and for queries from the user. + +Additionally, there is a `Flush` method that is called on every connection, +and an `Echo` method that is just for debugging. + +More details on managing state across connections can be found in the section on +[ABCI Applications](apps.md). + +## Errors + +Some methods (`Echo, Info, InitChain, BeginBlock, EndBlock, Commit`), +don't return errors because an error would indicate a critical failure +in the application and there's nothing Tendermint can do. The problem +should be addressed and both Tendermint and the application restarted. + +All other methods (`SetOption, Query, CheckTx, DeliverTx`) return an +application-specific response `Code uint32`, where only `0` is reserved +for `OK`. + +Finally, `Query`, `CheckTx`, and `DeliverTx` include a `Codespace string`, whose +intended use is to disambiguate `Code` values returned by different domains of the +application. The `Codespace` is a namespace for the `Code`. + +## Events + +Some methods (`CheckTx, BeginBlock, DeliverTx, EndBlock`) +include an `Events` field in their `Response*`. Each event contains a type and a +list of attributes, which are key-value pairs denoting something about what happened +during the method's execution. + +Events can be used to index transactions and blocks according to what happened +during their execution. Note that the set of events returned for a block from +`BeginBlock` and `EndBlock` are merged. In case both methods return the same +tag, only the value defined in `EndBlock` is used. + +Each event has a `type` which is meant to categorize the event for a particular +`Response*` or tx. A `Response*` or tx may contain multiple events with duplicate +`type` values, where each distinct entry is meant to categorize attributes for a +particular event. Every key and value in an event's attributes must be UTF-8 +encoded strings along with the even type itself. + +Example: + +```go + abci.ResponseDeliverTx{ + // ... + Events: []abci.Event{ + { + Type: "validator.provisions", + Attributes: cmn.KVPairs{ + cmn.KVPair{Key: []byte("address"), Value: []byte("...")}, + cmn.KVPair{Key: []byte("amount"), Value: []byte("...")}, + cmn.KVPair{Key: []byte("balance"), Value: []byte("...")}, + }, + }, + { + Type: "validator.provisions", + Attributes: cmn.KVPairs{ + cmn.KVPair{Key: []byte("address"), Value: []byte("...")}, + cmn.KVPair{Key: []byte("amount"), Value: []byte("...")}, + cmn.KVPair{Key: []byte("balance"), Value: []byte("...")}, + }, + }, + { + Type: "validator.slashed", + Attributes: cmn.KVPairs{ + cmn.KVPair{Key: []byte("address"), Value: []byte("...")}, + cmn.KVPair{Key: []byte("amount"), Value: []byte("...")}, + cmn.KVPair{Key: []byte("reason"), Value: []byte("...")}, + }, + }, + // ... + }, +} +``` + +## Determinism + +ABCI applications must implement deterministic finite-state machines to be +securely replicated by the Tendermint consensus. This means block execution +over the Consensus Connection must be strictly deterministic: given the same +ordered set of requests, all nodes will compute identical responses, for all +BeginBlock, DeliverTx, EndBlock, and Commit. This is critical, because the +responses are included in the header of the next block, either via a Merkle root +or directly, so all nodes must agree on exactly what they are. + +For this reason, it is recommended that applications not be exposed to any +external user or process except via the ABCI connections to a consensus engine +like Tendermint Core. The application must only change its state based on input +from block execution (BeginBlock, DeliverTx, EndBlock, Commit), and not through +any other kind of request. This is the only way to ensure all nodes see the same +transactions and compute the same results. + +If there is some non-determinism in the state machine, consensus will eventually +fail as nodes disagree over the correct values for the block header. The +non-determinism must be fixed and the nodes restarted. + +Sources of non-determinism in applications may include: + +- Hardware failures + - Cosmic rays, overheating, etc. +- Node-dependent state + - Random numbers + - Time +- Underspecification + - Library version changes + - Race conditions + - Floating point numbers + - JSON serialization + - Iterating through hash-tables/maps/dictionaries +- External Sources + - Filesystem + - Network calls (eg. some external REST API service) + +See [#56](https://github.com/tendermint/abci/issues/56) for original discussion. + +Note that some methods (`SetOption, Query, CheckTx, DeliverTx`) return +explicitly non-deterministic data in the form of `Info` and `Log` fields. The `Log` is +intended for the literal output from the application's logger, while the +`Info` is any additional info that should be returned. These are the only fields +that are not included in block header computations, so we don't need agreement +on them. All other fields in the `Response*` must be strictly deterministic. + +## Block Execution + +The first time a new blockchain is started, Tendermint calls +`InitChain`. From then on, the following sequence of methods is executed for each +block: + +`BeginBlock, [DeliverTx], EndBlock, Commit` + +where one `DeliverTx` is called for each transaction in the block. +The result is an updated application state. +Cryptographic commitments to the results of DeliverTx, EndBlock, and +Commit are included in the header of the next block. + +## Messages + +### Echo + +- **Request**: + - `Message (string)`: A string to echo back +- **Response**: + - `Message (string)`: The input string +- **Usage**: + - Echo a string to test an abci client/server implementation + +### Flush + +- **Usage**: + - Signals that messages queued on the client should be flushed to + the server. It is called periodically by the client + implementation to ensure asynchronous requests are actually + sent, and is called immediately to make a synchronous request, + which returns when the Flush response comes back. + +### Info + +- **Request**: + - `Version (string)`: The Tendermint software semantic version + - `BlockVersion (uint64)`: The Tendermint Block Protocol version + - `P2PVersion (uint64)`: The Tendermint P2P Protocol version +- **Response**: + - `Data (string)`: Some arbitrary information + - `Version (string)`: The application software semantic version + - `AppVersion (uint64)`: The application protocol version + - `LastBlockHeight (int64)`: Latest block for which the app has + called Commit + - `LastBlockAppHash ([]byte)`: Latest result of Commit +- **Usage**: + - Return information about the application state. + - Used to sync Tendermint with the application during a handshake + that happens on startup. + - The returned `AppVersion` will be included in the Header of every block. + - Tendermint expects `LastBlockAppHash` and `LastBlockHeight` to + be updated during `Commit`, ensuring that `Commit` is never + called twice for the same block height. + +### SetOption + +- **Request**: + - `Key (string)`: Key to set + - `Value (string)`: Value to set for key +- **Response**: + - `Code (uint32)`: Response code + - `Log (string)`: The output of the application's logger. May + be non-deterministic. + - `Info (string)`: Additional information. May + be non-deterministic. +- **Usage**: + - Set non-consensus critical application specific options. + - e.g. Key="min-fee", Value="100fermion" could set the minimum fee + required for CheckTx (but not DeliverTx - that would be + consensus critical). + +### InitChain + +- **Request**: + - `Time (google.protobuf.Timestamp)`: Genesis time. + - `ChainID (string)`: ID of the blockchain. + - `ConsensusParams (ConsensusParams)`: Initial consensus-critical parameters. + - `Validators ([]ValidatorUpdate)`: Initial genesis validators. + - `AppStateBytes ([]byte)`: Serialized initial application state. Amino-encoded JSON bytes. +- **Response**: + - `ConsensusParams (ConsensusParams)`: Initial + consensus-critical parameters. + - `Validators ([]ValidatorUpdate)`: Initial validator set (if non empty). +- **Usage**: + - Called once upon genesis. + - If ResponseInitChain.Validators is empty, the initial validator set will be the RequestInitChain.Validators + - If ResponseInitChain.Validators is not empty, the initial validator set will be the + ResponseInitChain.Validators (regardless of what is in RequestInitChain.Validators). + - This allows the app to decide if it wants to accept the initial validator + set proposed by tendermint (ie. in the genesis file), or if it wants to use + a different one (perhaps computed based on some application specific + information in the genesis file). + +### Query + +- **Request**: + - `Data ([]byte)`: Raw query bytes. Can be used with or in lieu + of Path. + - `Path (string)`: Path of request, like an HTTP GET path. Can be + used with or in liue of Data. + - Apps MUST interpret '/store' as a query by key on the + underlying store. The key SHOULD be specified in the Data field. + - Apps SHOULD allow queries over specific types like + '/accounts/...' or '/votes/...' + - `Height (int64)`: The block height for which you want the query + (default=0 returns data for the latest committed block). Note + that this is the height of the block containing the + application's Merkle root hash, which represents the state as it + was after committing the block at Height-1 + - `Prove (bool)`: Return Merkle proof with response if possible +- **Response**: + - `Code (uint32)`: Response code. + - `Log (string)`: The output of the application's logger. May + be non-deterministic. + - `Info (string)`: Additional information. May + be non-deterministic. + - `Index (int64)`: The index of the key in the tree. + - `Key ([]byte)`: The key of the matching data. + - `Value ([]byte)`: The value of the matching data. + - `Proof (Proof)`: Serialized proof for the value data, if requested, to be + verified against the `AppHash` for the given Height. + - `Height (int64)`: The block height from which data was derived. + Note that this is the height of the block containing the + application's Merkle root hash, which represents the state as it + was after committing the block at Height-1 + - `Codespace (string)`: Namespace for the `Code`. +- **Usage**: + - Query for data from the application at current or past height. + - Optionally return Merkle proof. + - Merkle proof includes self-describing `type` field to support many types + of Merkle trees and encoding formats. + +### BeginBlock + +- **Request**: + - `Hash ([]byte)`: The block's hash. This can be derived from the + block header. + - `Header (struct{})`: The block header. + - `LastCommitInfo (LastCommitInfo)`: Info about the last commit, including the + round, and the list of validators and which ones signed the last block. + - `ByzantineValidators ([]Evidence)`: List of evidence of + validators that acted maliciously. +- **Response**: + - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing +- **Usage**: + - Signals the beginning of a new block. Called prior to + any DeliverTxs. + - The header contains the height, timestamp, and more - it exactly matches the + Tendermint block header. We may seek to generalize this in the future. + - The `LastCommitInfo` and `ByzantineValidators` can be used to determine + rewards and punishments for the validators. NOTE validators here do not + include pubkeys. + +### CheckTx + +- **Request**: + - `Tx ([]byte)`: The request transaction bytes + - `Type (CheckTxType)`: What type of `CheckTx` request is this? At present, + there are two possible values: `CheckTx_New` (the default, which says + that a full check is required), and `CheckTx_Recheck` (when the mempool is + initiating a normal recheck of a transaction). +- **Response**: + - `Code (uint32)`: Response code + - `Data ([]byte)`: Result bytes, if any. + - `Log (string)`: The output of the application's logger. May + be non-deterministic. + - `Info (string)`: Additional information. May + be non-deterministic. + - `GasWanted (int64)`: Amount of gas requested for transaction. + - `GasUsed (int64)`: Amount of gas consumed by transaction. + - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing + transactions (eg. by account). + - `Codespace (string)`: Namespace for the `Code`. +- **Usage**: + - Technically optional - not involved in processing blocks. + - Guardian of the mempool: every node runs CheckTx before letting a + transaction into its local mempool. + - The transaction may come from an external user or another node + - CheckTx need not execute the transaction in full, but rather a light-weight + yet stateful validation, like checking signatures and account balances, but + not running code in a virtual machine. + - Transactions where `ResponseCheckTx.Code != 0` will be rejected - they will not be broadcast to + other nodes or included in a proposal block. + - Tendermint attributes no other value to the response code + +### DeliverTx + +- **Request**: + - `Tx ([]byte)`: The request transaction bytes. +- **Response**: + - `Code (uint32)`: Response code. + - `Data ([]byte)`: Result bytes, if any. + - `Log (string)`: The output of the application's logger. May + be non-deterministic. + - `Info (string)`: Additional information. May + be non-deterministic. + - `GasWanted (int64)`: Amount of gas requested for transaction. + - `GasUsed (int64)`: Amount of gas consumed by transaction. + - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing + transactions (eg. by account). + - `Codespace (string)`: Namespace for the `Code`. +- **Usage**: + - The workhorse of the application - non-optional. + - Execute the transaction in full. + - `ResponseDeliverTx.Code == 0` only if the transaction is fully valid. + +### EndBlock + +- **Request**: + - `Height (int64)`: Height of the block just executed. +- **Response**: + - `ValidatorUpdates ([]ValidatorUpdate)`: Changes to validator set (set + voting power to 0 to remove). + - `ConsensusParamUpdates (ConsensusParams)`: Changes to + consensus-critical time, size, and other parameters. + - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing +- **Usage**: + - Signals the end of a block. + - Called after all transactions, prior to each Commit. + - Validator updates returned by block `H` impact blocks `H+1`, `H+2`, and + `H+3`, but only effects changes on the validator set of `H+2`: + - `H+1`: NextValidatorsHash + - `H+2`: ValidatorsHash (and thus the validator set) + - `H+3`: LastCommitInfo (ie. the last validator set) + - Consensus params returned for block `H` apply for block `H+1` + +### Commit + +- **Response**: + - `Data ([]byte)`: The Merkle root hash of the application state +- **Usage**: + - Persist the application state. + - Return an (optional) Merkle root hash of the application state + - `ResponseCommit.Data` is included as the `Header.AppHash` in the next block + - it may be empty + - Later calls to `Query` can return proofs about the application state anchored + in this Merkle root hash + - Note developers can return whatever they want here (could be nothing, or a + constant string, etc.), so long as it is deterministic - it must not be a + function of anything that did not come from the + BeginBlock/DeliverTx/EndBlock methods. + +## Data Types + +### Header + +- **Fields**: + - `Version (Version)`: Version of the blockchain and the application + - `ChainID (string)`: ID of the blockchain + - `Height (int64)`: Height of the block in the chain + - `Time (google.protobuf.Timestamp)`: Time of the previous block. + For heights > 1, it's the weighted median of the timestamps of the valid + votes in the block.LastCommit. + For height == 1, it's genesis time. + - `NumTxs (int32)`: Number of transactions in the block + - `TotalTxs (int64)`: Total number of transactions in the blockchain until + now + - `LastBlockID (BlockID)`: Hash of the previous (parent) block + - `LastCommitHash ([]byte)`: Hash of the previous block's commit + - `ValidatorsHash ([]byte)`: Hash of the validator set for this block + - `NextValidatorsHash ([]byte)`: Hash of the validator set for the next block + - `ConsensusHash ([]byte)`: Hash of the consensus parameters for this block + - `AppHash ([]byte)`: Data returned by the last call to `Commit` - typically the + Merkle root of the application state after executing the previous block's + transactions + - `LastResultsHash ([]byte)`: Hash of the ABCI results returned by the last block + - `EvidenceHash ([]byte)`: Hash of the evidence included in this block + - `ProposerAddress ([]byte)`: Original proposer for the block +- **Usage**: + - Provided in RequestBeginBlock + - Provides important context about the current state of the blockchain - + especially height and time. + - Provides the proposer of the current block, for use in proposer-based + reward mechanisms. + +### Version + +- **Fields**: + - `Block (uint64)`: Protocol version of the blockchain data structures. + - `App (uint64)`: Protocol version of the application. +- **Usage**: + - Block version should be static in the life of a blockchain. + - App version may be updated over time by the application. + +### Validator + +- **Fields**: + - `Address ([]byte)`: Address of the validator (hash of the public key) + - `Power (int64)`: Voting power of the validator +- **Usage**: + - Validator identified by address + - Used in RequestBeginBlock as part of VoteInfo + - Does not include PubKey to avoid sending potentially large quantum pubkeys + over the ABCI + +### ValidatorUpdate + +- **Fields**: + - `PubKey (PubKey)`: Public key of the validator + - `Power (int64)`: Voting power of the validator +- **Usage**: + - Validator identified by PubKey + - Used to tell Tendermint to update the validator set + +### VoteInfo + +- **Fields**: + - `Validator (Validator)`: A validator + - `SignedLastBlock (bool)`: Indicates whether or not the validator signed + the last block +- **Usage**: + - Indicates whether a validator signed the last block, allowing for rewards + based on validator availability + +### PubKey + +- **Fields**: + - `Type (string)`: Type of the public key. A simple string like `"ed25519"`. + In the future, may indicate a serialization algorithm to parse the `Data`, + for instance `"amino"`. + - `Data ([]byte)`: Public key data. For a simple public key, it's just the + raw bytes. If the `Type` indicates an encoding algorithm, this is the + encoded public key. +- **Usage**: + - A generic and extensible typed public key + +### Evidence + +- **Fields**: + - `Type (string)`: Type of the evidence. A hierarchical path like + "duplicate/vote". + - `Validator (Validator`: The offending validator + - `Height (int64)`: Height when the offense was committed + - `Time (google.protobuf.Timestamp)`: Time of the block at height `Height`. + It is the proposer's local time when block was created. + - `TotalVotingPower (int64)`: Total voting power of the validator set at + height `Height` + +### LastCommitInfo + +- **Fields**: + - `Round (int32)`: Commit round. + - `Votes ([]VoteInfo)`: List of validators addresses in the last validator set + with their voting power and whether or not they signed a vote. + +### ConsensusParams + +- **Fields**: + - `Block (BlockParams)`: Parameters limiting the size of a block and time between consecutive blocks. + - `Evidence (EvidenceParams)`: Parameters limiting the validity of + evidence of byzantine behaviour. + - `Validator (ValidatorParams)`: Parameters limitng the types of pubkeys validators can use. + +### BlockParams + +- **Fields**: + - `MaxBytes (int64)`: Max size of a block, in bytes. + - `MaxGas (int64)`: Max sum of `GasWanted` in a proposed block. + - NOTE: blocks that violate this may be committed if there are Byzantine proposers. + It's the application's responsibility to handle this when processing a + block! + +### EvidenceParams + +- **Fields**: + - `MaxAge (int64)`: Max age of evidence, in blocks. Evidence older than this + is considered stale and ignored. + - This should correspond with an app's "unbonding period" or other + similar mechanism for handling Nothing-At-Stake attacks. + - NOTE: this should change to time (instead of blocks)! + +### ValidatorParams + +- **Fields**: + - `PubKeyTypes ([]string)`: List of accepted pubkey types. Uses same + naming as `PubKey.Type`. + +### Proof + +- **Fields**: + - `Ops ([]ProofOp)`: List of chained Merkle proofs, of possibly different types + - The Merkle root of one op is the value being proven in the next op. + - The Merkle root of the final op should equal the ultimate root hash being + verified against. + +### ProofOp + +- **Fields**: + - `Type (string)`: Type of Merkle proof and how it's encoded. + - `Key ([]byte)`: Key in the Merkle tree that this proof is for. + - `Data ([]byte)`: Encoded Merkle proof for the key. diff --git a/spec/abci/apps.md b/spec/abci/apps.md new file mode 100644 index 000000000..528f92634 --- /dev/null +++ b/spec/abci/apps.md @@ -0,0 +1,453 @@ +# Applications + +Please ensure you've first read the spec for [ABCI Methods and Types](abci.md) + +Here we cover the following components of ABCI applications: + +- [Connection State](#state) - the interplay between ABCI connections and application state + and the differences between `CheckTx` and `DeliverTx`. +- [Transaction Results](#transaction-results) - rules around transaction + results and validity +- [Validator Set Updates](#validator-updates) - how validator sets are + changed during `InitChain` and `EndBlock` +- [Query](#query) - standards for using the `Query` method and proofs about the + application state +- [Crash Recovery](#crash-recovery) - handshake protocol to synchronize + Tendermint and the application on startup. + +## State + +Since Tendermint maintains three concurrent ABCI connections, it is typical +for an application to maintain a distinct state for each, and for the states to +be synchronized during `Commit`. + +### Commit + +Application state should only be persisted to disk during `Commit`. + +Before `Commit` is called, Tendermint locks and flushes the mempool so that no new messages will +be received on the mempool connection. This provides an opportunity to safely update all three +states to the latest committed state at once. + +When `Commit` completes, it unlocks the mempool. + +WARNING: if the ABCI app logic processing the `Commit` message sends a +`/broadcast_tx_sync` or `/broadcast_tx_commit` and waits for the response +before proceeding, it will deadlock. Executing those `broadcast_tx` calls +involves acquiring a lock that is held during the `Commit` call, so it's not +possible. If you make the call to the `broadcast_tx` endpoints concurrently, +that's no problem, it just can't be part of the sequential logic of the +`Commit` function. + +### Consensus Connection + +The Consensus Connection should maintain a `DeliverTxState` - +the working state for block execution. It should be updated by the calls to +`BeginBlock`, `DeliverTx`, and `EndBlock` during block execution and committed to +disk as the "latest committed state" during `Commit`. + +Updates made to the DeliverTxState by each method call must be readable by each subsequent method - +ie. the updates are linearizable. + +### Mempool Connection + +The Mempool Connection should maintain a `CheckTxState` +to sequentially process pending transactions in the mempool that have +not yet been committed. It should be initialized to the latest committed state +at the end of every `Commit`. + +The CheckTxState may be updated concurrently with the DeliverTxState, as +messages may be sent concurrently on the Consensus and Mempool connections. However, +before calling `Commit`, Tendermint will lock and flush the mempool connection, +ensuring that all existing CheckTx are responded to and no new ones can +begin. + +After `Commit`, CheckTx is run again on all transactions that remain in the +node's local mempool after filtering those included in the block. To prevent the +mempool from rechecking all transactions every time a block is committed, set +the configuration option `mempool.recheck=false`. As of Tendermint v0.32.1, +an additional `Type` parameter is made available to the CheckTx function that +indicates whether an incoming transaction is new (`CheckTxType_New`), or a +recheck (`CheckTxType_Recheck`). + +Finally, the mempool will unlock and new transactions can be processed through CheckTx again. + +Note that CheckTx doesn't have to check everything that affects transaction validity; the +expensive things can be skipped. In fact, CheckTx doesn't have to check +anything; it might say that any transaction is a valid transaction. +Unlike DeliverTx, CheckTx is just there as +a sort of weak filter to keep invalid transactions out of the blockchain. It's +weak, because a Byzantine node doesn't care about CheckTx; it can propose a +block full of invalid transactions if it wants. + +### Info Connection + +The Info Connection should maintain a `QueryState` for answering queries from the user, +and for initialization when Tendermint first starts up (both described further +below). +It should always contain the latest committed state associated with the +latest committed block. + +QueryState should be set to the latest `DeliverTxState` at the end of every `Commit`, +ie. after the full block has been processed and the state committed to disk. +Otherwise it should never be modified. + +## Transaction Results + +`ResponseCheckTx` and `ResponseDeliverTx` contain the same fields. + +The `Info` and `Log` fields are non-deterministic values for debugging/convenience purposes +that are otherwise ignored. + +The `Data` field must be strictly deterministic, but can be arbitrary data. + +### Gas + +Ethereum introduced the notion of `gas` as an abstract representation of the +cost of resources used by nodes when processing transactions. Every operation in the +Ethereum Virtual Machine uses some amount of gas, and gas can be accepted at a market-variable price. +Users propose a maximum amount of gas for their transaction; if the tx uses less, they get +the difference credited back. Tendermint adopts a similar abstraction, +though uses it only optionally and weakly, allowing applications to define +their own sense of the cost of execution. + +In Tendermint, the `ConsensusParams.Block.MaxGas` limits the amount of `gas` that can be used in a block. +The default value is `-1`, meaning no limit, or that the concept of gas is +meaningless. + +Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum +amount of gas the sender of a tx is willing to use, and the later is how much it actually +used. Applications should enforce that `GasUsed <= GasWanted` - ie. tx execution +should halt before it can use more resources than it requested. + +When `MaxGas > -1`, Tendermint enforces the following rules: + +- `GasWanted <= MaxGas` for all txs in the mempool +- `(sum of GasWanted in a block) <= MaxGas` when proposing a block + +If `MaxGas == -1`, no rules about gas are enforced. + +Note that Tendermint does not currently enforce anything about Gas in the consensus, only the mempool. +This means it does not guarantee that committed blocks satisfy these rules! +It is the application's responsibility to return non-zero response codes when gas limits are exceeded. + +The `GasUsed` field is ignored completely by Tendermint. That said, applications should enforce: + +- `GasUsed <= GasWanted` for any given transaction +- `(sum of GasUsed in a block) <= MaxGas` for every block + +In the future, we intend to add a `Priority` field to the responses that can be +used to explicitly prioritize txs in the mempool for inclusion in a block +proposal. See [#1861](https://github.com/tendermint/tendermint/issues/1861). + +### CheckTx + +If `Code != 0`, it will be rejected from the mempool and hence +not broadcasted to other peers and not included in a proposal block. + +`Data` contains the result of the CheckTx transaction execution, if any. It is +semantically meaningless to Tendermint. + +`Tags` include any tags for the execution, though since the transaction has not +been committed yet, they are effectively ignored by Tendermint. + +### DeliverTx + +If DeliverTx returns `Code != 0`, the transaction will be considered invalid, +though it is still included in the block. + +`Data` contains the result of the CheckTx transaction execution, if any. It is +semantically meaningless to Tendermint. + +Both the `Code` and `Data` are included in a structure that is hashed into the +`LastResultsHash` of the next block header. + +`Tags` include any tags for the execution, which Tendermint will use to index +the transaction by. This allows transactions to be queried according to what +events took place during their execution. + +See issue [#1007](https://github.com/tendermint/tendermint/issues/1007) for how +the tags will be hashed into the next block header. + +## Validator Updates + +The application may set the validator set during InitChain, and update it during +EndBlock. + +Note that the maximum total power of the validator set is bounded by +`MaxTotalVotingPower = MaxInt64 / 8`. Applications are responsible for ensuring +they do not make changes to the validator set that cause it to exceed this +limit. + +Additionally, applications must ensure that a single set of updates does not contain any duplicates - +a given public key can only appear in an update once. If an update includes +duplicates, the block execution will fail irrecoverably. + +### InitChain + +ResponseInitChain can return a list of validators. +If the list is empty, Tendermint will use the validators loaded in the genesis +file. +If the list is not empty, Tendermint will use it for the validator set. +This way the application can determine the initial validator set for the +blockchain. + +### EndBlock + +Updates to the Tendermint validator set can be made by returning +`ValidatorUpdate` objects in the `ResponseEndBlock`: + +``` +message ValidatorUpdate { + PubKey pub_key + int64 power +} + +message PubKey { + string type + bytes data +} +``` + +The `pub_key` currently supports only one type: + +- `type = "ed25519"` and `data = ` + +The `power` is the new voting power for the validator, with the +following rules: + +- power must be non-negative +- if power is 0, the validator must already exist, and will be removed from the + validator set +- if power is non-0: + - if the validator does not already exist, it will be added to the validator + set with the given power + - if the validator does already exist, its power will be adjusted to the given power +- the total power of the new validator set must not exceed MaxTotalVotingPower + +Note the updates returned in block `H` will only take effect at block `H+2`. + +## Consensus Parameters + +ConsensusParams enforce certain limits in the blockchain, like the maximum size +of blocks, amount of gas used in a block, and the maximum acceptable age of +evidence. They can be set in InitChain and updated in EndBlock. + +### Block.MaxBytes + +The maximum size of a complete Amino encoded block. +This is enforced by Tendermint consensus. + +This implies a maximum tx size that is this MaxBytes, less the expected size of +the header, the validator set, and any included evidence in the block. + +Must have `0 < MaxBytes < 100 MB`. + +### Block.MaxGas + +The maximum of the sum of `GasWanted` in a proposed block. +This is *not* enforced by Tendermint consensus. +It is left to the app to enforce (ie. if txs are included past the +limit, they should return non-zero codes). It is used by Tendermint to limit the +txs included in a proposed block. + +Must have `MaxGas >= -1`. +If `MaxGas == -1`, no limit is enforced. + +### Block.TimeIotaMs + +The minimum time between consecutive blocks (in milliseconds). +This is enforced by Tendermint consensus. + +Must have `TimeIotaMs > 0` to ensure time monotonicity. + +### EvidenceParams.MaxAge + +This is the maximum age of evidence. +This is enforced by Tendermint consensus. +If a block includes evidence older than this, the block will be rejected +(validators won't vote for it). + +Must have `MaxAge > 0`. + +### Updates + +The application may set the ConsensusParams during InitChain, and update them during +EndBlock. If the ConsensusParams is empty, it will be ignored. Each field +that is not empty will be applied in full. For instance, if updating the +Block.MaxBytes, applications must also set the other Block fields (like +Block.MaxGas), even if they are unchanged, as they will otherwise cause the +value to be updated to 0. + +#### InitChain + +ResponseInitChain includes a ConsensusParams. +If its nil, Tendermint will use the params loaded in the genesis +file. If it's not nil, Tendermint will use it. +This way the application can determine the initial consensus params for the +blockchain. + +#### EndBlock + +ResponseEndBlock includes a ConsensusParams. +If its nil, Tendermint will do nothing. +If it's not nil, Tendermint will use it. +This way the application can update the consensus params over time. + +Note the updates returned in block `H` will take effect right away for block +`H+1`. + +## Query + +Query is a generic method with lots of flexibility to enable diverse sets +of queries on application state. Tendermint makes use of Query to filter new peers +based on ID and IP, and exposes Query to the user over RPC. + +Note that calls to Query are not replicated across nodes, but rather query the +local node's state - hence they may return stale reads. For reads that require +consensus, use a transaction. + +The most important use of Query is to return Merkle proofs of the application state at some height +that can be used for efficient application-specific lite-clients. + +Note Tendermint has technically no requirements from the Query +message for normal operation - that is, the ABCI app developer need not implement +Query functionality if they do not wish too. + +### Query Proofs + +The Tendermint block header includes a number of hashes, each providing an +anchor for some type of proof about the blockchain. The `ValidatorsHash` enables +quick verification of the validator set, the `DataHash` gives quick +verification of the transactions included in the block, etc. + +The `AppHash` is unique in that it is application specific, and allows for +application-specific Merkle proofs about the state of the application. +While some applications keep all relevant state in the transactions themselves +(like Bitcoin and its UTXOs), others maintain a separated state that is +computed deterministically *from* transactions, but is not contained directly in +the transactions themselves (like Ethereum contracts and accounts). +For such applications, the `AppHash` provides a much more efficient way to verify lite-client proofs. + +ABCI applications can take advantage of more efficient lite-client proofs for +their state as follows: + +- return the Merkle root of the deterministic application state in +`ResponseCommit.Data`. +- it will be included as the `AppHash` in the next block. +- return efficient Merkle proofs about that application state in `ResponseQuery.Proof` + that can be verified using the `AppHash` of the corresponding block. + +For instance, this allows an application's lite-client to verify proofs of +absence in the application state, something which is much less efficient to do using the block hash. + +Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees, +where the leaves of one tree are the root hashes of others. To support this, and +the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure: + +``` +message Proof { + repeated ProofOp ops +} + +message ProofOp { + string type = 1; + bytes key = 2; + bytes data = 3; +} +``` + +Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`. +This allows ABCI to support many different kinds of Merkle trees, encoding +formats, and proofs (eg. of presence and absence) just by varying the `type`. +The `data` contains the actual encoded proof, encoded according to the `type`. +When verifying the full proof, the root hash for one ProofOp is the value being +verified for the next ProofOp in the list. The root hash of the final ProofOp in +the list should match the `AppHash` being verified against. + +### Peer Filtering + +When Tendermint connects to a peer, it sends two queries to the ABCI application +using the following paths, with no additional data: + +- `/p2p/filter/addr/`, where `` denote the IP address and + the port of the connection +- `p2p/filter/id/`, where `` is the peer node ID (ie. the + pubkey.Address() for the peer's PubKey) + +If either of these queries return a non-zero ABCI code, Tendermint will refuse +to connect to the peer. + +### Paths + +Queries are directed at paths, and may optionally include additional data. + +The expectation is for there to be some number of high level paths +differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently, +Tendermint only uses `/p2p`, for filtering peers. For more advanced use, see the +implementation of +[Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/v0.23.1/baseapp/baseapp.go#L333). + +## Crash Recovery + +On startup, Tendermint calls the `Info` method on the Info Connection to get the latest +committed state of the app. The app MUST return information consistent with the +last block it succesfully completed Commit for. + +If the app succesfully committed block H but not H+1, then `last_block_height = H` and `last_block_app_hash = `. If the app +failed during the Commit of block H, then `last_block_height = H-1` and +`last_block_app_hash = `. + +We now distinguish three heights, and describe how Tendermint syncs itself with +the app. + +``` +storeBlockHeight = height of the last block Tendermint saw a commit for +stateBlockHeight = height of the last block for which Tendermint completed all + block processing and saved all ABCI results to disk +appBlockHeight = height of the last block for which ABCI app succesfully + completed Commit +``` + +Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight` +Note also we never call Commit on an ABCI app twice for the same height. + +The procedure is as follows. + +First, some simple start conditions: + +If `appBlockHeight == 0`, then call InitChain. + +If `storeBlockHeight == 0`, we're done. + +Now, some sanity checks: + +If `storeBlockHeight < appBlockHeight`, error +If `storeBlockHeight < stateBlockHeight`, panic +If `storeBlockHeight > stateBlockHeight+1`, panic + +Now, the meat: + +If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`, +replay all blocks in full from `appBlockHeight` to `storeBlockHeight`. +This happens if we completed processing the block, but the app forgot its height. + +If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done. +This happens if we crashed at an opportune spot. + +If `storeBlockHeight == stateBlockHeight+1` +This happens if we started processing the block but didn't finish. + +If `appBlockHeight < stateBlockHeight` + replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`, + and replay the block at `storeBlockHeight` using the WAL. +This happens if the app forgot the last block it committed. + +If `appBlockHeight == stateBlockHeight`, + replay the last block (storeBlockHeight) in full. +This happens if we crashed before the app finished Commit + +If `appBlockHeight == storeBlockHeight` + update the state using the saved ABCI responses but dont run the block against the real app. +This happens if we crashed after the app finished Commit but before Tendermint saved the state. + diff --git a/spec/abci/client-server.md b/spec/abci/client-server.md new file mode 100644 index 000000000..94485f0d9 --- /dev/null +++ b/spec/abci/client-server.md @@ -0,0 +1,109 @@ +# Client and Server + +This section is for those looking to implement their own ABCI Server, perhaps in +a new programming language. + +You are expected to have read [ABCI Methods and Types](./abci.md) and [ABCI +Applications](./apps.md). + +## Message Protocol + +The message protocol consists of pairs of requests and responses defined in the +[protobuf file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto). + +Some messages have no fields, while others may include byte-arrays, strings, integers, +or custom protobuf types. + +For more details on protobuf, see the [documentation](https://developers.google.com/protocol-buffers/docs/overview). + +For each request, a server should respond with the corresponding +response, where the order of requests is preserved in the order of +responses. + +## Server Implementations + +To use ABCI in your programming language of choice, there must be a ABCI +server in that language. Tendermint supports three implementations of the ABCI, written in Go: + +- In-process (Golang only) +- ABCI-socket +- GRPC + +The latter two can be tested using the `abci-cli` by setting the `--abci` flag +appropriately (ie. to `socket` or `grpc`). + +See examples, in various stages of maintenance, in +[Go](https://github.com/tendermint/tendermint/tree/master/abci/server), +[JavaScript](https://github.com/tendermint/js-abci), +[Python](https://github.com/tendermint/tendermint/tree/master/abci/example/python3/abci), +[C++](https://github.com/mdyring/cpp-tmsp), and +[Java](https://github.com/jTendermint/jabci). + +### In Process + +The simplest implementation uses function calls within Golang. +This means ABCI applications written in Golang can be compiled with TendermintCore and run as a single binary. + +### GRPC + +If GRPC is available in your language, this is the easiest approach, +though it will have significant performance overhead. + +To get started with GRPC, copy in the [protobuf +file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto) +and compile it using the GRPC plugin for your language. For instance, +for golang, the command is `protoc --go_out=plugins=grpc:. types.proto`. +See the [grpc documentation for more details](http://www.grpc.io/docs/). +`protoc` will autogenerate all the necessary code for ABCI client and +server in your language, including whatever interface your application +must satisfy to be used by the ABCI server for handling requests. + +Note the length-prefixing used in the socket implementation (TSP) does not apply for GRPC. + +### TSP + +Tendermint Socket Protocol is an asynchronous, raw socket server which provides ordered message passing over unix or tcp. +Messages are serialized using Protobuf3 and length-prefixed with a [signed Varint](https://developers.google.com/protocol-buffers/docs/encoding?csw=1#signed-integers) + +If GRPC is not available in your language, or you require higher +performance, or otherwise enjoy programming, you may implement your own +ABCI server using the Tendermint Socket Protocol. The first step is still to auto-generate the relevant data +types and codec in your language using `protoc`. In addition to being proto3 encoded, messages coming over +the socket are length-prefixed to facilitate use as a streaming protocol. proto3 doesn't have an +official length-prefix standard, so we use our own. The first byte in +the prefix represents the length of the Big Endian encoded length. The +remaining bytes in the prefix are the Big Endian encoded length. + +For example, if the proto3 encoded ABCI message is 0xDEADBEEF (4 +bytes), the length-prefixed message is 0x0104DEADBEEF. If the proto3 +encoded ABCI message is 65535 bytes long, the length-prefixed message +would be like 0x02FFFF.... + +The benefit of using this `varint` encoding over the old version (where integers were encoded as `` is that +it is the standard way to encode integers in Protobuf. It is also generally shorter. + +As noted above, this prefixing does not apply for GRPC. + +An ABCI server must also be able to support multiple connections, as +Tendermint uses three connections. + +### Async vs Sync + +The main ABCI server (ie. non-GRPC) provides ordered asynchronous messages. +This is useful for DeliverTx and CheckTx, since it allows Tendermint to forward +transactions to the app before it's finished processing previous ones. + +Thus, DeliverTx and CheckTx messages are sent asynchronously, while all other +messages are sent synchronously. + +## Client + +There are currently two use-cases for an ABCI client. One is a testing +tool, as in the `abci-cli`, which allows ABCI requests to be sent via +command line. The other is a consensus engine, such as Tendermint Core, +which makes requests to the application every time a new transaction is +received or a block is committed. + +It is unlikely that you will need to implement a client. For details of +our client, see +[here](https://github.com/tendermint/tendermint/tree/master/abci/client).