Add ABCI SPEC (#51)

- move the abci spec from tendermint to spec repo Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
5 years ago · 95cf253b6d
--- a/spec/README.md
+++ b/spec/README.md
@ -3,3 +3,7 @@
 This folder houses the spec of Tendermint the Protocol.

 **Note: We are currently working on expanding the spec and will slowly be migrating it from Tendermint the repo**

 ### Table of Contents

 - [ABCI Spec](./abci/README.md)
--- a/spec/abci/README.md
+++ b/spec/abci/README.md
@ -0,0 +1,19 @@
 # Overview

 ABCI is the interface between Tendermint (a state-machine replication engine)
 and your application (the actual state machine). It consists of a set of
 _methods_, where each method has a corresponding `Request` and `Response`
 message type. Tendermint calls the ABCI methods on the ABCI application by sending the `Request*`
 messages and receiving the `Response*` messages in return.

 All message types are defined in a [protobuf file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto).
 This allows Tendermint to run applications written in any programming language.

 This specification is split as follows:

 - [Methods and Types](./abci.md) - complete details on all ABCI methods and
  message types
 - [Applications](./apps.md) - how to manage ABCI application state and other
  details about building ABCI applications
 - [Client and Server](./client-server.md) - for those looking to implement their
  own ABCI application servers
--- a/spec/abci/abci.md
+++ b/spec/abci/abci.md
@ -0,0 +1,532 @@
 # Methods and Types

 ## Overview

 The ABCI message types are defined in a [protobuf
 file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto).

 ABCI methods are split across 3 separate ABCI _connections_:

 - `Consensus Connection`: `InitChain, BeginBlock, DeliverTx, EndBlock, Commit`
 - `Mempool Connection`: `CheckTx`
 - `Info Connection`: `Info, SetOption, Query`

 The `Consensus Connection` is driven by a consensus protocol and is responsible
 for block execution.
 The `Mempool Connection` is for validating new transactions, before they're
 shared or included in a block.
 The `Info Connection` is for initialization and for queries from the user.

 Additionally, there is a `Flush` method that is called on every connection,
 and an `Echo` method that is just for debugging.

 More details on managing state across connections can be found in the section on
 [ABCI Applications](apps.md).

 ## Errors

 Some methods (`Echo, Info, InitChain, BeginBlock, EndBlock, Commit`),
 don't return errors because an error would indicate a critical failure
 in the application and there's nothing Tendermint can do. The problem
 should be addressed and both Tendermint and the application restarted.

 All other methods (`SetOption, Query, CheckTx, DeliverTx`) return an
 application-specific response `Code uint32`, where only `0` is reserved
 for `OK`.

 Finally, `Query`, `CheckTx`, and `DeliverTx` include a `Codespace string`, whose
 intended use is to disambiguate `Code` values returned by different domains of the
 application. The `Codespace` is a namespace for the `Code`.

 ## Events

 Some methods (`CheckTx, BeginBlock, DeliverTx, EndBlock`)
 include an `Events` field in their `Response*`. Each event contains a type and a
 list of attributes, which are key-value pairs denoting something about what happened
 during the method's execution.

 Events can be used to index transactions and blocks according to what happened
 during their execution. Note that the set of events returned for a block from
 `BeginBlock` and `EndBlock` are merged. In case both methods return the same
 tag, only the value defined in `EndBlock` is used.

 Each event has a `type` which is meant to categorize the event for a particular
 `Response*` or tx. A `Response*` or tx may contain multiple events with duplicate
 `type` values, where each distinct entry is meant to categorize attributes for a
 particular event. Every key and value in an event's attributes must be UTF-8
 encoded strings along with the even type itself.

 Example:

 ```go
 abci.ResponseDeliverTx{
 	// ...
 	Events: []abci.Event{
 		{
 			Type: "validator.provisions",
 			Attributes: cmn.KVPairs{
 				cmn.KVPair{Key: []byte("address"), Value: []byte("...")},
 				cmn.KVPair{Key: []byte("amount"), Value: []byte("...")},
 				cmn.KVPair{Key: []byte("balance"), Value: []byte("...")},
 			},
 		},
 		{
 			Type: "validator.provisions",
 			Attributes: cmn.KVPairs{
 				cmn.KVPair{Key: []byte("address"), Value: []byte("...")},
 				cmn.KVPair{Key: []byte("amount"), Value: []byte("...")},
 				cmn.KVPair{Key: []byte("balance"), Value: []byte("...")},
 			},
 		},
 		{
 			Type: "validator.slashed",
 			Attributes: cmn.KVPairs{
 				cmn.KVPair{Key: []byte("address"), Value: []byte("...")},
 				cmn.KVPair{Key: []byte("amount"), Value: []byte("...")},
 				cmn.KVPair{Key: []byte("reason"), Value: []byte("...")},
 			},
 		},
 		// ...
 	},
 }
 ```

 ## Determinism

 ABCI applications must implement deterministic finite-state machines to be
 securely replicated by the Tendermint consensus. This means block execution
 over the Consensus Connection must be strictly deterministic: given the same
 ordered set of requests, all nodes will compute identical responses, for all
 BeginBlock, DeliverTx, EndBlock, and Commit. This is critical, because the
 responses are included in the header of the next block, either via a Merkle root
 or directly, so all nodes must agree on exactly what they are.

 For this reason, it is recommended that applications not be exposed to any
 external user or process except via the ABCI connections to a consensus engine
 like Tendermint Core. The application must only change its state based on input
 from block execution (BeginBlock, DeliverTx, EndBlock, Commit), and not through
 any other kind of request. This is the only way to ensure all nodes see the same
 transactions and compute the same results.

 If there is some non-determinism in the state machine, consensus will eventually
 fail as nodes disagree over the correct values for the block header. The
 non-determinism must be fixed and the nodes restarted.

 Sources of non-determinism in applications may include:

 - Hardware failures
  - Cosmic rays, overheating, etc.
 - Node-dependent state
  - Random numbers
  - Time
 - Underspecification
  - Library version changes
  - Race conditions
  - Floating point numbers
  - JSON serialization
  - Iterating through hash-tables/maps/dictionaries
 - External Sources
  - Filesystem
  - Network calls (eg. some external REST API service)

 See [#56](https://github.com/tendermint/abci/issues/56) for original discussion.

 Note that some methods (`SetOption, Query, CheckTx, DeliverTx`) return
 explicitly non-deterministic data in the form of `Info` and `Log` fields. The `Log` is
 intended for the literal output from the application's logger, while the
 `Info` is any additional info that should be returned. These are the only fields
 that are not included in block header computations, so we don't need agreement
 on them. All other fields in the `Response*` must be strictly deterministic.

 ## Block Execution

 The first time a new blockchain is started, Tendermint calls
 `InitChain`. From then on, the following sequence of methods is executed for each
 block:

 `BeginBlock, [DeliverTx], EndBlock, Commit`

 where one `DeliverTx` is called for each transaction in the block.
 The result is an updated application state.
 Cryptographic commitments to the results of DeliverTx, EndBlock, and
 Commit are included in the header of the next block.

 ## Messages

 ### Echo

 - **Request**:
  - `Message (string)`: A string to echo back
 - **Response**:
  - `Message (string)`: The input string
 - **Usage**:
  - Echo a string to test an abci client/server implementation

 ### Flush

 - **Usage**:
  - Signals that messages queued on the client should be flushed to
    the server. It is called periodically by the client
    implementation to ensure asynchronous requests are actually
    sent, and is called immediately to make a synchronous request,
    which returns when the Flush response comes back.

 ### Info

 - **Request**:
  - `Version (string)`: The Tendermint software semantic version
  - `BlockVersion (uint64)`: The Tendermint Block Protocol version
  - `P2PVersion (uint64)`: The Tendermint P2P Protocol version
 - **Response**:
  - `Data (string)`: Some arbitrary information
  - `Version (string)`: The application software semantic version
  - `AppVersion (uint64)`: The application protocol version
  - `LastBlockHeight (int64)`: Latest block for which the app has
    called Commit
  - `LastBlockAppHash ([]byte)`: Latest result of Commit
 - **Usage**:
  - Return information about the application state.
  - Used to sync Tendermint with the application during a handshake
    that happens on startup.
  - The returned `AppVersion` will be included in the Header of every block.
  - Tendermint expects `LastBlockAppHash` and `LastBlockHeight` to
    be updated during `Commit`, ensuring that `Commit` is never
    called twice for the same block height.

 ### SetOption

 - **Request**:
  - `Key (string)`: Key to set
  - `Value (string)`: Value to set for key
 - **Response**:
  - `Code (uint32)`: Response code
  - `Log (string)`: The output of the application's logger. May
    be non-deterministic.
  - `Info (string)`: Additional information. May
    be non-deterministic.
 - **Usage**:
  - Set non-consensus critical application specific options.
  - e.g. Key="min-fee", Value="100fermion" could set the minimum fee
    required for CheckTx (but not DeliverTx - that would be
    consensus critical).

 ### InitChain

 - **Request**:
  - `Time (google.protobuf.Timestamp)`: Genesis time.
  - `ChainID (string)`: ID of the blockchain.
  - `ConsensusParams (ConsensusParams)`: Initial consensus-critical parameters.
  - `Validators ([]ValidatorUpdate)`: Initial genesis validators.
  - `AppStateBytes ([]byte)`: Serialized initial application state. Amino-encoded JSON bytes.
 - **Response**:
  - `ConsensusParams (ConsensusParams)`: Initial
    consensus-critical parameters.
  - `Validators ([]ValidatorUpdate)`: Initial validator set (if non empty).
 - **Usage**:
  - Called once upon genesis.
  - If ResponseInitChain.Validators is empty, the initial validator set will be the RequestInitChain.Validators
  - If ResponseInitChain.Validators is not empty, the initial validator set will be the
    ResponseInitChain.Validators (regardless of what is in RequestInitChain.Validators).
  - This allows the app to decide if it wants to accept the initial validator
    set proposed by tendermint (ie. in the genesis file), or if it wants to use
    a different one (perhaps computed based on some application specific
    information in the genesis file).

 ### Query

 - **Request**:
  - `Data ([]byte)`: Raw query bytes. Can be used with or in lieu
    of Path.
  - `Path (string)`: Path of request, like an HTTP GET path. Can be
    used with or in liue of Data.
    - Apps MUST interpret '/store' as a query by key on the
      underlying store. The key SHOULD be specified in the Data field.
    - Apps SHOULD allow queries over specific types like
      '/accounts/...' or '/votes/...'
  - `Height (int64)`: The block height for which you want the query
    (default=0 returns data for the latest committed block). Note
    that this is the height of the block containing the
    application's Merkle root hash, which represents the state as it
    was after committing the block at Height-1
  - `Prove (bool)`: Return Merkle proof with response if possible
 - **Response**:
  - `Code (uint32)`: Response code.
  - `Log (string)`: The output of the application's logger. May
    be non-deterministic.
  - `Info (string)`: Additional information. May
    be non-deterministic.
  - `Index (int64)`: The index of the key in the tree.
  - `Key ([]byte)`: The key of the matching data.
  - `Value ([]byte)`: The value of the matching data.
  - `Proof (Proof)`: Serialized proof for the value data, if requested, to be
    verified against the `AppHash` for the given Height.
  - `Height (int64)`: The block height from which data was derived.
    Note that this is the height of the block containing the
    application's Merkle root hash, which represents the state as it
    was after committing the block at Height-1
  - `Codespace (string)`: Namespace for the `Code`.
 - **Usage**:
  - Query for data from the application at current or past height.
  - Optionally return Merkle proof.
  - Merkle proof includes self-describing `type` field to support many types
    of Merkle trees and encoding formats.

 ### BeginBlock

 - **Request**:
  - `Hash ([]byte)`: The block's hash. This can be derived from the
    block header.
  - `Header (struct{})`: The block header.
  - `LastCommitInfo (LastCommitInfo)`: Info about the last commit, including the
    round, and the list of validators and which ones signed the last block.
  - `ByzantineValidators ([]Evidence)`: List of evidence of
    validators that acted maliciously.
 - **Response**:
  - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing
 - **Usage**:
  - Signals the beginning of a new block. Called prior to
    any DeliverTxs.
  - The header contains the height, timestamp, and more - it exactly matches the
    Tendermint block header. We may seek to generalize this in the future.
  - The `LastCommitInfo` and `ByzantineValidators` can be used to determine
    rewards and punishments for the validators. NOTE validators here do not
    include pubkeys.

 ### CheckTx

 - **Request**:
  - `Tx ([]byte)`: The request transaction bytes
  - `Type (CheckTxType)`: What type of `CheckTx` request is this? At present,
    there are two possible values: `CheckTx_New` (the default, which says
    that a full check is required), and `CheckTx_Recheck` (when the mempool is
    initiating a normal recheck of a transaction).
 - **Response**:
  - `Code (uint32)`: Response code
  - `Data ([]byte)`: Result bytes, if any.
  - `Log (string)`: The output of the application's logger. May
    be non-deterministic.
  - `Info (string)`: Additional information. May
    be non-deterministic.
  - `GasWanted (int64)`: Amount of gas requested for transaction.
  - `GasUsed (int64)`: Amount of gas consumed by transaction.
  - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing
    transactions (eg. by account).
  - `Codespace (string)`: Namespace for the `Code`.
 - **Usage**:
  - Technically optional - not involved in processing blocks.
  - Guardian of the mempool: every node runs CheckTx before letting a
    transaction into its local mempool.
  - The transaction may come from an external user or another node
  - CheckTx need not execute the transaction in full, but rather a light-weight
    yet stateful validation, like checking signatures and account balances, but
    not running code in a virtual machine.
  - Transactions where `ResponseCheckTx.Code != 0` will be rejected - they will not be broadcast to
    other nodes or included in a proposal block.
  - Tendermint attributes no other value to the response code

 ### DeliverTx

 - **Request**:
  - `Tx ([]byte)`: The request transaction bytes.
 - **Response**:
  - `Code (uint32)`: Response code.
  - `Data ([]byte)`: Result bytes, if any.
  - `Log (string)`: The output of the application's logger. May
    be non-deterministic.
  - `Info (string)`: Additional information. May
    be non-deterministic.
  - `GasWanted (int64)`: Amount of gas requested for transaction.
  - `GasUsed (int64)`: Amount of gas consumed by transaction.
  - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing
    transactions (eg. by account).
  - `Codespace (string)`: Namespace for the `Code`.
 - **Usage**:
  - The workhorse of the application - non-optional.
  - Execute the transaction in full.
  - `ResponseDeliverTx.Code == 0` only if the transaction is fully valid.

 ### EndBlock

 - **Request**:
  - `Height (int64)`: Height of the block just executed.
 - **Response**:
  - `ValidatorUpdates ([]ValidatorUpdate)`: Changes to validator set (set
    voting power to 0 to remove).
  - `ConsensusParamUpdates (ConsensusParams)`: Changes to
    consensus-critical time, size, and other parameters.
  - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing
 - **Usage**:
  - Signals the end of a block.
  - Called after all transactions, prior to each Commit.
  - Validator updates returned by block `H` impact blocks `H+1`, `H+2`, and
    `H+3`, but only effects changes on the validator set of `H+2`:
    - `H+1`: NextValidatorsHash
    - `H+2`: ValidatorsHash (and thus the validator set)
    - `H+3`: LastCommitInfo (ie. the last validator set)
  - Consensus params returned for block `H` apply for block `H+1`

 ### Commit

 - **Response**:
  - `Data ([]byte)`: The Merkle root hash of the application state
 - **Usage**:
  - Persist the application state.
  - Return an (optional) Merkle root hash of the application state
  - `ResponseCommit.Data` is included as the `Header.AppHash` in the next block
    - it may be empty
  - Later calls to `Query` can return proofs about the application state anchored
    in this Merkle root hash
  - Note developers can return whatever they want here (could be nothing, or a
    constant string, etc.), so long as it is deterministic - it must not be a
    function of anything that did not come from the
    BeginBlock/DeliverTx/EndBlock methods.

 ## Data Types

 ### Header

 - **Fields**:
  - `Version (Version)`: Version of the blockchain and the application
  - `ChainID (string)`: ID of the blockchain
  - `Height (int64)`: Height of the block in the chain
  - `Time (google.protobuf.Timestamp)`: Time of the previous block.
    For heights > 1, it's the weighted median of the timestamps of the valid
    votes in the block.LastCommit.
    For height == 1, it's genesis time.
  - `NumTxs (int32)`: Number of transactions in the block
  - `TotalTxs (int64)`: Total number of transactions in the blockchain until
    now
  - `LastBlockID (BlockID)`: Hash of the previous (parent) block
  - `LastCommitHash ([]byte)`: Hash of the previous block's commit
  - `ValidatorsHash ([]byte)`: Hash of the validator set for this block
  - `NextValidatorsHash ([]byte)`: Hash of the validator set for the next block
  - `ConsensusHash ([]byte)`: Hash of the consensus parameters for this block
  - `AppHash ([]byte)`: Data returned by the last call to `Commit` - typically the
    Merkle root of the application state after executing the previous block's
    transactions
  - `LastResultsHash ([]byte)`: Hash of the ABCI results returned by the last block
  - `EvidenceHash ([]byte)`: Hash of the evidence included in this block
  - `ProposerAddress ([]byte)`: Original proposer for the block
 - **Usage**:
  - Provided in RequestBeginBlock
  - Provides important context about the current state of the blockchain -
    especially height and time.
  - Provides the proposer of the current block, for use in proposer-based
    reward mechanisms.

 ### Version

 - **Fields**:
  - `Block (uint64)`: Protocol version of the blockchain data structures.
  - `App (uint64)`: Protocol version of the application.
 - **Usage**:
  - Block version should be static in the life of a blockchain.
  - App version may be updated over time by the application.

 ### Validator

 - **Fields**:
  - `Address ([]byte)`: Address of the validator (hash of the public key)
  - `Power (int64)`: Voting power of the validator
 - **Usage**:
  - Validator identified by address
  - Used in RequestBeginBlock as part of VoteInfo
  - Does not include PubKey to avoid sending potentially large quantum pubkeys
    over the ABCI

 ### ValidatorUpdate

 - **Fields**:
  - `PubKey (PubKey)`: Public key of the validator
  - `Power (int64)`: Voting power of the validator
 - **Usage**:
  - Validator identified by PubKey
  - Used to tell Tendermint to update the validator set

 ### VoteInfo

 - **Fields**:
  - `Validator (Validator)`: A validator
  - `SignedLastBlock (bool)`: Indicates whether or not the validator signed
    the last block
 - **Usage**:
  - Indicates whether a validator signed the last block, allowing for rewards
    based on validator availability

 ### PubKey

 - **Fields**:
  - `Type (string)`: Type of the public key. A simple string like `"ed25519"`.
    In the future, may indicate a serialization algorithm to parse the `Data`,
    for instance `"amino"`.
  - `Data ([]byte)`: Public key data. For a simple public key, it's just the
    raw bytes. If the `Type` indicates an encoding algorithm, this is the
    encoded public key.
 - **Usage**:
  - A generic and extensible typed public key

 ### Evidence

 - **Fields**:
  - `Type (string)`: Type of the evidence. A hierarchical path like
    "duplicate/vote".
  - `Validator (Validator`: The offending validator
  - `Height (int64)`: Height when the offense was committed
  - `Time (google.protobuf.Timestamp)`: Time of the block at height `Height`.
    It is the proposer's local time when block was created.
  - `TotalVotingPower (int64)`: Total voting power of the validator set at
    height `Height`

 ### LastCommitInfo

 - **Fields**:
  - `Round (int32)`: Commit round.
  - `Votes ([]VoteInfo)`: List of validators addresses in the last validator set
    with their voting power and whether or not they signed a vote.

 ### ConsensusParams

 - **Fields**:
  - `Block (BlockParams)`: Parameters limiting the size of a block and time between consecutive blocks.
  - `Evidence (EvidenceParams)`: Parameters limiting the validity of
    evidence of byzantine behaviour.
  - `Validator (ValidatorParams)`: Parameters limitng the types of pubkeys validators can use.

 ### BlockParams

 - **Fields**:
  - `MaxBytes (int64)`: Max size of a block, in bytes.
  - `MaxGas (int64)`: Max sum of `GasWanted` in a proposed block.
    - NOTE: blocks that violate this may be committed if there are Byzantine proposers.
      It's the application's responsibility to handle this when processing a
      block!

 ### EvidenceParams

 - **Fields**:
  - `MaxAge (int64)`: Max age of evidence, in blocks. Evidence older than this
    is considered stale and ignored.
    - This should correspond with an app's "unbonding period" or other
      similar mechanism for handling Nothing-At-Stake attacks.
    - NOTE: this should change to time (instead of blocks)!

 ### ValidatorParams

 - **Fields**:
  - `PubKeyTypes ([]string)`: List of accepted pubkey types. Uses same
    naming as `PubKey.Type`.

 ### Proof

 - **Fields**:
  - `Ops ([]ProofOp)`: List of chained Merkle proofs, of possibly different types
    - The Merkle root of one op is the value being proven in the next op.
    - The Merkle root of the final op should equal the ultimate root hash being
      verified against.

 ### ProofOp

 - **Fields**:
  - `Type (string)`: Type of Merkle proof and how it's encoded.
  - `Key ([]byte)`: Key in the Merkle tree that this proof is for.
  - `Data ([]byte)`: Encoded Merkle proof for the key.
--- a/spec/abci/apps.md
+++ b/spec/abci/apps.md
@ -0,0 +1,453 @@
 # Applications

 Please ensure you've first read the spec for [ABCI Methods and Types](abci.md)

 Here we cover the following components of ABCI applications:

 - [Connection State](#state) - the interplay between ABCI connections and application state
  and the differences between `CheckTx` and `DeliverTx`.
 - [Transaction Results](#transaction-results) - rules around transaction
  results and validity
 - [Validator Set Updates](#validator-updates) - how validator sets are
  changed during `InitChain` and `EndBlock`
 - [Query](#query) - standards for using the `Query` method and proofs about the
  application state
 - [Crash Recovery](#crash-recovery) - handshake protocol to synchronize
  Tendermint and the application on startup.

 ## State

 Since Tendermint maintains three concurrent ABCI connections, it is typical
 for an application to maintain a distinct state for each, and for the states to
 be synchronized during `Commit`.

 ### Commit

 Application state should only be persisted to disk during `Commit`.

 Before `Commit` is called, Tendermint locks and flushes the mempool so that no new messages will
 be received on the mempool connection. This provides an opportunity to safely update all three
 states to the latest committed state at once.

 When `Commit` completes, it unlocks the mempool.

 WARNING: if the ABCI app logic processing the `Commit` message sends a
 `/broadcast_tx_sync` or `/broadcast_tx_commit` and waits for the response
 before proceeding, it will deadlock. Executing those `broadcast_tx` calls
 involves acquiring a lock that is held during the `Commit` call, so it's not
 possible. If you make the call to the `broadcast_tx` endpoints concurrently,
 that's no problem, it just can't be part of the sequential logic of the
 `Commit` function.

 ### Consensus Connection

 The Consensus Connection should maintain a `DeliverTxState` -
 the working state for block execution. It should be updated by the calls to
 `BeginBlock`, `DeliverTx`, and `EndBlock` during block execution and committed to
 disk as the "latest committed state" during `Commit`.

 Updates made to the DeliverTxState by each method call must be readable by each subsequent method -
 ie. the updates are linearizable.

 ### Mempool Connection

 The Mempool Connection should maintain a `CheckTxState`
 to sequentially process pending transactions in the mempool that have
 not yet been committed. It should be initialized to the latest committed state
 at the end of every `Commit`.

 The CheckTxState may be updated concurrently with the DeliverTxState, as
 messages may be sent concurrently on the Consensus and Mempool connections. However,
 before calling `Commit`, Tendermint will lock and flush the mempool connection,
 ensuring that all existing CheckTx are responded to and no new ones can
 begin.

 After `Commit`, CheckTx is run again on all transactions that remain in the
 node's local mempool after filtering those included in the block. To prevent the
 mempool from rechecking all transactions every time a block is committed, set
 the configuration option `mempool.recheck=false`. As of Tendermint v0.32.1,
 an additional `Type` parameter is made available to the CheckTx function that
 indicates whether an incoming transaction is new (`CheckTxType_New`), or a
 recheck (`CheckTxType_Recheck`).

 Finally, the mempool will unlock and new transactions can be processed through CheckTx again.

 Note that CheckTx doesn't have to check everything that affects transaction validity; the
 expensive things can be skipped. In fact, CheckTx doesn't have to check
 anything; it might say that any transaction is a valid transaction.
 Unlike DeliverTx, CheckTx is just there as
 a sort of weak filter to keep invalid transactions out of the blockchain. It's
 weak, because a Byzantine node doesn't care about CheckTx; it can propose a
 block full of invalid transactions if it wants.

 ### Info Connection

 The Info Connection should maintain a `QueryState` for answering queries from the user,
 and for initialization when Tendermint first starts up (both described further
 below).
 It should always contain the latest committed state associated with the
 latest committed block.

 QueryState should be set to the latest `DeliverTxState` at the end of every `Commit`,
 ie. after the full block has been processed and the state committed to disk.
 Otherwise it should never be modified.

 ## Transaction Results

 `ResponseCheckTx` and `ResponseDeliverTx` contain the same fields.

 The `Info` and `Log` fields are non-deterministic values for debugging/convenience purposes
 that are otherwise ignored.

 The `Data` field must be strictly deterministic, but can be arbitrary data.

 ### Gas

 Ethereum introduced the notion of `gas` as an abstract representation of the
 cost of resources used by nodes when processing transactions. Every operation in the
 Ethereum Virtual Machine uses some amount of gas, and gas can be accepted at a market-variable price.
 Users propose a maximum amount of gas for their transaction; if the tx uses less, they get
 the difference credited back. Tendermint adopts a similar abstraction,
 though uses it only optionally and weakly, allowing applications to define
 their own sense of the cost of execution.

 In Tendermint, the `ConsensusParams.Block.MaxGas` limits the amount of `gas` that can be used in a block.
 The default value is `-1`, meaning no limit, or that the concept of gas is
 meaningless.

 Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum
 amount of gas the sender of a tx is willing to use, and the later is how much it actually
 used. Applications should enforce that `GasUsed <= GasWanted` - ie. tx execution
 should halt before it can use more resources than it requested.

 When `MaxGas > -1`, Tendermint enforces the following rules:

 - `GasWanted <= MaxGas` for all txs in the mempool
 - `(sum of GasWanted in a block) <= MaxGas` when proposing a block

 If `MaxGas == -1`, no rules about gas are enforced.

 Note that Tendermint does not currently enforce anything about Gas in the consensus, only the mempool.
 This means it does not guarantee that committed blocks satisfy these rules!
 It is the application's responsibility to return non-zero response codes when gas limits are exceeded.

 The `GasUsed` field is ignored completely by Tendermint. That said, applications should enforce:

 - `GasUsed <= GasWanted` for any given transaction
 - `(sum of GasUsed in a block) <= MaxGas` for every block

 In the future, we intend to add a `Priority` field to the responses that can be
 used to explicitly prioritize txs in the mempool for inclusion in a block
 proposal. See [#1861](https://github.com/tendermint/tendermint/issues/1861).

 ### CheckTx

 If `Code != 0`, it will be rejected from the mempool and hence
 not broadcasted to other peers and not included in a proposal block.

 `Data` contains the result of the CheckTx transaction execution, if any. It is
 semantically meaningless to Tendermint.

 `Tags` include any tags for the execution, though since the transaction has not
 been committed yet, they are effectively ignored by Tendermint.

 ### DeliverTx

 If DeliverTx returns `Code != 0`, the transaction will be considered invalid,
 though it is still included in the block.

 `Data` contains the result of the CheckTx transaction execution, if any. It is
 semantically meaningless to Tendermint.

 Both the `Code` and `Data` are included in a structure that is hashed into the
 `LastResultsHash` of the next block header.

 `Tags` include any tags for the execution, which Tendermint will use to index
 the transaction by. This allows transactions to be queried according to what
 events took place during their execution.

 See issue [#1007](https://github.com/tendermint/tendermint/issues/1007) for how
 the tags will be hashed into the next block header.

 ## Validator Updates

 The application may set the validator set during InitChain, and update it during
 EndBlock.

 Note that the maximum total power of the validator set is bounded by
 `MaxTotalVotingPower = MaxInt64 / 8`. Applications are responsible for ensuring
 they do not make changes to the validator set that cause it to exceed this
 limit.

 Additionally, applications must ensure that a single set of updates does not contain any duplicates -
 a given public key can only appear in an update once. If an update includes
 duplicates, the block execution will fail irrecoverably.

 ### InitChain

 ResponseInitChain can return a list of validators.
 If the list is empty, Tendermint will use the validators loaded in the genesis
 file.
 If the list is not empty, Tendermint will use it for the validator set.
 This way the application can determine the initial validator set for the
 blockchain.

 ### EndBlock

 Updates to the Tendermint validator set can be made by returning
 `ValidatorUpdate` objects in the `ResponseEndBlock`:

 ```
 message ValidatorUpdate {
  PubKey pub_key
  int64 power
 }

 message PubKey {
  string type
  bytes  data
 }
 ```

 The `pub_key` currently supports only one type:

 - `type = "ed25519"` and `data = <raw 32-byte public key>`

 The `power` is the new voting power for the validator, with the
 following rules:

 - power must be non-negative
 - if power is 0, the validator must already exist, and will be removed from the
  validator set
 - if power is non-0:
  - if the validator does not already exist, it will be added to the validator
    set with the given power
  - if the validator does already exist, its power will be adjusted to the given power
 - the total power of the new validator set must not exceed MaxTotalVotingPower

 Note the updates returned in block `H` will only take effect at block `H+2`.

 ## Consensus Parameters

 ConsensusParams enforce certain limits in the blockchain, like the maximum size
 of blocks, amount of gas used in a block, and the maximum acceptable age of
 evidence. They can be set in InitChain and updated in EndBlock.

 ### Block.MaxBytes

 The maximum size of a complete Amino encoded block.
 This is enforced by Tendermint consensus.

 This implies a maximum tx size that is this MaxBytes, less the expected size of
 the header, the validator set, and any included evidence in the block.

 Must have `0 < MaxBytes < 100 MB`.

 ### Block.MaxGas

 The maximum of the sum of `GasWanted` in a proposed block.
 This is *not* enforced by Tendermint consensus.
 It is left to the app to enforce (ie. if txs are included past the
 limit, they should return non-zero codes). It is used by Tendermint to limit the
 txs included in a proposed block.

 Must have `MaxGas >= -1`.
 If `MaxGas == -1`, no limit is enforced.

 ### Block.TimeIotaMs

 The minimum time between consecutive blocks (in milliseconds).
 This is enforced by Tendermint consensus.

 Must have `TimeIotaMs > 0` to ensure time monotonicity.

 ### EvidenceParams.MaxAge

 This is the maximum age of evidence.
 This is enforced by Tendermint consensus.
 If a block includes evidence older than this, the block will be rejected
 (validators won't vote for it).

 Must have `MaxAge > 0`.

 ### Updates

 The application may set the ConsensusParams during InitChain, and update them during
 EndBlock. If the ConsensusParams is empty, it will be ignored. Each field
 that is not empty will be applied in full. For instance, if updating the
 Block.MaxBytes, applications must also set the other Block fields (like
 Block.MaxGas), even if they are unchanged, as they will otherwise cause the
 value to be updated to 0.

 #### InitChain

 ResponseInitChain includes a ConsensusParams.
 If its nil, Tendermint will use the params loaded in the genesis
 file. If it's not nil, Tendermint will use it.
 This way the application can determine the initial consensus params for the
 blockchain.

 #### EndBlock

 ResponseEndBlock includes a ConsensusParams.
 If its nil, Tendermint will do nothing.
 If it's not nil, Tendermint will use it.
 This way the application can update the consensus params over time.

 Note the updates returned in block `H` will take effect right away for block
 `H+1`.

 ## Query

 Query is a generic method with lots of flexibility to enable diverse sets
 of queries on application state. Tendermint makes use of Query to filter new peers
 based on ID and IP, and exposes Query to the user over RPC.

 Note that calls to Query are not replicated across nodes, but rather query the
 local node's state - hence they may return stale reads. For reads that require
 consensus, use a transaction.

 The most important use of Query is to return Merkle proofs of the application state at some height
 that can be used for efficient application-specific lite-clients.

 Note Tendermint has technically no requirements from the Query
 message for normal operation - that is, the ABCI app developer need not implement
 Query functionality if they do not wish too.

 ### Query Proofs

 The Tendermint block header includes a number of hashes, each providing an
 anchor for some type of proof about the blockchain. The `ValidatorsHash` enables
 quick verification of the validator set, the `DataHash` gives quick
 verification of the transactions included in the block, etc.

 The `AppHash` is unique in that it is application specific, and allows for
 application-specific Merkle proofs about the state of the application.
 While some applications keep all relevant state in the transactions themselves
 (like Bitcoin and its UTXOs), others maintain a separated state that is
 computed deterministically *from* transactions, but is not contained directly in
 the transactions themselves (like Ethereum contracts and accounts).
 For such applications, the `AppHash` provides a much more efficient way to verify lite-client proofs.

 ABCI applications can take advantage of more efficient lite-client proofs for
 their state as follows:

 - return the Merkle root of the deterministic application state in
 `ResponseCommit.Data`.
 - it will be included as the `AppHash` in the next block.
 - return efficient Merkle proofs about that application state in `ResponseQuery.Proof`
  that can be verified using the `AppHash` of the corresponding block.

 For instance, this allows an application's lite-client to verify proofs of
 absence in the application state, something which is much less efficient to do using the block hash.

 Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees,
 where the leaves of one tree are the root hashes of others. To support this, and
 the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure:

 ```
 message Proof {
  repeated ProofOp ops
 }

 message ProofOp {
  string type = 1;
  bytes key = 2;
  bytes data = 3;
 }
 ```

 Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`.
 This allows ABCI to support many different kinds of Merkle trees, encoding
 formats, and proofs (eg. of presence and absence) just by varying the `type`.
 The `data` contains the actual encoded proof, encoded according to the `type`.
 When verifying the full proof, the root hash for one ProofOp is the value being
 verified for the next ProofOp in the list. The root hash of the final ProofOp in
 the list should match the `AppHash` being verified against.

 ### Peer Filtering

 When Tendermint connects to a peer, it sends two queries to the ABCI application
 using the following paths, with no additional data:

 - `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and
  the port of the connection
 - `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the
  pubkey.Address() for the peer's PubKey)

 If either of these queries return a non-zero ABCI code, Tendermint will refuse
 to connect to the peer.

 ### Paths

 Queries are directed at paths, and may optionally include additional data.

 The expectation is for there to be some number of high level paths
 differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently,
 Tendermint only uses `/p2p`, for filtering peers. For more advanced use, see the
 implementation of
 [Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/v0.23.1/baseapp/baseapp.go#L333).

 ## Crash Recovery

 On startup, Tendermint calls the `Info` method on the Info Connection to get the latest
 committed state of the app. The app MUST return information consistent with the
 last block it succesfully completed Commit for.

 If the app succesfully committed block H but not H+1, then `last_block_height = H` and `last_block_app_hash = <hash returned by Commit for block H>`. If the app
 failed during the Commit of block H, then `last_block_height = H-1` and
 `last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>`.

 We now distinguish three heights, and describe how Tendermint syncs itself with
 the app.

 ```
 storeBlockHeight = height of the last block Tendermint saw a commit for
 stateBlockHeight = height of the last block for which Tendermint completed all
    block processing and saved all ABCI results to disk
 appBlockHeight = height of the last block for which ABCI app succesfully
    completed Commit
 ```

 Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight`
 Note also we never call Commit on an ABCI app twice for the same height.

 The procedure is as follows.

 First, some simple start conditions:

 If `appBlockHeight == 0`, then call InitChain.

 If `storeBlockHeight == 0`, we're done.

 Now, some sanity checks:

 If `storeBlockHeight < appBlockHeight`, error
 If `storeBlockHeight < stateBlockHeight`, panic
 If `storeBlockHeight > stateBlockHeight+1`, panic

 Now, the meat:

 If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`,
 replay all blocks in full from `appBlockHeight` to `storeBlockHeight`.
 This happens if we completed processing the block, but the app forgot its height.

 If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done.
 This happens if we crashed at an opportune spot.

 If `storeBlockHeight == stateBlockHeight+1`
 This happens if we started processing the block but didn't finish.

 If `appBlockHeight < stateBlockHeight`
    replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`,
    and replay the block at `storeBlockHeight` using the WAL.
 This happens if the app forgot the last block it committed.

 If `appBlockHeight == stateBlockHeight`,
    replay the last block (storeBlockHeight) in full.
 This happens if we crashed before the app finished Commit

 If `appBlockHeight == storeBlockHeight`
    update the state using the saved ABCI responses but dont run the block against the real app.
 This happens if we crashed after the app finished Commit but before Tendermint saved the state.

--- a/spec/abci/client-server.md
+++ b/spec/abci/client-server.md
@ -0,0 +1,109 @@
 # Client and Server

 This section is for those looking to implement their own ABCI Server, perhaps in
 a new programming language.

 You are expected to have read [ABCI Methods and Types](./abci.md) and [ABCI
 Applications](./apps.md).

 ## Message Protocol

 The message protocol consists of pairs of requests and responses defined in the
 [protobuf file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto).

 Some messages have no fields, while others may include byte-arrays, strings, integers,
 or custom protobuf types.

 For more details on protobuf, see the [documentation](https://developers.google.com/protocol-buffers/docs/overview).

 For each request, a server should respond with the corresponding
 response, where the order of requests is preserved in the order of
 responses.

 ## Server Implementations

 To use ABCI in your programming language of choice, there must be a ABCI
 server in that language. Tendermint supports three implementations of the ABCI, written in Go:

 - In-process (Golang only)
 - ABCI-socket
 - GRPC

 The latter two can be tested using the `abci-cli` by setting the `--abci` flag
 appropriately (ie. to `socket` or `grpc`).

 See examples, in various stages of maintenance, in
 [Go](https://github.com/tendermint/tendermint/tree/master/abci/server),
 [JavaScript](https://github.com/tendermint/js-abci),
 [Python](https://github.com/tendermint/tendermint/tree/master/abci/example/python3/abci),
 [C++](https://github.com/mdyring/cpp-tmsp), and
 [Java](https://github.com/jTendermint/jabci).

 ### In Process

 The simplest implementation uses function calls within Golang.
 This means ABCI applications written in Golang can be compiled with TendermintCore and run as a single binary.

 ### GRPC

 If GRPC is available in your language, this is the easiest approach,
 though it will have significant performance overhead.

 To get started with GRPC, copy in the [protobuf
 file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto)
 and compile it using the GRPC plugin for your language. For instance,
 for golang, the command is `protoc --go_out=plugins=grpc:. types.proto`.
 See the [grpc documentation for more details](http://www.grpc.io/docs/).
 `protoc` will autogenerate all the necessary code for ABCI client and
 server in your language, including whatever interface your application
 must satisfy to be used by the ABCI server for handling requests.

 Note the length-prefixing used in the socket implementation (TSP) does not apply for GRPC.

 ### TSP

 Tendermint Socket Protocol is an asynchronous, raw socket server which provides ordered message passing over unix or tcp.
 Messages are serialized using Protobuf3 and length-prefixed with a [signed Varint](https://developers.google.com/protocol-buffers/docs/encoding?csw=1#signed-integers)

 If GRPC is not available in your language, or you require higher
 performance, or otherwise enjoy programming, you may implement your own
 ABCI server using the Tendermint Socket Protocol. The first step is still to auto-generate the relevant data
 types and codec in your language using `protoc`. In addition to being proto3 encoded, messages coming over
 the socket are length-prefixed to facilitate use as a streaming protocol. proto3 doesn't have an
 official length-prefix standard, so we use our own. The first byte in
 the prefix represents the length of the Big Endian encoded length. The
 remaining bytes in the prefix are the Big Endian encoded length.

 For example, if the proto3 encoded ABCI message is 0xDEADBEEF (4
 bytes), the length-prefixed message is 0x0104DEADBEEF. If the proto3
 encoded ABCI message is 65535 bytes long, the length-prefixed message
 would be like 0x02FFFF....

 The benefit of using this `varint` encoding over the old version (where integers were encoded as `<len of len><big endian len>` is that
 it is the standard way to encode integers in Protobuf. It is also generally shorter.

 As noted above, this prefixing does not apply for GRPC.

 An ABCI server must also be able to support multiple connections, as
 Tendermint uses three connections.

 ### Async vs Sync

 The main ABCI server (ie. non-GRPC) provides ordered asynchronous messages.
 This is useful for DeliverTx and CheckTx, since it allows Tendermint to forward
 transactions to the app before it's finished processing previous ones.

 Thus, DeliverTx and CheckTx messages are sent asynchronously, while all other
 messages are sent synchronously.

 ## Client

 There are currently two use-cases for an ABCI client. One is a testing
 tool, as in the `abci-cli`, which allows ABCI requests to be sent via
 command line. The other is a consensus engine, such as Tendermint Core,
 which makes requests to the application every time a new transaction is
 received or a block is committed.

 It is unlikely that you will need to implement a client. For details of
 our client, see
 [here](https://github.com/tendermint/tendermint/tree/master/abci/client).