Add consesnus and blockchain specs, (#52)

- Open questions - Do we want to split lite client work from consesnsus - From the blockchain spec, is encoding nessecary in the spec Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
5 years ago · 9b3531d7d6
--- a/spec/blockchain/blockchain.md
+++ b/spec/blockchain/blockchain.md
@ -0,0 +1,504 @@
 # Blockchain

 Here we describe the data structures in the Tendermint blockchain and the rules for validating them.

 ## Data Structures

 The Tendermint blockchains consists of a short list of basic data types:

 - `Block`
 - `Header`
 - `Version`
 - `BlockID`
 - `Time`
 - `Data` (for transactions)
 - `Commit` and `Vote`
 - `EvidenceData` and `Evidence`

 ## Block

 A block consists of a header, transactions, votes (the commit),
 and a list of evidence of malfeasance (ie. signing conflicting votes).

 ```go
 type Block struct {
    Header      Header
    Txs         Data
    Evidence    EvidenceData
    LastCommit  Commit
 }
 ```

 Note the `LastCommit` is the set of votes that committed the last block.

 ## Header

 A block header contains metadata about the block and about the consensus, as well as commitments to
 the data in the current block, the previous block, and the results returned by the application:

 ```go
 type Header struct {
 	// basic block info
 	Version  Version
 	ChainID  string
 	Height   int64
 	Time     Time
 	NumTxs   int64
 	TotalTxs int64

 	// prev block info
 	LastBlockID BlockID

 	// hashes of block data
 	LastCommitHash []byte // commit from validators from the last block
 	DataHash       []byte // MerkleRoot of transaction hashes

 	// hashes from the app output from the prev block
 	ValidatorsHash     []byte // validators for the current block
 	NextValidatorsHash []byte // validators for the next block
 	ConsensusHash      []byte // consensus params for current block
 	AppHash            []byte // state after txs from the previous block
 	LastResultsHash    []byte // root hash of all results from the txs from the previous block

 	// consensus info
 	EvidenceHash    []byte // evidence included in the block
 	ProposerAddress []byte // original proposer of the block
 ```

 Further details on each of these fields is described below.

 ## Version

 The `Version` contains the protocol version for the blockchain and the
 application as two `uint64` values:

 ```go
 type Version struct {
    Block   uint64
    App     uint64
 }
 ```

 ## BlockID

 The `BlockID` contains two distinct Merkle roots of the block.
 The first, used as the block's main hash, is the MerkleRoot
 of all the fields in the header (ie. `MerkleRoot(header)`.
 The second, used for secure gossipping of the block during consensus,
 is the MerkleRoot of the complete serialized block
 cut into parts (ie. `MerkleRoot(MakeParts(block))`).
 The `BlockID` includes these two hashes, as well as the number of
 parts (ie. `len(MakeParts(block))`)

 ```go
 type BlockID struct {
    Hash []byte
    PartsHeader PartSetHeader
 }

 type PartSetHeader struct {
    Total int32
    Hash []byte
 }
 ```

 See [MerkleRoot](./encoding.md#MerkleRoot) for details.

 ## Time

 Tendermint uses the
 [Google.Protobuf.WellKnownTypes.Timestamp](https://developers.google.com/protocol-buffers/docs/reference/csharp/class/google/protobuf/well-known-types/timestamp)
 format, which uses two integers, one for Seconds and for Nanoseconds.

 ## Data

 Data is just a wrapper for a list of transactions, where transactions are
 arbitrary byte arrays:

 ```
 type Data struct {
    Txs [][]byte
 }
 ```

 ## Commit

 Commit is a simple wrapper for a list of votes, with one vote for each
 validator. It also contains the relevant BlockID:

 ```
 type Commit struct {
    BlockID     BlockID
    Precommits  []Vote
 }
 ```

 NOTE: this will likely change to reduce the commit size by eliminating redundant
 information - see [issue #1648](https://github.com/tendermint/tendermint/issues/1648).

 ## Vote

 A vote is a signed message from a validator for a particular block.
 The vote includes information about the validator signing it.

 ```go
 type Vote struct {
 	Type             byte
 	Height           int64
 	Round            int
 	BlockID          BlockID
 	Timestamp        Time
 	ValidatorAddress []byte
 	ValidatorIndex   int
 	Signature        []byte
 }
 ```

 There are two types of votes:
 a _prevote_ has `vote.Type == 1` and
 a _precommit_ has `vote.Type == 2`.

 ## Signature

 Signatures in Tendermint are raw bytes representing the underlying signature.

 See the [signature spec](./encoding.md#key-types) for more.

 ## EvidenceData

 EvidenceData is a simple wrapper for a list of evidence:

 ```go
 type EvidenceData struct {
    Evidence []Evidence
 }
 ```

 ## Evidence

 Evidence in Tendermint is implemented as an interface.
 This means any evidence is encoded using its Amino prefix.
 There is currently only a single type, the `DuplicateVoteEvidence`.

 ```
 // amino name: "tendermint/DuplicateVoteEvidence"
 type DuplicateVoteEvidence struct {
 	PubKey PubKey
 	VoteA  Vote
 	VoteB  Vote
 }
 ```

 See the [pubkey spec](./encoding.md#key-types) for more.

 ## Validation

 Here we describe the validation rules for every element in a block.
 Blocks which do not satisfy these rules are considered invalid.

 We abuse notation by using something that looks like Go, supplemented with English.
 A statement such as `x == y` is an assertion - if it fails, the item is invalid.

 We refer to certain globally available objects:
 `block` is the block under consideration,
 `prevBlock` is the `block` at the previous height,
 and `state` keeps track of the validator set, the consensus parameters
 and other results from the application. At the point when `block` is the block under consideration,
 the current version of the `state` corresponds to the state
 after executing transactions from the `prevBlock`.
 Elements of an object are accessed as expected,
 ie. `block.Header`.
 See the [definition of `State`](./state.md).

 ### Header

 A Header is valid if its corresponding fields are valid.

 ### Version

 ```
 block.Version.Block == state.Version.Block
 block.Version.App == state.Version.App
 ```

 The block version must match the state version.

 ### ChainID

 ```
 len(block.ChainID) < 50
 ```

 ChainID must be less than 50 bytes.

 ### Height

 ```go
 block.Header.Height > 0
 block.Header.Height == prevBlock.Header.Height + 1
 ```

 The height is an incrementing integer. The first block has `block.Header.Height == 1`.

 ### Time

 ```
 block.Header.Timestamp >= prevBlock.Header.Timestamp + state.consensusParams.Block.TimeIotaMs
 block.Header.Timestamp == MedianTime(block.LastCommit, state.LastValidators)
 ```

 The block timestamp must be monotonic.
 It must equal the weighted median of the timestamps of the valid votes in the block.LastCommit.

 Note: the timestamp of a vote must be greater by at least one millisecond than that of the
 block being voted on.

 The timestamp of the first block must be equal to the genesis time (since
 there's no votes to compute the median).

 ```
 if block.Header.Height == 1 {
    block.Header.Timestamp == genesisTime
 }
 ```

 See the section on [BFT time](../consensus/bft-time.md) for more details.

 ### NumTxs

 ```go
 block.Header.NumTxs == len(block.Txs.Txs)
 ```

 Number of transactions included in the block.

 ### TotalTxs

 ```go
 block.Header.TotalTxs == prevBlock.Header.TotalTxs + block.Header.NumTxs
 ```

 The cumulative sum of all transactions included in this blockchain.

 The first block has `block.Header.TotalTxs = block.Header.NumberTxs`.

 ### LastBlockID

 LastBlockID is the previous block's BlockID:

 ```go
 prevBlockParts := MakeParts(prevBlock)
 block.Header.LastBlockID == BlockID {
    Hash: MerkleRoot(prevBlock.Header),
    PartsHeader{
        Hash: MerkleRoot(prevBlockParts),
        Total: len(prevBlockParts),
    },
 }
 ```

 The first block has `block.Header.LastBlockID == BlockID{}`.

 ### LastCommitHash

 ```go
 block.Header.LastCommitHash == MerkleRoot(block.LastCommit.Precommits)
 ```

 MerkleRoot of the votes included in the block.
 These are the votes that committed the previous block.

 The first block has `block.Header.LastCommitHash == []byte{}`

 ### DataHash

 ```go
 block.Header.DataHash == MerkleRoot(Hashes(block.Txs.Txs))
 ```

 MerkleRoot of the hashes of transactions included in the block.

 Note the transactions are hashed before being included in the Merkle tree,
 so the leaves of the Merkle tree are the hashes, not the transactions
 themselves. This is because transaction hashes are regularly used as identifiers for
 transactions.

 ### ValidatorsHash

 ```go
 block.ValidatorsHash == MerkleRoot(state.Validators)
 ```

 MerkleRoot of the current validator set that is committing the block.
 This can be used to validate the `LastCommit` included in the next block.
 Note the validators are sorted by their address before computing the MerkleRoot.

 ### NextValidatorsHash

 ```go
 block.NextValidatorsHash == MerkleRoot(state.NextValidators)
 ```

 MerkleRoot of the next validator set that will be the validator set that commits the next block.
 This is included so that the current validator set gets a chance to sign the
 next validator sets Merkle root.
 Note the validators are sorted by their address before computing the MerkleRoot.

 ### ConsensusHash

 ```go
 block.ConsensusHash == state.ConsensusParams.Hash()
 ```

 Hash of the amino-encoding of a subset of the consensus parameters.

 ### AppHash

 ```go
 block.AppHash == state.AppHash
 ```

 Arbitrary byte array returned by the application after executing and commiting the previous block. It serves as the basis for validating any merkle proofs that comes from the ABCI application and represents the state of the actual application rather than the state of the blockchain itself.

 The first block has `block.Header.AppHash == []byte{}`.

 ### LastResultsHash

 ```go
 block.ResultsHash == MerkleRoot(state.LastResults)
 ```

 MerkleRoot of the results of the transactions in the previous block.

 The first block has `block.Header.ResultsHash == []byte{}`.

 ## EvidenceHash

 ```go
 block.EvidenceHash == MerkleRoot(block.Evidence)
 ```

 MerkleRoot of the evidence of Byzantine behaviour included in this block.

 ### ProposerAddress

 ```go
 block.Header.ProposerAddress in state.Validators
 ```

 Address of the original proposer of the block. Must be a current validator.

 ## Txs

 Arbitrary length array of arbitrary length byte-arrays.

 ## LastCommit

 The first height is an exception - it requires the LastCommit to be empty:

 ```go
 if block.Header.Height == 1 {
    len(b.LastCommit) == 0
 }
 ```

 Otherwise, we require:

 ```go
 len(block.LastCommit) == len(state.LastValidators)
 talliedVotingPower := 0
 for i, vote := range block.LastCommit{
    if vote == nil{
        continue
    }
    vote.Type == 2
    vote.Height == block.LastCommit.Height()
    vote.Round == block.LastCommit.Round()
    vote.BlockID == block.LastBlockID

    val := state.LastValidators[i]
    vote.Verify(block.ChainID, val.PubKey) == true

    talliedVotingPower += val.VotingPower
 }

 talliedVotingPower > (2/3) * TotalVotingPower(state.LastValidators)
 ```

 Includes one (possibly nil) vote for every current validator.
 Non-nil votes must be Precommits.
 All votes must be for the same height and round.
 All votes must be for the previous block.
 All votes must have a valid signature from the corresponding validator.
 The sum total of the voting power of the validators that voted
 must be greater than 2/3 of the total voting power of the complete validator set.

 ### Vote

 A vote is a signed message broadcast in the consensus for a particular block at a particular height and round.
 When stored in the blockchain or propagated over the network, votes are encoded in Amino.
 For signing, votes are represented via `CanonicalVote` and also encoded using amino (protobuf compatible) via
 `Vote.SignBytes` which includes the `ChainID`, and uses a different ordering of
 the fields.

 We define a method `Verify` that returns `true` if the signature verifies against the pubkey for the `SignBytes`
 using the given ChainID:

 ```go
 func (vote *Vote) Verify(chainID string, pubKey crypto.PubKey) error {
 	if !bytes.Equal(pubKey.Address(), vote.ValidatorAddress) {
 		return ErrVoteInvalidValidatorAddress
 	}

 	if !pubKey.VerifyBytes(vote.SignBytes(chainID), vote.Signature) {
 		return ErrVoteInvalidSignature
 	}
 	return nil
 }
 ```

 where `pubKey.Verify` performs the appropriate digital signature verification of the `pubKey`
 against the given signature and message bytes.

 ## Evidence

 There is currently only one kind of evidence, `DuplicateVoteEvidence`.

 DuplicateVoteEvidence `ev` is valid if

 - `ev.VoteA` and `ev.VoteB` can be verified with `ev.PubKey`
 - `ev.VoteA` and `ev.VoteB` have the same `Height, Round, Address, Index, Type`
 - `ev.VoteA.BlockID != ev.VoteB.BlockID`
 - `(block.Height - ev.VoteA.Height) < MAX_EVIDENCE_AGE`

 # Execution

 Once a block is validated, it can be executed against the state.

 The state follows this recursive equation:

 ```go
 state(1) = InitialState
 state(h+1) <- Execute(state(h), ABCIApp, block(h))
 ```

 where `InitialState` includes the initial consensus parameters and validator set,
 and `ABCIApp` is an ABCI application that can return results and changes to the validator
 set (TODO). Execute is defined as:

 ```go
 Execute(s State, app ABCIApp, block Block) State {
    // Fuction ApplyBlock executes block of transactions against the app and returns the new root hash of the app state,
    // modifications to the validator set and the changes of the consensus parameters.
    AppHash, ValidatorChanges, ConsensusParamChanges := app.ApplyBlock(block)

    return State{
        LastResults: abciResponses.DeliverTxResults,
        AppHash: AppHash,
        LastValidators: state.Validators,
        Validators: state.NextValidators,
        NextValidators: UpdateValidators(state.NextValidators, ValidatorChanges),
        ConsensusParams: UpdateConsensusParams(state.ConsensusParams, ConsensusParamChanges),
    }
 }
 ```
--- a/spec/blockchain/encoding.md
+++ b/spec/blockchain/encoding.md
@ -0,0 +1,344 @@
 # Encoding

 ## Amino

 Tendermint uses the proto3 derivative [Amino](https://github.com/tendermint/go-amino) for all data structures.
 Think of Amino as an object-oriented proto3 with native JSON support.
 The goal of the Amino encoding protocol is to bring parity between application
 logic objects and persistence objects.

 Please see the [Amino
 specification](https://github.com/tendermint/go-amino#amino-encoding-for-go) for
 more details.

 Notably, every object that satisfies an interface (eg. a particular kind of p2p message,
 or a particular kind of pubkey) is registered with a global name, the hash of
 which is included in the object's encoding as the so-called "prefix bytes".

 We define the `func AminoEncode(obj interface{}) []byte` function to take an
 arbitrary object and return the Amino encoded bytes.

 ## Byte Arrays

 The encoding of a byte array is simply the raw-bytes prefixed with the length of
 the array as a `UVarint` (what proto calls a `Varint`).

 For details on varints, see the [protobuf
 spec](https://developers.google.com/protocol-buffers/docs/encoding#varints).

 For example, the byte-array `[0xA, 0xB]` would be encoded as `0x020A0B`,
 while a byte-array containing 300 entires beginning with `[0xA, 0xB, ...]` would
 be encoded as `0xAC020A0B...` where `0xAC02` is the UVarint encoding of 300.

 ## Hashing

 Tendermint uses `SHA256` as its hash function.
 Objects are always Amino encoded before being hashed.
 So `SHA256(obj)` is short for `SHA256(AminoEncode(obj))`.

 ## Public Key Cryptography

 Tendermint uses Amino to distinguish between different types of private keys,
 public keys, and signatures. Additionally, for each public key, Tendermint
 defines an Address function that can be used as a more compact identifier in
 place of the public key. Here we list the concrete types, their names,
 and prefix bytes for public keys and signatures, as well as the address schemes
 for each PubKey. Note for brevity we don't
 include details of the private keys beyond their type and name, as they can be
 derived the same way as the others using Amino.

 All registered objects are encoded by Amino using a 4-byte PrefixBytes that
 uniquely identifies the object and includes information about its underlying
 type. For details on how PrefixBytes are computed, see the [Amino
 spec](https://github.com/tendermint/go-amino#computing-the-prefix-and-disambiguation-bytes).

 In what follows, we provide the type names and prefix bytes directly.
 Notice that when encoding byte-arrays, the length of the byte-array is appended
 to the PrefixBytes. Thus the encoding of a byte array becomes `<PrefixBytes> <Length> <ByteArray>`. In other words, to encode any type listed below you do not need to be
 familiar with amino encoding.
 You can simply use below table and concatenate Prefix || Length (of raw bytes) || raw bytes
 ( while || stands for byte concatenation here).

 | Type                    | Name                               | Prefix     | Length   | Notes |
 | ----------------------- | ---------------------------------- | ---------- | -------- | ----- |
 | PubKeyEd25519           | tendermint/PubKeyEd25519           | 0x1624DE64 | 0x20     |       |
 | PubKeySecp256k1         | tendermint/PubKeySecp256k1         | 0xEB5AE987 | 0x21     |       |
 | PrivKeyEd25519          | tendermint/PrivKeyEd25519          | 0xA3288910 | 0x40     |       |
 | PrivKeySecp256k1        | tendermint/PrivKeySecp256k1        | 0xE1B0F79B | 0x20     |       |
 | PubKeyMultisigThreshold | tendermint/PubKeyMultisigThreshold | 0x22C1F7E2 | variable |       |

 ### Example

 For example, the 33-byte (or 0x21-byte in hex) Secp256k1 pubkey
 `020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9`
 would be encoded as
 `EB5AE98721020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9`

 ### Key Types

 Each type specifies it's own pubkey, address, and signature format.

 #### Ed25519

 TODO: pubkey

 The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key:

 ```
 address = SHA256(pubkey)[:20]
 ```

 The signature is the raw 64-byte ED25519 signature.

 #### Secp256k1

 TODO: pubkey

 The address is the RIPEMD160 hash of the SHA256 hash of the OpenSSL compressed public key:

 ```
 address = RIPEMD160(SHA256(pubkey))
 ```

 This is the same as Bitcoin.

 The signature is the 64-byte concatenation of ECDSA `r` and `s` (ie. `r || s`),
 where `s` is lexicographically less than its inverse, to prevent malleability.
 This is like Ethereum, but without the extra byte for pubkey recovery, since
 Tendermint assumes the pubkey is always provided anyway.

 #### Multisig

 TODO

 ## Other Common Types

 ### BitArray

 The BitArray is used in some consensus messages to represent votes received from
 validators, or parts received in a block. It is represented
 with a struct containing the number of bits (`Bits`) and the bit-array itself
 encoded in base64 (`Elems`).

 ```go
 type BitArray struct {
    Bits  int
    Elems []uint64
 }
 ```

 This type is easily encoded directly by Amino.

 Note BitArray receives a special JSON encoding in the form of `x` and `_`
 representing `1` and `0`. Ie. the BitArray `10110` would be JSON encoded as
 `"x_xx_"`

 ### Part

 Part is used to break up blocks into pieces that can be gossiped in parallel
 and securely verified using a Merkle tree of the parts.

 Part contains the index of the part (`Index`), the actual
 underlying data of the part (`Bytes`), and a Merkle proof that the part is contained in
 the set (`Proof`).

 ```go
 type Part struct {
    Index int
    Bytes []byte
    Proof SimpleProof
 }
 ```

 See details of SimpleProof, below.

 ### MakeParts

 Encode an object using Amino and slice it into parts.
 Tendermint uses a part size of 65536 bytes.

 ```go
 func MakeParts(block Block) []Part
 ```

 ## Merkle Trees

 For an overview of Merkle trees, see
 [wikipedia](https://en.wikipedia.org/wiki/Merkle_tree)

 We use the RFC 6962 specification of a merkle tree, with sha256 as the hash function.
 Merkle trees are used throughout Tendermint to compute a cryptographic digest of a data structure.
 The differences between RFC 6962 and the simplest form a merkle tree are that:

 1. leaf nodes and inner nodes have different hashes.
   This is for "second pre-image resistance", to prevent the proof to an inner node being valid as the proof of a leaf.
   The leaf nodes are `SHA256(0x00 || leaf_data)`, and inner nodes are `SHA256(0x01 || left_hash || right_hash)`.

 2. When the number of items isn't a power of two, the left half of the tree is as big as it could be.
   (The largest power of two less than the number of items) This allows new leaves to be added with less
   recomputation. For example:

 ```
   Simple Tree with 6 items           Simple Tree with 7 items

              *                                  *
             / \                                / \
           /     \                            /     \
         /         \                        /         \
       /             \                    /             \
      *               *                  *               *
     / \             / \                / \             / \
    /   \           /   \              /   \           /   \
   /     \         /     \            /     \         /     \
  *       *       h4     h5          *       *       *       h6
 / \     / \                        / \     / \     / \
 h0  h1  h2 h3                      h0  h1  h2  h3  h4  h5
 ```

 ### MerkleRoot

 The function `MerkleRoot` is a simple recursive function defined as follows:

 ```go
 // SHA256(0x00 || leaf)
 func leafHash(leaf []byte) []byte {
 	return tmhash.Sum(append(0x00, leaf...))
 }

 // SHA256(0x01 || left || right)
 func innerHash(left []byte, right []byte) []byte {
 	return tmhash.Sum(append(0x01, append(left, right...)...))
 }

 // largest power of 2 less than k
 func getSplitPoint(k int) { ... }

 func MerkleRoot(items [][]byte) []byte{
 	switch len(items) {
 	case 0:
 		return nil
 	case 1:
 		return leafHash(items[0])
 	default:
 		k := getSplitPoint(len(items))
 		left := MerkleRoot(items[:k])
 		right := MerkleRoot(items[k:])
 		return innerHash(left, right)
 	}
 }
 ```

 Note: `MerkleRoot` operates on items which are arbitrary byte arrays, not
 necessarily hashes. For items which need to be hashed first, we introduce the
 `Hashes` function:

 ```
 func Hashes(items [][]byte) [][]byte {
    return SHA256 of each item
 }
 ```

 Note: we will abuse notion and invoke `MerkleRoot` with arguments of type `struct` or type `[]struct`.
 For `struct` arguments, we compute a `[][]byte` containing the amino encoding of each
 field in the struct, in the same order the fields appear in the struct.
 For `[]struct` arguments, we compute a `[][]byte` by amino encoding the individual `struct` elements.

 ### Simple Merkle Proof

 Proof that a leaf is in a Merkle tree is composed as follows:

 ```golang
 type SimpleProof struct {
        Total int
        Index int
        LeafHash []byte
        Aunts [][]byte
 }
 ```

 Which is verified as follows:

 ```golang
 func (proof SimpleProof) Verify(rootHash []byte, leaf []byte) bool {
 	assert(proof.LeafHash, leafHash(leaf)

 	computedHash := computeHashFromAunts(proof.Index, proof.Total, proof.LeafHash, proof.Aunts)
    return computedHash == rootHash
 }

 func computeHashFromAunts(index, total int, leafHash []byte, innerHashes [][]byte) []byte{
 	assert(index < total && index >= 0 && total > 0)

 	if total == 1{
 		assert(len(proof.Aunts) == 0)
 		return leafHash
 	}

 	assert(len(innerHashes) > 0)

 	numLeft := getSplitPoint(total) // largest power of 2 less than total
 	if index < numLeft {
 		leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1])
 		assert(leftHash != nil)
 		return innerHash(leftHash, innerHashes[len(innerHashes)-1])
 	}
 	rightHash := computeHashFromAunts(index-numLeft, total-numLeft, leafHash, innerHashes[:len(innerHashes)-1])
 	assert(rightHash != nil)
 	return innerHash(innerHashes[len(innerHashes)-1], rightHash)
 }
 ```

 ### IAVL+ Tree

 Because Tendermint only uses a Simple Merkle Tree, application developers are expect to use their own Merkle tree in their applications. For example, the IAVL+ Tree - an immutable self-balancing binary tree for persisting application state is used by the [Cosmos SDK](https://github.com/cosmos/cosmos-sdk/blob/master/docs/clients/lite/specification.md)

 ## JSON

 ### Amino

 Amino also supports JSON encoding - registered types are simply encoded as:

 ```
 {
  "type": "<amino type name>",
  "value": <JSON>
 }
 ```

 For instance, an ED25519 PubKey would look like:

 ```
 {
  "type": "tendermint/PubKeyEd25519",
  "value": "uZ4h63OFWuQ36ZZ4Bd6NF+/w9fWUwrOncrQsackrsTk="
 }
 ```

 Where the `"value"` is the base64 encoding of the raw pubkey bytes, and the
 `"type"` is the amino name for Ed25519 pubkeys.

 ### Signed Messages

 Signed messages (eg. votes, proposals) in the consensus are encoded using Amino.

 When signing, the elements of a message are re-ordered so the fixed-length fields
 are first, making it easy to quickly check the type, height, and round.
 The `ChainID` is also appended to the end.
 We call this encoding the SignBytes. For instance, SignBytes for a vote is the Amino encoding of the following struct:

 ```go
 type CanonicalVote struct {
 	Type      byte
 	Height    int64            `binary:"fixed64"`
 	Round     int64            `binary:"fixed64"`
 	BlockID   CanonicalBlockID
 	Timestamp time.Time
 	ChainID   string
 }
 ```

 The field ordering and the fixed sized encoding for the first three fields is optimized to ease parsing of SignBytes
 in HSMs. It creates fixed offsets for relevant fields that need to be read in this context.
 For more details, see the [signing spec](../consensus/signing.md).
 Also, see the motivating discussion in
 [#1622](https://github.com/tendermint/tendermint/issues/1622).
--- a/spec/blockchain/state.md
+++ b/spec/blockchain/state.md
@ -0,0 +1,141 @@
 # State

 ## State

 The state contains information whose cryptographic digest is included in block headers, and thus is
 necessary for validating new blocks. For instance, the validators set and the results of
 transactions are never included in blocks, but their Merkle roots are - the state keeps track of them.

 Note that the `State` object itself is an implementation detail, since it is never
 included in a block or gossipped over the network, and we never compute
 its hash. Thus we do not include here details of how the `State` object is
 persisted or queried. That said, the types it contains are part of the specification, since
 their Merkle roots are included in blocks and their values are used in
 validation.

 ```go
 type State struct {
    Version     Version
    LastResults []Result
    AppHash []byte

    LastValidators []Validator
    Validators []Validator
    NextValidators []Validator

    ConsensusParams ConsensusParams
 }
 ```

 ### Result

 ```go
 type Result struct {
    Code uint32
    Data []byte
 }
 ```

 `Result` is the result of executing a transaction against the application.
 It returns a result code and an arbitrary byte array (ie. a return value).

 NOTE: the Result needs to be updated to include more fields returned from
 processing transactions, like gas variables and tags - see
 [issue 1007](https://github.com/tendermint/tendermint/issues/1007).

 ### Validator

 A validator is an active participant in the consensus with a public key and a voting power.
 Validator's also contain an address field, which is a hash digest of the PubKey.

 ```go
 type Validator struct {
    Address     []byte
    PubKey      PubKey
    VotingPower int64
 }
 ```

 When hashing the Validator struct, the address is not included,
 because it is redundant with the pubkey.

 The `state.Validators`, `state.LastValidators`, and `state.NextValidators`, must always be sorted by validator address,
 so that there is a canonical order for computing the MerkleRoot.

 We also define a `TotalVotingPower` function, to return the total voting power:

 ```go
 func TotalVotingPower(vals []Validators) int64{
    sum := 0
    for v := range vals{
        sum += v.VotingPower
    }
    return sum
 }
 ```

 ### ConsensusParams

 ConsensusParams define various limits for blockchain data structures.
 Like validator sets, they are set during genesis and can be updated by the application through ABCI.
 When hashed, only a subset of the params are included, to allow the params to
 evolve without breaking the header.

 ```go
 type ConsensusParams struct {
 	Block
 	Evidence
 	Validator
 }

 type hashedParams struct {
    BlockMaxBytes int64
    BlockMaxGas   int64
 }

 func (params ConsensusParams) Hash() []byte {
    SHA256(hashedParams{
        BlockMaxBytes: params.Block.MaxBytes,
        BlockMaxGas: params.Block.MaxGas,
    })
 }

 type BlockParams struct {
 	MaxBytes        int64
 	MaxGas          int64
  TimeIotaMs      int64
 }

 type EvidenceParams struct {
 	MaxAge int64
 }

 type ValidatorParams struct {
 	PubKeyTypes []string
 }
 ```

 #### Block

 The total size of a block is limited in bytes by the `ConsensusParams.Block.MaxBytes`.
 Proposed blocks must be less than this size, and will be considered invalid
 otherwise.

 Blocks should additionally be limited by the amount of "gas" consumed by the
 transactions in the block, though this is not yet implemented.

 The minimal time between consecutive blocks is controlled by the
 `ConsensusParams.Block.TimeIotaMs`.

 #### Evidence

 For evidence in a block to be valid, it must satisfy:

 ```
 block.Header.Height - evidence.Height < ConsensusParams.Evidence.MaxAge
 ```

 #### Validator

 Validators from genesis file and `ResponseEndBlock` must have pubkeys of type ∈
 `ConsensusParams.Validator.PubKeyTypes`.
--- a/spec/consensus/bft-time.md
+++ b/spec/consensus/bft-time.md
@ -0,0 +1,54 @@
 # BFT Time

 Tendermint provides a deterministic, Byzantine fault-tolerant, source of time. 
 Time in Tendermint is defined with the Time field of the block header. 

 It satisfies the following properties:

 - Time Monotonicity: Time is monotonically increasing, i.e., given 
 a header H1 for height h1 and a header H2 for height `h2 = h1 + 1`, `H1.Time < H2.Time`. 
 - Time Validity: Given a set of Commit votes that forms the `block.LastCommit` field, a range of 
 valid values for the Time field of the block header is defined only by  
 Precommit messages (from the LastCommit field) sent by correct processes, i.e., 
 a faulty process cannot arbitrarily increase the Time value.  

 In the context of Tendermint, time is of type int64 and denotes UNIX time in milliseconds, i.e., 
 corresponds to the number of milliseconds since January 1, 1970. Before defining rules that need to be enforced by the 
 Tendermint consensus protocol, so the properties above holds, we introduce the following definition:

 - median of a Commit is equal to the median of `Vote.Time` fields of the `Vote` messages,
 where the value of `Vote.Time` is counted number of times proportional to the process voting power. As in Tendermint
 the voting power is not uniform (one process one vote), a vote message is actually an aggregator of the same votes whose 
 number is equal to the voting power of the process that has casted the corresponding votes message.

 Let's consider the following example:
 - we have four processes p1, p2, p3 and p4, with the following voting power distribution (p1, 23), (p2, 27), (p3, 10)
 and (p4, 10). The total voting power is 70 (`N = 3f+1`, where `N` is the total voting power, and `f` is the maximum voting 
 power of the faulty processes), so we assume that the faulty processes have at most 23 of voting power. 
 Furthermore, we have the following vote messages in some LastCommit field (we ignore all fields except Time field): 
      - (p1, 100), (p2, 98), (p3, 1000), (p4, 500). We assume that p3 and p4 are faulty processes. Let's assume that the 
      `block.LastCommit` message contains votes of processes p2, p3 and p4. Median is then chosen the following way: 
      the value 98 is counted 27 times, the value 1000 is counted 10 times and the value 500 is counted also 10 times. 
      So the median value will be the value 98. No matter what set of messages with at least `2f+1` voting power we 
      choose, the median value will always be between the values sent by correct processes.   

 We ensure Time Monotonicity and Time Validity properties by the following rules: 
  
 - let rs denotes `RoundState` (consensus internal state) of some process. Then 
 `rs.ProposalBlock.Header.Time == median(rs.LastCommit) &&
 rs.Proposal.Timestamp == rs.ProposalBlock.Header.Time`.

 - Furthermore, when creating the `vote` message, the following rules for determining `vote.Time` field should hold: 

    - if `rs.LockedBlock` is defined then
    `vote.Time = max(rs.LockedBlock.Timestamp + config.BlockTimeIota, time.Now())`, where `time.Now()` 
        denotes local Unix time in milliseconds, and `config.BlockTimeIota` is a configuration parameter that corresponds
        to the minimum timestamp increment of the next block.
        
    - else if `rs.Proposal` is defined then 
    `vote.Time = max(rs.Proposal.Timestamp + config.BlockTimeIota, time.Now())`,
    
    - otherwise, `vote.Time = time.Now())`. In this case vote is for `nil` so it is not taken into account for 
    the timestamp of the next block. 


--- a/spec/consensus/consensus-paper/IEEEtran.bst
+++ b/spec/consensus/consensus-paper/IEEEtran.bst
--- a/spec/consensus/consensus-paper/IEEEtran.cls
+++ b/spec/consensus/consensus-paper/IEEEtran.cls
--- a/spec/consensus/consensus-paper/README.md
+++ b/spec/consensus/consensus-paper/README.md
--- a/spec/consensus/consensus-paper/algorithmicplus.sty
+++ b/spec/consensus/consensus-paper/algorithmicplus.sty
--- a/spec/consensus/consensus-paper/conclusion.tex
+++ b/spec/consensus/consensus-paper/conclusion.tex
--- a/spec/consensus/consensus-paper/consensus.tex
+++ b/spec/consensus/consensus-paper/consensus.tex
--- a/spec/consensus/consensus-paper/definitions.tex
+++ b/spec/consensus/consensus-paper/definitions.tex
--- a/spec/consensus/consensus-paper/homodel.sty
+++ b/spec/consensus/consensus-paper/homodel.sty
--- a/spec/consensus/consensus-paper/intro.tex
+++ b/spec/consensus/consensus-paper/intro.tex
--- a/spec/consensus/consensus-paper/latex8.bst
+++ b/spec/consensus/consensus-paper/latex8.bst
--- a/spec/consensus/consensus-paper/latex8.sty
+++ b/spec/consensus/consensus-paper/latex8.sty
--- a/spec/consensus/consensus-paper/lit.bib
+++ b/spec/consensus/consensus-paper/lit.bib
--- a/spec/consensus/consensus-paper/paper.tex
+++ b/spec/consensus/consensus-paper/paper.tex
--- a/spec/consensus/consensus-paper/proof.tex
+++ b/spec/consensus/consensus-paper/proof.tex
--- a/spec/consensus/consensus-paper/rounddiag.sty
+++ b/spec/consensus/consensus-paper/rounddiag.sty
--- a/spec/consensus/consensus-paper/technote.sty
+++ b/spec/consensus/consensus-paper/technote.sty
--- a/spec/consensus/consensus.md
+++ b/spec/consensus/consensus.md
@ -0,0 +1,339 @@
 # Byzantine Consensus Algorithm

 ## Terms

 - The network is composed of optionally connected _nodes_. Nodes
  directly connected to a particular node are called _peers_.
 - The consensus process in deciding the next block (at some _height_
  `H`) is composed of one or many _rounds_.
 - `NewHeight`, `Propose`, `Prevote`, `Precommit`, and `Commit`
  represent state machine states of a round. (aka `RoundStep` or
  just "step").
 - A node is said to be _at_ a given height, round, and step, or at
  `(H,R,S)`, or at `(H,R)` in short to omit the step.
 - To _prevote_ or _precommit_ something means to broadcast a [prevote
  vote](https://godoc.org/github.com/tendermint/tendermint/types#Vote)
  or [first precommit
  vote](https://godoc.org/github.com/tendermint/tendermint/types#FirstPrecommit)
  for something.
 - A vote _at_ `(H,R)` is a vote signed with the bytes for `H` and `R`
  included in its [sign-bytes](../blockchain/blockchain.md#vote).
 - _+2/3_ is short for "more than 2/3"
 - _1/3+_ is short for "1/3 or more"
 - A set of +2/3 of prevotes for a particular block or `<nil>` at
  `(H,R)` is called a _proof-of-lock-change_ or _PoLC_ for short.

 ## State Machine Overview

 At each height of the blockchain a round-based protocol is run to
 determine the next block. Each round is composed of three _steps_
 (`Propose`, `Prevote`, and `Precommit`), along with two special steps
 `Commit` and `NewHeight`.

 In the optimal scenario, the order of steps is:

 ```
 NewHeight -> (Propose -> Prevote -> Precommit)+ -> Commit -> NewHeight ->...
 ```

 The sequence `(Propose -> Prevote -> Precommit)` is called a _round_.
 There may be more than one round required to commit a block at a given
 height. Examples for why more rounds may be required include:

 - The designated proposer was not online.
 - The block proposed by the designated proposer was not valid.
 - The block proposed by the designated proposer did not propagate
  in time.
 - The block proposed was valid, but +2/3 of prevotes for the proposed
  block were not received in time for enough validator nodes by the
  time they reached the `Precommit` step. Even though +2/3 of prevotes
  are necessary to progress to the next step, at least one validator
  may have voted `<nil>` or maliciously voted for something else.
 - The block proposed was valid, and +2/3 of prevotes were received for
  enough nodes, but +2/3 of precommits for the proposed block were not
  received for enough validator nodes.

 Some of these problems are resolved by moving onto the next round &
 proposer. Others are resolved by increasing certain round timeout
 parameters over each successive round.

 ## State Machine Diagram

 ```
                         +-------------------------------------+
                         v                                     |(Wait til `CommmitTime+timeoutCommit`)
                   +-----------+                         +-----+-----+
      +----------> |  Propose  +--------------+          | NewHeight |
      |            +-----------+              |          +-----------+
      |                                       |                ^
      |(Else, after timeoutPrecommit)         v                |
 +-----+-----+                           +-----------+          |
 | Precommit |  <------------------------+  Prevote  |          |
 +-----+-----+                           +-----------+          |
      |(When +2/3 Precommits for block found)                  |
      v                                                        |
 +--------------------------------------------------------------------+
 |  Commit                                                            |
 |                                                                    |
 |  * Set CommitTime = now;                                           |
 |  * Wait for block, then stage/save/commit block;                   |
 +--------------------------------------------------------------------+
 ```

 # Background Gossip

 A node may not have a corresponding validator private key, but it
 nevertheless plays an active role in the consensus process by relaying
 relevant meta-data, proposals, blocks, and votes to its peers. A node
 that has the private keys of an active validator and is engaged in
 signing votes is called a _validator-node_. All nodes (not just
 validator-nodes) have an associated state (the current height, round,
 and step) and work to make progress.

 Between two nodes there exists a `Connection`, and multiplexed on top of
 this connection are fairly throttled `Channel`s of information. An
 epidemic gossip protocol is implemented among some of these channels to
 bring peers up to speed on the most recent state of consensus. For
 example,

 - Nodes gossip `PartSet` parts of the current round's proposer's
  proposed block. A LibSwift inspired algorithm is used to quickly
  broadcast blocks across the gossip network.
 - Nodes gossip prevote/precommit votes. A node `NODE_A` that is ahead
  of `NODE_B` can send `NODE_B` prevotes or precommits for `NODE_B`'s
  current (or future) round to enable it to progress forward.
 - Nodes gossip prevotes for the proposed PoLC (proof-of-lock-change)
  round if one is proposed.
 - Nodes gossip to nodes lagging in blockchain height with block
  [commits](https://godoc.org/github.com/tendermint/tendermint/types#Commit)
  for older blocks.
 - Nodes opportunistically gossip `HasVote` messages to hint peers what
  votes it already has.
 - Nodes broadcast their current state to all neighboring peers. (but
  is not gossiped further)

 There's more, but let's not get ahead of ourselves here.

 ## Proposals

 A proposal is signed and published by the designated proposer at each
 round. The proposer is chosen by a deterministic and non-choking round
 robin selection algorithm that selects proposers in proportion to their
 voting power (see
 [implementation](https://github.com/tendermint/tendermint/blob/master/types/validator_set.go)).

 A proposal at `(H,R)` is composed of a block and an optional latest
 `PoLC-Round < R` which is included iff the proposer knows of one. This
 hints the network to allow nodes to unlock (when safe) to ensure the
 liveness property.

 ## State Machine Spec

 ### Propose Step (height:H,round:R)

 Upon entering `Propose`:

 - The designated proposer proposes a block at `(H,R)`.

 The `Propose` step ends:

 - After `timeoutProposeR` after entering `Propose`. --> goto
  `Prevote(H,R)`
 - After receiving proposal block and all prevotes at `PoLC-Round`. -->
  goto `Prevote(H,R)`
 - After [common exit conditions](#common-exit-conditions)

 ### Prevote Step (height:H,round:R)

 Upon entering `Prevote`, each validator broadcasts its prevote vote.

 - First, if the validator is locked on a block since `LastLockRound`
  but now has a PoLC for something else at round `PoLC-Round` where
  `LastLockRound < PoLC-Round < R`, then it unlocks.
 - If the validator is still locked on a block, it prevotes that.
 - Else, if the proposed block from `Propose(H,R)` is good, it
  prevotes that.
 - Else, if the proposal is invalid or wasn't received on time, it
  prevotes `<nil>`.

 The `Prevote` step ends:

 - After +2/3 prevotes for a particular block or `<nil>`. -->; goto
  `Precommit(H,R)`
 - After `timeoutPrevote` after receiving any +2/3 prevotes. --> goto
  `Precommit(H,R)`
 - After [common exit conditions](#common-exit-conditions)

 ### Precommit Step (height:H,round:R)

 Upon entering `Precommit`, each validator broadcasts its precommit vote.

 - If the validator has a PoLC at `(H,R)` for a particular block `B`, it
  (re)locks (or changes lock to) and precommits `B` and sets
  `LastLockRound = R`.
 - Else, if the validator has a PoLC at `(H,R)` for `<nil>`, it unlocks
  and precommits `<nil>`.
 - Else, it keeps the lock unchanged and precommits `<nil>`.

 A precommit for `<nil>` means "I didn’t see a PoLC for this round, but I
 did get +2/3 prevotes and waited a bit".

 The Precommit step ends:

 - After +2/3 precommits for `<nil>`. --> goto `Propose(H,R+1)`
 - After `timeoutPrecommit` after receiving any +2/3 precommits. --> goto
  `Propose(H,R+1)`
 - After [common exit conditions](#common-exit-conditions)

 ### Common exit conditions

 - After +2/3 precommits for a particular block. --> goto
  `Commit(H)`
 - After any +2/3 prevotes received at `(H,R+x)`. --> goto
  `Prevote(H,R+x)`
 - After any +2/3 precommits received at `(H,R+x)`. --> goto
  `Precommit(H,R+x)`

 ### Commit Step (height:H)

 - Set `CommitTime = now()`
 - Wait until block is received. --> goto `NewHeight(H+1)`

 ### NewHeight Step (height:H)

 - Move `Precommits` to `LastCommit` and increment height.
 - Set `StartTime = CommitTime+timeoutCommit`
 - Wait until `StartTime` to receive straggler commits. --> goto
  `Propose(H,0)`

 ## Proofs

 ### Proof of Safety

 Assume that at most -1/3 of the voting power of validators is byzantine.
 If a validator commits block `B` at round `R`, it's because it saw +2/3
 of precommits at round `R`. This implies that 1/3+ of honest nodes are
 still locked at round `R' > R`. These locked validators will remain
 locked until they see a PoLC at `R' > R`, but this won't happen because
 1/3+ are locked and honest, so at most -2/3 are available to vote for
 anything other than `B`.

 ### Proof of Liveness

 If 1/3+ honest validators are locked on two different blocks from
 different rounds, a proposers' `PoLC-Round` will eventually cause nodes
 locked from the earlier round to unlock. Eventually, the designated
 proposer will be one that is aware of a PoLC at the later round. Also,
 `timeoutProposalR` increments with round `R`, while the size of a
 proposal are capped, so eventually the network is able to "fully gossip"
 the whole proposal (e.g. the block & PoLC).

 ### Proof of Fork Accountability

 Define the JSet (justification-vote-set) at height `H` of a validator
 `V1` to be all the votes signed by the validator at `H` along with
 justification PoLC prevotes for each lock change. For example, if `V1`
 signed the following precommits: `Precommit(B1 @ round 0)`,
 `Precommit(<nil> @ round 1)`, `Precommit(B2 @ round 4)` (note that no
 precommits were signed for rounds 2 and 3, and that's ok),
 `Precommit(B1 @ round 0)` must be justified by a PoLC at round 0, and
 `Precommit(B2 @ round 4)` must be justified by a PoLC at round 4; but
 the precommit for `<nil>` at round 1 is not a lock-change by definition
 so the JSet for `V1` need not include any prevotes at round 1, 2, or 3
 (unless `V1` happened to have prevoted for those rounds).

 Further, define the JSet at height `H` of a set of validators `VSet` to
 be the union of the JSets for each validator in `VSet`. For a given
 commit by honest validators at round `R` for block `B` we can construct
 a JSet to justify the commit for `B` at `R`. We say that a JSet
 _justifies_ a commit at `(H,R)` if all the committers (validators in the
 commit-set) are each justified in the JSet with no duplicitous vote
 signatures (by the committers).

 - **Lemma**: When a fork is detected by the existence of two
  conflicting [commits](../blockchain/blockchain.md#commit), the
  union of the JSets for both commits (if they can be compiled) must
  include double-signing by at least 1/3+ of the validator set.
  **Proof**: The commit cannot be at the same round, because that
  would immediately imply double-signing by 1/3+. Take the union of
  the JSets of both commits. If there is no double-signing by at least
  1/3+ of the validator set in the union, then no honest validator
  could have precommitted any different block after the first commit.
  Yet, +2/3 did. Reductio ad absurdum.

 As a corollary, when there is a fork, an external process can determine
 the blame by requiring each validator to justify all of its round votes.
 Either we will find 1/3+ who cannot justify at least one of their votes,
 and/or, we will find 1/3+ who had double-signed.

 ### Alternative algorithm

 Alternatively, we can take the JSet of a commit to be the "full commit".
 That is, if light clients and validators do not consider a block to be
 committed unless the JSet of the commit is also known, then we get the
 desirable property that if there ever is a fork (e.g. there are two
 conflicting "full commits"), then 1/3+ of the validators are immediately
 punishable for double-signing.

 There are many ways to ensure that the gossip network efficiently share
 the JSet of a commit. One solution is to add a new message type that
 tells peers that this node has (or does not have) a +2/3 majority for B
 (or) at (H,R), and a bitarray of which votes contributed towards that
 majority. Peers can react by responding with appropriate votes.

 We will implement such an algorithm for the next iteration of the
 Tendermint consensus protocol.

 Other potential improvements include adding more data in votes such as
 the last known PoLC round that caused a lock change, and the last voted
 round/step (or, we may require that validators not skip any votes). This
 may make JSet verification/gossip logic easier to implement.

 ### Censorship Attacks

 Due to the definition of a block
 [commit](https://github.com/tendermint/tendermint/blob/master/docs/tendermint-core/validators.md), any 1/3+ coalition of
 validators can halt the blockchain by not broadcasting their votes. Such
 a coalition can also censor particular transactions by rejecting blocks
 that include these transactions, though this would result in a
 significant proportion of block proposals to be rejected, which would
 slow down the rate of block commits of the blockchain, reducing its
 utility and value. The malicious coalition might also broadcast votes in
 a trickle so as to grind blockchain block commits to a near halt, or
 engage in any combination of these attacks.

 If a global active adversary were also involved, it can partition the
 network in such a way that it may appear that the wrong subset of
 validators were responsible for the slowdown. This is not just a
 limitation of Tendermint, but rather a limitation of all consensus
 protocols whose network is potentially controlled by an active
 adversary.

 ### Overcoming Forks and Censorship Attacks

 For these types of attacks, a subset of the validators through external
 means should coordinate to sign a reorg-proposal that chooses a fork
 (and any evidence thereof) and the initial subset of validators with
 their signatures. Validators who sign such a reorg-proposal forego its
 collateral on all other forks. Clients should verify the signatures on
 the reorg-proposal, verify any evidence, and make a judgement or prompt
 the end-user for a decision. For example, a phone wallet app may prompt
 the user with a security warning, while a refrigerator may accept any
 reorg-proposal signed by +1/2 of the original validators.

 No non-synchronous Byzantine fault-tolerant algorithm can come to
 consensus when 1/3+ of validators are dishonest, yet a fork assumes that
 1/3+ of validators have already been dishonest by double-signing or
 lock-changing without justification. So, signing the reorg-proposal is a
 coordination problem that cannot be solved by any non-synchronous
 protocol (i.e. automatically, and without making assumptions about the
 reliability of the underlying network). It must be provided by means
 external to the weakly-synchronous Tendermint consensus algorithm. For
 now, we leave the problem of reorg-proposal coordination to human
 coordination via internet media. Validators must take care to ensure
 that there are no significant network partitions, to avoid situations
 where two conflicting reorg-proposals are signed.

 Assuming that the external coordination medium and protocol is robust,
 it follows that forks are less of a concern than [censorship
 attacks](#censorship-attacks).
--- a/spec/consensus/creating-proposal.md
+++ b/spec/consensus/creating-proposal.md
@ -0,0 +1,42 @@
 # Creating a proposal

 A block consists of a header, transactions, votes (the commit),
 and a list of evidence of malfeasance (ie. signing conflicting votes).

 We include no more than 1/10th of the maximum block size
 (`ConsensusParams.Block.MaxBytes`) of evidence with each block.

 ## Reaping transactions from the mempool

 When we reap transactions from the mempool, we calculate maximum data
 size by subtracting maximum header size (`MaxHeaderBytes`), the maximum
 amino overhead for a block (`MaxAminoOverheadForBlock`), the size of
 the last commit (if present) and evidence (if present). While reaping
 we account for amino overhead for each transaction.

 ```go
 func MaxDataBytes(maxBytes int64, valsCount, evidenceCount int) int64 {
 	return maxBytes -
 		MaxAminoOverheadForBlock -
 		MaxHeaderBytes -
 		int64(valsCount)*MaxVoteBytes -
 		int64(evidenceCount)*MaxEvidenceBytes
 }
 ```

 ## Validating transactions in the mempool

 Before we accept a transaction in the mempool, we check if it's size is no more
 than {MaxDataSize}. {MaxDataSize} is calculated using the same formula as
 above, except because the evidence size is unknown at the moment, we subtract
 maximum evidence size (1/10th of the maximum block size).

 ```go
 func MaxDataBytesUnknownEvidence(maxBytes int64, valsCount int) int64 {
 	return maxBytes -
 		MaxAminoOverheadForBlock -
 		MaxHeaderBytes -
 		int64(valsCount)*MaxVoteBytes -
 		MaxEvidenceBytesPerBlock(maxBytes)
 }
 ```
--- a/spec/consensus/fork-accountability.md
+++ b/spec/consensus/fork-accountability.md
@ -0,0 +1,319 @@
 # Fork accountability -- Problem statement and attacks

 ## Problem Statement

 Tendermint consensus guarantees the following specifications for all heights:
 * agreement -- no two correct full nodes decide differently.
 * validity -- the decided block satisfies the predefined predicate *valid()*.
 * termination -- all correct full nodes eventually decide,

 if the
 faulty validators have at most 1/3 of voting power in the current validator set. In the case where this assumption
 does not hold, each of the specification may be violated.

 The agreement property says that for a given height, any two correct validators that decide on a block for that height decide on the same block. That the block was indeed generated by the blockchain, can be verified starting from a trusted (genesis) block, and checking that all subsequent blocks are properly signed.


 However, faulty nodes may forge blocks and try to convince users (lite clients) that the blocks had been correctly generated. In addition, Tendermint agreement might be violated in the case where more than 1/3 of the voting power belongs to faulty validators: Two correct validators decide on different blocks. The latter case motivates the term "fork": as Tendermint consensus also agrees on the next validator set, correct validators may have decided on disjoint next validator sets, and the chain branches into two or more partitions (possibly having faulty validators in common) and each branch continues to generate blocks independently of the other.

 We say that a fork is a case in which there are two commits for different blocks at the same height of the blockchain. The proplem is to ensure that in those cases we are able to detect faulty validators (and not mistakenly accuse correct validators), and incentivize therefore validators to behave according to the protocol specification.

 **Conceptual Limit.** In order to prove misbehavior of a node, we have to show that the behavior deviates from correct behavior with respect to a given algorithm. Thus, an algorithm that detects misbehavior of nodes executing some algorithm *A* must be defined with respect to algorithm *A*. In our case, *A* is Tendermint consensus (+ other protocols in the infrastructure; e.g.,full nodes and the Lite Client). If the consensus algorithm is changed/updated/optimized in the future, we have to check whether changes to the accountability algorithm are also required. All the discussions in this document are thus inherently specific to Tendermint consensus and the Lite Client specification.

 **Q:** Should we distinguish agreement for validators and full nodes for agreement? The case where all correct validators agree on a block, but a correct full node decides on a different block seems to be slightly less severe that the case where two correct validators decide on different blocks. Still, if a contaminated full node becomes validator that may be problematic later on. Also it is not clear how gossiping is impaired if a contaminated full node is on a different branch.

 *Remark.* In the case more than 1/3 of the voting power belongs to faulty validators, also validity and termination can be broken. Termination can be broken if faulty processes just do not send the messages that are needed to make progress. Due to asynchrony, this is not punishable, because faulty validators can always claim they never received the messages that would have forced them to send messages.


 ## The Misbehavior of Faulty Validators

 Forks are the result of faulty validators deviating from the protocol. In principle several such deviations can be detected without a fork actually occurring:

 1. double proposal: A faulty proposer proposes two different values (blocks) for the same height and the same round in Tendermint consensus.

 2. double signing: Tendermint consensus forces correct validators to prevote and precommit for at most one value per round. In case a faulty validator sends multiple prevote and/or precommit messages for different values for the same height/round, this is a misbehavior.

 3. lunatic validator: Tendermint consensus forces correct validators to prevote and precommit only for values *v* that satisfy *valid(v)*. If faulty validators prevote and precommit for *v* although *valid(v)=false* this is misbehavior.

 *Remark.* In isolation, Point 3 is an attack on validity (rather than agreement). However, the prevotes and precommits can then also be used to forge blocks.

 1. amnesia: Tendermint consensus has a locking mechanism. If a validator has some value v locked, then it can only prevote/precommit for v or nil. Sending prevote/precomit message for a different value v' (that is not nil) while holding lock on value v is misbehavior.   

 2. spurious messages: In Tendermint consensus most of the message send instructions are guarded by threshold guards, e.g., one needs to receive *2f + 1* prevote messages to send precommit. Faulty validators may send precommit without having received the prevote messages.


 Independently of a fork happening, punishing this behavior might be important to prevent forks altogether. This should keep attackers from misbehaving: if at most 1/3 of the voting power is faulty, this misbehavior is detectable but will not lead to a safety violation. Thus, unless they have more than 1/3 (or in some cases more than 2/3) of the voting power attackers have the incentive to not misbehave. If attackers control too much voting power, we have to deal with forks, as discussed in this document.   


 ## Two types of forks

 * Fork-Full. Two correct validators decide on different blocks for the same height. Since also the next validator sets are decided upon, the correct validators may be partitioned to participate in two distinct branches of the forked chain.

 As in this case we have two different blocks (both having the same right/no right to exist), a central system invariant (one block per height decided by correct validators) is violated. As full nodes are contaminated in this case, the contamination can spread also to lite clients. However, even without breaking this system invariant, lite clients can be subject to a fork:

 * Fork-Lite. All correct validators decide on the same block for height *h*, but faulty processes (validators or not), forge a different block for that height, in order to fool users (who use the lite client).


 # Attack scenarios

 ## On-chain attacks

 ### Equivocation (one round)

 There are several scenarios in which forks might happen. The first is double signing within a round.

 * F1. Equivocation: faulty validators sign multiple vote messages (prevote and/or precommit) for different values *during the same round r* at a given height h.


 ### Flip-flopping

 Tendermint consensus implements a locking mechanism: If a correct validator *p* receives proposal for value v and *2f + 1* prevotes for a value *id(v)* in round *r*, it locks *v* and remembers *r*. In this case, *p* also sends a precommit message for *id(v)*, which later may serve as proof that *p* locked *v*.
 In subsequent rounds, *p* only sends prevote messages for a value it had previously locked. However, it is possible to change the locked value if in a future round *r' > r*, if the process receives proposal and *2f + 1* prevotes for a different value *v'*. In this case, *p* could send a prevote/precommit for *id(v')*. This algorithmic feature can be exploited in two ways:



 * F2. Faulty Flip-flopping (Amnesia): faulty validators precommit some value *id(v)* in round *r* (value *v* is locked in round *r*) and then prevote for different value *id(v')* in higher round *r' > r* without previously correctly unlocking value *v*. In this case faulty processes "forget" that they have locked value *v* and prevote some other value in the following rounds.
 Some correct validators might have decided on *v* in *r*, and other correct validators decide on *v'* in *r'*. Here we can have branching on the main chain (Fork-Full).


 * F3. Correct Flip-flopping (Back to the past): There are some precommit messages signed by (correct) validators for value *id(v)* in  round *r*. Still, *v* is not decided upon, and all processes move on to the next round. Then correct validators (correctly) lock and decide a different value *v'* in some round *r' > r*. And the correct validators continue; there is no branching on the main chain.
 However, faulty validators may use the correct precommit messages from round *r* together with a posteriori generated faulty precommit messages for round *r* to forge a block for a value that was not decided on the main chain (Fork-Lite).





 ## Off-chain attacks

 F1-F3 may contaminate the state of full nodes (and even validators). Contaminated (but otherwise correct) full nodes may thus communicate faulty blocks to lite clients.
 Similarly, without actually interfering with the main chain, we can have the following:

 * F4. Phantom validators: faulty validators vote (sign prevote and precommit messages) in heights in which they are not part of the validator sets (at the main chain).

 * F5. Lunatic validator: faulty validator that sign vote messages to support (arbitrary) application state that is different from the application state that resulted from valid state transitions.

 ## Types of victims

 We consider three types of potential attack victims:


 - FN: full node
 - LCS: lite client with sequential header verification
 - LCB: lite client with bisection based header verification

 F1 and F2 can be used by faulty validators to actually create multiple branches on the blockchain. That means that correctly operating full nodes decide on different blocks for the same height. Until a fork is detected locally by a full node (by receiving evidence from others or by some other local check that fails), the full node can spread corrupted blocks to lite clients.

 *Remark.* If full nodes take a branch different from the one taken by the validators, it may be that the liveness of the gossip protocol may be affected. We should eventually look at this more closely. However, as it does not influence safety it is not a primary concern.

 F3 is similar to F1, except that no two correct validators decide on different blocks. It may still be the case that full nodes become affected.

 In addition, without creating a fork on the main chain, lite clients can be contaminated by more than a third of validators that are faulty and sign a forged header
 F4 cannot fool correct full nodes as they know the current validator set. Similarly, LCS know who the validators are. Hence, F4 is an attack against LCB that do not necessarily know the complete prefix of headers (Fork-Lite), as they trust a header that is signed by at least one correct validator (trusting period method).






 The following table gives an overview of how the different attacks may affect different nodes. F1-F3 are *on-chain* attacks so they can corrupt the state of full nodes. Then if a lite client (LCS or LCB) contacts a full node to obtain headers (or blocks), the corrupted state may propagate to the lite client.  

 F4 and F5 are *off-chain*, that is, these attacks cannot be used to corrupt the state of full nodes (which have sufficient knowledge on the state of the chain to not be fooled). 


 | Attack |  FN | LCS | LCB |
 |:------:|:------:|:------:|:------:|
 | F1 |  direct  | FN |     FN |
 | F2 |   direct | FN |     FN |
 | F3 |  direct  | FN |     FN |
 | F4 |          |    | direct |
 | F5 |          |    | direct |



 **Q:** Lite clients are more vulnerable than full nodes, because the former do only verify headers but do not execute transactions. What kind of certainty is gained by a full node that executes a transaction?

 As a full node verifies all transactions, it can only be
 contaminated by an attack if the blockchain itself violates its invariant (one block per height), that is, in case of a fork that leads to branching.




 ## Detailed Attack Scenarios

 ### Equivocation based attacks

 In case of equivocation based attacks, faulty validators sign multiple votes (prevote and/or precommit) in the same
 round of some height. This attack can be executed on both full nodes and lite clients. It requires more than 1/3 of voting power to be executed.

 #### Scenario 1: Equivocation on the main chain

 Validators:
 * CA - a set of correct validators with less than 1/3 of the voting power
 * CB - a set of correct validators with less than 1/3 of the voting power
 * CA and CB are disjoint
 * F - a set of faulty validators with more than 1/3 voting power

 Observe that this setting violates the Tendermint failure model.

 Execution:

 * A faulty proposer proposes block A to CA
 * A faulty proposer proposes block B to CB
 * Validators from the set CA and CB prevote for A and B, respectively.
 * Faulty validators from the set F prevote both for A and B.
 * The faulty prevote messages
  - for A arrive at CA long before the B messages
  - for B arrive at CB long before the A messages
 * Therefore correct validators from set CA and CB will observe
 more than 2/3 of prevotes for A and B and precommit for A and B, respectively.
 * Faulty validators from the set F precommit both values A and B.
 * Thus, we have more than 2/3 commits for both A and B.

 Consequences:
 * Creating evidence of misbehavior is simple in this case as we have multiple messages signed by the same faulty processes for different values in the same round.

 * We have to ensure that these different messages reach a correct process (full node, monitor?), which can submit evidence.

 * This is an attack on the full node level (Fork-Full).
 * It extends also to the lite clients,
 * For both we need a detection and recovery mechanism.

 #### Scenario 2: Equivocation to a lite client (LCS)


 Validators:
 * a set F of faulty validators with more than 2/3 of the voting power.

 Execution:
 * for the main chain F behaves nicely
 * F coordinates to sign a block B that is different from the one on the main chain.
 * the lite clients obtains B and trusts at as it is signed by more and 2/3 of the voting power.

 Consequences:

 Once equivocation is used to attack lite client it opens space
 for different kind of attacks as application state can be diverged in any direction. For example, it can modify validator set such that it contains only validators that do not have any stake bonded. Note that after a lite client is fooled by a fork, that means that an attacker can change application state and validator set arbitrarily.

 In order to detect such (equivocation-based attack), the lite client would need to cross check its state with some correct validator (or to obtain a hash of the state from the main chain using out of band channels).

 *Remark.* The lite client would be able to create evidence of misbehavior, but this would require to pull potentially a lot of data from correct full nodes. Maybe we need to figure out different architecture where a lite client that is attacked will push all its data for the current unbonding period to a correct node that will inspect this data and submit corresponding evidence. There are also architectures that assumes a special role (sometimes called fisherman) whose goal is to collect as much as possible useful data from the network, to do analysis and create evidence transactions. That functionality is outside the scope of this document.

 *Remark.* The difference between LCS and LCB might only be in the amount of voting power needed to convince lite client about arbitrary state. In case of LCB where security threshold is at minimum, an attacker can arbitrarily modify application state with  more than 1/3 of voting power, while in case of LCS it requires more than 2/3 of the voting power.

 ### Flip-flopping: Amnesia based attacks



 In case of amnesia, faulty validators lock some value *v* in some round *r*, and then vote for different value *v'* in higher rounds without correctly unlocking value *v*. This attack can be used both on full nodes and lite clients.

 #### Scenario 3: At most 2/3 of faults

 Validators:

 * a set F of faulty validators with more than 1/3 but at most 2/3 of the voting power
 * a set C of correct validators

 Execution:

 * Faulty validators commit (without exposing it on the main chain) a block A in round *r* by collecting more than 2/3 of the 
  voting power (containing correct and faulty validators).
 * All validators (correct and faulty) reach a round *r' > r*.
 * Some correct validators in C do not lock any value before round *r'*.
 * The faulty validators in F deviate from Tendermint consensus by ignoring that they locked A in *r*, and propose a different block B in *r'*.
 * As the validators in C that have not locked any value find B acceptable, they accept the proposal for B and commit a block B.

 *Remark.* In this case, the more than 1/3 of faulty validators do not need to commit an equivocation (F1) as they only vote once per round in the execution.

 Detecting faulty validators in the case of such an attack can be done by the fork accountability mechanism described in: https://docs.google.com/document/d/11ZhMsCj3y7zIZz4udO9l25xqb0kl7gmWqNpGVRzOeyY/edit?usp=sharing.

 If a lite client is attacked using this attack with more than 1/3 of voting power (and less than 2/3), the attacker cannot change the application state arbitrarily. Rather, the attacker is limited to a state a correct validator finds acceptable: In the execution above, correct validators still find the value acceptable, however, the block the lite client trusts deviates from the one on the main chain.

 #### Scenario 4: More than 2/3 of faults

 In case there is an attack with more than 2/3 of the voting power, an attacker can arbitrarily change application state.

 Validators:

 * a set F1 of faulty validators with more than 1/3 of the voting power
 * a set F2 of faulty validators with at most 1/3 of the voting power

 Execution

 * Similar to Scenario 3 (however, messages by correct validators are not needed)
 * The faulty validators in F1 lock value A in round *r*
 * They sign a different value in follow-up rounds
 * F2 does not lock A in round *r*

 Consequences:

 * The validators in F1 will be detectable by the the fork accountability mechanisms.
 * The validators in F2 cannot be detected using this mechanism.
 Only in case they signed something which conflicts with the application this can be used against them. Otherwise they do not do anything incorrect.
 * This case is not covered by the report https://docs.google.com/document/d/11ZhMsCj3y7zIZz4udO9l25xqb0kl7gmWqNpGVRzOeyY/edit?usp=sharing as it only assumes at most 2/3 of faulty validators.

 **Q:** do we need to define a special kind of attack for the case where a validator sign arbitrarily state? It seems that detecting such attack requires different mechanism that would require as an evidence a sequence of blocks that lead to that state. This might be very tricky to implement.

 ### Back to the past

 In this kind of attacks faulty validators take advantage of the fact that they did not sign messages in some of the past rounds. Due to the asynchronous network in which Tendermint operates, we cannot easily differentiate between such an attack and delayed message. This kind of attack can be used at both full nodes and lite clients.

 #### Scenario 5:

 Validators:
 * C1 -  a set of correct validators with 1/3 of the voting power
 * C2 - a set of correct validators with 1/3 of the voting power
 * C1 and C2 are disjoint
 * F - a set of faulty validators with 1/3 voting power
 * one additional faulty process *q*
 * F and *q* violate the Tendermint failure model.


 Execution:

 * in a round *r* of height *h* we have C1 precommitting a value A,
 * C2  precommits nil,
 * F does not send any message
 * *q* precommits nil.
 * In some round *r' > r*, F and *q* and C2 commit some other value B different from A.
 * F and *fp* "go back to the past" and sign precommit message for value A in round *r*.
 * Together with precomit messages of C1 this is sufficient for a commit for value A.


 Consequences:

 * Only a single faulty validator that previously precommited nil did equivocation, while the other 1/3 of faulty validators actually executed an attack that has exactly the same sequence of messages as part of amnesia attack. Detecting this kind of attack boil down to mechanisms for equivocation and amnesia.

 **Q:** should we keep this as a separate kind of attack? It seems that equivocation, amnesia and phantom validators are the only kind of attack we need to support and this gives us security also in other cases. This would not be surprising as equivocation and amnesia are attacks that followed from the protocol and phantom attack is not really an attack to Tendermint but more to the Proof of stake module.

 ### Phantom validators

 In case of phantom validators, processes that are not part of the current validator set but are still bonded (as attack happen during their unbonding period) can be part of the attack by signing vote messages. This attack can be executed against both full nodes and lite clients.

 #### Scenario 6:

 Validators:
 * F -- a set of faulty validators that are not part of the validator set on the main chain at height *h + k*

 Execution:

 * There is a fork, and there exists two different headers for height *h + k*, with different validator sets:
  - VS2 on the main chain
  - forged header VS2', signed by F (and others)

 * a lite client has a trust in a header for height *h* (and the corresponding validator set VS1).
 * As part of bisection header verification, it verifies header at height *h + k* with new validator set VS2'.

 Consequences:

 * To detect this, a node needs to see both, the forged header and the canonical header from the chain.
 * If this is the case, detecting these kind of attacks is easy as it just requires verifying if processes are signing messages in heights in which they are not part of the validator set.  

 **Remark.** We can have phantom-validator-based attacks as a follow up of equivocation or amnesia based attack where forked state contains validators that are not part of the validator set at the main chain. In this case, they keep signing messages contributed to a forked chain (the wrong branch) although they are not part of the validator set on the main chain. This attack can also be used to attack full node during a period of time he is eclipsed.

 ### Lunatic validator

 Lunatic validator agrees to sign commit messages for arbitrary application state. It is used to attack lite clients.
 Note that detecting this behavior require application knowledge. Detecting this behavior can probably be done by
 referring to the block before the one in which height happen.  

 **Q:** can we say that in this case a validator ignore to check if proposed value is valid before voting for it?
--- a/spec/consensus/light-client.md
+++ b/spec/consensus/light-client.md
@ -0,0 +1,329 @@
 # Lite client

 A lite client is a process that connects to Tendermint full nodes and then tries to verify application data using the Merkle proofs.

 ## Context of this document

 In order to make sure that full nodes have the incentive to follow the protocol, we have to address the following three Issues

 1) The lite client needs a method to verify headers it obtains from full nodes according to trust assumptions -- this document.

 2) The lite client must be able to connect to one correct full node to detect and report on failures in the trust assumptions (i.e., conflicting headers) -- a future document.

 3) In the event the trust assumption fails (i.e., a lite client is fooled by a conflicting header), the Tendermint fork accountability protocol must account for the evidence -- see #3840

 ## Problem statement


 We assume that the lite client knows a (base) header *inithead* it trusts (by social consensus or because the lite client has decided to trust the header before). The goal is to check whether another header *newhead* can be trusted based on the data in *inithead*.

 The correctness of the protocol is based on the assumption that *inithead* was generated by an instance of Tendermint consensus. The term "trusting" above indicates that the correctness on the protocol depends on this assumption. It is in the responsibility of the user that runs the lite client to make sure that the risk of trusting a corrupted/forged *inithead* is negligible.


 ## Definitions

 ### Data structures

 In the following, only the details of the data structures needed for this specification are given.

  * header fields
    - *height*
    - *bfttime*: the chain time when the header (block) was generated
    - *V*: validator set containing validators for this block.
    - *NextV*: validator set for next block.
    - *commit*: evidence that block with height *height* - 1 was committed by a set of validators (canonical commit). We will use ```signers(commit)``` to refer to the set of validators that committed the block.

  * signed header fields: contains a header and a  *commit* for the current header; a "seen commit". In the Tendermint consensus the "canonical commit" is stored in header *height* + 1.

  * For each header *h* it has locally stored, the lite client stores whether
    it trusts *h*. We write *trust(h) = true*, if this is the case.

  * Validator fields. We will write a validator as a tuple *(v,p)* such that
    + *v* is the identifier (we assume identifiers are unique in each validator set)
    + *p* is its voting power


 ### Functions

 For the purpose of this lite client specification, we assume that the Tendermint Full Node exposes the following function over Tendermint RPC:
 ```go
    func Commit(height int64) (SignedHeader, error)
      // returns signed header: header (with the fields from
      // above) with Commit that include signatures of
      // validators that signed the header


    type SignedHeader struct {
      Header        Header
      Commit        Commit
    }
 ```

 ### Definitions

 * *tp*: trusting period   
 * for realtime *t*, the predicate *correct(v,t)* is true if the validator *v*
  follows the protocol until time *t* (we will see about recovery later).




 ### Tendermint Failure Model

 If a block *h* is generated at time *bfttime* (and this time is stored in the block), then a set of validators that hold more than 2/3 of the voting power in h.Header.NextV is correct until time h.Header.bfttime + tp.

 Formally,
 \[
 \sum_{(v,p) \in h.Header.NextV \wedge correct(v,h.Header.bfttime + tp)} p >
 2/3 \sum_{(v,p) \in h.Header.NextV} p
 \]

 *Assumption*: "correct" is defined w.r.t. realtime (some Newtonian global notion of time, i.e., wall time), while *bfttime* corresponds to the reading of the local clock of a validator (how this time is computed may change when the Tendermint consensus is modified). In this note, we assume that all clocks are synchronized to realtime. We can make this more precise eventually (incorporating clock drift, accuracy, precision, etc.). Right now, we consider this assumption sufficient, as clock synchronization (under NTP) is in the order of milliseconds and *tp* is in the order of weeks.  

 *Remark*: This failure model might change to a hybrid version that takes heights into account in the future.

 The specification in this document considers an implementation of the lite client under this assumption. Issues like *counter-factual signing* and *fork accountability* and *evidence submission* are mechanisms that justify this assumption by incentivizing validators to follow the protocol.
 If they don't, and we have more that 1/3 faults, safety may be violated. Our approach then is to *detect* these cases (after the fact), and take suitable repair actions (automatic and social). This is discussed in an upcoming document on "Fork accountability". (These safety violations include the lite client wrongly trusting a header, a fork in the blockchain, etc.)


 ## Lite Client Trusting Spec

 The lite client communicates with a full node and learns new headers. The goal is to locally decide whether to trust a header. Our implementation needs to ensure the following two properties:

 - Lite Client Completeness: If header *h* was correctly generated by an instance of Tendermint consensus (and its age is less than the trusting period), then the lite client should eventually set *trust(h)* to true.

 - Lite Client Accuracy:  If header *h* was *not generated* by an instance of Tendermint consensus, then the lite client should never set *trust(h)* to true.

 *Remark*: If in the course of the computation, the lite client obtains certainty that some headers were forged by adversaries (that is were not generated by an instance of Tendermint consensus), it may submit (a subset of) the headers it has seen as evidence of misbehavior.

 *Remark*: In Completeness we use "eventually", while in practice *trust(h)* should be set to true before *h.Header.bfttime + tp*. If not, the block cannot be trusted because it is too old.

 *Remark*: If a header *h* is marked with *trust(h)*, but it is too old (its bfttime is more than *tp* ago), then the lite client should set *trust(h)* to false again.

 *Assumption*: Initially, the lite client has a header *inithead* that it trusts correctly, that is, *inithead* was correctly generated by the Tendermint consensus.

 To reason about the correctness, we may prove the following invariant.

 *Verification Condition: Lite Client Invariant.*
 For each lite client *l* and each header *h*:
 if *l* has set *trust(h) = true*,
  then validators that are correct until time *h.Header.bfttime + tp* have more than two thirds of the voting power in *h.Header.NextV*.

  Formally,
  \[
  \sum_{(v,p) \in h.Header.NextV \wedge correct(v,h.Header.bfttime + tp)} p >
  2/3 \sum_{(v,p) \in h.Header.NextV} p
  \]

 *Remark.* To prove the invariant, we will have to prove that the lite client only trusts headers that were correctly generated by Tendermint consensus, then the formula above follows from the Tendermint failure model.


 ## High Level Solution

 Upon initialization, the lite client is given a header *inithead* it trusts (by
 social consensus). It is assumed that *inithead* satisfies the lite client invariant. (If *inithead* has been correctly generated by Tendermint consensus, the invariant follows from the Tendermint Failure Model.)

 When a lite clients sees a signed new header *snh*, it has to decide whether to trust the new
 header. Trust can be obtained by (possibly) the combination of three methods.

 1. **Uninterrupted sequence of proof.** If a block is appended to the chain, where the last block
 is trusted (and properly committed by the old validator set in the next block),
 and the new block contains a new validator set, the new block is trusted if the lite client knows all headers in the prefix.
 Intuitively, a trusted validator set is assumed to only chose a new validator set that will obey the Tendermint Failure Model.

 2. **Trusting period.** Based on a trusted block *h*, and the lite client
 invariant, which ensures the fault assumption during the trusting period, we can check whether at least one  validator, that has been continuously correct from *h.Header.bfttime* until now, has signed *snh*.
 If this is the case, similarly to above, the chosen validator set in *snh* does not violate the Tendermint Failure Model.

 3. **Bisection.** If a check according to the trusting period fails, the lite client can try to obtain a header *hp* whose height lies between *h* and *snh* in order to check whether *h* can be used to get trust for *hp*, and *hp* can be used to get trust for *snh*. If this is the case we can trust *snh*; if not, we may continue recursively.

 ## How to use it

 We consider the following use case:
 the lite client wants to verify a header for some given height *k*. Thus:
  - it requests the signed header for height *k* from a full node
  - it tries to verify this header with the methods described here.

 This can be used in several settings:
  - someone tells the lite client that application data that is relevant for it can be read in the block of height *k*.
  - the lite clients wants the latest state. It asks a full nude for the current height, and uses the response for *k*.


 ## Details

 *Assumptions*

 1. *tp < unbonding period*.
 2. *snh.Header.bfttime < now*
 3. *snh.Header.bfttime < h.Header.bfttime+tp*
 4. *trust(h)=true*


 **Observation 1.** If *h.Header.bfttime + tp > now*, we trust the old
 validator set *h.Header.NextV*.

 When we say we trust *h.Header.NextV* we do *not* trust that each individual validator in *h.Header.NextV* is correct, but we only trust the fact that at most 1/3 of them are faulty (more precisely, the faulty ones have at most 1/3 of the total voting power).



 ### Functions

 The function *Bisection* checks whether to trust header *h2* based on the trusted header *h1*. It does so by calling
 the function *CheckSupport* in the process of
 bisection/recursion. *CheckSupport* implements the trusted period method and, for two adjacent headers (in term of heights), it checks uninterrupted sequence of proof.

 *Assumption*: In the following, we assume that *h2.Header.height > h1.Header.height*. We will quickly discuss the other case in the next section.

 We consider the following set-up:
 - the lite client communicates with one full node
 - the lite client locally stores all the signed headers it obtained (trusted or not). In the pseudo code below we write *Store(header)* for this.
 - If *Bisection* returns *false*, then the lite client has seen a  forged header.
  * However, it does not know which header(s) is/are the problematic one(s).
  * In this case, the lite client can submit (some of) the headers it has seen as evidence. As the lite client communicates with one full node only when executing Bisection, there are two cases
    - the full node is faulty
    - the full node is correct and there was a fork in Tendermint consensus. Header *h1* is from a different branch than the one taken by the full node. This case is not focus of this document, but will be treated in the document on fork accountability.

 - the lite client must retry to retrieve correct headers from another full node
  * it picks a new full node
  * it restarts *Bisection*
  * there might be optimizations; a lite client may not need to call *Commit(k)*, for a height *k* for which it already has a signed header it trusts.
  * how to make sure that a lite client can communicate with a correct full node will be the focus of a separate document (recall Issue 3 from "Context of this document").

 **Auxiliary Functions.** We will use the  function ```votingpower_in(V1,V2)``` to compute the voting power the validators in set V1 have according to their voting power in set V2;
 we will write ```totalVotingPower(V)``` for ```votingpower_in(V,V)```, which returns the total voting power in V.
 We further use the function ```signers(Commit)``` that returns the set of validators that signed the Commit.

 **CheckSupport.** The following function checks whether we can trust the header h2 based on header h1 following the trusting period method.

 ```go
  func CheckSupport(h1,h2,trustlevel) bool {
    if h1.Header.bfttime + tp < now { // Observation 1
      return false // old header was once trusted but it is expired
    }  
    vp_all := totalVotingPower(h1.Header.NextV)
      // total sum of voting power of validators in h2

    if h2.Header.height == h1.Header.height + 1 {
      // specific check for adjacent headers; everything must be
      // properly signed.
      // also check that h2.Header.V == h1.Header.NextV
      // Plus the following check that 2/3 of the voting power
      // in h1 signed h2
      return (votingpower_in(signers(h2.Commit),h1.Header.NextV) >
              2/3 * vp_all)
        // signing validators are more than two third in h1.
    }

    return (votingpower_in(signers(h2.Commit),h1.Header.NextV) >
            max(1/3,trustlevel) * vp_all)
      // get validators in h1 that signed h2
      // sum of voting powers in h1 of
      // validators that signed h2
      // is more than a third in h1
  }
 ```

  *Remark*: Basic header verification must be done for *h2*. Similar checks are done in:  
  https://github.com/tendermint/tendermint/blob/master/types/validator_set.go#L591-L633

  *Remark*: There are some sanity checks which are not in the code:
  *h2.Header.height > h1.Header.height* and *h2.Header.bfttime > h1.Header.bfttime* and *h2.Header.bfttime < now*.

  *Remark*: ```return (votingpower_in(signers(h2.Commit),h1.Header.NextV) > max(1/3,trustlevel) * vp_all)``` may return false even if *h2* was properly generated by Tendermint consensus in the case of big changes in the validator sets. However, the check ```return (votingpower_in(signers(h2.Commit),h1.Header.NextV) >
          2/3 * vp_all)``` must return true if *h1* and *h2* were generated by Tendermint consensus.

 *Remark*: The 1/3 check differs from a previously proposed method that was based on intersecting validator sets and checking that the new validator set contains "enough" correct validators. We found that the old check is not suited for realistic changes in the validator sets. The new method is not only based on cardinalities, but also exploits that we can trust what is signed by a correct validator (i.e., signed by more than 1/3 of the voting power).

 *Correctness arguments*

 Towards Lite Client Accuracy:
 - Assume by contradiction that *h2* was not generated correctly and the lite client sets trust to true because *CheckSupport* returns true.
 - h1 is trusted and sufficiently new
 - by Tendermint Fault Model, less than 1/3 of voting power held by faulty validators => at least one correct validator *v* has signed *h2*.
 - as *v* is correct up to now, it followed the Tendermint consensus protocol at least up to signing *h2* => *h2* was correctly generated, we arrive at the required contradiction.


 Towards Lite Client Completeness:
 - The check is successful if sufficiently many validators of *h1* are still validators in *h2* and signed *h2*.
 - If *h2.Header.height = h1.Header.height + 1*, and both headers were generated correctly, the test passes

 *Verification Condition:* We may need a Tendermint invariant stating that if *h2.Header.height = h1.Header.height + 1* then *signers(h2.Commit) \subseteq h1.Header.NextV*.

 *Remark*: The variable *trustlevel* can be used if the user believes that relying on one correct validator is not sufficient. However, in case of (frequent) changes in the validator set, the higher the *trustlevel* is chosen, the more unlikely it becomes that CheckSupport returns true for non-adjacent headers.

 **Bisection.** The following function uses CheckSupport in a recursion to find intermediate headers that allow to establish a sequence of trust.




 ```go
 func Bisection(h1,h2,trustlevel) bool{
  if CheckSupport(h1,h2,trustlevel) {
    return true
  }
  if h2.Header.height == h1.Header.height + 1 {
    // we have adjacent headers that are not matching (failed
    // the CheckSupport)
    // we could submit evidence here
    return false
  }
  pivot := (h1.Header.height + h2.Header.height) / 2
  hp := Commit(pivot)
    // ask a full node for header of height pivot
  Store(hp)  
    // store header hp locally
  if Bisection(h1,hp,trustlevel) {
    // only check right branch if hp is trusted
    // (otherwise a lot of unnecessary computation may be done)
    return Bisection(hp,h2,trustlevel)
  }
  else {
    return false
  }
 }
 ```  




 *Correctness arguments (sketch)*

 Lite Client Accuracy:
 - Assume by contradiction that *h2* was not generated correctly and the lite client sets trust to true because Bisection returns true.
 - Bisection returns true only if all calls to CheckSupport in the recursion return true.
 - Thus we have a sequence of headers that all satisfied the CheckSupport
 - again a contradiction

 Lite Client Completeness:

 This is only ensured if upon *Commit(pivot)* the lite client is always provided with a correctly generated header.

 *Stalling*

 With Bisection, a faulty full node could stall a lite client by creating a long sequence of headers that are queried one-by-one by the lite client and look OK, before the lite client eventually detects a problem. There are several ways to address this:
 * Each call to ```Commit``` could be issued to a different full node
 * Instead of querying header by header, the lite client tells a full node which header it trusts, and the height of the header it needs. The full node responds with the header along with a proof consisting of intermediate headers that the light client can use to verify. Roughly, Bisection would then be executed at the full node.
 * We may set a timeout how long bisection may take.


 ### The case *h2.Header.height < h1.Header.height*

 In the use case where someone tells the lite client that application data that is relevant for it can be read in the block of height *k* and the lite client trusts a more recent header, we can use the hashes to verify headers "down the chain." That is, we iterate down the heights and check the hashes in each step.

 *Remark.* For the case were the lite client trusts two headers *i* and *j* with *i < k < j*, we should discuss/experiment whether the forward or the backward method is more effective.

 ```go
 func Backwards(h1,h2) bool {
  assert (h2.Header.height < h1.Header.height)
  old := h1
  for i := h1.Header.height - 1; i > h2.Header.height; i-- {
    new := Commit(i)
    Store(new)
    if (hash(new) != old.Header.hash) {
      return false
    }
    old := new
  }
  return (hash(h2) == old.Header.hash)
 }
 ```
--- a/spec/consensus/signing.md
+++ b/spec/consensus/signing.md
@ -0,0 +1,205 @@
 # Validator Signing

 Here we specify the rules for validating a proposal and vote before signing.
 First we include some general notes on validating data structures common to both types.
 We then provide specific validation rules for each. Finally, we include validation rules to prevent double-sigining.

 ## SignedMsgType

 The `SignedMsgType` is a single byte that refers to the type of the message
 being signed. It is defined in Go as follows:

 ```go
 // SignedMsgType is a type of signed message in the consensus.
 type SignedMsgType byte

 const (
 	// Votes
 	PrevoteType   SignedMsgType = 0x01
 	PrecommitType SignedMsgType = 0x02

 	// Proposals
 	ProposalType SignedMsgType = 0x20
 )
 ```

 All signed messages must correspond to one of these types.

 ## Timestamp

 Timestamp validation is subtle and there are currently no bounds placed on the
 timestamp included in a proposal or vote. It is expected that validators will honestly
 report their local clock time. The median of all timestamps
 included in a commit is used as the timestamp for the next block height.

 Timestamps are expected to be strictly monotonic for a given validator, though
 this is not currently enforced.

 ## ChainID

 ChainID is an unstructured string with a max length of 50-bytes.
 In the future, the ChainID may become structured, and may take on longer lengths.
 For now, it is recommended that signers be configured for a particular ChainID,
 and to only sign votes and proposals corresponding to that ChainID.

 ## BlockID

 BlockID is the structure used to represent the block:

 ```go
 type BlockID struct {
 	Hash        []byte
 	PartsHeader PartSetHeader
 }

 type PartSetHeader struct {
 	Hash  []byte
 	Total int
 }
 ```

 To be included in a valid vote or proposal, BlockID must either represent a `nil` block, or a complete one.
 We introduce two methods, `BlockID.IsZero()` and `BlockID.IsComplete()` for these cases, respectively.

 `BlockID.IsZero()` returns true for BlockID `b` if each of the following
 are true:

 ```
 b.Hash == nil
 b.PartsHeader.Total == 0
 b.PartsHeader.Hash == nil
 ```

 `BlockID.IsComplete()` returns true for BlockID `b` if each of the following
 are true:

 ```
 len(b.Hash) == 32
 b.PartsHeader.Total > 0
 len(b.PartsHeader.Hash) == 32
 ```

 ## Proposals

 The structure of a proposal for signing looks like:

 ```go
 type CanonicalProposal struct {
 	Type      SignedMsgType // type alias for byte
 	Height    int64         `binary:"fixed64"`
 	Round     int64         `binary:"fixed64"`
 	POLRound  int64         `binary:"fixed64"`
 	BlockID   BlockID
 	Timestamp time.Time
 	ChainID   string
 }
 ```

 A proposal is valid if each of the following lines evaluates to true for proposal `p`:

 ```go
 p.Type == 0x20
 p.Height > 0
 p.Round >= 0
 p.POLRound >= -1
 p.BlockID.IsComplete()
 ```

 In other words, a proposal is valid for signing if it contains the type of a Proposal
 (0x20), has a positive, non-zero height, a
 non-negative round, a POLRound not less than -1, and a complete BlockID.

 ## Votes

 The structure of a vote for signing looks like:

 ```go
 type CanonicalVote struct {
 	Type      SignedMsgType // type alias for byte
 	Height    int64         `binary:"fixed64"`
 	Round     int64         `binary:"fixed64"`
 	BlockID   BlockID
 	Timestamp time.Time
 	ChainID   string
 }
 ```

 A vote is valid if each of the following lines evaluates to true for vote `v`:

 ```
 v.Type == 0x1 || v.Type == 0x2
 v.Height > 0
 v.Round >= 0
 v.BlockID.IsZero() || v.BlockID.IsComplete()
 ```

 In other words, a vote is valid for signing if it contains the type of a Prevote
 or Precommit (0x1 or 0x2, respectively), has a positive, non-zero height, a
 non-negative round, and an empty or valid BlockID.

 ## Invalid Votes and Proposals

 Votes and proposals which do not satisfy the above rules are considered invalid.
 Peers gossipping invalid votes and proposals may be disconnected from other peers on the network.
 Note, however, that there is not currently any explicit mechanism to punish validators signing votes or proposals that fail
 these basic validation rules.

 ## Double Signing

 Signers must be careful not to sign conflicting messages, also known as "double signing" or "equivocating".
 Tendermint has mechanisms to publish evidence of validators that signed conflicting votes, so they can be punished
 by the application. Note Tendermint does not currently handle evidence of conflciting proposals, though it may in the future.

 ### State

 To prevent such double signing, signers must track the height, round, and type of the last message signed.
 Assume the signer keeps the following state, `s`:

 ```go
 type LastSigned struct {
 	Height	int64
 	Round	int64
 	Type	SignedMsgType // byte
 }
 ```

 After signing a vote or proposal `m`, the signer sets:

 ```go
 s.Height = m.Height
 s.Round = m.Round
 s.Type = m.Type
 ```

 ### Proposals

 A signer should only sign a proposal `p` if any of the following lines are true:

 ```
 p.Height > s.Height
 p.Height == s.Height && p.Round > s.Round
 ```

 In other words, a proposal should only be signed if it's at a higher height, or a higher round for the same height.
 Once a proposal or vote has been signed for a given height and round, a proposal should never be signed for the same height and round.

 ### Votes

 A signer should only sign a vote `v` if any of the following lines are true:

 ```
 v.Height > s.Height
 v.Height == s.Height && v.Round > s.Round
 v.Height == s.Height && v.Round == s.Round && v.Step == 0x1 && s.Step == 0x20
 v.Height == s.Height && v.Round == s.Round && v.Step == 0x2 && s.Step != 0x2
 ```

 In other words, a vote should only be signed if it's:

 - at a higher height
 - at a higher round for the same height
 - a prevote for the same height and round where we haven't signed a prevote or precommit (but have signed a proposal)
 - a precommit for the same height and round where we haven't signed a precommit (but have signed a proposal and/or a prevote)

 This means that once a validator signs a prevote for a given height and round, the only other message it can sign for that height and round is a precommit.
 And once a validator signs a precommit for a given height and round, it must not sign any other message for that same height and round.
--- a/spec/consensus/wal.md
+++ b/spec/consensus/wal.md
@ -0,0 +1,32 @@
 # WAL

 Consensus module writes every message to the WAL (write-ahead log).

 It also issues fsync syscall through
 [File#Sync](https://golang.org/pkg/os/#File.Sync) for messages signed by this
 node (to prevent double signing).

 Under the hood, it uses
 [autofile.Group](https://godoc.org/github.com/tendermint/tmlibs/autofile#Group),
 which rotates files when those get too big (> 10MB).

 The total maximum size is 1GB. We only need the latest block and the block before it,
 but if the former is dragging on across many rounds, we want all those rounds.

 ## Replay

 Consensus module will replay all the messages of the last height written to WAL
 before a crash (if such occurs).

 The private validator may try to sign messages during replay because it runs
 somewhat autonomously and does not know about replay process.

 For example, if we got all the way to precommit in the WAL and then crash,
 after we replay the proposal message, the private validator will try to sign a
 prevote. But it will fail. That's ok because we’ll see the prevote later in the
 WAL. Then it will go to precommit, and that time it will work because the
 private validator contains the `LastSignBytes` and then we’ll replay the
 precommit from the WAL.

 Make sure to read about [WAL corruption](https://github.com/tendermint/tendermint/blob/master/docs/tendermint-core/running-in-production.md#wal-corruptionn)
 and recovery strategies.