From ff2104ec0b7d944d0711c4f2fb62f49a6de55687 Mon Sep 17 00:00:00 2001 From: Sergio Mena Date: Tue, 21 Dec 2021 10:24:30 +0100 Subject: [PATCH] Moved the `same_block` boolean to ConsensusParams. Revamped the "Tendermint's behavior" section --- .../abci++/abci++_basic_concepts_002_draft.md | 12 +- spec/abci++/abci++_methods_002_draft.md | 26 +- ...bci++_tmint_expected_behavior_002_draft.md | 350 ++++++++++-------- 3 files changed, 207 insertions(+), 181 deletions(-) diff --git a/spec/abci++/abci++_basic_concepts_002_draft.md b/spec/abci++/abci++_basic_concepts_002_draft.md index c0f431167..cd1575d46 100644 --- a/spec/abci++/abci++_basic_concepts_002_draft.md +++ b/spec/abci++/abci++_basic_concepts_002_draft.md @@ -322,9 +322,9 @@ namely: * the consensus parameter updates * the validator updates -With ABCI++, an Application may decide to keep using the next-block execution model, +With ABCI++, an Application may decide to keep using the next-block execution model; however the new methods introduced, `PrepareProposal` and `ProcessProposal` allow -for a new execution model, called same-block execution. An Application implementing +for a new execution model, called _same-block execution_. An Application implementing this execution model, upon receiving a raw proposal via `PrepareProposal` and potentially modifying the list of transactions, the Application fully executes the resulting prepared proposal as though it was the decided block. The results of the @@ -341,13 +341,19 @@ block execution are used as follows: in the prepared proposal's header. * the consensus parameter updates and validator updates are also provided in `ResponsePrepareProposal` and reflect the result of the prepared proposal's - execution. They come into force in the next height (as opposed to the H+2 rule + execution. They come into force in height H+1 (as opposed to the H+2 rule in next-block execution model). If the Application decides to keep the next-block execution model, it will not provide any data in `ResponsePrepareProposal`, other than an optionally modified transaction list. +The execution model is set in boolean parameter _same_block_ in ConsensusParameters. +It should **not** be changed once the blockchain has started, unless the Application +developers _really_ know what they are doing. + +>**TODO**: Update ConsensusParams struct with "same_block" + ## State Sync State sync allows new nodes to rapidly bootstrap by discovering, fetching, and applying diff --git a/spec/abci++/abci++_methods_002_draft.md b/spec/abci++/abci++_methods_002_draft.md index 73431108e..29dadd323 100644 --- a/spec/abci++/abci++_methods_002_draft.md +++ b/spec/abci++/abci++_methods_002_draft.md @@ -182,7 +182,7 @@ title: Methods | Name | Type | Description | Field Number | |--------|--------|-----------------------------------------------------------------------|--------------| - | height | uint64 | The height of the snapshot the chunks belongs to. | 1 | + | height | uint64 | The height of the snapshot the chunk belongs to. | 1 | | format | uint32 | The application-specific format of the snapshot the chunk belongs to. | 2 | | chunk | uint32 | The chunk index, starting from `0` for the initial chunk. | 3 | @@ -304,11 +304,10 @@ From the App's perspective, they'll probably skip ProcessProposal |-------------------------|--------------------------------------------------|---------------------------------------------------------------------------------------------|--------------| | modified_tx | bool | The Application sets it to true to denote it made changes to transactions | 1 | | tx | repeated [TransactionRecord](#transactionrecord) | Possibly modified list of transactions that have been picked as part of the proposed block. | 2 | - | same_block | bool | If true, Application is in same-block execution mode | 3 | - | data | bytes | The Merkle root hash of the application state. | 4 | - | tx_result | repeated [DeliverTxResult](#delivertxresult) | List of structures containing the data resulting from executing the transactions | 5 | - | validator_updates | repeated [ValidatorUpdate](#validatorupdate) | Changes to validator set (set voting power to 0 to remove). | 6 | - | consensus_param_updates | [ConsensusParams](#consensusparams) | Changes to consensus-critical gas, size, and other parameters. | 7 | + | data | bytes | The Merkle root hash of the application state. | 3 | + | tx_result | repeated [DeliverTxResult](#delivertxresult) | List of structures containing the data resulting from executing the transactions | 4 | + | validator_updates | repeated [ValidatorUpdate](#validatorupdate) | Changes to validator set (set voting power to 0 to remove). | 5 | + | consensus_param_updates | [ConsensusParams](#consensusparams) | Changes to consensus-critical gas, size, and other parameters. | 6 | * **Usage**: * Contains a preliminary block to be proposed, which the Application can modify. @@ -320,10 +319,9 @@ From the App's perspective, they'll probably skip ProcessProposal them in `ResponsePrepareProposal`. In that case, `ResponsePrepareProposal.modified_tx` is set to true. * If `ResponsePrepareProposal.modified_tx` is false, then Tendermint will ignore the contents of `ResponsePrepareProposal.tx`. - * In same-block execution mode, the Application must set `ResponsePrepareProposal.same_block` to true, and, - as a result of executing the block, provide values for `ResponsePrepareProposal.data`, + * In same-block execution mode, the Application must provide values for `ResponsePrepareProposal.data`, `ResponsePrepareProposal.tx_result`, `ResponsePrepareProposal.validator_updates`, and - `ResponsePrepareProposal.consensus_param_updates`. + `ResponsePrepareProposal.consensus_param_updates`, as a result of executing the block. * The values for `ResponsePrepareProposal.validator_updates`, or `ResponsePrepareProposal.consensus_param_updates` may be empty. In this case, Tendermint will keep the current values. @@ -335,8 +333,7 @@ From the App's perspective, they'll probably skip ProcessProposal * `ResponseFinalizeBlock.consensus_param_updates` returned for block `H` apply to the consensus params for the same block `H`. For more information on the consensus parameters, see the [application spec entry on consensus parameters](../abci/apps.md#consensus-parameters). - * In next-block execution mode, the Application must set `ResponsePrepareProposal.same_block` to false. - Tendermint will ignore parameters `ResponsePrepareProposal.tx_result`, + * In next-block execution mode, Tendermint will ignore parameters `ResponsePrepareProposal.tx_result`, `ResponsePrepareProposal.validator_updates`, and `ResponsePrepareProposal.consensus_param_updates`. * As a result of executing the prepared proposal, the Application may produce header events or transaction events. The Application must keep those events until a block is decided and then pass them on to Tendermint via @@ -437,7 +434,7 @@ Note that, if _p_ has a non-`nil` _validValue_, Tendermint will use it as propos * The implementation of `ProcessProposal` MUST be deterministic. Moreover, the value of `ResponseProcessProposal.accept` MUST *exclusively* depend on the parameters passed in the call to `RequestProcessProposal`, and the last committed Application state - (see [Properties](#properties) section below). + (see [Requirements](abci++_app_requirements_002_draft.md) section). * Moreover, application implementors SHOULD always set `ResponseProcessProposal.accept` to _true_, unless they _really_ know what the potential liveness implications of returning _false_ are. @@ -525,11 +522,12 @@ In the cases when _p_'s Tendermint is to broadcast `precommit nil` messages (eit * **Usage**: * If `ResponseVerifyVoteExtension.accept` is _false_, Tendermint will reject the whole received vote. - See the [Properties](#properties) section to understand the potential liveness implications of this. + See the [Requirements](abci++_app_requirements_002_draft.md) section to understand the potential + liveness implications of this. * The implementation of `VerifyVoteExtension` MUST be deterministic. Moreover, the value of `ResponseVerifyVoteExtension.accept` MUST *exclusively* depend on the parameters passed in the call to `RequestVerifyVoteExtension`, and the last committed Application state - (see [Properties](#properties) section below). + (see [Requirements](abci++_app_requirements_002_draft.md) section). * Moreover, application implementors SHOULD always set `ResponseVerifyVoteExtension.accept` to _true_, unless they _really_ know what the potential liveness implications of returning _false_ are. diff --git a/spec/abci++/abci++_tmint_expected_behavior_002_draft.md b/spec/abci++/abci++_tmint_expected_behavior_002_draft.md index bb9aafc28..a36580f05 100644 --- a/spec/abci++/abci++_tmint_expected_behavior_002_draft.md +++ b/spec/abci++/abci++_tmint_expected_behavior_002_draft.md @@ -5,180 +5,202 @@ title: Tendermint's expected behavior # Tendermint's expected behavior ->**TODO**: Try the regex-based explanation for this section. -> -> First summarize the current text into: all good vs all bad -> Then set out the rules (regex? grammar?) - -This section describes what the Application can expect from Tendermint - -The following section uses these definitions: - -* We define the _optimal case_ as a run in which (a) the system behaves synchronously, and (b) there are no Byzantine processes. - The optimal case captures the conditions that hold most of the time, but not always. -* We define the _suboptimal case_ as a run in which (a) the system behaves asynchronously in round 0, height h -- messages may be delayed, - timeouts may be triggered --, (b) it behaves synchronously in all subsequent rounds (_r>0_), and (c) there are no Byzantine processes. - The _suboptimal case_ captures a possible glitch in the network, or some sudden, sporadic performance issue in some validator - (network card glitch, thrashing peak, etc.). -* We define the _general case_ as a run in which (a) the system behaves asynchronously for a number of rounds, - (b) it behaves synchronously after some time, denoted _GST_, which is unknown to the processes, - and (c) there may be up to _f_ Byzantine processes. - -### `PrepareProposal`: Application's expectations - -Given a block height _h_, process _p_'s Tendermint calls `RequestPrepareProposal` under -the following conditions: - -* _p_'s Tendermint may decide at height _h_, without calling `RequestPrepareProposal`. - In the optimal case, this will happen often, as there will only be one proposer for height _h_ - (the one for round 0). -* _p_'s Tendermint may call `RequestPrepareProposal` with a block with no transactions, if - _ConsensusParams_ is configured to produce empty blocks when there are outstanding transactions. - If _ConsensusParams_ is configured to avoid empty blocks, any block passed with a call to - `RequestPrepareProposal` will contain at lesat one transaction. -* In the optimal case, _p_'s Tendermint will call `RequestPrepareProposal` at most once, - as there is only one round. -* In the suboptimal case, _p_'s Tendermint - * will not call `RequestPrepareProposal` for height _h_, if _p_ is neither the proposer - of round 0, nor round 1 - * will call `RequestPrepareProposal` once for height _h_, if _p_ is either the proposer - of round 0, or round 1 (but not both). - * will call `RequestPrepareProposal` twice for height _h_, in the unlikely case that - _p_ is the proposer of round 0, and round 1. -* In the general case, _p_'s Tendermint will potentially span many rounds. So, it will call - `RequestPrepareProposal` a number of times which is impossible to predict. Thus, the - Application will to return numerous _amended_ blocks via `ResponsePrepareProposal`. - - If the application is fully executing the blocks it returns via `ResponsePrepareProposal`, - it should be careful about its usage of resources (memory, disk, etc.) as the number - of `PrepareProposal` for a particular height is not bounded. - -If `PrepareProposal` is called more than once for a height _h_, the Application _is_ allowed -to return different blocks each time. - -### `ProcessProposal`: Application's expectations - -Given a block height _h_, process _p_'s Tendermint calls `RequestProcessProposal` depending on the case: - -* In the optimal case, all Tendermint processes decide in round _r=0_. Let's call _v_ the block proposed by the proposer of round _r=0_, height _h_. - Process _p_'s Application will receive - * exactly one call to `RequestPrepareProposal` if _p_ is the proposer of round 0. If that is the case, _p_ will return _v_ in its call to - `ResponsePrepareProposal` - * exactly one call to `RequestProcessProposal` for block height _h_ containing _v_. - - In addition, the Application may execute _v_ as part of handling the `RequestProcessProposal` call, keep the resulting state as a candidate state, - and apply it when the call to `RequestFinalizeBlock` for _h_ confirms that _v_ is indeed the block decided in height _h_. -* In the suboptimal case, Tendermint processes may decide in round _r=0_ or in round _r=1_. - Therefore, the Application of a process _q_ may receive zero, one or two calls to `RequestProcessProposal`, depending on - whether _q_ is the proposer of round 0 or 1 of height _h_. - Likewise, the Application of a process _q_ may receive one or two calls to `RequestProcessProposal` - with two different values _v0_ and _v1_ while in height _h_. - * There is no way for the Application to predict whether proposed block _v0_ or _v1_ will be decided before `RequestFinalizeBlock` is called. - * The Application may choose to execute either (or both) proposed block as part of handling the `RequestProcessProposal` - call and keep the resulting state as a _candidate state_. - At decision time (i.e. when Tendermint calls `RequestFinalizeBlock`), if the block decided corresponds to one of the candidate states, - it can be committed, otherwise the Application will have to execute the block at this point. -* In the general case, the round in which a Tendermint processes _p_ will decide cannot be forecast. - The number of time _p_'s Tendermint may call `RequestProcessProposal` with different proposed blocks for a given height is *unbounded*. - As a result, the Application may need to deal with an unbounded number of different proposals for a given height, - and also an unbounded number of candidate states if it is fully executing the proposed blocks upon `PrepareProposal` or `ProcessProposal`. - In order protects the processes' stability (memory, CPU), the Application has to: - * Be ready to discard candidate states if they become too many. In other words, the set of candidate states should be managed like a cache. - * Be able to execute the blocks upon `FinalizeBlock` if the block decided was one whose candidate state was discarded. - -### `ExtendVote`: expectations from the Application - ->**TODO** (in different rounds). [Finish first discussion above] - --- - -If `ProcessProposal`'s outcome is _Reject_ for some proposed block. Tendermint guarantees that the block will not be the decision. - --- - -The validity of every transaction in a block (from the App's point of view), as well as the hashes in its header can be guaranteed if: - -* `ProcessProposal` *synchronously* handles every proposed block as though Tendermint had already decided on it. -* All the properties of `ProcessProposal` and `FinalizeBlock` mentioned above hold. - -## [Points to discuss further] - -### Byzantine proposer - ->**TODO** [From Josef] We should understand the influence of equivocation on proposals in ABCI++ -> ->N.B.#1: If Byzantine proposer proposes both A and B in the same round, today we might only receive A (proposal) and not B (due to p2p implementation) -> ->Sergio: some thoughts: -> ->[Thought 1] If Tendermint delivers all proposals from a Byzantine proposer _p_ in _h_ and _r_ to the App, we're vulnerable to DoS attacks, ->as _p_ can send as many as it wishes: -> ->* Assuming N.B.#1 above gets "fixed" (A and B get delivered by p2p) ->* Assuming the App fully executes the proposed block at _ProcessProposal_ time -> ->So, whenever N.B.#1 is "fixed" at p2p level, we need to ensure that only the the first proposal in round _r_, height _h_ ->gets delivered to the Application. Any subsequent proposal for that round should just be used to submit evidence, but not given to the App. -> ->[Thought 2] In terms of the consensus's safety properties, as far as pure equivocation on proposals is concerned, ->I think we are OK as long as _f" +offer-snapshot = %s"" +apply-chunk = %s"" +info = %s"" +prepare-proposal = %s"" +process-proposal = %s"" +extend-vote = %s"" +got-vote = %s"" +decide = %s"" +``` + +>**TODO** We need to get the grammar reviewed by the people that know block-execution inside out. + +>**TODO** Still hesitating... introduce _n_ as total number of validators, so that we can bound the occurrences of +>`got-vote` in a round. + +We have kept some of the ABCI++ methods out of the grammar, in order to keep it as clear and concise as possible. +A common reason for keeping all these methods out is that they all can be called at any point in a sequence defined +by the grammar above. Other reasons depend on the method in question: + +* `Echo` and `Flush` are only used for debugging purposes. Further, their handling by the Application should be trivial. +* `CheckTx` is detached from the main method call sequence that drives block execution. +* `Query` provides read-only access to the current Application state, so handling it should also be independent from + block execution. +* Similarly, `ListSnapshots` and `LoadSnapshotChunk` provide read-only access to the Application's previously created + snapshots (if any), and help populate the parameters of `OfferSnapshot` and `ApplySnapshotChunk` at a process performing + state-sync while bootstrapping. Unlike `ListSnapshots` and `LoadSnapshotChunk`, both `OfferSnapshot` + and `ApplySnapshotChunk` _are_ included in the grammar. + +Finally, method `Info` is a special case. The method's purpose is three-fold, it can be used + +1. as part of handling an RPC call from an external client, +2. as a handshake between Tendermint and the Application upon recovery to check whether any blocks need + to be replayed, and +3. at the end of _state-sync_ to verify that the correct state has been reached. + +We have left `Info`'s first purpose out of the grammar for the same reasons as all the others: it can happen +at any time, and has nothing to do with the block execution sequence. The second and third purposes, on the other +hand, are present in the grammar. + +Let us now examine the grammar line by line, providing further details. + +* When a process starts, it may do so for the first time or after a crash (it is recovering). +>```abnf +>start = clean-start / recovery +>``` + +* If the process is starting from scratch, Tendermint first calls `InitChain`, then it may optionally + start a _state-sync_ mechanism to catch up with other processes. Finally, it enters normal + consensus execution, which is a sequence of zero or more consensus heights. +>```abnf +>clean-start = init-chain [state-sync] *consensus-height +>``` + +* In _state-sync_ mode, Tendermint makes one or more attempts at synchronizing the Application's state. + At the beginning of each attempt, it offers the Application a snapshot found at another process. + If the Application accepts the snapshop, at sequence of calls to `ApplySnapshotChunk` method follow + to provide the Application with all the snapshots needed, in order, to reconstruct the state locally. + At the end of a successful attempt, Tendermint calls `Info` to make sure the recontructed state's + _AppHash_ matches the one in the block header at the corresponding height. +>```abnf +>state-sync = 1*state-sync-attempt info +>state-sync-attempt = offer-snapshot *apply-chunk +>``` + +* In recovery mode, Tendermint first calls `Info` to know from which height it needs to replay decisions + to the Application. To replay a decision, Tendermint simply calls `FinalizeBlock` with the decided + block at that height. After this, Tendermint enters nomal consensus execution: zero or more consensus heights. +>```abnf +>recovery = info *consensus-replay *consensus-height +>consensus-replay = decide +>``` + +* A consensus height consists of zero or more rounds before deciding via a call to `FinalizeBlock`. + In each round, the sequence of method calls depends on whether the local process is the proposer or not. +>```abnf +>consensus-height = *consensus-round decide +>consensus-round = proposer / non-proposer +>``` + +* If the local process is the proposer of the current round, Tendermint starts by calling `PrepareProposal`. + No calls to methods related to vote extensions (`ExtendVote`, `VerifyVoteExtension`) can be called + in the present round before `PrepareProposal`. Once `PrepareProposal` is called, calls to + `ExtendVote` and `VerifyVoteExtension` can come in any order, although the former will be called + at most once in this round. +>```abnf +>proposer = prepare-proposal extend-proposer +>extend-proposer = *got-vote [extend-vote] *got-vote +>``` + +* If the local process is not the proposer of the current round, Tendermint will call `ProcessProposal` + at most once. At most one call to `ExtendVote` can occur only after `ProcessProposal` is called. + A number of calls to `VerifyVoteExtension` can occur in any order with respect to `ProcessProposal` + and `ExtendVote` throughout the round. +>```abnf +>non-proposer = *got-vote [extend-non-proposer] *got-vote +>extend-non-proposer = process-proposal *got-vote [extend-vote] +>``` + +* Finally, the grammar describes all its terminal symbols, which denote the different ABCI++ method calls that + may appear in a sequence. +>```abnf +>init-chain = %s"" +>offer-snapshot = %s"" +>apply-chunk = %s"" +>info = %s"" +>prepare-proposal = %s"" +>process-proposal = %s"" +>extend-vote = %s"" +>got-vote = %s"" +>decide = %s"" +>``` + +## `ProcessProposal`'s timeout (a.k.a. Zarko's Github comment in Issue#351) >**TODO** (to discuss): `PrepareProposal` is called synchronously. `ProcessProposal` may also want to fully process the block synchronously. >However, they stand on Tendermint's critical path, so the Tendermint's Propose timeout needs to accomodate that. > >Idea: Make propose timestamp (currently hardcoded to 3 secs in the Tendermint Go implementation) part of ConsensusParams, >so the App can adjust it with its knowledge of the time may take to prepare/process the proposal. +> +>This should probably go elsewhere in the spec. -# Failure modes - ->**TODO** Is it worth explaining the failure modes? Since we're going for halt, and can't configure them. - -# Application modes - -[This is a sketch ATM] - -Mode 1: Simple mode (equivalent to ABCI). - -**TODO**: Define _candidate block_ - -Definition: "candidate state" is the App state resulting from the optimistic exectution of a block that is not decided yet by consensus. An application managing candidate states needs to be able to discard them and recover the previous state - -* PrepareProposal: Set `ResponsePrepareProposal.modified` to false and return -* ProcessProposal: keep the block data in memory (indexed by app hash) as _candidate block_ and return Accept -* ExtendVote: return empty byte array -* VerifyVoteExtension: if the byte array is empty, return Accept; else, return Reject -* Finalize block: look up the block by the app hash and fully execute it. Discard all other candidate blocks - -Mode 2: Basic checks on proposals. - -* PrepareProposal: Go through the transactions, apply basic app-dependent checks based on the block and last committed state. Remove/reorder invalid transactions in the block -* ProcessProposal: Same as PrepareProposal; return Reject if invalid transactions are detected. If Accept, keep the block data in memory (indexed by app hash) as _candidate block_. -* ExtendVote: return empty byte array -* VerifyVoteExtension: if the byte array is empty, return Accept; else, return Reject -* Finalize block: look up the block by the app hash and fully execute it. Discard all other candidate blocks - -Mode 3: Full check of proposals. Optimistic (or immediate) execution. No candidate state management. - -* PrepareProposal: fully execute the block, but discard the resulting state. Remove/reorder invalid transactions in the block. -* ProcessProposal: Same as PrepareProposal. Return Reject if invalid transactions are detected. If Accept, keep the block data in memory (indexed by app hash) as _candidate block_. -* ExtendVote: return empty byte array -* VerifyVoteExtension: if the byte array is empty, return Accept; else, return Reject -* Finalize block: look up the block by the app hash and fully execute it. Discard all other candidate blocks +## Failure modes -This mode guarantees that no invalid transactions will ever make it into a committed block +>**TODO** Is it worth explaining the failure modes? Since we're going for halt, and can't configure them... -Mode 4: Mode 3 + candidate state management. +## Adapting existing Applications that use ABCI -* PrepareProposal: First Remove/reorder invalid transactions in the block. If _v_ is not in the set of candidate states, fully execute the block, add the resulting state as candidate state for value _v_, height _h_. -* ProcessProposal: Same as PrepareProposal. Return Reject if invalid transactions are detected -* ExtendVote: return empty byte array -* VerifyVoteExtension: if the byte array is empty, return Accept; else, return Reject -* Finalize block: commit candidate state corresponding to _v_, discard all other candidate states. +In some cases, an existing Application using the legacy ABCI may need to be adapted to work with ABCI++ +with as minimal changes as possible. In this case, of course, ABCI++ will not provide any advange with respect +to the existing implementation, but will keep the same guarantees already provided by ABCI. +Here is how ABCI++ methods should be implemented. -Mode 5: Mode 4 + AppHash for heigh _h_ is in _h_'s header +First of all, all the methods that did not change from ABCI to ABCI++, namely `Echo`, `Flush`, `Info`, `InitChain`, +`Query`, `CheckTx`, `ListSnapshots`, `LoadSnapshotChunk`, `OfferSnapshot`, and `ApplySnapshotChunk`, do not need +to undergo any changes in their implementation. ->**TODO** Mode 6: With vote extensions (?) +As for the new methods: ->**TODO**: Explain the two workflows discussed with Callum: App hash on N+1 vs App hash on N. How to capture it in ABCI++ ? +* `PrepareProposal` should set `ResponsePrepareProposal.modified_tx` to _false_ and return. +* `ProcessProposal` should set `ResponseProcessProposal.accept` to _true_ and return. +* `ExtendVote` should set `ResponseExtendVote.extension` to an empty byte array and return. +* `VerifyVoteExtension` should set `ResponseVerifyVoteExtension.accept` to _true_ if the extension is an empty byte array + and _false_ otherwise, then return. +* `FinalizeBlock` should coalesce the implementation of methods `BeginBlock`, `DeliverTx`, `EndBlock`, and `Commit`. + The logic extracted from `DeliverTx` should be wrappped by a loop that will execute as many times as + transactions exist in `RequestFinalizeBlock.tx`.