Update spec so that Tendermnit can propose more txs than the size limit in

spec/abci++/abci++_app_requirements_002_draft.md

@@ -46,33 +46,43 @@ Full execution of blocks at `PrepareProposal` time stands on Tendermint's critic
 Requirement 3 ensures the Application will set a value for _TimeoutPropose_ such that the time it takes
 to fully execute blocks in `PrepareProposal` does not interfere with Tendermint's propose timer.
 
-* Requirement 4 [`PrepareProposal`, `ProcessProposal`, coherence]: For any two correct processes $p$ and $q$,
+* Requirement 4 [`PrepareProposal`, tx-size] When $p$'s Application calls `ResponsePrepareProposal`, the
+  total size in bytes of the transactions returned does not exceed `RequestPrepareProposal.max_tx_bytes`.
+
+Busy blockchains might seek to maximize the amount of transactions included in each block. Under those conditions,
+Tendermint might choose to increase the transactions passed to the Application via `RequestPrepareProposal.txs`
+beyond the `RequestPrepareProposal.max_tx_bytes` limit. The idea is that, if the Application drops some of
+those transactions, it can still return a transaction list whose byte size is as close to
+`RequestPrepareProposal.max_tx_bytes` as possible. Thus, Requirement 4 ensures that the size in bytes of the
+transaction list returned by the application will never cause the resulting block to go beyond its byte limit.
+
+* Requirement 5 [`PrepareProposal`, `ProcessProposal`, coherence]: For any two correct processes $p$ and $q$,
   if $q$'s Tendermint calls `RequestProcessProposal` on $v'_p$,
   $q$'s Application returns Accept in `ResponseProcessProposal`.
 
-Requirement 4 makes sure that blocks proposed by correct processes _always_ pass the correct receiving process's
+Requirement 5 makes sure that blocks proposed by correct processes _always_ pass the correct receiving process's
 `ProcessProposal` check.
 On the other hand, if there is a deterministic bug in `PrepareProposal` or `ProcessProposal` (or in both),
 strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice,
 as very often validators are running the Application from the same codebase, so potentially _all_ would
 likely hit the bug at the same time. This would result in most (or all) processes prevoting `nil`, with the
-serious consequences on Tendermint's liveness that this entails. Due to its criticality, Requirement 4 is a
+serious consequences on Tendermint's liveness that this entails. Due to its criticality, Requirement 5 is a
 target for extensive testing and automated verification.
 
-* Requirement 5 [`ProcessProposal`, determinism-1]: `ProcessProposal` is a (deterministic) function of the current
+* Requirement 6 [`ProcessProposal`, determinism-1]: `ProcessProposal` is a (deterministic) function of the current
   state and the block that is about to be applied. In other words, for any correct process $p$, and any arbitrary block $v'$,
   if $p$'s Tendermint calls `RequestProcessProposal` on $v'$ at height $h$,
   then $p$'s Application's acceptance or rejection **exclusively** depends on $v'$ and $s_{p,h-1}$.
 
-* Requirement 6 [`ProcessProposal`, determinism-2]: For any two correct processes $p$ and $q$, and any arbitrary block $v'$,
+* Requirement 7 [`ProcessProposal`, determinism-2]: For any two correct processes $p$ and $q$, and any arbitrary block $v'$,
   if $p$'s (resp. $q$'s) Tendermint calls `RequestProcessProposal` on $v'$ at height $h$,
   then $p$'s Application accepts $v'$ if and only if $q$'s Application accepts $v'$.
-  Note that this requirement follows from Requirement 5 and the Agreement property of consensus.
+  Note that this requirement follows from Requirement 6 and the Agreement property of consensus.
 
-Requirements 5 and 6 ensure that all correct processes will react in the same way to a proposed block, even
+Requirements 6 and 7 ensure that all correct processes will react in the same way to a proposed block, even
 if the proposer is Byzantine. However, `ProcessProposal` may contain a bug that renders the
 acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting
-the bug from fulfilling Requirements 5 or 6 (effectively making those processes Byzantine).
+the bug from fulfilling Requirements 6 or 7 (effectively making those processes Byzantine).
 In such a scenario, Tendermint's liveness cannot be guaranteed.
 Again, this is a problem in practice if most validators are running the same software, as they are likely
 to hit the bug at the same point. There is currently no clear solution to help with this situation, so
@@ -85,43 +95,43 @@ is about to broadcast a non-`nil` precommit message, a correct process can only
 Let $e^r_p$ be the vote extension that the Application of a correct process $p$ returns via `ResponseExtendVote` in round $r$, height $h$.
 Let $w^r_p$ be the proposed block that $p$'s Tendermint passes to the Application via `RequestExtendVote` in round $r$, height $h$.
 
-* Requirement 7 [`ExtendVote`, `VerifyVoteExtension`, coherence]: For any two correct processes $p$ and $q$, if $q$ receives $e^r_p$
+* Requirement 8 [`ExtendVote`, `VerifyVoteExtension`, coherence]: For any two correct processes $p$ and $q$, if $q$ receives $e^r_p$
   from $p$ in height $h$, $q$'s Application returns Accept in `ResponseVerifyVoteExtension`.
 
-Requirement 7 constrains the creation and handling of vote extensions in a similar way as Requirement 4
+Requirement 8 constrains the creation and handling of vote extensions in a similar way as Requirement 5
 contrains the creation and handling of proposed blocks.
-Requirement 7 ensures that extensions created by correct processes _always_ pass the `VerifyVoteExtension`
+Requirement 8 ensures that extensions created by correct processes _always_ pass the `VerifyVoteExtension`
 checks performed by correct processes receiving those extensions.
 However, if there is a (deterministic) bug in `ExtendVote` or `VerifyVoteExtension` (or in both),
-we will face the same liveness issues as described for Requirement 4, as Precommit messages with invalid vote
+we will face the same liveness issues as described for Requirement 5, as Precommit messages with invalid vote
 extensions will be discarded.
 
-* Requirement 8 [`VerifyVoteExtension`, determinism-1]: `VerifyVoteExtension` is a (deterministic) function of
+* Requirement 9 [`VerifyVoteExtension`, determinism-1]: `VerifyVoteExtension` is a (deterministic) function of
   the current state, the vote extension received, and the prepared proposal that the extension refers to.
   In other words, for any correct process $p$, and any arbitrary vote extension $e$, and any arbitrary
   block $w$, if $p$'s (resp. $q$'s) Tendermint calls `RequestVerifyVoteExtension` on $e$ and $w$ at height $h$,
   then $p$'s Application's acceptance or rejection **exclusively** depends on $e$, $w$ and $s_{p,h-1}$.
 
-* Requirement 9 [`VerifyVoteExtension`, determinism-2]: For any two correct processes $p$ and $q$,
+* Requirement 10 [`VerifyVoteExtension`, determinism-2]: For any two correct processes $p$ and $q$,
   and any arbitrary vote extension $e$, and any arbitrary block $w$,
   if $p$'s (resp. $q$'s) Tendermint calls `RequestVerifyVoteExtension` on $e$ and $w$ at height $h$,
   then $p$'s Application accepts $e$ if and only if $q$'s Application accepts $e$.
-  Note that this requirement follows from Requirement 8 and the Agreement property of consensus.
+  Note that this requirement follows from Requirement 9 and the Agreement property of consensus.
 
-Requirements 8 and 9 ensure that the validation of vote extensions will be deterministic at all
+Requirements 9 and 10 ensure that the validation of vote extensions will be deterministic at all
 correct processes.
-Requirements 8 and 9 protect against arbitrary vote extension data from Byzantine processes
-similarly to Requirements 5 and 6 and proposed blocks.
-Requirements 8 and 9 can be violated by a bug inducing non-determinism in
+Requirements 9 and 10 protect against arbitrary vote extension data from Byzantine processes
+similarly to Requirements 6 and 7 and proposed blocks.
+Requirements 9 and 10 can be violated by a bug inducing non-determinism in
 `VerifyVoteExtension`. In this case liveness can be compromised.
 Extra care should be put in the implementation of `ExtendVote` and `VerifyVoteExtension` and,
 as a general rule, `VerifyVoteExtension` _should_ always accept the vote extension.
 
-* Requirement 10 [_all_, no-side-effects]: $p$'s calls to `RequestPrepareProposal`,
+* Requirement 11 [_all_, no-side-effects]: $p$'s calls to `RequestPrepareProposal`,
   `RequestProcessProposal`, `RequestExtendVote`, and `RequestVerifyVoteExtension` at height $h$ do
   not modify $s_{p,h-1}$.
 
-* Requirement 11 [`ExtendVote`, `FinalizeBlock`, non-dependency]: for any correct process $p$,
+* Requirement 12 [`ExtendVote`, `FinalizeBlock`, non-dependency]: for any correct process $p$,
 and any vote extension $e$ that $p$ received at height $h$, the computation of
 $s_{p,h}$ does not depend on $e$.
 
@@ -133,16 +143,13 @@ Additionally,
 * in same-block execution mode, $p$'s `PrepareProposal` creates a set of transaction results $T_{p,h}$
   if $p$ was the proposer of $v_{p,h}$, otherwise `FinalizeBlock` creates $T_{p,h}$.
 
->**TODO** I have left out all the "events" as they don't have any impact in safety or liveness
->(same for consensus params, and validator set)
-
-* Requirement 12 [`FinalizeBlock`, determinism-1]: For any correct process $p$,
+* Requirement 13 [`FinalizeBlock`, determinism-1]: For any correct process $p$,
   $s_{p,h}$ exclusively depends on $s_{p,h-1}$ and $v_{p,h}$.
 
-* Requirement 13 [`FinalizeBlock`, determinism-2]: For any correct process $p$,
+* Requirement 14 [`FinalizeBlock`, determinism-2]: For any correct process $p$,
   the contents of $T_{p,h}$ exclusively depend on $s_{p,h-1}$ and $v_{p,h}$.
 
-Note that Requirements 12 and 13, combined with Agreement property of consensus ensure
+Note that Requirements 13 and 14, combined with Agreement property of consensus ensure
 the Application state evolves consistently at all correct processes.
 
 Finally, notice that neither `PrepareProposal` nor `ExtendVote` have determinism-related

spec/abci++/abci++_methods_002_draft.md

@@ -318,8 +318,11 @@ title: Methods
             * If the Application wants to add a new transaction, then the Application should include it in `tx_records` and _mark_ it as `ADD`. In this case, Tendermint will add it to the mempool.
         * The Application should be aware that removing and adding transactions may compromise _traceability_.
           > Consider the following example: the Application transforms a client-submitted transaction `t1` into a second transaction `t2`, i.e., the Application asks Tendermint to remove `t1` and add `t2` to the mempool. If a client wants to eventually check what happened to `t1`, it will discover that `t_1` is not in the mempool or in a committed block, getting the wrong idea that `t_1` did not make it into a block. Note that `t_2` _will be_ in a committed block, but unless the Application tracks this information, no component will be aware of it. Thus, if the Application wants traceability, it is its responsability to support it. For instance, the Application could attach to a transformed transaction a list with the hashes of the transactions it derives from. 
-        * If the Application modifies the set of transactions, the modified transactions MUST NOT exceed the configured maximum size `RequestPrepareProposal.max_tx_bytes`.
     * If the Application does not modify the preliminary set of transactions `txs`, then it sets `ResponsePrepareProposal.modified_tx_status` to `UNMODIFIED`. In this case, Tendermint will ignore the contents of `ResponsePrepareProposal.tx_records`.
+    * In order to optimize the block size in busy blockchains, Tendermint might include a list of transactions in `RequestPrepareProposal.txs` whose size
+      in bytes exceeds `RequestPrepareProposal.max_tx_bytes`. In this case:
+        * The Application MUST still abide by the `RequestPrepareProposal.max_tx_bytes` limit in the list of transaction records it returns (excluding those marked as `REMOVE`).
+        * It is _invalid_ to return `ResponsePrepareProposal.modified_tx_status` set to `UNMODIFIED`: Tendermint will panic.
     * In same-block execution mode, the Application must provide values for `ResponsePrepareProposal.app_hash`,
       `ResponsePrepareProposal.tx_results`, `ResponsePrepareProposal.validator_updates`, and
       `ResponsePrepareProposal.consensus_param_updates`, as a result of fully executing the block.

spec/abci++/abci++_tmint_expected_behavior_002_draft.md

@@ -202,7 +202,10 @@ to undergo any changes in their implementation.
 
 As for the new methods:
 
-* `PrepareProposal` should set `ResponsePrepareProposal.modified_tx` to _false_ and return.
+* `PrepareProposal` should check whether the size of transactions exceeds the byte limit.
+    * If it does: remove transactions at the end of the list until the total byte size conforms to the limit,
+      then set `ResponsePrepareProposal.modified_tx_status` to `MODIFIED` and return.
+    * Else, set `ResponsePrepareProposal.modified_tx_status` to `UNMODIFIED` and return.
 * `ProcessProposal` should set `ResponseProcessProposal.accept` to _true_ and return.
 * `ExtendVote` should set `ResponseExtendVote.extension` to an empty byte array and return.
 * `VerifyVoteExtension` should set `ResponseVerifyVoteExtension.accept` to _true_ if the extension is an empty byte array