You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

298 lines
18 KiB

  1. # ADR 059: Evidence Composition and Lifecycle
  2. ## Changelog
  3. - 04/09/2020: Initial Draft (Unabridged)
  4. - 07/09/2020: First Version
  5. ## Scope
  6. This document is designed to collate together and surface some predicaments involving evidence in Tendermint: both its composition and lifecycle. It then aims to find a solution to these. The scope does not extend to the verification nor detection of certain types of evidence but concerns itself mainly with the general form of evidence and how it moves from inception to application.
  7. ## Background
  8. For a long time `DuplicateVoteEvidence`, formed in the consensus reactor, was the only evidence Tendermint had. It was produced whenever two votes from the same validator in the same round
  9. was observed and thus it was designed that each evidence was for a single validator. It was predicted that there may come more forms of evidence and thus `DuplicateVoteEvidence` was used as the model for the `Evidence` interface and also for the form of the evidence data sent to the application. It is important to note that Tendermint concerns itself just with the detection and reporting of evidence and it is the responsibility of the application to exercise punishment.
  10. ```go
  11. type Evidence interface { //existing
  12. Height() int64 // height of the offense
  13. Time() time.Time // time of the offense
  14. Address() []byte // address of the offending validator
  15. Bytes() []byte // bytes which comprise the evidence
  16. Hash() []byte // hash of the evidence
  17. Verify(chainID string, pubKey crypto.PubKey) error // verify the evidence
  18. Equal(Evidence) bool // check equality of evidence
  19. ValidateBasic() error
  20. String() string
  21. }
  22. ```
  23. ```go
  24. type DuplicateVoteEvidence struct {
  25. VoteA *Vote
  26. VoteB *Vote
  27. timestamp time.Time // taken from the block time
  28. }
  29. ```
  30. Tendermint has now introduced a new type of evidence to protect light clients from being attacked. This `LightClientAttackEvidence` (see [here](https://github.com/informalsystems/tendermint-rs/blob/31ca3e64ce90786c1734caf186e30595832297a4/docs/spec/lightclient/attacks/evidence-handling.md) for more information) is vastly different to `DuplicateVoteEvidence` in that it is physically a much different size containing a complete signed header and validator set. It is formed within the light client, not the consensus reactor and requires a lot more information from state to verify (`VerifyLightClientAttack(commonHeader, trustedHeader *SignedHeader, commonVals *ValidatorSet)` vs `VerifyDuplicateVote(chainID string, pubKey PubKey)`). Finally it batches validators together (a single piece of evidence that implicates multiple malicious validators at a height) as opposed to having individual evidence (each piece of evidence is per validator per height). This evidence stretches the existing mould that was used to accommodate new types of evidence and has thus caused us to reconsider how evidence should be formatted and processed.
  31. ```go
  32. type LightClientAttackEvidence struct { // proposed struct in spec
  33. ConflictingBlock *LightBlock
  34. CommonHeight int64
  35. Type AttackType // enum: {Lunatic|Equivocation|Amnesia}
  36. timestamp time.Time // taken from the block time at the common height
  37. }
  38. ```
  39. *Note: These three attack types have been proven by the research team to be exhaustive*
  40. ## Possible Approaches for Evidence Composition
  41. ### Individual framework
  42. Evidence remains on a per validator basis. This causes the least disruption to the current processes but requires that we break `LightClientAttackEvidence` into several pieces of evidence for each malicious validator. This not only has performance consequences in that there are n times as many database operations and that the gossiping of evidence will require more bandwidth then necessary (by requiring a header for each piece) but it potentially impacts our ability to validate it. In batch form, the full node can run the same process the light client did to see that 1/3 validating power was present in both the common block and the conflicting block whereas this becomes more difficult to verify individually without opening the possibility that malicious validators forge evidence against innocent . Not only that, but `LightClientAttackEvidence` also deals with amnesia attacks which unfortunately have the characteristic where we know the set of validators involved but not the subset that were actually malicious (more to be said about this later). And finally splitting the evidence into individual pieces makes it difficult to understand the severity of the attack (i.e. the total voting power involved in the attack)
  43. #### An example of a possible implementation path
  44. We would ignore amnesia evidence (as individually it's hard to make) and revert to the initial split we had before where `DuplicateVoteEvidence` is also used for light client equivocation attacks and thus we only need `LunaticEvidence`. We would also most likely need to remove `Verify` from the interface as this isn't really something that can be used.
  45. ``` go
  46. type LunaticEvidence struct { // individual lunatic attack
  47. header *Header
  48. commonHeight int64
  49. vote *Vote
  50. timestamp time.Time // once again taken from the block time at the height of the common header
  51. }
  52. ```
  53. ### Batch Framework
  54. The last approach of this category would be to consider batch only evidence. This works fine with `LightClientAttackEvidence` but would require alterations to `DuplicateVoteEvidence` which would most likely mean that the consensus would send conflicting votes to a buffer in the evidence module which would then wrap all the votes together per height before gossiping them to other nodes and trying to commit it on chain. At a glance this may improve IO and verification speed and perhaps more importantly grouping validators gives the application and Tendermint a better overview of the severity of the attack.
  55. However individual evidence has the advantage that it is easy to check if a node already has that evidence meaning we just need to check hashes to know that we've already verified this evidence before. Batching evidence would imply that each node may have a different combination of duplicate votes which may complicate things.
  56. #### An example of a possible implementation path
  57. `LightClientAttackEvidence` won't change but the evidence interface will need to look like the proposed one above and `DuplicateVoteEvidence` will need to change to encompass multiple double votes. A problem with batch evidence is that it needs to be unique to avoid people from submitting different permutations.
  58. ## Decision
  59. The decision is to adopt a hybrid design.
  60. We allow individual and batch evidence to coexist together, meaning that verification is done depending on the evidence type and that the bulk of the work is done in the evidence pool itself (including forming the evidence to be sent to the application).
  61. ## Detailed Design
  62. Evidence has the following simple interface:
  63. ```go
  64. type Evidence interface { //proposed
  65. Height() int64 // height of the offense
  66. Bytes() []byte // bytes which comprise the evidence
  67. Hash() []byte // hash of the evidence
  68. ValidateBasic() error
  69. String() string
  70. }
  71. ```
  72. The changing of the interface is backwards compatible as these methods are all present in the previous version of the interface. However, networks will need to upgrade to be able to process the new evidence as verification has changed.
  73. We have two concrete types of evidence that fulfil this interface
  74. ```go
  75. type LightClientAttackEvidence struct {
  76. ConflictingBlock *LightBlock
  77. CommonHeight int64 // the last height at which the primary provider and witness provider had the same header
  78. }
  79. ```
  80. where the `Hash()` is the hash of the header and commonHeight.
  81. Note: It was also discussed whether to include the commit hash which captures the validators that signed the header. However this would open the opportunity for someone to propose multiple permutations of the same evidence (through different commit signatures) hence it was omitted. Consequentially, when it comes to verifying evidence in a block, for `LightClientAttackEvidence` we can't just check the hashes because someone could have the same hash as us but a different commit where less than 1/3 validators voted which would be an invalid version of the evidence. (see `fastCheck` for more details)
  82. ```go
  83. type DuplicateVoteEvidence {
  84. VoteA *Vote
  85. VoteB *Vote
  86. }
  87. ```
  88. where the `Hash()` is the hash of the two votes
  89. For both of these types of evidence, `Bytes()` represents the proto-encoded byte array format of the evidence and `ValidateBasic` is
  90. an initial consistency check to make sure the evidence has a valid structure.
  91. ### The Evidence Pool
  92. `LightClientAttackEvidence` is generated in the light client and `DuplicateVoteEvidence` in consensus. Both are sent to the evidence pool through `AddEvidence(ev Evidence) error`. The evidence pool's primary purpose is to verify evidence. It also gossips evidence to other peers' evidence pool and serves it to consensus so it can be committed on chain and the relevant information can be sent to the application in order to exercise punishment. When evidence is added, the pool first runs `Has(ev Evidence)` to check if it has already received it (by comparing hashes) and then `Verify(ev Evidence) error`. Once verified the evidence pool stores it it's pending database. There are two databases: one for pending evidence that is not yet committed and another of the committed evidence (to avoid committing evidence twice)
  93. #### Verification
  94. `Verify()` does the following:
  95. - Use the hash to see if we already have this evidence in our committed database.
  96. - Use the height to check if the evidence hasn't expired.
  97. - If it has expired then use the height to find the block header and check if the time has also expired in which case we drop the evidence
  98. - Then proceed with switch statement for each of the two evidence:
  99. For `DuplicateVote`:
  100. - Check that height, round, type and validator address are the same
  101. - Check that the Block ID is different
  102. - Check the look up table for addresses to make sure there already isn't evidence against this validator
  103. - Fetch the validator set and confirm that the address is in the set at the height of the attack
  104. - Check that the chain ID and signature is valid.
  105. For `LightClientAttack`
  106. - Fetch the common signed header and val set from the common height and use skipping verification to verify the conflicting header
  107. - Fetch the trusted signed header at the same height as the conflicting header and compare with the conflicting header to work out which type of attack it is and in doing so return the malicious validators.
  108. - If equivocation, return the validators that signed for the commits of both the trusted and signed header
  109. - If lunatic, return the validators from the common val set that signed in the conflicting block
  110. - If amnesia, return no validators (since we can't know which validators are malicious). This also means that we don't currently send amnesia evidence to the application, although we will introduce more robust amnesia evidence handling in future Tendermint Core releases
  111. - For each validator, check the look up table to make sure there already isn't evidence against this validator
  112. After verification we persist the evidence with the key `height/hash` to the pending evidence database in the evidence pool with the following format:
  113. ```go
  114. type EvidenceInfo struct {
  115. ev Evidence
  116. time time.Time
  117. validators []Validator
  118. totalVotingPower int64
  119. }
  120. ```
  121. `time`, `validators` and `totalVotingPower` are need to form the `abci.Evidence` that we send to the application layer. More in this to come later.
  122. #### Broadcasting and receiving evidence
  123. The evidence pool also runs a reactor that broadcasts the newly validated
  124. evidence to all connected peers.
  125. Receiving evidence from other evidence reactors works in the same manner as receiving evidence from the consensus reactor or a light client.
  126. #### Proposing evidence on the block
  127. When it comes to prevoting and precomitting a proposal that contains evidence, the full node will once again
  128. call upon the evidence pool to verify the evidence using `CheckEvidence(ev []Evidence)`:
  129. This performs the following actions:
  130. 1. Loops through all the evidence to check that nothing has been duplicated
  131. 2. For each evidence, run `fastCheck(ev evidence)` which works similar to `Has` but instead for `LightClientAttackEvidence` if it has the
  132. same hash it then goes on to check that the validators it has are all signers in the commit of the conflicting header. If it doesn't pass fast check (because it hasn't seen the evidence before) then it will have to verify the evidence.
  133. 3. runs `Verify(ev Evidence)` - Note: this also saves the evidence to the db as mentioned before.
  134. #### Updating application and pool
  135. The final part of the lifecycle is when the block is committed and the `BlockExecutor` then updates state. As part of this process, the `BlockExecutor` gets the evidence pool to create a simplified format for the evidence to be sent to the application. This happens in `ApplyBlock` where the executor calls `Update(Block, State) []abci.Evidence`.
  136. ```go
  137. abciResponses.BeginBlock.ByzantineValidators = evpool.Update(block, state)
  138. ```
  139. Here is the format of the evidence that the application will receive. As seen above, this is stored as an array within `BeginBlock`.
  140. The changes to the application are minimal (it is still formed one for each malicious validator) with the exception of using an enum instead of a string for the evidence type.
  141. ```go
  142. type Evidence struct {
  143. // either LightClientAttackEvidence or DuplicateVoteEvidence as an enum (abci.EvidenceType)
  144. Type EvidenceType `protobuf:"varint,1,opt,name=type,proto3,enum=tendermint.abci.EvidenceType" json:"type,omitempty"`
  145. // The offending validator
  146. Validator Validator `protobuf:"bytes,2,opt,name=validator,proto3" json:"validator"`
  147. // The height when the offense occurred
  148. Height int64 `protobuf:"varint,3,opt,name=height,proto3" json:"height,omitempty"`
  149. // The corresponding time where the offense occurred
  150. Time time.Time `protobuf:"bytes,4,opt,name=time,proto3,stdtime" json:"time"`
  151. // Total voting power of the validator set in case the ABCI application does
  152. // not store historical validators.
  153. // https://github.com/tendermint/tendermint/issues/4581
  154. TotalVotingPower int64 `protobuf:"varint,5,opt,name=total_voting_power,json=totalVotingPower,proto3" json:"total_voting_power,omitempty"`
  155. }
  156. ```
  157. This `Update()` function does the following:
  158. - Increments state which keeps track of both the current time and height used for measuring expiry
  159. - Marks evidence as committed and saves to db. This prevents validators from proposing committed evidence in the future
  160. Note: the db just saves the height and the hash. There is no need to save the entire committed evidence
  161. - Forms ABCI evidence as such: (note for `DuplicateVoteEvidence` the validators array size is 1)
  162. ```go
  163. for _, val := range evInfo.Validators {
  164. abciEv = append(abciEv, &abci.Evidence{
  165. Type: evType, // either DuplicateVote or LightClientAttack
  166. Validator: val, // the offending validator (which includes the address, pubkey and power)
  167. Height: evInfo.ev.Height(), // the height when the offense happened
  168. Time: evInfo.time, // the time when the offense happened
  169. TotalVotingPower: evInfo.totalVotingPower // the total voting power of the validator set
  170. })
  171. }
  172. ```
  173. - Removes expired evidence from both pending and committed databases
  174. The ABCI evidence is then sent via the `BlockExecutor` to the application.
  175. #### Summary
  176. To summarize, we can see the lifecycle of evidence as such:
  177. ![evidence_lifecycle](../imgs/evidence_lifecycle.png)
  178. Evidence is first detected and created in the light client and consensus reactor. It is verified and stored as `EvidenceInfo` and gossiped to the evidence pools in other nodes. The consensus reactor later communicates with the evidence pool to either retrieve evidence to be put into a block, or verify the evidence the consensus reactor has retrieved in a block. Lastly when a block is added to the chain, the block executor sends the committed evidence back to the evidence pool so a pointer to the evidence can be stored in the evidence pool and it can update it's height and time. Finally, it turns the committed evidence into ABCI evidence and through the block executor passes the evidence to the application so the application can handle it.
  179. ## Status
  180. Implemented
  181. ## Consequences
  182. <!-- > This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. -->
  183. ### Positive
  184. - Evidence is better contained to the evidence pool / module
  185. - LightClientAttack is kept together (easier for verification and bandwidth)
  186. - Variations on commit sigs in LightClientAttack doesn't lead to multiple permutations and multiple evidence
  187. - Address to evidence map prevents DOS attacks, where a single validator could DOS the network by flooding it with evidence submissions
  188. ### Negative
  189. - Changes the `Evidence` interface and thus is a block breaking change
  190. - Changes the ABCI `Evidence` and is thus a ABCI breaking change
  191. - Unable to query evidence for address / time without evidence pool
  192. ### Neutral
  193. ## References
  194. <!-- > Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! -->
  195. - [LightClientAttackEvidence](https://github.com/informalsystems/tendermint-rs/blob/31ca3e64ce90786c1734caf186e30595832297a4/docs/spec/lightclient/attacks/evidence-handling.md)