You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

352 lines
15 KiB

  1. # Consensus Reactor
  2. Consensus Reactor defines a reactor for the consensus service. It contains the ConsensusState service that
  3. manages the state of the Tendermint consensus internal state machine.
  4. When Consensus Reactor is started, it starts Broadcast Routine which starts ConsensusState service.
  5. Furthermore, for each peer that is added to the Consensus Reactor, it creates (and manages) the known peer state
  6. (that is used extensively in gossip routines) and starts the following three routines for the peer p:
  7. Gossip Data Routine, Gossip Votes Routine and QueryMaj23Routine. Finally, Consensus Reactor is responsible
  8. for decoding messages received from a peer and for adequate processing of the message depending on its type and content.
  9. The processing normally consists of updating the known peer state and for some messages
  10. (`ProposalMessage`, `BlockPartMessage` and `VoteMessage`) also forwarding message to ConsensusState module
  11. for further processing. In the following text we specify the core functionality of those separate unit of executions
  12. that are part of the Consensus Reactor.
  13. ## ConsensusState service
  14. Consensus State handles execution of the Tendermint BFT consensus algorithm. It processes votes and proposals,
  15. and upon reaching agreement, commits blocks to the chain and executes them against the application.
  16. The internal state machine receives input from peers, the internal validator and from a timer.
  17. Inside Consensus State we have the following units of execution: Timeout Ticker and Receive Routine.
  18. Timeout Ticker is a timer that schedules timeouts conditional on the height/round/step that are processed
  19. by the Receive Routine.
  20. ### Receive Routine of the ConsensusState service
  21. Receive Routine of the ConsensusState handles messages which may cause internal consensus state transitions.
  22. It is the only routine that updates RoundState that contains internal consensus state.
  23. Updates (state transitions) happen on timeouts, complete proposals, and 2/3 majorities.
  24. It receives messages from peers, internal validators and from Timeout Ticker
  25. and invokes the corresponding handlers, potentially updating the RoundState.
  26. The details of the protocol (together with formal proofs of correctness) implemented by the Receive Routine are
  27. discussed in separate document. For understanding of this document
  28. it is sufficient to understand that the Receive Routine manages and updates RoundState data structure that is
  29. then extensively used by the gossip routines to determine what information should be sent to peer processes.
  30. ## Round State
  31. RoundState defines the internal consensus state. It contains height, round, round step, a current validator set,
  32. a proposal and proposal block for the current round, locked round and block (if some block is being locked), set of
  33. received votes and last commit and last validators set.
  34. ```golang
  35. type RoundState struct {
  36. Height int64
  37. Round int
  38. Step RoundStepType
  39. Validators ValidatorSet
  40. Proposal Proposal
  41. ProposalBlock Block
  42. ProposalBlockParts PartSet
  43. LockedRound int
  44. LockedBlock Block
  45. LockedBlockParts PartSet
  46. Votes HeightVoteSet
  47. LastCommit VoteSet
  48. LastValidators ValidatorSet
  49. }
  50. ```
  51. Internally, consensus will run as a state machine with the following states:
  52. - RoundStepNewHeight
  53. - RoundStepNewRound
  54. - RoundStepPropose
  55. - RoundStepProposeWait
  56. - RoundStepPrevote
  57. - RoundStepPrevoteWait
  58. - RoundStepPrecommit
  59. - RoundStepPrecommitWait
  60. - RoundStepCommit
  61. ## Peer Round State
  62. Peer round state contains the known state of a peer. It is being updated by the Receive routine of
  63. Consensus Reactor and by the gossip routines upon sending a message to the peer.
  64. ```golang
  65. type PeerRoundState struct {
  66. Height int64 // Height peer is at
  67. Round int // Round peer is at, -1 if unknown.
  68. Step RoundStepType // Step peer is at
  69. Proposal bool // True if peer has proposal for this round
  70. ProposalBlockPartsHeader PartSetHeader
  71. ProposalBlockParts BitArray
  72. ProposalPOLRound int // Proposal's POL round. -1 if none.
  73. ProposalPOL BitArray // nil until ProposalPOLMessage received.
  74. Prevotes BitArray // All votes peer has for this round
  75. Precommits BitArray // All precommits peer has for this round
  76. LastCommitRound int // Round of commit for last height. -1 if none.
  77. LastCommit BitArray // All commit precommits of commit for last height.
  78. CatchupCommitRound int // Round that we have commit for. Not necessarily unique. -1 if none.
  79. CatchupCommit BitArray // All commit precommits peer has for this height & CatchupCommitRound
  80. }
  81. ```
  82. ## Receive method of Consensus reactor
  83. The entry point of the Consensus reactor is a receive method. When a message is received from a peer p,
  84. normally the peer round state is updated correspondingly, and some messages
  85. are passed for further processing, for example to ConsensusState service. We now specify the processing of messages
  86. in the receive method of Consensus reactor for each message type. In the following message handler, `rs` and `prs` denote
  87. `RoundState` and `PeerRoundState`, respectively.
  88. ### NewRoundStepMessage handler
  89. ```
  90. handleMessage(msg):
  91. if msg is from smaller height/round/step then return
  92. // Just remember these values.
  93. prsHeight = prs.Height
  94. prsRound = prs.Round
  95. prsCatchupCommitRound = prs.CatchupCommitRound
  96. prsCatchupCommit = prs.CatchupCommit
  97. Update prs with values from msg
  98. if prs.Height or prs.Round has been updated then
  99. reset Proposal related fields of the peer state
  100. if prs.Round has been updated and msg.Round == prsCatchupCommitRound then
  101. prs.Precommits = psCatchupCommit
  102. if prs.Height has been updated then
  103. if prsHeight+1 == msg.Height && prsRound == msg.LastCommitRound then
  104. prs.LastCommitRound = msg.LastCommitRound
  105. prs.LastCommit = prs.Precommits
  106. } else {
  107. prs.LastCommitRound = msg.LastCommitRound
  108. prs.LastCommit = nil
  109. }
  110. Reset prs.CatchupCommitRound and prs.CatchupCommit
  111. ```
  112. ### CommitStepMessage handler
  113. ```
  114. handleMessage(msg):
  115. if prs.Height == msg.Height then
  116. prs.ProposalBlockPartsHeader = msg.BlockPartsHeader
  117. prs.ProposalBlockParts = msg.BlockParts
  118. ```
  119. ### HasVoteMessage handler
  120. ```
  121. handleMessage(msg):
  122. if prs.Height == msg.Height then
  123. prs.setHasVote(msg.Height, msg.Round, msg.Type, msg.Index)
  124. ```
  125. ### VoteSetMaj23Message handler
  126. ```
  127. handleMessage(msg):
  128. if prs.Height == msg.Height then
  129. Record in rs that a peer claim to have ⅔ majority for msg.BlockID
  130. Send VoteSetBitsMessage showing votes node has for that BlockId
  131. ```
  132. ### ProposalMessage handler
  133. ```
  134. handleMessage(msg):
  135. if prs.Height != msg.Height || prs.Round != msg.Round || prs.Proposal then return
  136. prs.Proposal = true
  137. prs.ProposalBlockPartsHeader = msg.BlockPartsHeader
  138. prs.ProposalBlockParts = empty set
  139. prs.ProposalPOLRound = msg.POLRound
  140. prs.ProposalPOL = nil
  141. Send msg through internal peerMsgQueue to ConsensusState service
  142. ```
  143. ### ProposalPOLMessage handler
  144. ```
  145. handleMessage(msg):
  146. if prs.Height != msg.Height or prs.ProposalPOLRound != msg.ProposalPOLRound then return
  147. prs.ProposalPOL = msg.ProposalPOL
  148. ```
  149. ### BlockPartMessage handler
  150. ```
  151. handleMessage(msg):
  152. if prs.Height != msg.Height || prs.Round != msg.Round then return
  153. Record in prs that peer has block part msg.Part.Index
  154. Send msg trough internal peerMsgQueue to ConsensusState service
  155. ```
  156. ### VoteMessage handler
  157. ```
  158. handleMessage(msg):
  159. Record in prs that a peer knows vote with index msg.vote.ValidatorIndex for particular height and round
  160. Send msg trough internal peerMsgQueue to ConsensusState service
  161. ```
  162. ### VoteSetBitsMessage handler
  163. ```
  164. handleMessage(msg):
  165. Update prs for the bit-array of votes peer claims to have for the msg.BlockID
  166. ```
  167. ## Gossip Data Routine
  168. It is used to send the following messages to the peer: `BlockPartMessage`, `ProposalMessage` and
  169. `ProposalPOLMessage` on the DataChannel. The gossip data routine is based on the local RoundState (`rs`)
  170. and the known PeerRoundState (`prs`). The routine repeats forever the logic shown below:
  171. ```
  172. 1a) if rs.ProposalBlockPartsHeader == prs.ProposalBlockPartsHeader and the peer does not have all the proposal parts then
  173. Part = pick a random proposal block part the peer does not have
  174. Send BlockPartMessage(rs.Height, rs.Round, Part) to the peer on the DataChannel
  175. if send returns true, record that the peer knows the corresponding block Part
  176. Continue
  177. 1b) if (0 < prs.Height) and (prs.Height < rs.Height) then
  178. help peer catch up using gossipDataForCatchup function
  179. Continue
  180. 1c) if (rs.Height != prs.Height) or (rs.Round != prs.Round) then
  181. Sleep PeerGossipSleepDuration
  182. Continue
  183. // at this point rs.Height == prs.Height and rs.Round == prs.Round
  184. 1d) if (rs.Proposal != nil and !prs.Proposal) then
  185. Send ProposalMessage(rs.Proposal) to the peer
  186. if send returns true, record that the peer knows Proposal
  187. if 0 <= rs.Proposal.POLRound then
  188. polRound = rs.Proposal.POLRound
  189. prevotesBitArray = rs.Votes.Prevotes(polRound).BitArray()
  190. Send ProposalPOLMessage(rs.Height, polRound, prevotesBitArray)
  191. Continue
  192. 2) Sleep PeerGossipSleepDuration
  193. ```
  194. ### Gossip Data For Catchup
  195. This function is responsible for helping peer catch up if it is at the smaller height (prs.Height < rs.Height).
  196. The function executes the following logic:
  197. if peer does not have all block parts for prs.ProposalBlockPart then
  198. blockMeta = Load Block Metadata for height prs.Height from blockStore
  199. if (!blockMeta.BlockID.PartsHeader == prs.ProposalBlockPartsHeader) then
  200. Sleep PeerGossipSleepDuration
  201. return
  202. Part = pick a random proposal block part the peer does not have
  203. Send BlockPartMessage(prs.Height, prs.Round, Part) to the peer on the DataChannel
  204. if send returns true, record that the peer knows the corresponding block Part
  205. return
  206. else Sleep PeerGossipSleepDuration
  207. ## Gossip Votes Routine
  208. It is used to send the following message: `VoteMessage` on the VoteChannel.
  209. The gossip votes routine is based on the local RoundState (`rs`)
  210. and the known PeerRoundState (`prs`). The routine repeats forever the logic shown below:
  211. ```
  212. 1a) if rs.Height == prs.Height then
  213. if prs.Step == RoundStepNewHeight then
  214. vote = random vote from rs.LastCommit the peer does not have
  215. Send VoteMessage(vote) to the peer
  216. if send returns true, continue
  217. if prs.Step <= RoundStepPrevote and prs.Round != -1 and prs.Round <= rs.Round then
  218. Prevotes = rs.Votes.Prevotes(prs.Round)
  219. vote = random vote from Prevotes the peer does not have
  220. Send VoteMessage(vote) to the peer
  221. if send returns true, continue
  222. if prs.Step <= RoundStepPrecommit and prs.Round != -1 and prs.Round <= rs.Round then
  223. Precommits = rs.Votes.Precommits(prs.Round)
  224. vote = random vote from Precommits the peer does not have
  225. Send VoteMessage(vote) to the peer
  226. if send returns true, continue
  227. if prs.ProposalPOLRound != -1 then
  228. PolPrevotes = rs.Votes.Prevotes(prs.ProposalPOLRound)
  229. vote = random vote from PolPrevotes the peer does not have
  230. Send VoteMessage(vote) to the peer
  231. if send returns true, continue
  232. 1b) if prs.Height != 0 and rs.Height == prs.Height+1 then
  233. vote = random vote from rs.LastCommit peer does not have
  234. Send VoteMessage(vote) to the peer
  235. if send returns true, continue
  236. 1c) if prs.Height != 0 and rs.Height >= prs.Height+2 then
  237. Commit = get commit from BlockStore for prs.Height
  238. vote = random vote from Commit the peer does not have
  239. Send VoteMessage(vote) to the peer
  240. if send returns true, continue
  241. 2) Sleep PeerGossipSleepDuration
  242. ```
  243. ## QueryMaj23Routine
  244. It is used to send the following message: `VoteSetMaj23Message`. `VoteSetMaj23Message` is sent to indicate that a given
  245. BlockID has seen +2/3 votes. This routine is based on the local RoundState (`rs`) and the known PeerRoundState
  246. (`prs`). The routine repeats forever the logic shown below.
  247. ```
  248. 1a) if rs.Height == prs.Height then
  249. Prevotes = rs.Votes.Prevotes(prs.Round)
  250. if there is a ⅔ majority for some blockId in Prevotes then
  251. m = VoteSetMaj23Message(prs.Height, prs.Round, Prevote, blockId)
  252. Send m to peer
  253. Sleep PeerQueryMaj23SleepDuration
  254. 1b) if rs.Height == prs.Height then
  255. Precommits = rs.Votes.Precommits(prs.Round)
  256. if there is a ⅔ majority for some blockId in Precommits then
  257. m = VoteSetMaj23Message(prs.Height,prs.Round,Precommit,blockId)
  258. Send m to peer
  259. Sleep PeerQueryMaj23SleepDuration
  260. 1c) if rs.Height == prs.Height and prs.ProposalPOLRound >= 0 then
  261. Prevotes = rs.Votes.Prevotes(prs.ProposalPOLRound)
  262. if there is a ⅔ majority for some blockId in Prevotes then
  263. m = VoteSetMaj23Message(prs.Height,prs.ProposalPOLRound,Prevotes,blockId)
  264. Send m to peer
  265. Sleep PeerQueryMaj23SleepDuration
  266. 1d) if prs.CatchupCommitRound != -1 and 0 < prs.Height and
  267. prs.Height <= blockStore.Height() then
  268. Commit = LoadCommit(prs.Height)
  269. m = VoteSetMaj23Message(prs.Height,Commit.Round,Precommit,Commit.blockId)
  270. Send m to peer
  271. Sleep PeerQueryMaj23SleepDuration
  272. 2) Sleep PeerQueryMaj23SleepDuration
  273. ```
  274. ## Broadcast routine
  275. The Broadcast routine subscribes to an internal event bus to receive new round steps, votes messages and proposal
  276. heartbeat messages, and broadcasts messages to peers upon receiving those events.
  277. It broadcasts `NewRoundStepMessage` or `CommitStepMessage` upon new round state event. Note that
  278. broadcasting these messages does not depend on the PeerRoundState; it is sent on the StateChannel.
  279. Upon receiving VoteMessage it broadcasts `HasVoteMessage` message to its peers on the StateChannel.
  280. `ProposalHeartbeatMessage` is sent the same way on the StateChannel.
  281. ## Channels
  282. Defines 4 channels: state, data, vote and vote_set_bits. Each channel
  283. has `SendQueueCapacity` and `RecvBufferCapacity` and
  284. `RecvMessageCapacity` set to `maxMsgSize`.
  285. Sending incorrectly encoded data will result in stopping the peer.