You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

187 lines
4.5 KiB

Blockchain v2 Scheduler (#4043) * Add processor prototype * Change processor API + expose a simple `handle` function which mutates internal state * schedule event handling * rename schedule -> scheduler * fill in handle function * processor tests * fix gofmt and ohter golangci issues * scopelint var on range scope * add check for short block received * small test reorg * ci fix changes * go.mod revert * some cleanup and review comments * scheduler fixes and unit tests, also small processor changes. changed scPeerPruned to include a list of pruned peers touchPeer to check peer state and remove the blocks from blockStates if the peer removal causes the max peer height to be lower. remove the block at sc.initHeight changed peersInactiveSince, peersSlowerThan, getPeersAtHeight check peer state prunablePeers to return a sorted list of peers lastRate in markReceived() attempted to divide by 0, temp fix. fixed allBlocksProcessed conditions maxHeight() and minHeight() to return sc.initHeight if no ready peers present make selectPeer() deterministic. added handleBlockProcessError() added termination cond. (sc.allBlocksProcessed()) to handleTryPrunePeer() and others. changed pcBlockVerificationFailure to include peer of H+2 block along with the one for H+1 changed the processor to call purgePeer on block verification failure. fixed processor tests added scheduler tests. * typo and ci fixes * remove height from scBlockRequest, golangci fixes * limit on blockState map, updated tests * remove unused * separate test for maxHeight(), used for sched. validation * use Math.Min * fix golangci * Document the semantics of blockStates in the scheduler * better docs * distinguish between unknown and invalid blockstate * Standardize peer filtering methods * feedback * s/getPeersAtHeight/getPeersAtHeightOrAbove * small notes * Update blockchain/v2/scheduler.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update comments based on feedback * Add enum offset * panic on nil block in processor * remove unused max height calculation * format shorter line
5 years ago
Blockchain v2 Scheduler (#4043) * Add processor prototype * Change processor API + expose a simple `handle` function which mutates internal state * schedule event handling * rename schedule -> scheduler * fill in handle function * processor tests * fix gofmt and ohter golangci issues * scopelint var on range scope * add check for short block received * small test reorg * ci fix changes * go.mod revert * some cleanup and review comments * scheduler fixes and unit tests, also small processor changes. changed scPeerPruned to include a list of pruned peers touchPeer to check peer state and remove the blocks from blockStates if the peer removal causes the max peer height to be lower. remove the block at sc.initHeight changed peersInactiveSince, peersSlowerThan, getPeersAtHeight check peer state prunablePeers to return a sorted list of peers lastRate in markReceived() attempted to divide by 0, temp fix. fixed allBlocksProcessed conditions maxHeight() and minHeight() to return sc.initHeight if no ready peers present make selectPeer() deterministic. added handleBlockProcessError() added termination cond. (sc.allBlocksProcessed()) to handleTryPrunePeer() and others. changed pcBlockVerificationFailure to include peer of H+2 block along with the one for H+1 changed the processor to call purgePeer on block verification failure. fixed processor tests added scheduler tests. * typo and ci fixes * remove height from scBlockRequest, golangci fixes * limit on blockState map, updated tests * remove unused * separate test for maxHeight(), used for sched. validation * use Math.Min * fix golangci * Document the semantics of blockStates in the scheduler * better docs * distinguish between unknown and invalid blockstate * Standardize peer filtering methods * feedback * s/getPeersAtHeight/getPeersAtHeightOrAbove * small notes * Update blockchain/v2/scheduler.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update comments based on feedback * Add enum offset * panic on nil block in processor * remove unused max height calculation * format shorter line
5 years ago
Blockchain v2 Scheduler (#4043) * Add processor prototype * Change processor API + expose a simple `handle` function which mutates internal state * schedule event handling * rename schedule -> scheduler * fill in handle function * processor tests * fix gofmt and ohter golangci issues * scopelint var on range scope * add check for short block received * small test reorg * ci fix changes * go.mod revert * some cleanup and review comments * scheduler fixes and unit tests, also small processor changes. changed scPeerPruned to include a list of pruned peers touchPeer to check peer state and remove the blocks from blockStates if the peer removal causes the max peer height to be lower. remove the block at sc.initHeight changed peersInactiveSince, peersSlowerThan, getPeersAtHeight check peer state prunablePeers to return a sorted list of peers lastRate in markReceived() attempted to divide by 0, temp fix. fixed allBlocksProcessed conditions maxHeight() and minHeight() to return sc.initHeight if no ready peers present make selectPeer() deterministic. added handleBlockProcessError() added termination cond. (sc.allBlocksProcessed()) to handleTryPrunePeer() and others. changed pcBlockVerificationFailure to include peer of H+2 block along with the one for H+1 changed the processor to call purgePeer on block verification failure. fixed processor tests added scheduler tests. * typo and ci fixes * remove height from scBlockRequest, golangci fixes * limit on blockState map, updated tests * remove unused * separate test for maxHeight(), used for sched. validation * use Math.Min * fix golangci * Document the semantics of blockStates in the scheduler * better docs * distinguish between unknown and invalid blockstate * Standardize peer filtering methods * feedback * s/getPeersAtHeight/getPeersAtHeightOrAbove * small notes * Update blockchain/v2/scheduler.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update comments based on feedback * Add enum offset * panic on nil block in processor * remove unused max height calculation * format shorter line
5 years ago
Blockchain v2 Scheduler (#4043) * Add processor prototype * Change processor API + expose a simple `handle` function which mutates internal state * schedule event handling * rename schedule -> scheduler * fill in handle function * processor tests * fix gofmt and ohter golangci issues * scopelint var on range scope * add check for short block received * small test reorg * ci fix changes * go.mod revert * some cleanup and review comments * scheduler fixes and unit tests, also small processor changes. changed scPeerPruned to include a list of pruned peers touchPeer to check peer state and remove the blocks from blockStates if the peer removal causes the max peer height to be lower. remove the block at sc.initHeight changed peersInactiveSince, peersSlowerThan, getPeersAtHeight check peer state prunablePeers to return a sorted list of peers lastRate in markReceived() attempted to divide by 0, temp fix. fixed allBlocksProcessed conditions maxHeight() and minHeight() to return sc.initHeight if no ready peers present make selectPeer() deterministic. added handleBlockProcessError() added termination cond. (sc.allBlocksProcessed()) to handleTryPrunePeer() and others. changed pcBlockVerificationFailure to include peer of H+2 block along with the one for H+1 changed the processor to call purgePeer on block verification failure. fixed processor tests added scheduler tests. * typo and ci fixes * remove height from scBlockRequest, golangci fixes * limit on blockState map, updated tests * remove unused * separate test for maxHeight(), used for sched. validation * use Math.Min * fix golangci * Document the semantics of blockStates in the scheduler * better docs * distinguish between unknown and invalid blockstate * Standardize peer filtering methods * feedback * s/getPeersAtHeight/getPeersAtHeightOrAbove * small notes * Update blockchain/v2/scheduler.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update comments based on feedback * Add enum offset * panic on nil block in processor * remove unused max height calculation * format shorter line
5 years ago
  1. package v2
  2. import (
  3. "fmt"
  4. "github.com/tendermint/tendermint/p2p"
  5. tdState "github.com/tendermint/tendermint/state"
  6. "github.com/tendermint/tendermint/types"
  7. )
  8. type peerError struct {
  9. priorityHigh
  10. peerID p2p.ID
  11. }
  12. type pcDuplicateBlock struct {
  13. priorityNormal
  14. }
  15. type pcShortBlock struct {
  16. priorityNormal
  17. }
  18. type pcBlockVerificationFailure struct {
  19. priorityNormal
  20. height int64
  21. firstPeerID p2p.ID
  22. secondPeerID p2p.ID
  23. }
  24. type pcBlockProcessed struct {
  25. priorityNormal
  26. height int64
  27. peerID p2p.ID
  28. }
  29. type pcProcessBlock struct {
  30. priorityNormal
  31. }
  32. type pcStop struct {
  33. priorityNormal
  34. }
  35. type pcFinished struct {
  36. priorityNormal
  37. height int64
  38. blocksSynced int64
  39. }
  40. func (p pcFinished) Error() string {
  41. return "finished"
  42. }
  43. type queueItem struct {
  44. block *types.Block
  45. peerID p2p.ID
  46. }
  47. type blockQueue map[int64]queueItem
  48. type pcState struct {
  49. height int64 // height of the last synced block
  50. queue blockQueue // blocks waiting to be processed
  51. chainID string
  52. blocksSynced int64
  53. draining bool
  54. tdState tdState.State
  55. context processorContext
  56. }
  57. func (state *pcState) String() string {
  58. return fmt.Sprintf("height: %d queue length: %d draining: %v blocks synced: %d",
  59. state.height, len(state.queue), state.draining, state.blocksSynced)
  60. }
  61. // newPcState returns a pcState initialized with the last verified block enqueued
  62. func newPcState(initHeight int64, tdState tdState.State, chainID string, context processorContext) *pcState {
  63. return &pcState{
  64. height: initHeight,
  65. queue: blockQueue{},
  66. chainID: chainID,
  67. draining: false,
  68. blocksSynced: 0,
  69. context: context,
  70. tdState: tdState,
  71. }
  72. }
  73. // nextTwo returns the next two unverified blocks
  74. func (state *pcState) nextTwo() (queueItem, queueItem, error) {
  75. if first, ok := state.queue[state.height+1]; ok {
  76. if second, ok := state.queue[state.height+2]; ok {
  77. return first, second, nil
  78. }
  79. }
  80. return queueItem{}, queueItem{}, fmt.Errorf("not found")
  81. }
  82. // synced returns true when at most the last verified block remains in the queue
  83. func (state *pcState) synced() bool {
  84. return len(state.queue) <= 1
  85. }
  86. func (state *pcState) advance() {
  87. state.height++
  88. delete(state.queue, state.height)
  89. state.blocksSynced++
  90. }
  91. func (state *pcState) enqueue(peerID p2p.ID, block *types.Block, height int64) error {
  92. if _, ok := state.queue[height]; ok {
  93. return fmt.Errorf("duplicate queue item")
  94. }
  95. state.queue[height] = queueItem{block: block, peerID: peerID}
  96. return nil
  97. }
  98. // purgePeer moves all unprocessed blocks from the queue
  99. func (state *pcState) purgePeer(peerID p2p.ID) {
  100. // what if height is less than state.height?
  101. for height, item := range state.queue {
  102. if item.peerID == peerID {
  103. delete(state.queue, height)
  104. }
  105. }
  106. }
  107. // handle processes FSM events
  108. func (state *pcState) handle(event Event) (Event, error) {
  109. switch event := event.(type) {
  110. case *scBlockReceived:
  111. if event.block == nil {
  112. panic("processor received an event with a nil block")
  113. }
  114. if event.block.Height <= state.height {
  115. return pcShortBlock{}, nil
  116. }
  117. err := state.enqueue(event.peerID, event.block, event.block.Height)
  118. if err != nil {
  119. return pcDuplicateBlock{}, nil
  120. }
  121. case pcProcessBlock:
  122. firstItem, secondItem, err := state.nextTwo()
  123. if err != nil {
  124. if state.draining {
  125. return noOp, pcFinished{height: state.height}
  126. }
  127. return noOp, nil
  128. }
  129. first, second := firstItem.block, secondItem.block
  130. firstParts := first.MakePartSet(types.BlockPartSizeBytes)
  131. firstPartsHeader := firstParts.Header()
  132. firstID := types.BlockID{Hash: first.Hash(), PartsHeader: firstPartsHeader}
  133. err = state.context.verifyCommit(state.chainID, firstID, first.Height, second.LastCommit)
  134. if err != nil {
  135. state.purgePeer(firstItem.peerID)
  136. state.purgePeer(secondItem.peerID)
  137. return pcBlockVerificationFailure{
  138. height: first.Height, firstPeerID: firstItem.peerID, secondPeerID: secondItem.peerID},
  139. nil
  140. }
  141. state.context.saveBlock(first, firstParts, second.LastCommit)
  142. state.tdState, err = state.context.applyBlock(state.tdState, firstID, first)
  143. if err != nil {
  144. panic(fmt.Sprintf("failed to process committed block (%d:%X): %v", first.Height, first.Hash(), err))
  145. }
  146. state.advance()
  147. return pcBlockProcessed{height: first.Height, peerID: firstItem.peerID}, nil
  148. case *peerError:
  149. state.purgePeer(event.peerID)
  150. case pcStop:
  151. if state.synced() {
  152. return noOp, pcFinished{height: state.height, blocksSynced: state.blocksSynced}
  153. }
  154. state.draining = true
  155. }
  156. return noOp, nil
  157. }