You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

642 lines
17 KiB

cleanup: Reduce and normalize import path aliasing. (#6975) The code in the Tendermint repository makes heavy use of import aliasing. This is made necessary by our extensive reuse of common base package names, and by repetition of similar names across different subdirectories. Unfortunately we have not been very consistent about which packages we alias in various circumstances, and the aliases we use vary. In the spirit of the advice in the style guide and https://github.com/golang/go/wiki/CodeReviewComments#imports, his change makes an effort to clean up and normalize import aliasing. This change makes no API or behavioral changes. It is a pure cleanup intended o help make the code more readable to developers (including myself) trying to understand what is being imported where. Only unexported names have been modified, and the changes were generated and applied mechanically with gofmt -r and comby, respecting the lexical and syntactic rules of Go. Even so, I did not fix every inconsistency. Where the changes would be too disruptive, I left it alone. The principles I followed in this cleanup are: - Remove aliases that restate the package name. - Remove aliases where the base package name is unambiguous. - Move overly-terse abbreviations from the import to the usage site. - Fix lexical issues (remove underscores, remove capitalization). - Fix import groupings to more closely match the style guide. - Group blank (side-effecting) imports and ensure they are commented. - Add aliases to multiple imports with the same base package name.
3 years ago
cleanup: Reduce and normalize import path aliasing. (#6975) The code in the Tendermint repository makes heavy use of import aliasing. This is made necessary by our extensive reuse of common base package names, and by repetition of similar names across different subdirectories. Unfortunately we have not been very consistent about which packages we alias in various circumstances, and the aliases we use vary. In the spirit of the advice in the style guide and https://github.com/golang/go/wiki/CodeReviewComments#imports, his change makes an effort to clean up and normalize import aliasing. This change makes no API or behavioral changes. It is a pure cleanup intended o help make the code more readable to developers (including myself) trying to understand what is being imported where. Only unexported names have been modified, and the changes were generated and applied mechanically with gofmt -r and comby, respecting the lexical and syntactic rules of Go. Even so, I did not fix every inconsistency. Where the changes would be too disruptive, I left it alone. The principles I followed in this cleanup are: - Remove aliases that restate the package name. - Remove aliases where the base package name is unambiguous. - Move overly-terse abbreviations from the import to the usage site. - Fix lexical issues (remove underscores, remove capitalization). - Fix import groupings to more closely match the style guide. - Group blank (side-effecting) imports and ensure they are commented. - Add aliases to multiple imports with the same base package name.
3 years ago
cleanup: Reduce and normalize import path aliasing. (#6975) The code in the Tendermint repository makes heavy use of import aliasing. This is made necessary by our extensive reuse of common base package names, and by repetition of similar names across different subdirectories. Unfortunately we have not been very consistent about which packages we alias in various circumstances, and the aliases we use vary. In the spirit of the advice in the style guide and https://github.com/golang/go/wiki/CodeReviewComments#imports, his change makes an effort to clean up and normalize import aliasing. This change makes no API or behavioral changes. It is a pure cleanup intended o help make the code more readable to developers (including myself) trying to understand what is being imported where. Only unexported names have been modified, and the changes were generated and applied mechanically with gofmt -r and comby, respecting the lexical and syntactic rules of Go. Even so, I did not fix every inconsistency. Where the changes would be too disruptive, I left it alone. The principles I followed in this cleanup are: - Remove aliases that restate the package name. - Remove aliases where the base package name is unambiguous. - Move overly-terse abbreviations from the import to the usage site. - Fix lexical issues (remove underscores, remove capitalization). - Fix import groupings to more closely match the style guide. - Group blank (side-effecting) imports and ensure they are commented. - Add aliases to multiple imports with the same base package name.
3 years ago
cleanup: Reduce and normalize import path aliasing. (#6975) The code in the Tendermint repository makes heavy use of import aliasing. This is made necessary by our extensive reuse of common base package names, and by repetition of similar names across different subdirectories. Unfortunately we have not been very consistent about which packages we alias in various circumstances, and the aliases we use vary. In the spirit of the advice in the style guide and https://github.com/golang/go/wiki/CodeReviewComments#imports, his change makes an effort to clean up and normalize import aliasing. This change makes no API or behavioral changes. It is a pure cleanup intended o help make the code more readable to developers (including myself) trying to understand what is being imported where. Only unexported names have been modified, and the changes were generated and applied mechanically with gofmt -r and comby, respecting the lexical and syntactic rules of Go. Even so, I did not fix every inconsistency. Where the changes would be too disruptive, I left it alone. The principles I followed in this cleanup are: - Remove aliases that restate the package name. - Remove aliases where the base package name is unambiguous. - Move overly-terse abbreviations from the import to the usage site. - Fix lexical issues (remove underscores, remove capitalization). - Fix import groupings to more closely match the style guide. - Group blank (side-effecting) imports and ensure they are commented. - Add aliases to multiple imports with the same base package name.
3 years ago
cleanup: Reduce and normalize import path aliasing. (#6975) The code in the Tendermint repository makes heavy use of import aliasing. This is made necessary by our extensive reuse of common base package names, and by repetition of similar names across different subdirectories. Unfortunately we have not been very consistent about which packages we alias in various circumstances, and the aliases we use vary. In the spirit of the advice in the style guide and https://github.com/golang/go/wiki/CodeReviewComments#imports, his change makes an effort to clean up and normalize import aliasing. This change makes no API or behavioral changes. It is a pure cleanup intended o help make the code more readable to developers (including myself) trying to understand what is being imported where. Only unexported names have been modified, and the changes were generated and applied mechanically with gofmt -r and comby, respecting the lexical and syntactic rules of Go. Even so, I did not fix every inconsistency. Where the changes would be too disruptive, I left it alone. The principles I followed in this cleanup are: - Remove aliases that restate the package name. - Remove aliases where the base package name is unambiguous. - Move overly-terse abbreviations from the import to the usage site. - Fix lexical issues (remove underscores, remove capitalization). - Fix import groupings to more closely match the style guide. - Group blank (side-effecting) imports and ensure they are commented. - Add aliases to multiple imports with the same base package name.
3 years ago
cleanup: Reduce and normalize import path aliasing. (#6975) The code in the Tendermint repository makes heavy use of import aliasing. This is made necessary by our extensive reuse of common base package names, and by repetition of similar names across different subdirectories. Unfortunately we have not been very consistent about which packages we alias in various circumstances, and the aliases we use vary. In the spirit of the advice in the style guide and https://github.com/golang/go/wiki/CodeReviewComments#imports, his change makes an effort to clean up and normalize import aliasing. This change makes no API or behavioral changes. It is a pure cleanup intended o help make the code more readable to developers (including myself) trying to understand what is being imported where. Only unexported names have been modified, and the changes were generated and applied mechanically with gofmt -r and comby, respecting the lexical and syntactic rules of Go. Even so, I did not fix every inconsistency. Where the changes would be too disruptive, I left it alone. The principles I followed in this cleanup are: - Remove aliases that restate the package name. - Remove aliases where the base package name is unambiguous. - Move overly-terse abbreviations from the import to the usage site. - Fix lexical issues (remove underscores, remove capitalization). - Fix import groupings to more closely match the style guide. - Group blank (side-effecting) imports and ensure they are commented. - Add aliases to multiple imports with the same base package name.
3 years ago
blocksync: fix shutdown deadlock issue (#7030) When shutting down blocksync, it is observed that the process can hang completely. A dump of running goroutines reveals that this is due to goroutines not listening on the correct shutdown signal. Namely, the `poolRoutine` goroutine does not wait on `pool.Quit`. The `poolRoutine` does not receive any other shutdown signal during `OnStop` becuase it must stop before the `r.closeCh` is closed. Currently the `poolRoutine` listens in the `closeCh` which will not close until the `poolRoutine` stops and calls `poolWG.Done()`. This change also puts the `requestRoutine()` in the `OnStart` method to make it more visible since it does not rely on anything that is spawned in the `poolRoutine`. ``` goroutine 183 [semacquire]: sync.runtime_Semacquire(0xc0000d3bd8) runtime/sema.go:56 +0x45 sync.(*WaitGroup).Wait(0xc0000d3bd0) sync/waitgroup.go:130 +0x65 github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStop(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:193 +0x47 github.com/tendermint/tendermint/libs/service.(*BaseService).Stop(0xc0000d3a00, 0x0, 0x0) github.com/tendermint/tendermint/libs/service/service.go:171 +0x323 github.com/tendermint/tendermint/node.(*nodeImpl).OnStop(0xc00052c000) github.com/tendermint/tendermint/node/node.go:758 +0xc62 github.com/tendermint/tendermint/libs/service.(*BaseService).Stop(0xc00052c000, 0x0, 0x0) github.com/tendermint/tendermint/libs/service/service.go:171 +0x323 github.com/tendermint/tendermint/cmd/tendermint/commands.NewRunNodeCmd.func1.1() github.com/tendermint/tendermint/cmd/tendermint/commands/run_node.go:143 +0x62 github.com/tendermint/tendermint/libs/os.TrapSignal.func1(0xc000df6d20, 0x7f04a68da900, 0xc0004a8930, 0xc0005a72d8) github.com/tendermint/tendermint/libs/os/os.go:26 +0x102 created by github.com/tendermint/tendermint/libs/os.TrapSignal github.com/tendermint/tendermint/libs/os/os.go:22 +0xe6 goroutine 161 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).poolRoutine(0xc0000d3a00, 0x0) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:464 +0x2b3 created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:174 +0xf1 goroutine 162 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).processBlockSyncCh(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:310 +0x151 created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:177 +0x54 goroutine 163 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).processPeerUpdates(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:363 +0x12b created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:178 +0x76 ```
3 years ago
blocksync: fix shutdown deadlock issue (#7030) When shutting down blocksync, it is observed that the process can hang completely. A dump of running goroutines reveals that this is due to goroutines not listening on the correct shutdown signal. Namely, the `poolRoutine` goroutine does not wait on `pool.Quit`. The `poolRoutine` does not receive any other shutdown signal during `OnStop` becuase it must stop before the `r.closeCh` is closed. Currently the `poolRoutine` listens in the `closeCh` which will not close until the `poolRoutine` stops and calls `poolWG.Done()`. This change also puts the `requestRoutine()` in the `OnStart` method to make it more visible since it does not rely on anything that is spawned in the `poolRoutine`. ``` goroutine 183 [semacquire]: sync.runtime_Semacquire(0xc0000d3bd8) runtime/sema.go:56 +0x45 sync.(*WaitGroup).Wait(0xc0000d3bd0) sync/waitgroup.go:130 +0x65 github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStop(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:193 +0x47 github.com/tendermint/tendermint/libs/service.(*BaseService).Stop(0xc0000d3a00, 0x0, 0x0) github.com/tendermint/tendermint/libs/service/service.go:171 +0x323 github.com/tendermint/tendermint/node.(*nodeImpl).OnStop(0xc00052c000) github.com/tendermint/tendermint/node/node.go:758 +0xc62 github.com/tendermint/tendermint/libs/service.(*BaseService).Stop(0xc00052c000, 0x0, 0x0) github.com/tendermint/tendermint/libs/service/service.go:171 +0x323 github.com/tendermint/tendermint/cmd/tendermint/commands.NewRunNodeCmd.func1.1() github.com/tendermint/tendermint/cmd/tendermint/commands/run_node.go:143 +0x62 github.com/tendermint/tendermint/libs/os.TrapSignal.func1(0xc000df6d20, 0x7f04a68da900, 0xc0004a8930, 0xc0005a72d8) github.com/tendermint/tendermint/libs/os/os.go:26 +0x102 created by github.com/tendermint/tendermint/libs/os.TrapSignal github.com/tendermint/tendermint/libs/os/os.go:22 +0xe6 goroutine 161 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).poolRoutine(0xc0000d3a00, 0x0) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:464 +0x2b3 created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:174 +0xf1 goroutine 162 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).processBlockSyncCh(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:310 +0x151 created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:177 +0x54 goroutine 163 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).processPeerUpdates(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:363 +0x12b created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:178 +0x76 ```
3 years ago
blocksync: fix shutdown deadlock issue (#7030) When shutting down blocksync, it is observed that the process can hang completely. A dump of running goroutines reveals that this is due to goroutines not listening on the correct shutdown signal. Namely, the `poolRoutine` goroutine does not wait on `pool.Quit`. The `poolRoutine` does not receive any other shutdown signal during `OnStop` becuase it must stop before the `r.closeCh` is closed. Currently the `poolRoutine` listens in the `closeCh` which will not close until the `poolRoutine` stops and calls `poolWG.Done()`. This change also puts the `requestRoutine()` in the `OnStart` method to make it more visible since it does not rely on anything that is spawned in the `poolRoutine`. ``` goroutine 183 [semacquire]: sync.runtime_Semacquire(0xc0000d3bd8) runtime/sema.go:56 +0x45 sync.(*WaitGroup).Wait(0xc0000d3bd0) sync/waitgroup.go:130 +0x65 github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStop(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:193 +0x47 github.com/tendermint/tendermint/libs/service.(*BaseService).Stop(0xc0000d3a00, 0x0, 0x0) github.com/tendermint/tendermint/libs/service/service.go:171 +0x323 github.com/tendermint/tendermint/node.(*nodeImpl).OnStop(0xc00052c000) github.com/tendermint/tendermint/node/node.go:758 +0xc62 github.com/tendermint/tendermint/libs/service.(*BaseService).Stop(0xc00052c000, 0x0, 0x0) github.com/tendermint/tendermint/libs/service/service.go:171 +0x323 github.com/tendermint/tendermint/cmd/tendermint/commands.NewRunNodeCmd.func1.1() github.com/tendermint/tendermint/cmd/tendermint/commands/run_node.go:143 +0x62 github.com/tendermint/tendermint/libs/os.TrapSignal.func1(0xc000df6d20, 0x7f04a68da900, 0xc0004a8930, 0xc0005a72d8) github.com/tendermint/tendermint/libs/os/os.go:26 +0x102 created by github.com/tendermint/tendermint/libs/os.TrapSignal github.com/tendermint/tendermint/libs/os/os.go:22 +0xe6 goroutine 161 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).poolRoutine(0xc0000d3a00, 0x0) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:464 +0x2b3 created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:174 +0xf1 goroutine 162 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).processBlockSyncCh(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:310 +0x151 created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:177 +0x54 goroutine 163 [select]: github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).processPeerUpdates(0xc0000d3a00) github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:363 +0x12b created by github.com/tendermint/tendermint/internal/blocksync/v0.(*Reactor).OnStart github.com/tendermint/tendermint/internal/blocksync/v0/reactor.go:178 +0x76 ```
3 years ago
  1. package blocksync
  2. import (
  3. "context"
  4. "fmt"
  5. "runtime/debug"
  6. "sync"
  7. "time"
  8. "github.com/tendermint/tendermint/internal/consensus"
  9. "github.com/tendermint/tendermint/internal/p2p"
  10. sm "github.com/tendermint/tendermint/internal/state"
  11. "github.com/tendermint/tendermint/internal/store"
  12. "github.com/tendermint/tendermint/libs/log"
  13. "github.com/tendermint/tendermint/libs/service"
  14. tmsync "github.com/tendermint/tendermint/libs/sync"
  15. bcproto "github.com/tendermint/tendermint/proto/tendermint/blocksync"
  16. "github.com/tendermint/tendermint/types"
  17. )
  18. var _ service.Service = (*Reactor)(nil)
  19. const (
  20. // BlockSyncChannel is a channel for blocks and status updates
  21. BlockSyncChannel = p2p.ChannelID(0x40)
  22. trySyncIntervalMS = 10
  23. // ask for best height every 10s
  24. statusUpdateIntervalSeconds = 10
  25. // check if we should switch to consensus reactor
  26. switchToConsensusIntervalSeconds = 1
  27. // switch to consensus after this duration of inactivity
  28. syncTimeout = 60 * time.Second
  29. )
  30. func GetChannelDescriptor() *p2p.ChannelDescriptor {
  31. return &p2p.ChannelDescriptor{
  32. ID: BlockSyncChannel,
  33. MessageType: new(bcproto.Message),
  34. Priority: 5,
  35. SendQueueCapacity: 1000,
  36. RecvBufferCapacity: 1024,
  37. RecvMessageCapacity: MaxMsgSize,
  38. }
  39. }
  40. type consensusReactor interface {
  41. // For when we switch from block sync reactor to the consensus
  42. // machine.
  43. SwitchToConsensus(ctx context.Context, state sm.State, skipWAL bool)
  44. }
  45. type peerError struct {
  46. err error
  47. peerID types.NodeID
  48. }
  49. func (e peerError) Error() string {
  50. return fmt.Sprintf("error with peer %v: %s", e.peerID, e.err.Error())
  51. }
  52. // Reactor handles long-term catchup syncing.
  53. type Reactor struct {
  54. service.BaseService
  55. // immutable
  56. initialState sm.State
  57. blockExec *sm.BlockExecutor
  58. store *store.BlockStore
  59. pool *BlockPool
  60. consReactor consensusReactor
  61. blockSync *tmsync.AtomicBool
  62. blockSyncCh *p2p.Channel
  63. // blockSyncOutBridgeCh defines a channel that acts as a bridge between sending Envelope
  64. // messages that the reactor will consume in processBlockSyncCh and receiving messages
  65. // from the peer updates channel and other goroutines. We do this instead of directly
  66. // sending on blockSyncCh.Out to avoid race conditions in the case where other goroutines
  67. // send Envelopes directly to the to blockSyncCh.Out channel, since processBlockSyncCh
  68. // may close the blockSyncCh.Out channel at the same time that other goroutines send to
  69. // blockSyncCh.Out.
  70. blockSyncOutBridgeCh chan p2p.Envelope
  71. peerUpdates *p2p.PeerUpdates
  72. closeCh chan struct{}
  73. requestsCh <-chan BlockRequest
  74. errorsCh <-chan peerError
  75. // poolWG is used to synchronize the graceful shutdown of the poolRoutine and
  76. // requestRoutine spawned goroutines when stopping the reactor and before
  77. // stopping the p2p Channel(s).
  78. poolWG sync.WaitGroup
  79. metrics *consensus.Metrics
  80. syncStartTime time.Time
  81. }
  82. // NewReactor returns new reactor instance.
  83. func NewReactor(
  84. logger log.Logger,
  85. state sm.State,
  86. blockExec *sm.BlockExecutor,
  87. store *store.BlockStore,
  88. consReactor consensusReactor,
  89. blockSyncCh *p2p.Channel,
  90. peerUpdates *p2p.PeerUpdates,
  91. blockSync bool,
  92. metrics *consensus.Metrics,
  93. ) (*Reactor, error) {
  94. if state.LastBlockHeight != store.Height() {
  95. return nil, fmt.Errorf("state (%v) and store (%v) height mismatch", state.LastBlockHeight, store.Height())
  96. }
  97. startHeight := store.Height() + 1
  98. if startHeight == 1 {
  99. startHeight = state.InitialHeight
  100. }
  101. requestsCh := make(chan BlockRequest, maxTotalRequesters)
  102. errorsCh := make(chan peerError, maxPeerErrBuffer) // NOTE: The capacity should be larger than the peer count.
  103. r := &Reactor{
  104. initialState: state,
  105. blockExec: blockExec,
  106. store: store,
  107. pool: NewBlockPool(logger, startHeight, requestsCh, errorsCh),
  108. consReactor: consReactor,
  109. blockSync: tmsync.NewBool(blockSync),
  110. requestsCh: requestsCh,
  111. errorsCh: errorsCh,
  112. blockSyncCh: blockSyncCh,
  113. blockSyncOutBridgeCh: make(chan p2p.Envelope),
  114. peerUpdates: peerUpdates,
  115. closeCh: make(chan struct{}),
  116. metrics: metrics,
  117. syncStartTime: time.Time{},
  118. }
  119. r.BaseService = *service.NewBaseService(logger, "BlockSync", r)
  120. return r, nil
  121. }
  122. // OnStart starts separate go routines for each p2p Channel and listens for
  123. // envelopes on each. In addition, it also listens for peer updates and handles
  124. // messages on that p2p channel accordingly. The caller must be sure to execute
  125. // OnStop to ensure the outbound p2p Channels are closed.
  126. //
  127. // If blockSync is enabled, we also start the pool and the pool processing
  128. // goroutine. If the pool fails to start, an error is returned.
  129. func (r *Reactor) OnStart(ctx context.Context) error {
  130. if r.blockSync.IsSet() {
  131. if err := r.pool.Start(ctx); err != nil {
  132. return err
  133. }
  134. r.poolWG.Add(1)
  135. go r.requestRoutine()
  136. r.poolWG.Add(1)
  137. go r.poolRoutine(false)
  138. }
  139. go r.processBlockSyncCh()
  140. go r.processPeerUpdates()
  141. return nil
  142. }
  143. // OnStop stops the reactor by signaling to all spawned goroutines to exit and
  144. // blocking until they all exit.
  145. func (r *Reactor) OnStop() {
  146. if r.blockSync.IsSet() {
  147. if err := r.pool.Stop(); err != nil {
  148. r.Logger.Error("failed to stop pool", "err", err)
  149. }
  150. }
  151. // wait for the poolRoutine and requestRoutine goroutines to gracefully exit
  152. r.poolWG.Wait()
  153. // Close closeCh to signal to all spawned goroutines to gracefully exit. All
  154. // p2p Channels should execute Close().
  155. close(r.closeCh)
  156. // Wait for all p2p Channels to be closed before returning. This ensures we
  157. // can easily reason about synchronization of all p2p Channels and ensure no
  158. // panics will occur.
  159. <-r.blockSyncCh.Done()
  160. <-r.peerUpdates.Done()
  161. }
  162. // respondToPeer loads a block and sends it to the requesting peer, if we have it.
  163. // Otherwise, we'll respond saying we do not have it.
  164. func (r *Reactor) respondToPeer(msg *bcproto.BlockRequest, peerID types.NodeID) {
  165. block := r.store.LoadBlock(msg.Height)
  166. if block != nil {
  167. blockProto, err := block.ToProto()
  168. if err != nil {
  169. r.Logger.Error("failed to convert msg to protobuf", "err", err)
  170. return
  171. }
  172. r.blockSyncCh.Out <- p2p.Envelope{
  173. To: peerID,
  174. Message: &bcproto.BlockResponse{Block: blockProto},
  175. }
  176. return
  177. }
  178. r.Logger.Info("peer requesting a block we do not have", "peer", peerID, "height", msg.Height)
  179. r.blockSyncCh.Out <- p2p.Envelope{
  180. To: peerID,
  181. Message: &bcproto.NoBlockResponse{Height: msg.Height},
  182. }
  183. }
  184. // handleBlockSyncMessage handles envelopes sent from peers on the
  185. // BlockSyncChannel. It returns an error only if the Envelope.Message is unknown
  186. // for this channel. This should never be called outside of handleMessage.
  187. func (r *Reactor) handleBlockSyncMessage(envelope p2p.Envelope) error {
  188. logger := r.Logger.With("peer", envelope.From)
  189. switch msg := envelope.Message.(type) {
  190. case *bcproto.BlockRequest:
  191. r.respondToPeer(msg, envelope.From)
  192. case *bcproto.BlockResponse:
  193. block, err := types.BlockFromProto(msg.Block)
  194. if err != nil {
  195. logger.Error("failed to convert block from proto", "err", err)
  196. return err
  197. }
  198. r.pool.AddBlock(envelope.From, block, block.Size())
  199. case *bcproto.StatusRequest:
  200. r.blockSyncCh.Out <- p2p.Envelope{
  201. To: envelope.From,
  202. Message: &bcproto.StatusResponse{
  203. Height: r.store.Height(),
  204. Base: r.store.Base(),
  205. },
  206. }
  207. case *bcproto.StatusResponse:
  208. r.pool.SetPeerRange(envelope.From, msg.Base, msg.Height)
  209. case *bcproto.NoBlockResponse:
  210. logger.Debug("peer does not have the requested block", "height", msg.Height)
  211. default:
  212. return fmt.Errorf("received unknown message: %T", msg)
  213. }
  214. return nil
  215. }
  216. // handleMessage handles an Envelope sent from a peer on a specific p2p Channel.
  217. // It will handle errors and any possible panics gracefully. A caller can handle
  218. // any error returned by sending a PeerError on the respective channel.
  219. func (r *Reactor) handleMessage(chID p2p.ChannelID, envelope p2p.Envelope) (err error) {
  220. defer func() {
  221. if e := recover(); e != nil {
  222. err = fmt.Errorf("panic in processing message: %v", e)
  223. r.Logger.Error(
  224. "recovering from processing message panic",
  225. "err", err,
  226. "stack", string(debug.Stack()),
  227. )
  228. }
  229. }()
  230. r.Logger.Debug("received message", "message", envelope.Message, "peer", envelope.From)
  231. switch chID {
  232. case BlockSyncChannel:
  233. err = r.handleBlockSyncMessage(envelope)
  234. default:
  235. err = fmt.Errorf("unknown channel ID (%d) for envelope (%v)", chID, envelope)
  236. }
  237. return err
  238. }
  239. // processBlockSyncCh initiates a blocking process where we listen for and handle
  240. // envelopes on the BlockSyncChannel and blockSyncOutBridgeCh. Any error encountered during
  241. // message execution will result in a PeerError being sent on the BlockSyncChannel.
  242. // When the reactor is stopped, we will catch the signal and close the p2p Channel
  243. // gracefully.
  244. func (r *Reactor) processBlockSyncCh() {
  245. defer r.blockSyncCh.Close()
  246. for {
  247. select {
  248. case envelope := <-r.blockSyncCh.In:
  249. if err := r.handleMessage(r.blockSyncCh.ID, envelope); err != nil {
  250. r.Logger.Error("failed to process message", "ch_id", r.blockSyncCh.ID, "envelope", envelope, "err", err)
  251. r.blockSyncCh.Error <- p2p.PeerError{
  252. NodeID: envelope.From,
  253. Err: err,
  254. }
  255. }
  256. case envelope := <-r.blockSyncOutBridgeCh:
  257. r.blockSyncCh.Out <- envelope
  258. case <-r.closeCh:
  259. r.Logger.Debug("stopped listening on block sync channel; closing...")
  260. return
  261. }
  262. }
  263. }
  264. // processPeerUpdate processes a PeerUpdate.
  265. func (r *Reactor) processPeerUpdate(peerUpdate p2p.PeerUpdate) {
  266. r.Logger.Debug("received peer update", "peer", peerUpdate.NodeID, "status", peerUpdate.Status)
  267. // XXX: Pool#RedoRequest can sometimes give us an empty peer.
  268. if len(peerUpdate.NodeID) == 0 {
  269. return
  270. }
  271. switch peerUpdate.Status {
  272. case p2p.PeerStatusUp:
  273. // send a status update the newly added peer
  274. r.blockSyncOutBridgeCh <- p2p.Envelope{
  275. To: peerUpdate.NodeID,
  276. Message: &bcproto.StatusResponse{
  277. Base: r.store.Base(),
  278. Height: r.store.Height(),
  279. },
  280. }
  281. case p2p.PeerStatusDown:
  282. r.pool.RemovePeer(peerUpdate.NodeID)
  283. }
  284. }
  285. // processPeerUpdates initiates a blocking process where we listen for and handle
  286. // PeerUpdate messages. When the reactor is stopped, we will catch the signal and
  287. // close the p2p PeerUpdatesCh gracefully.
  288. func (r *Reactor) processPeerUpdates() {
  289. defer r.peerUpdates.Close()
  290. for {
  291. select {
  292. case peerUpdate := <-r.peerUpdates.Updates():
  293. r.processPeerUpdate(peerUpdate)
  294. case <-r.closeCh:
  295. r.Logger.Debug("stopped listening on peer updates channel; closing...")
  296. return
  297. }
  298. }
  299. }
  300. // SwitchToBlockSync is called by the state sync reactor when switching to fast
  301. // sync.
  302. func (r *Reactor) SwitchToBlockSync(ctx context.Context, state sm.State) error {
  303. r.blockSync.Set()
  304. r.initialState = state
  305. r.pool.height = state.LastBlockHeight + 1
  306. if err := r.pool.Start(ctx); err != nil {
  307. return err
  308. }
  309. r.syncStartTime = time.Now()
  310. r.poolWG.Add(1)
  311. go r.requestRoutine()
  312. r.poolWG.Add(1)
  313. go r.poolRoutine(true)
  314. return nil
  315. }
  316. func (r *Reactor) requestRoutine() {
  317. statusUpdateTicker := time.NewTicker(statusUpdateIntervalSeconds * time.Second)
  318. defer statusUpdateTicker.Stop()
  319. defer r.poolWG.Done()
  320. for {
  321. select {
  322. case <-r.closeCh:
  323. return
  324. case <-r.pool.Quit():
  325. return
  326. case request := <-r.requestsCh:
  327. r.blockSyncOutBridgeCh <- p2p.Envelope{
  328. To: request.PeerID,
  329. Message: &bcproto.BlockRequest{Height: request.Height},
  330. }
  331. case pErr := <-r.errorsCh:
  332. r.blockSyncCh.Error <- p2p.PeerError{
  333. NodeID: pErr.peerID,
  334. Err: pErr.err,
  335. }
  336. case <-statusUpdateTicker.C:
  337. r.poolWG.Add(1)
  338. go func() {
  339. defer r.poolWG.Done()
  340. r.blockSyncOutBridgeCh <- p2p.Envelope{
  341. Broadcast: true,
  342. Message: &bcproto.StatusRequest{},
  343. }
  344. }()
  345. }
  346. }
  347. }
  348. func (r *Reactor) stopCtx() context.Context {
  349. ctx, cancel := context.WithCancel(context.Background())
  350. go func() {
  351. <-r.closeCh
  352. cancel()
  353. }()
  354. return ctx
  355. }
  356. // poolRoutine handles messages from the poolReactor telling the reactor what to
  357. // do.
  358. //
  359. // NOTE: Don't sleep in the FOR_LOOP or otherwise slow it down!
  360. func (r *Reactor) poolRoutine(stateSynced bool) {
  361. var (
  362. trySyncTicker = time.NewTicker(trySyncIntervalMS * time.Millisecond)
  363. switchToConsensusTicker = time.NewTicker(switchToConsensusIntervalSeconds * time.Second)
  364. blocksSynced = uint64(0)
  365. chainID = r.initialState.ChainID
  366. state = r.initialState
  367. lastHundred = time.Now()
  368. lastRate = 0.0
  369. didProcessCh = make(chan struct{}, 1)
  370. ctx = r.stopCtx()
  371. )
  372. defer trySyncTicker.Stop()
  373. defer switchToConsensusTicker.Stop()
  374. defer r.poolWG.Done()
  375. FOR_LOOP:
  376. for {
  377. select {
  378. case <-switchToConsensusTicker.C:
  379. var (
  380. height, numPending, lenRequesters = r.pool.GetStatus()
  381. lastAdvance = r.pool.LastAdvance()
  382. )
  383. r.Logger.Debug(
  384. "consensus ticker",
  385. "num_pending", numPending,
  386. "total", lenRequesters,
  387. "height", height,
  388. )
  389. switch {
  390. case r.pool.IsCaughtUp():
  391. r.Logger.Info("switching to consensus reactor", "height", height)
  392. case time.Since(lastAdvance) > syncTimeout:
  393. r.Logger.Error("no progress since last advance", "last_advance", lastAdvance)
  394. default:
  395. r.Logger.Info(
  396. "not caught up yet",
  397. "height", height,
  398. "max_peer_height", r.pool.MaxPeerHeight(),
  399. "timeout_in", syncTimeout-time.Since(lastAdvance),
  400. )
  401. continue
  402. }
  403. if err := r.pool.Stop(); err != nil {
  404. r.Logger.Error("failed to stop pool", "err", err)
  405. }
  406. r.blockSync.UnSet()
  407. if r.consReactor != nil {
  408. r.consReactor.SwitchToConsensus(ctx, state, blocksSynced > 0 || stateSynced)
  409. }
  410. break FOR_LOOP
  411. case <-trySyncTicker.C:
  412. select {
  413. case didProcessCh <- struct{}{}:
  414. default:
  415. }
  416. case <-didProcessCh:
  417. // NOTE: It is a subtle mistake to process more than a single block at a
  418. // time (e.g. 10) here, because we only send one BlockRequest per loop
  419. // iteration. The ratio mismatch can result in starving of blocks, i.e. a
  420. // sudden burst of requests and responses, and repeat. Consequently, it is
  421. // better to split these routines rather than coupling them as it is
  422. // written here.
  423. //
  424. // TODO: Uncouple from request routine.
  425. // see if there are any blocks to sync
  426. first, second := r.pool.PeekTwoBlocks()
  427. if first == nil || second == nil {
  428. // we need both to sync the first block
  429. continue FOR_LOOP
  430. } else {
  431. // try again quickly next loop
  432. didProcessCh <- struct{}{}
  433. }
  434. var (
  435. firstParts = first.MakePartSet(types.BlockPartSizeBytes)
  436. firstPartSetHeader = firstParts.Header()
  437. firstID = types.BlockID{Hash: first.Hash(), PartSetHeader: firstPartSetHeader}
  438. )
  439. // Finally, verify the first block using the second's commit.
  440. //
  441. // NOTE: We can probably make this more efficient, but note that calling
  442. // first.Hash() doesn't verify the tx contents, so MakePartSet() is
  443. // currently necessary.
  444. err := state.Validators.VerifyCommitLight(chainID, firstID, first.Height, second.LastCommit)
  445. if err != nil {
  446. err = fmt.Errorf("invalid last commit: %w", err)
  447. r.Logger.Error(
  448. err.Error(),
  449. "last_commit", second.LastCommit,
  450. "block_id", firstID,
  451. "height", first.Height,
  452. )
  453. // NOTE: We've already removed the peer's request, but we still need
  454. // to clean up the rest.
  455. peerID := r.pool.RedoRequest(first.Height)
  456. r.blockSyncCh.Error <- p2p.PeerError{
  457. NodeID: peerID,
  458. Err: err,
  459. }
  460. peerID2 := r.pool.RedoRequest(second.Height)
  461. if peerID2 != peerID {
  462. r.blockSyncCh.Error <- p2p.PeerError{
  463. NodeID: peerID2,
  464. Err: err,
  465. }
  466. }
  467. continue FOR_LOOP
  468. } else {
  469. r.pool.PopRequest()
  470. // TODO: batch saves so we do not persist to disk every block
  471. r.store.SaveBlock(first, firstParts, second.LastCommit)
  472. var err error
  473. // TODO: Same thing for app - but we would need a way to get the hash
  474. // without persisting the state.
  475. state, err = r.blockExec.ApplyBlock(state, firstID, first)
  476. if err != nil {
  477. // TODO: This is bad, are we zombie?
  478. panic(fmt.Sprintf("failed to process committed block (%d:%X): %v", first.Height, first.Hash(), err))
  479. }
  480. r.metrics.RecordConsMetrics(first)
  481. blocksSynced++
  482. if blocksSynced%100 == 0 {
  483. lastRate = 0.9*lastRate + 0.1*(100/time.Since(lastHundred).Seconds())
  484. r.Logger.Info(
  485. "block sync rate",
  486. "height", r.pool.height,
  487. "max_peer_height", r.pool.MaxPeerHeight(),
  488. "blocks/s", lastRate,
  489. )
  490. lastHundred = time.Now()
  491. }
  492. }
  493. continue FOR_LOOP
  494. case <-r.closeCh:
  495. break FOR_LOOP
  496. case <-r.pool.Quit():
  497. break FOR_LOOP
  498. }
  499. }
  500. }
  501. func (r *Reactor) GetMaxPeerBlockHeight() int64 {
  502. return r.pool.MaxPeerHeight()
  503. }
  504. func (r *Reactor) GetTotalSyncedTime() time.Duration {
  505. if !r.blockSync.IsSet() || r.syncStartTime.IsZero() {
  506. return time.Duration(0)
  507. }
  508. return time.Since(r.syncStartTime)
  509. }
  510. func (r *Reactor) GetRemainingSyncTime() time.Duration {
  511. if !r.blockSync.IsSet() {
  512. return time.Duration(0)
  513. }
  514. targetSyncs := r.pool.targetSyncBlocks()
  515. currentSyncs := r.store.Height() - r.pool.startHeight + 1
  516. lastSyncRate := r.pool.getLastSyncRate()
  517. if currentSyncs < 0 || lastSyncRate < 0.001 {
  518. return time.Duration(0)
  519. }
  520. remain := float64(targetSyncs-currentSyncs) / lastSyncRate
  521. return time.Duration(int64(remain * float64(time.Second)))
  522. }