You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

209 lines
6.6 KiB

blockchain: Reorg reactor (#3561) * go routines in blockchain reactor * Added reference to the go routine diagram * Initial commit * cleanup * Undo testing_logger change, committed by mistake * Fix the test loggers * pulled some fsm code into pool.go * added pool tests * changes to the design added block requests under peer moved the request trigger in the reactor poolRoutine, triggered now by a ticker in general moved everything required for making block requests smarter in the poolRoutine added a simple map of heights to keep track of what will need to be requested next added a few more tests * send errors to FSM in a different channel than blocks send errors (RemovePeer) from switch on a different channel than the one receiving blocks renamed channels added more pool tests * more pool tests * lint errors * more tests * more tests * switch fast sync to new implementation * fixed data race in tests * cleanup * finished fsm tests * address golangci comments :) * address golangci comments :) * Added timeout on next block needed to advance * updating docs and cleanup * fix issue in test from previous cleanup * cleanup * Added termination scenarios, tests and more cleanup * small fixes to adr, comments and cleanup * Fix bug in sendRequest() If we tried to send a request to a peer not present in the switch, a missing continue statement caused the request to be blackholed in a peer that was removed and never retried. While this bug was manifesting, the reactor kept asking for other blocks that would be stored and never consumed. Added the number of unconsumed blocks in the math for requesting blocks ahead of current processing height so eventually there will be no more blocks requested until the already received ones are consumed. * remove bpPeer's didTimeout field * Use distinct err codes for peer timeout and FSM timeouts * Don't allow peers to update with lower height * review comments from Ethan and Zarko * some cleanup, renaming, comments * Move block execution in separate goroutine * Remove pool's numPending * review comments * fix lint, remove old blockchain reactor and duplicates in fsm tests * small reorg around peer after review comments * add the reactor spec * verify block only once * review comments * change to int for max number of pending requests * cleanup and godoc * Add configuration flag fast sync version * golangci fixes * fix config template * move both reactor versions under blockchain * cleanup, golint, renaming stuff * updated documentation, fixed more golint warnings * integrate with behavior package * sync with master * gofmt * add changelog_pending entry * move to improvments * suggestion to changelog entry
5 years ago
cs: sync WAL more frequently (#3300) As per #3043, this adds a ticker to sync the WAL every 2s while the WAL is running. * Flush WAL every 2s This adds a ticker that flushes the WAL every 2s while the WAL is running. This is related to #3043. * Fix spelling * Increase timeout to 2mins for slower build environments * Make WAL sync interval configurable * Add TODO to replace testChan with more comprehensive testBus * Remove extraneous debug statement * Remove testChan in favour of using system time As per https://github.com/tendermint/tendermint/pull/3300#discussion_r255886586, this removes the `testChan` WAL member and replaces the approach with a system time-oriented one. In this new approach, we keep track of the system time at which each flush and periodic flush successfully occurred. The naming of the various functions is also updated here to be more consistent with "flushing" as opposed to "sync'ing". * Update naming convention and ensure lock for timestamp update * Add Flush method as part of WAL interface Adds a `Flush` method as part of the WAL interface to enforce the idea that we can manually trigger a WAL flush from outside of the WAL. This is employed in the consensus state management to flush the WAL prior to signing votes/proposals, as per https://github.com/tendermint/tendermint/issues/3043#issuecomment-453853630 * Update CHANGELOG_PENDING * Remove mutex approach and replace with DI The dependency injection approach to dealing with testing concerns could allow similar effects to some kind of "testing bus"-based approach. This commit introduces an example of this, where instead of relying on (potentially fragile) timing of things between the code and the test, we inject code into the function under test that can signal the test through a channel. This allows us to avoid the `time.Sleep()`-based approach previously employed. * Update comment on WAL flushing during vote signing Co-Authored-By: thanethomson <connect@thanethomson.com> * Simplify flush interval definition Co-Authored-By: thanethomson <connect@thanethomson.com> * Expand commentary on WAL disk flushing Co-Authored-By: thanethomson <connect@thanethomson.com> * Add broken test to illustrate WAL sync test problem Removes test-related state (dependency injection code) from the WAL data structure and adds test code to illustrate the problem with using `WALGenerateNBlocks` and `wal.SearchForEndHeight` to test periodic sync'ing. * Fix test error messages * Use WAL group buffer size to check for flush A function is added to `libs/autofile/group.go#Group` in order to return the size of the buffered data (i.e. data that has not yet been flushed to disk). The test now checks that, prior to a `time.Sleep`, the group buffer has data in it. After the `time.Sleep` (during which time the periodic flush should have been called), the buffer should be empty. * Remove config root dir removal from #3291 * Add godoc for NewWAL mentioning periodic sync
6 years ago
cs/replay: execCommitBlock should not read from state.lastValidators (#3067) * execCommitBlock should not read from state.lastValidators * fix height 1 * fix blockchain/reactor_test * fix consensus/mempool_test * fix consensus/reactor_test * fix consensus/replay_test * add CHANGELOG * fix consensus/reactor_test * fix consensus/replay_test * add a test for replay validators change * fix mem_pool test * fix byzantine test * remove a redundant code * reduce validator change blocks to 6 * fix * return peer0 config * seperate testName * seperate testName 1 * seperate testName 2 * seperate app db path * seperate app db path 1 * add a lock before startNet * move the lock to reactor_test * simulate just once * try to find problem * handshake only saveState when app version changed * update gometalinter to 3.0.0 (#3233) in the attempt to fix https://circleci.com/gh/tendermint/tendermint/43165 also code is simplified by running gofmt -s . remove unused vars enable linters we're currently passing remove deprecated linters (cherry picked from commit d47094550315c094512a242445e0dde24b5a03f5) * gofmt code * goimport code * change the bool name to testValidatorsChange * adjust receive kvstore.ProtocolVersion * adjust receive kvstore.ProtocolVersion 1 * adjust receive kvstore.ProtocolVersion 3 * fix merge execution.go * fix merge develop * fix merge develop 1 * fix run cleanupFunc * adjust code according to reviewers' opinion * modify the func name match the convention * simplify simulate a chain containing some validator change txs 1 * test CI error * Merge remote-tracking branch 'upstream/develop' into fixReplay 1 * fix pubsub_test * subscribeUnbuffered vote channel
6 years ago
cs/replay: execCommitBlock should not read from state.lastValidators (#3067) * execCommitBlock should not read from state.lastValidators * fix height 1 * fix blockchain/reactor_test * fix consensus/mempool_test * fix consensus/reactor_test * fix consensus/replay_test * add CHANGELOG * fix consensus/reactor_test * fix consensus/replay_test * add a test for replay validators change * fix mem_pool test * fix byzantine test * remove a redundant code * reduce validator change blocks to 6 * fix * return peer0 config * seperate testName * seperate testName 1 * seperate testName 2 * seperate app db path * seperate app db path 1 * add a lock before startNet * move the lock to reactor_test * simulate just once * try to find problem * handshake only saveState when app version changed * update gometalinter to 3.0.0 (#3233) in the attempt to fix https://circleci.com/gh/tendermint/tendermint/43165 also code is simplified by running gofmt -s . remove unused vars enable linters we're currently passing remove deprecated linters (cherry picked from commit d47094550315c094512a242445e0dde24b5a03f5) * gofmt code * goimport code * change the bool name to testValidatorsChange * adjust receive kvstore.ProtocolVersion * adjust receive kvstore.ProtocolVersion 1 * adjust receive kvstore.ProtocolVersion 3 * fix merge execution.go * fix merge develop * fix merge develop 1 * fix run cleanupFunc * adjust code according to reviewers' opinion * modify the func name match the convention * simplify simulate a chain containing some validator change txs 1 * test CI error * Merge remote-tracking branch 'upstream/develop' into fixReplay 1 * fix pubsub_test * subscribeUnbuffered vote channel
6 years ago
blockchain: Reorg reactor (#3561) * go routines in blockchain reactor * Added reference to the go routine diagram * Initial commit * cleanup * Undo testing_logger change, committed by mistake * Fix the test loggers * pulled some fsm code into pool.go * added pool tests * changes to the design added block requests under peer moved the request trigger in the reactor poolRoutine, triggered now by a ticker in general moved everything required for making block requests smarter in the poolRoutine added a simple map of heights to keep track of what will need to be requested next added a few more tests * send errors to FSM in a different channel than blocks send errors (RemovePeer) from switch on a different channel than the one receiving blocks renamed channels added more pool tests * more pool tests * lint errors * more tests * more tests * switch fast sync to new implementation * fixed data race in tests * cleanup * finished fsm tests * address golangci comments :) * address golangci comments :) * Added timeout on next block needed to advance * updating docs and cleanup * fix issue in test from previous cleanup * cleanup * Added termination scenarios, tests and more cleanup * small fixes to adr, comments and cleanup * Fix bug in sendRequest() If we tried to send a request to a peer not present in the switch, a missing continue statement caused the request to be blackholed in a peer that was removed and never retried. While this bug was manifesting, the reactor kept asking for other blocks that would be stored and never consumed. Added the number of unconsumed blocks in the math for requesting blocks ahead of current processing height so eventually there will be no more blocks requested until the already received ones are consumed. * remove bpPeer's didTimeout field * Use distinct err codes for peer timeout and FSM timeouts * Don't allow peers to update with lower height * review comments from Ethan and Zarko * some cleanup, renaming, comments * Move block execution in separate goroutine * Remove pool's numPending * review comments * fix lint, remove old blockchain reactor and duplicates in fsm tests * small reorg around peer after review comments * add the reactor spec * verify block only once * review comments * change to int for max number of pending requests * cleanup and godoc * Add configuration flag fast sync version * golangci fixes * fix config template * move both reactor versions under blockchain * cleanup, golint, renaming stuff * updated documentation, fixed more golint warnings * integrate with behavior package * sync with master * gofmt * add changelog_pending entry * move to improvments * suggestion to changelog entry
5 years ago
lint: Enable Golint (#4212) * Fix many golint errors * Fix golint errors in the 'lite' package * Don't export Pool.store * Fix typo * Revert unwanted changes * Fix errors in counter package * Fix linter errors in kvstore package * Fix linter error in example package * Fix error in tests package * Fix linter errors in v2 package * Fix linter errors in consensus package * Fix linter errors in evidence package * Fix linter error in fail package * Fix linter errors in query package * Fix linter errors in core package * Fix linter errors in node package * Fix linter errors in mempool package * Fix linter error in conn package * Fix linter errors in pex package * Rename PEXReactor export to Reactor * Fix linter errors in trust package * Fix linter errors in upnp package * Fix linter errors in p2p package * Fix linter errors in proxy package * Fix linter errors in mock_test package * Fix linter error in client_test package * Fix linter errors in coretypes package * Fix linter errors in coregrpc package * Fix linter errors in rpcserver package * Fix linter errors in rpctypes package * Fix linter errors in rpctest package * Fix linter error in json2wal script * Fix linter error in wal2json script * Fix linter errors in kv package * Fix linter error in state package * Fix linter error in grpc_client * Fix linter errors in types package * Fix linter error in version package * Fix remaining errors * Address review comments * Fix broken tests * Reconcile package coregrpc * Fix golangci bot error * Fix new golint errors * Fix broken reference * Enable golint linter * minor changes to bring golint into line * fix failing test * fix pex reactor naming * address PR comments
5 years ago
  1. package consensus
  2. import (
  3. "bufio"
  4. "bytes"
  5. "fmt"
  6. "io"
  7. "path/filepath"
  8. "testing"
  9. "time"
  10. db "github.com/tendermint/tm-db"
  11. "github.com/tendermint/tendermint/abci/example/kvstore"
  12. cfg "github.com/tendermint/tendermint/config"
  13. "github.com/tendermint/tendermint/libs/log"
  14. tmrand "github.com/tendermint/tendermint/libs/rand"
  15. "github.com/tendermint/tendermint/privval"
  16. "github.com/tendermint/tendermint/proxy"
  17. sm "github.com/tendermint/tendermint/state"
  18. "github.com/tendermint/tendermint/store"
  19. "github.com/tendermint/tendermint/types"
  20. )
  21. // WALGenerateNBlocks generates a consensus WAL. It does this by spinning up a
  22. // stripped down version of node (proxy app, event bus, consensus state) with a
  23. // persistent kvstore application and special consensus wal instance
  24. // (byteBufferWAL) and waits until numBlocks are created.
  25. // If the node fails to produce given numBlocks, it returns an error.
  26. func WALGenerateNBlocks(t *testing.T, wr io.Writer, numBlocks int) (err error) {
  27. config := getConfig(t)
  28. app := kvstore.NewPersistentKVStoreApplication(filepath.Join(config.DBDir(), "wal_generator"))
  29. logger := log.TestingLogger().With("wal_generator", "wal_generator")
  30. logger.Info("generating WAL (last height msg excluded)", "numBlocks", numBlocks)
  31. /////////////////////////////////////////////////////////////////////////////
  32. // COPY PASTE FROM node.go WITH A FEW MODIFICATIONS
  33. // NOTE: we can't import node package because of circular dependency.
  34. // NOTE: we don't do handshake so need to set state.Version.Consensus.App directly.
  35. privValidatorKeyFile := config.PrivValidatorKeyFile()
  36. privValidatorStateFile := config.PrivValidatorStateFile()
  37. privValidator := privval.LoadOrGenFilePV(privValidatorKeyFile, privValidatorStateFile)
  38. genDoc, err := types.GenesisDocFromFile(config.GenesisFile())
  39. if err != nil {
  40. return fmt.Errorf("failed to read genesis file: %w", err)
  41. }
  42. blockStoreDB := db.NewMemDB()
  43. stateDB := blockStoreDB
  44. state, err := sm.MakeGenesisState(genDoc)
  45. if err != nil {
  46. return fmt.Errorf("failed to make genesis state: %w", err)
  47. }
  48. state.Version.Consensus.App = kvstore.ProtocolVersion
  49. sm.SaveState(stateDB, state)
  50. blockStore := store.NewBlockStore(blockStoreDB)
  51. proxyApp := proxy.NewAppConns(proxy.NewLocalClientCreator(app))
  52. proxyApp.SetLogger(logger.With("module", "proxy"))
  53. if err := proxyApp.Start(); err != nil {
  54. return fmt.Errorf("failed to start proxy app connections: %w", err)
  55. }
  56. defer proxyApp.Stop()
  57. eventBus := types.NewEventBus()
  58. eventBus.SetLogger(logger.With("module", "events"))
  59. if err := eventBus.Start(); err != nil {
  60. return fmt.Errorf("failed to start event bus: %w", err)
  61. }
  62. defer eventBus.Stop()
  63. mempool := emptyMempool{}
  64. evpool := emptyEvidencePool{}
  65. blockExec := sm.NewBlockExecutor(stateDB, log.TestingLogger(), proxyApp.Consensus(), mempool, evpool)
  66. consensusState := NewState(config.Consensus, state.Copy(), blockExec, blockStore, mempool, evpool)
  67. consensusState.SetLogger(logger)
  68. consensusState.SetEventBus(eventBus)
  69. if privValidator != nil {
  70. consensusState.SetPrivValidator(privValidator)
  71. }
  72. // END OF COPY PASTE
  73. /////////////////////////////////////////////////////////////////////////////
  74. // set consensus wal to buffered WAL, which will write all incoming msgs to buffer
  75. numBlocksWritten := make(chan struct{})
  76. wal := newByteBufferWAL(logger, NewWALEncoder(wr), int64(numBlocks), numBlocksWritten)
  77. // see wal.go#103
  78. wal.Write(EndHeightMessage{0})
  79. consensusState.wal = wal
  80. if err := consensusState.Start(); err != nil {
  81. return fmt.Errorf("failed to start consensus state: %w", err)
  82. }
  83. select {
  84. case <-numBlocksWritten:
  85. consensusState.Stop()
  86. return nil
  87. case <-time.After(1 * time.Minute):
  88. consensusState.Stop()
  89. return fmt.Errorf("waited too long for tendermint to produce %d blocks (grep logs for `wal_generator`)", numBlocks)
  90. }
  91. }
  92. //WALWithNBlocks returns a WAL content with numBlocks.
  93. func WALWithNBlocks(t *testing.T, numBlocks int) (data []byte, err error) {
  94. var b bytes.Buffer
  95. wr := bufio.NewWriter(&b)
  96. if err := WALGenerateNBlocks(t, wr, numBlocks); err != nil {
  97. return []byte{}, err
  98. }
  99. wr.Flush()
  100. return b.Bytes(), nil
  101. }
  102. func randPort() int {
  103. // returns between base and base + spread
  104. base, spread := 20000, 20000
  105. return base + tmrand.Intn(spread)
  106. }
  107. func makeAddrs() (string, string, string) {
  108. start := randPort()
  109. return fmt.Sprintf("tcp://127.0.0.1:%d", start),
  110. fmt.Sprintf("tcp://127.0.0.1:%d", start+1),
  111. fmt.Sprintf("tcp://127.0.0.1:%d", start+2)
  112. }
  113. // getConfig returns a config for test cases
  114. func getConfig(t *testing.T) *cfg.Config {
  115. c := cfg.ResetTestRoot(t.Name())
  116. // and we use random ports to run in parallel
  117. tm, rpc, grpc := makeAddrs()
  118. c.P2P.ListenAddress = tm
  119. c.RPC.ListenAddress = rpc
  120. c.RPC.GRPCListenAddress = grpc
  121. return c
  122. }
  123. // byteBufferWAL is a WAL which writes all msgs to a byte buffer. Writing stops
  124. // when the heightToStop is reached. Client will be notified via
  125. // signalWhenStopsTo channel.
  126. type byteBufferWAL struct {
  127. enc *WALEncoder
  128. stopped bool
  129. heightToStop int64
  130. signalWhenStopsTo chan<- struct{}
  131. logger log.Logger
  132. }
  133. // needed for determinism
  134. var fixedTime, _ = time.Parse(time.RFC3339, "2017-01-02T15:04:05Z")
  135. func newByteBufferWAL(logger log.Logger, enc *WALEncoder, nBlocks int64, signalStop chan<- struct{}) *byteBufferWAL {
  136. return &byteBufferWAL{
  137. enc: enc,
  138. heightToStop: nBlocks,
  139. signalWhenStopsTo: signalStop,
  140. logger: logger,
  141. }
  142. }
  143. // Save writes message to the internal buffer except when heightToStop is
  144. // reached, in which case it will signal the caller via signalWhenStopsTo and
  145. // skip writing.
  146. func (w *byteBufferWAL) Write(m WALMessage) error {
  147. if w.stopped {
  148. w.logger.Debug("WAL already stopped. Not writing message", "msg", m)
  149. return nil
  150. }
  151. if endMsg, ok := m.(EndHeightMessage); ok {
  152. w.logger.Debug("WAL write end height message", "height", endMsg.Height, "stopHeight", w.heightToStop)
  153. if endMsg.Height == w.heightToStop {
  154. w.logger.Debug("Stopping WAL at height", "height", endMsg.Height)
  155. w.signalWhenStopsTo <- struct{}{}
  156. w.stopped = true
  157. return nil
  158. }
  159. }
  160. w.logger.Debug("WAL Write Message", "msg", m)
  161. err := w.enc.Encode(&TimedWALMessage{fixedTime, m})
  162. if err != nil {
  163. panic(fmt.Sprintf("failed to encode the msg %v", m))
  164. }
  165. return nil
  166. }
  167. func (w *byteBufferWAL) WriteSync(m WALMessage) error {
  168. return w.Write(m)
  169. }
  170. func (w *byteBufferWAL) FlushAndSync() error { return nil }
  171. func (w *byteBufferWAL) SearchForEndHeight(
  172. height int64,
  173. options *WALSearchOptions) (rd io.ReadCloser, found bool, err error) {
  174. return nil, false, nil
  175. }
  176. func (w *byteBufferWAL) Start() error { return nil }
  177. func (w *byteBufferWAL) Stop() error { return nil }
  178. func (w *byteBufferWAL) Wait() {}