You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

229 lines
6.9 KiB

blockchain: Reorg reactor (#3561) * go routines in blockchain reactor * Added reference to the go routine diagram * Initial commit * cleanup * Undo testing_logger change, committed by mistake * Fix the test loggers * pulled some fsm code into pool.go * added pool tests * changes to the design added block requests under peer moved the request trigger in the reactor poolRoutine, triggered now by a ticker in general moved everything required for making block requests smarter in the poolRoutine added a simple map of heights to keep track of what will need to be requested next added a few more tests * send errors to FSM in a different channel than blocks send errors (RemovePeer) from switch on a different channel than the one receiving blocks renamed channels added more pool tests * more pool tests * lint errors * more tests * more tests * switch fast sync to new implementation * fixed data race in tests * cleanup * finished fsm tests * address golangci comments :) * address golangci comments :) * Added timeout on next block needed to advance * updating docs and cleanup * fix issue in test from previous cleanup * cleanup * Added termination scenarios, tests and more cleanup * small fixes to adr, comments and cleanup * Fix bug in sendRequest() If we tried to send a request to a peer not present in the switch, a missing continue statement caused the request to be blackholed in a peer that was removed and never retried. While this bug was manifesting, the reactor kept asking for other blocks that would be stored and never consumed. Added the number of unconsumed blocks in the math for requesting blocks ahead of current processing height so eventually there will be no more blocks requested until the already received ones are consumed. * remove bpPeer's didTimeout field * Use distinct err codes for peer timeout and FSM timeouts * Don't allow peers to update with lower height * review comments from Ethan and Zarko * some cleanup, renaming, comments * Move block execution in separate goroutine * Remove pool's numPending * review comments * fix lint, remove old blockchain reactor and duplicates in fsm tests * small reorg around peer after review comments * add the reactor spec * verify block only once * review comments * change to int for max number of pending requests * cleanup and godoc * Add configuration flag fast sync version * golangci fixes * fix config template * move both reactor versions under blockchain * cleanup, golint, renaming stuff * updated documentation, fixed more golint warnings * integrate with behavior package * sync with master * gofmt * add changelog_pending entry * move to improvments * suggestion to changelog entry
5 years ago
cs: sync WAL more frequently (#3300) As per #3043, this adds a ticker to sync the WAL every 2s while the WAL is running. * Flush WAL every 2s This adds a ticker that flushes the WAL every 2s while the WAL is running. This is related to #3043. * Fix spelling * Increase timeout to 2mins for slower build environments * Make WAL sync interval configurable * Add TODO to replace testChan with more comprehensive testBus * Remove extraneous debug statement * Remove testChan in favour of using system time As per https://github.com/tendermint/tendermint/pull/3300#discussion_r255886586, this removes the `testChan` WAL member and replaces the approach with a system time-oriented one. In this new approach, we keep track of the system time at which each flush and periodic flush successfully occurred. The naming of the various functions is also updated here to be more consistent with "flushing" as opposed to "sync'ing". * Update naming convention and ensure lock for timestamp update * Add Flush method as part of WAL interface Adds a `Flush` method as part of the WAL interface to enforce the idea that we can manually trigger a WAL flush from outside of the WAL. This is employed in the consensus state management to flush the WAL prior to signing votes/proposals, as per https://github.com/tendermint/tendermint/issues/3043#issuecomment-453853630 * Update CHANGELOG_PENDING * Remove mutex approach and replace with DI The dependency injection approach to dealing with testing concerns could allow similar effects to some kind of "testing bus"-based approach. This commit introduces an example of this, where instead of relying on (potentially fragile) timing of things between the code and the test, we inject code into the function under test that can signal the test through a channel. This allows us to avoid the `time.Sleep()`-based approach previously employed. * Update comment on WAL flushing during vote signing Co-Authored-By: thanethomson <connect@thanethomson.com> * Simplify flush interval definition Co-Authored-By: thanethomson <connect@thanethomson.com> * Expand commentary on WAL disk flushing Co-Authored-By: thanethomson <connect@thanethomson.com> * Add broken test to illustrate WAL sync test problem Removes test-related state (dependency injection code) from the WAL data structure and adds test code to illustrate the problem with using `WALGenerateNBlocks` and `wal.SearchForEndHeight` to test periodic sync'ing. * Fix test error messages * Use WAL group buffer size to check for flush A function is added to `libs/autofile/group.go#Group` in order to return the size of the buffered data (i.e. data that has not yet been flushed to disk). The test now checks that, prior to a `time.Sleep`, the group buffer has data in it. After the `time.Sleep` (during which time the periodic flush should have been called), the buffer should be empty. * Remove config root dir removal from #3291 * Add godoc for NewWAL mentioning periodic sync
6 years ago
cs/replay: execCommitBlock should not read from state.lastValidators (#3067) * execCommitBlock should not read from state.lastValidators * fix height 1 * fix blockchain/reactor_test * fix consensus/mempool_test * fix consensus/reactor_test * fix consensus/replay_test * add CHANGELOG * fix consensus/reactor_test * fix consensus/replay_test * add a test for replay validators change * fix mem_pool test * fix byzantine test * remove a redundant code * reduce validator change blocks to 6 * fix * return peer0 config * seperate testName * seperate testName 1 * seperate testName 2 * seperate app db path * seperate app db path 1 * add a lock before startNet * move the lock to reactor_test * simulate just once * try to find problem * handshake only saveState when app version changed * update gometalinter to 3.0.0 (#3233) in the attempt to fix https://circleci.com/gh/tendermint/tendermint/43165 also code is simplified by running gofmt -s . remove unused vars enable linters we're currently passing remove deprecated linters (cherry picked from commit d47094550315c094512a242445e0dde24b5a03f5) * gofmt code * goimport code * change the bool name to testValidatorsChange * adjust receive kvstore.ProtocolVersion * adjust receive kvstore.ProtocolVersion 1 * adjust receive kvstore.ProtocolVersion 3 * fix merge execution.go * fix merge develop * fix merge develop 1 * fix run cleanupFunc * adjust code according to reviewers' opinion * modify the func name match the convention * simplify simulate a chain containing some validator change txs 1 * test CI error * Merge remote-tracking branch 'upstream/develop' into fixReplay 1 * fix pubsub_test * subscribeUnbuffered vote channel
6 years ago
blockchain: Reorg reactor (#3561) * go routines in blockchain reactor * Added reference to the go routine diagram * Initial commit * cleanup * Undo testing_logger change, committed by mistake * Fix the test loggers * pulled some fsm code into pool.go * added pool tests * changes to the design added block requests under peer moved the request trigger in the reactor poolRoutine, triggered now by a ticker in general moved everything required for making block requests smarter in the poolRoutine added a simple map of heights to keep track of what will need to be requested next added a few more tests * send errors to FSM in a different channel than blocks send errors (RemovePeer) from switch on a different channel than the one receiving blocks renamed channels added more pool tests * more pool tests * lint errors * more tests * more tests * switch fast sync to new implementation * fixed data race in tests * cleanup * finished fsm tests * address golangci comments :) * address golangci comments :) * Added timeout on next block needed to advance * updating docs and cleanup * fix issue in test from previous cleanup * cleanup * Added termination scenarios, tests and more cleanup * small fixes to adr, comments and cleanup * Fix bug in sendRequest() If we tried to send a request to a peer not present in the switch, a missing continue statement caused the request to be blackholed in a peer that was removed and never retried. While this bug was manifesting, the reactor kept asking for other blocks that would be stored and never consumed. Added the number of unconsumed blocks in the math for requesting blocks ahead of current processing height so eventually there will be no more blocks requested until the already received ones are consumed. * remove bpPeer's didTimeout field * Use distinct err codes for peer timeout and FSM timeouts * Don't allow peers to update with lower height * review comments from Ethan and Zarko * some cleanup, renaming, comments * Move block execution in separate goroutine * Remove pool's numPending * review comments * fix lint, remove old blockchain reactor and duplicates in fsm tests * small reorg around peer after review comments * add the reactor spec * verify block only once * review comments * change to int for max number of pending requests * cleanup and godoc * Add configuration flag fast sync version * golangci fixes * fix config template * move both reactor versions under blockchain * cleanup, golint, renaming stuff * updated documentation, fixed more golint warnings * integrate with behavior package * sync with master * gofmt * add changelog_pending entry * move to improvments * suggestion to changelog entry
5 years ago
lint: Enable Golint (#4212) * Fix many golint errors * Fix golint errors in the 'lite' package * Don't export Pool.store * Fix typo * Revert unwanted changes * Fix errors in counter package * Fix linter errors in kvstore package * Fix linter error in example package * Fix error in tests package * Fix linter errors in v2 package * Fix linter errors in consensus package * Fix linter errors in evidence package * Fix linter error in fail package * Fix linter errors in query package * Fix linter errors in core package * Fix linter errors in node package * Fix linter errors in mempool package * Fix linter error in conn package * Fix linter errors in pex package * Rename PEXReactor export to Reactor * Fix linter errors in trust package * Fix linter errors in upnp package * Fix linter errors in p2p package * Fix linter errors in proxy package * Fix linter errors in mock_test package * Fix linter error in client_test package * Fix linter errors in coretypes package * Fix linter errors in coregrpc package * Fix linter errors in rpcserver package * Fix linter errors in rpctypes package * Fix linter errors in rpctest package * Fix linter error in json2wal script * Fix linter error in wal2json script * Fix linter errors in kv package * Fix linter error in state package * Fix linter error in grpc_client * Fix linter errors in types package * Fix linter error in version package * Fix remaining errors * Address review comments * Fix broken tests * Reconcile package coregrpc * Fix golangci bot error * Fix new golint errors * Fix broken reference * Enable golint linter * minor changes to bring golint into line * fix failing test * fix pex reactor naming * address PR comments
5 years ago
  1. package consensus
  2. import (
  3. "bufio"
  4. "bytes"
  5. "fmt"
  6. "io"
  7. "path/filepath"
  8. "testing"
  9. "time"
  10. db "github.com/tendermint/tm-db"
  11. "github.com/tendermint/tendermint/abci/example/kvstore"
  12. cfg "github.com/tendermint/tendermint/config"
  13. "github.com/tendermint/tendermint/libs/log"
  14. tmrand "github.com/tendermint/tendermint/libs/rand"
  15. "github.com/tendermint/tendermint/privval"
  16. "github.com/tendermint/tendermint/proxy"
  17. sm "github.com/tendermint/tendermint/state"
  18. "github.com/tendermint/tendermint/store"
  19. "github.com/tendermint/tendermint/types"
  20. )
  21. // WALGenerateNBlocks generates a consensus WAL. It does this by spinning up a
  22. // stripped down version of node (proxy app, event bus, consensus state) with a
  23. // persistent kvstore application and special consensus wal instance
  24. // (byteBufferWAL) and waits until numBlocks are created.
  25. // If the node fails to produce given numBlocks, it returns an error.
  26. func WALGenerateNBlocks(t *testing.T, wr io.Writer, numBlocks int) (err error) {
  27. config := getConfig(t)
  28. app := kvstore.NewPersistentKVStoreApplication(filepath.Join(config.DBDir(), "wal_generator"))
  29. logger := log.TestingLogger().With("wal_generator", "wal_generator")
  30. logger.Info("generating WAL (last height msg excluded)", "numBlocks", numBlocks)
  31. // COPY PASTE FROM node.go WITH A FEW MODIFICATIONS
  32. // NOTE: we can't import node package because of circular dependency.
  33. // NOTE: we don't do handshake so need to set state.Version.Consensus.App directly.
  34. privValidatorKeyFile := config.PrivValidatorKeyFile()
  35. privValidatorStateFile := config.PrivValidatorStateFile()
  36. privValidator, err := privval.LoadOrGenFilePV(privValidatorKeyFile, privValidatorStateFile)
  37. if err != nil {
  38. return err
  39. }
  40. genDoc, err := types.GenesisDocFromFile(config.GenesisFile())
  41. if err != nil {
  42. return fmt.Errorf("failed to read genesis file: %w", err)
  43. }
  44. blockStoreDB := db.NewMemDB()
  45. stateDB := blockStoreDB
  46. stateStore := sm.NewStore(stateDB)
  47. state, err := sm.MakeGenesisState(genDoc)
  48. if err != nil {
  49. return fmt.Errorf("failed to make genesis state: %w", err)
  50. }
  51. state.Version.Consensus.App = kvstore.ProtocolVersion
  52. if err = stateStore.Save(state); err != nil {
  53. t.Error(err)
  54. }
  55. blockStore := store.NewBlockStore(blockStoreDB)
  56. proxyApp := proxy.NewAppConns(proxy.NewLocalClientCreator(app))
  57. proxyApp.SetLogger(logger.With("module", "proxy"))
  58. if err := proxyApp.Start(); err != nil {
  59. return fmt.Errorf("failed to start proxy app connections: %w", err)
  60. }
  61. t.Cleanup(func() {
  62. if err := proxyApp.Stop(); err != nil {
  63. t.Error(err)
  64. }
  65. })
  66. eventBus := types.NewEventBus()
  67. eventBus.SetLogger(logger.With("module", "events"))
  68. if err := eventBus.Start(); err != nil {
  69. return fmt.Errorf("failed to start event bus: %w", err)
  70. }
  71. t.Cleanup(func() {
  72. if err := eventBus.Stop(); err != nil {
  73. t.Error(err)
  74. }
  75. })
  76. mempool := emptyMempool{}
  77. evpool := sm.EmptyEvidencePool{}
  78. blockExec := sm.NewBlockExecutor(stateStore, log.TestingLogger(), proxyApp.Consensus(), mempool, evpool)
  79. consensusState := NewState(config.Consensus, state.Copy(), blockExec, blockStore, mempool, evpool)
  80. consensusState.SetLogger(logger)
  81. consensusState.SetEventBus(eventBus)
  82. if privValidator != nil {
  83. consensusState.SetPrivValidator(privValidator)
  84. }
  85. // END OF COPY PASTE
  86. // set consensus wal to buffered WAL, which will write all incoming msgs to buffer
  87. numBlocksWritten := make(chan struct{})
  88. wal := newByteBufferWAL(logger, NewWALEncoder(wr), int64(numBlocks), numBlocksWritten)
  89. // see wal.go#103
  90. if err := wal.Write(EndHeightMessage{0}); err != nil {
  91. t.Error(err)
  92. }
  93. consensusState.wal = wal
  94. if err := consensusState.Start(); err != nil {
  95. return fmt.Errorf("failed to start consensus state: %w", err)
  96. }
  97. select {
  98. case <-numBlocksWritten:
  99. if err := consensusState.Stop(); err != nil {
  100. t.Error(err)
  101. }
  102. return nil
  103. case <-time.After(1 * time.Minute):
  104. if err := consensusState.Stop(); err != nil {
  105. t.Error(err)
  106. }
  107. return fmt.Errorf("waited too long for tendermint to produce %d blocks (grep logs for `wal_generator`)", numBlocks)
  108. }
  109. }
  110. // WALWithNBlocks returns a WAL content with numBlocks.
  111. func WALWithNBlocks(t *testing.T, numBlocks int) (data []byte, err error) {
  112. var b bytes.Buffer
  113. wr := bufio.NewWriter(&b)
  114. if err := WALGenerateNBlocks(t, wr, numBlocks); err != nil {
  115. return []byte{}, err
  116. }
  117. wr.Flush()
  118. return b.Bytes(), nil
  119. }
  120. func randPort() int {
  121. // returns between base and base + spread
  122. base, spread := 20000, 20000
  123. return base + tmrand.Intn(spread)
  124. }
  125. func makeAddrs() (string, string, string) {
  126. start := randPort()
  127. return fmt.Sprintf("tcp://127.0.0.1:%d", start),
  128. fmt.Sprintf("tcp://127.0.0.1:%d", start+1),
  129. fmt.Sprintf("tcp://127.0.0.1:%d", start+2)
  130. }
  131. // getConfig returns a config for test cases
  132. func getConfig(t *testing.T) *cfg.Config {
  133. c := cfg.ResetTestRoot(t.Name())
  134. // and we use random ports to run in parallel
  135. tm, rpc, grpc := makeAddrs()
  136. c.P2P.ListenAddress = tm
  137. c.RPC.ListenAddress = rpc
  138. c.RPC.GRPCListenAddress = grpc
  139. return c
  140. }
  141. // byteBufferWAL is a WAL which writes all msgs to a byte buffer. Writing stops
  142. // when the heightToStop is reached. Client will be notified via
  143. // signalWhenStopsTo channel.
  144. type byteBufferWAL struct {
  145. enc *WALEncoder
  146. stopped bool
  147. heightToStop int64
  148. signalWhenStopsTo chan<- struct{}
  149. logger log.Logger
  150. }
  151. // needed for determinism
  152. var fixedTime, _ = time.Parse(time.RFC3339, "2017-01-02T15:04:05Z")
  153. func newByteBufferWAL(logger log.Logger, enc *WALEncoder, nBlocks int64, signalStop chan<- struct{}) *byteBufferWAL {
  154. return &byteBufferWAL{
  155. enc: enc,
  156. heightToStop: nBlocks,
  157. signalWhenStopsTo: signalStop,
  158. logger: logger,
  159. }
  160. }
  161. // Save writes message to the internal buffer except when heightToStop is
  162. // reached, in which case it will signal the caller via signalWhenStopsTo and
  163. // skip writing.
  164. func (w *byteBufferWAL) Write(m WALMessage) error {
  165. if w.stopped {
  166. w.logger.Debug("WAL already stopped. Not writing message", "msg", m)
  167. return nil
  168. }
  169. if endMsg, ok := m.(EndHeightMessage); ok {
  170. w.logger.Debug("WAL write end height message", "height", endMsg.Height, "stopHeight", w.heightToStop)
  171. if endMsg.Height == w.heightToStop {
  172. w.logger.Debug("Stopping WAL at height", "height", endMsg.Height)
  173. w.signalWhenStopsTo <- struct{}{}
  174. w.stopped = true
  175. return nil
  176. }
  177. }
  178. w.logger.Debug("WAL Write Message", "msg", m)
  179. err := w.enc.Encode(&TimedWALMessage{fixedTime, m})
  180. if err != nil {
  181. panic(fmt.Sprintf("failed to encode the msg %v", m))
  182. }
  183. return nil
  184. }
  185. func (w *byteBufferWAL) WriteSync(m WALMessage) error {
  186. return w.Write(m)
  187. }
  188. func (w *byteBufferWAL) FlushAndSync() error { return nil }
  189. func (w *byteBufferWAL) SearchForEndHeight(
  190. height int64,
  191. options *WALSearchOptions) (rd io.ReadCloser, found bool, err error) {
  192. return nil, false, nil
  193. }
  194. func (w *byteBufferWAL) Start() error { return nil }
  195. func (w *byteBufferWAL) Stop() error { return nil }
  196. func (w *byteBufferWAL) Wait() {}