You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

1399 lines
39 KiB

lint: Enable Golint (#4212) * Fix many golint errors * Fix golint errors in the 'lite' package * Don't export Pool.store * Fix typo * Revert unwanted changes * Fix errors in counter package * Fix linter errors in kvstore package * Fix linter error in example package * Fix error in tests package * Fix linter errors in v2 package * Fix linter errors in consensus package * Fix linter errors in evidence package * Fix linter error in fail package * Fix linter errors in query package * Fix linter errors in core package * Fix linter errors in node package * Fix linter errors in mempool package * Fix linter error in conn package * Fix linter errors in pex package * Rename PEXReactor export to Reactor * Fix linter errors in trust package * Fix linter errors in upnp package * Fix linter errors in p2p package * Fix linter errors in proxy package * Fix linter errors in mock_test package * Fix linter error in client_test package * Fix linter errors in coretypes package * Fix linter errors in coregrpc package * Fix linter errors in rpcserver package * Fix linter errors in rpctypes package * Fix linter errors in rpctest package * Fix linter error in json2wal script * Fix linter error in wal2json script * Fix linter errors in kv package * Fix linter error in state package * Fix linter error in grpc_client * Fix linter errors in types package * Fix linter error in version package * Fix remaining errors * Address review comments * Fix broken tests * Reconcile package coregrpc * Fix golangci bot error * Fix new golint errors * Fix broken reference * Enable golint linter * minor changes to bring golint into line * fix failing test * fix pex reactor naming * address PR comments
5 years ago
10 years ago
new pubsub package comment out failing consensus tests for now rewrite rpc httpclient to use new pubsub package import pubsub as tmpubsub, query as tmquery make event IDs constants EventKey -> EventTypeKey rename EventsPubsub to PubSub mempool does not use pubsub rename eventsSub to pubsub new subscribe API fix channel size issues and consensus tests bugs refactor rpc client add missing discardFromChan method add mutex rename pubsub to eventBus remove IsRunning from WSRPCConnection interface (not needed) add a comment in broadcastNewRoundStepsAndVotes rename registerEventCallbacks to broadcastNewRoundStepsAndVotes See https://dave.cheney.net/2014/03/19/channel-axioms stop eventBuses after reactor tests remove unnecessary Unsubscribe return subscribe helper function move discardFromChan to where it is used subscribe now returns an err this gives us ability to refuse to subscribe if pubsub is at its max capacity. use context for control overflow cache queries handle err when subscribing in replay_test rename testClientID to testSubscriber extract var set channel buffer capacity to 1 in replay_file fix byzantine_test unsubscribe from single event, not all events refactor httpclient to return events to appropriate channels return failing testReplayCrashBeforeWriteVote test fix TestValidatorSetChanges refactor code a bit fix testReplayCrashBeforeWriteVote add comment fix TestValidatorSetChanges fixes from Bucky's review update comment [ci skip] test TxEventBuffer update changelog fix TestValidatorSetChanges (2nd attempt) only do wg.Done when no errors benchmark event bus create pubsub server inside NewEventBus only expose config params (later if needed) set buffer capacity to 0 so we are not testing cache new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ} This should allow to subscribe to all transactions! or a specific one using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'" use TimeoutCommit instead of afterPublishEventNewBlockTimeout TimeoutCommit is the time a node waits after committing a block, before it goes into the next height. So it will finish everything from the last block, but then wait a bit. The idea is this gives it time to hear more votes from other validators, to strengthen the commit it includes in the next block. But it also gives it time to hear about new transactions. waitForBlockWithUpdatedVals rewrite WAL crash tests Task: test that we can recover from any WAL crash. Solution: the old tests were relying on event hub being run in the same thread (we were injecting the private validator's last signature). when considering a rewrite, we considered two possible solutions: write a "fuzzy" testing system where WAL is crashing upon receiving a new message, or inject failures and trigger them in tests using something like https://github.com/coreos/gofail. remove sleep no cs.Lock around wal.Save test different cases (empty block, non-empty block, ...) comments add comments test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks fixes as per Bucky's last review reset subscriptions on UnsubscribeAll use a simple counter to track message for which we panicked also, set a smaller part size for all test cases
7 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
add support for block pruning via ABCI Commit response (#4588) * Added BlockStore.DeleteBlock() * Added initial block pruner prototype * wip * Added BlockStore.PruneBlocks() * Added consensus setting for block pruning * Added BlockStore base * Error on replay if base does not have blocks * Handle missing blocks when sending VoteSetMaj23Message * Error message tweak * Properly update blockstore state * Error message fix again * blockchain: ignore peer missing blocks * Added FIXME * Added test for block replay with truncated history * Handle peer base in blockchain reactor * Improved replay error handling * Added tests for Store.PruneBlocks() * Fix non-RPC handling of truncated block history * Panic on missing block meta in needProofBlock() * Updated changelog * Handle truncated block history in RPC layer * Added info about earliest block in /status RPC * Reorder height and base in blockchain reactor messages * Updated changelog * Fix tests * Appease linter * Minor review fixes * Non-empty BlockStores should always have base > 0 * Update code to assume base > 0 invariant * Added blockstore tests for pruning to 0 * Make sure we don't prune below the current base * Added BlockStore.Size() * config: added retain_blocks recommendations * Update v1 blockchain reactor to handle blockstore base * Added state database pruning * Propagate errors on missing validator sets * Comment tweaks * Improved error message Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * use ABCI field ResponseCommit.retain_height instead of retain-blocks config option * remove State.RetainHeight, return value instead * fix minor issues * rename pruneHeights() to pruneBlocks() * noop to fix GitHub borkage Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
5 years ago
abci: localClient improvements & bugfixes & pubsub Unsubscribe issues (#2748) * use READ lock/unlock in ConsensusState#GetLastHeight Refs #2721 * do not use defers when there's no need * fix peer formatting (output its address instead of the pointer) ``` [54310]: E[11-02|11:59:39.851] Connection failed @ sendRoutine module=p2p peer=0xb78f00 conn=MConn{74.207.236.148:26656} err="pong timeout" ``` https://github.com/tendermint/tendermint/issues/2721#issuecomment-435326581 * panic if peer has no state https://github.com/tendermint/tendermint/issues/2721#issuecomment-435347165 It's confusing that sometimes we check if peer has a state, but most of the times we expect it to be there 1. https://github.com/tendermint/tendermint/blob/add79700b5fe84417538202b6c927c8cc5383672/mempool/reactor.go#L138 2. https://github.com/tendermint/tendermint/blob/add79700b5fe84417538202b6c927c8cc5383672/rpc/core/consensus.go#L196 (edited) I will change everything to always assume peer has a state and panic otherwise that should help identify issues earlier * abci/localclient: extend lock on app callback App callback should be protected by lock as well (note this was already done for InitChainAsync, why not for others???). Otherwise, when we execute the block, tx might come in and call the callback in the same time we're updating it in execBlockOnProxyApp => DATA RACE Fixes #2721 Consensus state is locked ``` goroutine 113333 [semacquire, 309 minutes]: sync.runtime_SemacquireMutex(0xc00180009c, 0xc0000c7e00) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*RWMutex).RLock(0xc001800090) /usr/local/go/src/sync/rwmutex.go:50 +0x4e github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).GetRoundState(0xc001800000, 0x0) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:218 +0x46 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusReactor).queryMaj23Routine(0xc0017def80, 0x11104a0, 0xc0072488f0, 0xc007248 9c0) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/reactor.go:735 +0x16d created by github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusReactor).AddPeer /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/reactor.go:172 +0x236 ``` because localClient is locked ``` goroutine 1899 [semacquire, 309 minutes]: sync.runtime_SemacquireMutex(0xc00003363c, 0xc0000cb500) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc000033638) /usr/local/go/src/sync/mutex.go:134 +0xff github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).SetResponseCallback(0xc0001fb560, 0xc007868540) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:32 +0x33 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).SetResponseCallback(0xc00002f750, 0xc007868540) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:57 +0x40 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1104e20, 0xc002ca0ba0, 0x11092a0, 0xc00002f750, 0xc0001fe960, 0xc000bfc660, 0x110cfe0, 0xc000090330, 0xc9d12, 0xc000d9d5a0, ...) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/state/execution.go:230 +0x1fd github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc002c2a230, 0x7, 0x0, 0xc000eae880, 0x6, 0xc002e52c60, 0x16, 0x1f927, 0xc9d12, 0xc000d9d5a0, ...) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x142 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc001800000, 0x1f928) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:1339 +0xa3e github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc001800000, 0x1f928) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:1270 +0x451 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit.func1(0xc001800000, 0x0, 0x1f928) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:1218 +0x90 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit(0xc001800000, 0x1f928, 0x0) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:1247 +0x6b8 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc001800000, 0xc003d8dea0, 0xc000cf4cc0, 0x28, 0xf1, 0xc003bc7ad0, 0xc003bc7b10) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:1659 +0xbad github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc001800000, 0xc003d8dea0, 0xc000cf4cc0, 0x28, 0xf1, 0xf1, 0xf1) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:1517 +0x59 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc001800000, 0xd98200, 0xc0070dbed0, 0xc000cf4cc0, 0x28) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:660 +0x64b github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc001800000, 0x0) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:617 +0x670 created by github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/consensus/state.go:311 +0x132 ``` tx comes in and CheckTx is executed right when we execute the block ``` goroutine 111044 [semacquire, 309 minutes]: sync.runtime_SemacquireMutex(0xc00003363c, 0x0) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc000033638) /usr/local/go/src/sync/mutex.go:134 +0xff github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).CheckTxAsync(0xc0001fb0e0, 0xc002d94500, 0x13f, 0x280, 0x0) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:85 +0x47 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/proxy.(*appConnMempool).CheckTxAsync(0xc00002f720, 0xc002d94500, 0x13f, 0x280, 0x1) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:114 +0x51 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/mempool.(*Mempool).CheckTx(0xc002d3a320, 0xc002d94500, 0x13f, 0x280, 0xc0072355f0, 0x0, 0x0) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/mempool/mempool.go:316 +0x17b github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/rpc/core.BroadcastTxSync(0xc002d94500, 0x13f, 0x280, 0x0, 0x0, 0x0) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/rpc/core/mempool.go:93 +0xb8 reflect.Value.call(0xd85560, 0x10326c0, 0x13, 0xec7b8b, 0x4, 0xc00663f180, 0x1, 0x1, 0xc00663f180, 0xc00663f188, ...) /usr/local/go/src/reflect/value.go:447 +0x449 reflect.Value.Call(0xd85560, 0x10326c0, 0x13, 0xc00663f180, 0x1, 0x1, 0x0, 0x0, 0xc005cc9344) /usr/local/go/src/reflect/value.go:308 +0xa4 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/rpc/lib/server.makeHTTPHandler.func2(0x1102060, 0xc00663f100, 0xc0082d7900) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/rpc/lib/server/handlers.go:269 +0x188 net/http.HandlerFunc.ServeHTTP(0xc002c81f20, 0x1102060, 0xc00663f100, 0xc0082d7900) /usr/local/go/src/net/http/server.go:1964 +0x44 net/http.(*ServeMux).ServeHTTP(0xc002c81b60, 0x1102060, 0xc00663f100, 0xc0082d7900) /usr/local/go/src/net/http/server.go:2361 +0x127 github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/rpc/lib/server.maxBytesHandler.ServeHTTP(0x10f8a40, 0xc002c81b60, 0xf4240, 0x1102060, 0xc00663f100, 0xc0082d7900) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/rpc/lib/server/http_server.go:219 +0xcf github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1(0x1103220, 0xc00121e620, 0xc0082d7900) /root/go/src/github.com/MinterTeam/minter-go-node/vendor/github.com/tendermint/tendermint/rpc/lib/server/http_server.go:192 +0x394 net/http.HandlerFunc.ServeHTTP(0xc002c06ea0, 0x1103220, 0xc00121e620, 0xc0082d7900) /usr/local/go/src/net/http/server.go:1964 +0x44 net/http.serverHandler.ServeHTTP(0xc001a1aa90, 0x1103220, 0xc00121e620, 0xc0082d7900) /usr/local/go/src/net/http/server.go:2741 +0xab net/http.(*conn).serve(0xc00785a3c0, 0x11041a0, 0xc000f844c0) /usr/local/go/src/net/http/server.go:1847 +0x646 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2851 +0x2f5 ``` * consensus: use read lock in Receive#VoteMessage * use defer to unlock mutex because application might panic * use defer in every method of the localClient * add a changelog entry * drain channels before Unsubscribe(All) Read https://github.com/tendermint/tendermint/blob/55362ed76630f3e1ebec159a598f6a9fb5892cb1/libs/pubsub/pubsub.go#L13 for the detailed explanation of the issue. We'll need to fix it someday. Make sure to keep an eye on https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-033-pubsub.md * retry instead of panic when peer has no state in reactors other than consensus in /dump_consensus_state RPC endpoint, skip a peer with no state * rpc/core/mempool: simplify error messages * rpc/core/mempool: use time.After instead of timer also, do not log DeliverTx result (to be consistent with other memthods) * unlock before calling the callback in reqRes#SetCallback
6 years ago
7 years ago
fix data race Closes #1442 ``` WARNING: DATA RACE Write at 0x00c4209de7c8 by goroutine 23: github.com/tendermint/tendermint/types.(*Block).fillHeader() /home/vagrant/go/src/github.com/tendermint/tendermint/types/block.go:88 +0x157 github.com/tendermint/tendermint/types.(*Block).Hash() /home/vagrant/go/src/github.com/tendermint/tendermint/types/block.go:104 +0x121 github.com/tendermint/tendermint/types.(*Block).HashesTo() /home/vagrant/go/src/github.com/tendermint/tendermint/types/block.go:135 +0x4f github.com/tendermint/tendermint/consensus.(*ConsensusState).enterPrecommit() /home/vagrant/go/src/github.com/tendermint/tendermint/consensus/state.go:1037 +0x182d github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote() /home/vagrant/go/src/github.com/tendermint/tendermint/consensus/state.go:1425 +0x1a6c github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote() /home/vagrant/go/src/github.com/tendermint/tendermint/consensus/state.go:1318 +0x77 github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg() /home/vagrant/go/src/github.com/tendermint/tendermint/consensus/state.go:581 +0x7a9 github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine() /home/vagrant/go/src/github.com/tendermint/tendermint/consensus/state.go:539 +0x6c3 Previous read at 0x00c4209de7c8 by goroutine 47: github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common.(*HexBytes).MarshalJSON() <autogenerated>:1 +0x52 github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.invokeMarshalJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:433 +0x88 github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec)._encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:82 +0x8d2 github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:50 +0x10e github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSONStruct() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:348 +0x539 github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec)._encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:119 +0x83f github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:50 +0x10e github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSONStruct() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:348 +0x539 github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec)._encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:119 +0x83f github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:50 +0x10e github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSONStruct() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:348 +0x539 github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec)._encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:119 +0x83f github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:50 +0x10e github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSONStruct() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:348 +0x539 github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec)._encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:119 +0x83f github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).encodeReflectJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/json-encode.go:50 +0x10e github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino.(*Codec).MarshalJSON() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-amino/amino.go:296 +0x182 github.com/tendermint/tendermint/rpc/lib/types.NewRPCSuccessResponse() /home/vagrant/go/src/github.com/tendermint/tendermint/rpc/lib/types/types.go:100 +0x12c github.com/tendermint/tendermint/rpc/lib/server.makeJSONRPCHandler.func1() /home/vagrant/go/src/github.com/tendermint/tendermint/rpc/lib/server/handlers.go:152 +0xab7 net/http.HandlerFunc.ServeHTTP() /usr/lib/go-1.9/src/net/http/server.go:1918 +0x51 net/http.(*ServeMux).ServeHTTP() /usr/lib/go-1.9/src/net/http/server.go:2254 +0xa2 github.com/tendermint/tendermint/rpc/lib/server.RecoverAndLogHandler.func1() /home/vagrant/go/src/github.com/tendermint/tendermint/rpc/lib/server/http_server.go:138 +0x4fa net/http.HandlerFunc.ServeHTTP() /usr/lib/go-1.9/src/net/http/server.go:1918 +0x51 net/http.serverHandler.ServeHTTP() /usr/lib/go-1.9/src/net/http/server.go:2619 +0xbc net/http.(*conn).serve() /usr/lib/go-1.9/src/net/http/server.go:1801 +0x83b Goroutine 23 (running) created at: github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart() /home/vagrant/go/src/github.com/tendermint/tendermint/consensus/state.go:250 +0x35b github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common.(*BaseService).Start() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common/service.go:130 +0x5fc github.com/tendermint/tendermint/consensus.(*ConsensusReactor).OnStart() /home/vagrant/go/src/github.com/tendermint/tendermint/consensus/reactor.go:69 +0x1b4 github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common.(*BaseService).Start() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common/service.go:130 +0x5fc github.com/tendermint/tendermint/consensus.(*ConsensusReactor).Start() <autogenerated>:1 +0x43 github.com/tendermint/tendermint/p2p.(*Switch).OnStart() /home/vagrant/go/src/github.com/tendermint/tendermint/p2p/switch.go:177 +0x124 github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common.(*BaseService).Start() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common/service.go:130 +0x5fc github.com/tendermint/tendermint/node.(*Node).OnStart() /home/vagrant/go/src/github.com/tendermint/tendermint/node/node.go:416 +0xa1b github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common.(*BaseService).Start() /home/vagrant/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common/service.go:130 +0x5fc github.com/tendermint/tendermint/rpc/test.StartTendermint() /home/vagrant/go/src/github.com/tendermint/tendermint/rpc/test/helpers.go:100 +0x5b github.com/tendermint/tendermint/rpc/client_test.TestMain() /home/vagrant/go/src/github.com/tendermint/tendermint/rpc/client/main_test.go:17 +0x4c main.main() github.com/tendermint/tendermint/rpc/client/_test/_testmain.go:76 +0x1cd Goroutine 47 (running) created at: net/http.(*Server).Serve() /usr/lib/go-1.9/src/net/http/server.go:2720 +0x37c net/http.Serve() /usr/lib/go-1.9/src/net/http/server.go:2323 +0xe2 github.com/tendermint/tendermint/rpc/lib/server.StartHTTPServer.func1() /home/vagrant/go/src/github.com/tendermint/tendermint/rpc/lib/server/http_server.go:35 +0xb3 ```
7 years ago
10 years ago
protect Record* peerStateStats functions by mutex Fixes #1414 DATA RACE: ``` Read at 0x00c4214ee940 by goroutine 146: github.com/tendermint/tendermint/consensus.(*peerStateStats).String() <autogenerated>:1 +0x57 fmt.(*pp).handleMethods() /usr/local/go/src/fmt/print.go:596 +0x3f4 fmt.(*pp).printArg() /usr/local/go/src/fmt/print.go:679 +0x11f fmt.(*pp).doPrintf() /usr/local/go/src/fmt/print.go:996 +0x319 fmt.Sprintf() /usr/local/go/src/fmt/print.go:196 +0x73 github.com/tendermint/tendermint/consensus.(*PeerState).StringIndented() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:1426 +0x573 github.com/tendermint/tendermint/consensus.(*PeerState).String() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:1419 +0x66 github.com/go-logfmt/logfmt.safeString() /home/ubuntu/go/src/github.com/go-logfmt/logfmt/encode.go:299 +0x9d github.com/go-logfmt/logfmt.writeValue() /home/ubuntu/go/src/github.com/go-logfmt/logfmt/encode.go:217 +0x5a0 github.com/go-logfmt/logfmt.(*Encoder).EncodeKeyval() /home/ubuntu/go/src/github.com/go-logfmt/logfmt/encode.go:61 +0x1dd github.com/tendermint/tmlibs/log.tmfmtLogger.Log() /home/ubuntu/go/src/github.com/tendermint/tmlibs/log/tmfmt_logger.go:107 +0x1001 github.com/tendermint/tmlibs/log.(*tmfmtLogger).Log() <autogenerated>:1 +0x93 github.com/go-kit/kit/log.(*context).Log() /home/ubuntu/go/src/github.com/go-kit/kit/log/log.go:124 +0x248 github.com/tendermint/tmlibs/log.(*tmLogger).Debug() /home/ubuntu/go/src/github.com/tendermint/tmlibs/log/tm_logger.go:64 +0x1d0 github.com/tendermint/tendermint/consensus.(*PeerState).PickSendVote() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:1059 +0x242 github.com/tendermint/tendermint/consensus.(*ConsensusReactor).gossipVotesForHeight() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:789 +0x6ef github.com/tendermint/tendermint/consensus.(*ConsensusReactor).gossipVotesRoutine() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:723 +0x1039 Previous write at 0x00c4214ee940 by goroutine 21: github.com/tendermint/tendermint/consensus.(*PeerState).RecordVote() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:1242 +0x15a github.com/tendermint/tendermint/consensus.(*ConsensusReactor).Receive() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:309 +0x32e6 github.com/tendermint/tendermint/p2p.createMConnection.func1() /home/ubuntu/go/src/github.com/tendermint/tendermint/p2p/peer.go:365 +0xea github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine() /home/ubuntu/go/src/github.com/tendermint/tendermint/p2p/conn/connection.go:531 +0x779 ```
7 years ago
protect Record* peerStateStats functions by mutex Fixes #1414 DATA RACE: ``` Read at 0x00c4214ee940 by goroutine 146: github.com/tendermint/tendermint/consensus.(*peerStateStats).String() <autogenerated>:1 +0x57 fmt.(*pp).handleMethods() /usr/local/go/src/fmt/print.go:596 +0x3f4 fmt.(*pp).printArg() /usr/local/go/src/fmt/print.go:679 +0x11f fmt.(*pp).doPrintf() /usr/local/go/src/fmt/print.go:996 +0x319 fmt.Sprintf() /usr/local/go/src/fmt/print.go:196 +0x73 github.com/tendermint/tendermint/consensus.(*PeerState).StringIndented() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:1426 +0x573 github.com/tendermint/tendermint/consensus.(*PeerState).String() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:1419 +0x66 github.com/go-logfmt/logfmt.safeString() /home/ubuntu/go/src/github.com/go-logfmt/logfmt/encode.go:299 +0x9d github.com/go-logfmt/logfmt.writeValue() /home/ubuntu/go/src/github.com/go-logfmt/logfmt/encode.go:217 +0x5a0 github.com/go-logfmt/logfmt.(*Encoder).EncodeKeyval() /home/ubuntu/go/src/github.com/go-logfmt/logfmt/encode.go:61 +0x1dd github.com/tendermint/tmlibs/log.tmfmtLogger.Log() /home/ubuntu/go/src/github.com/tendermint/tmlibs/log/tmfmt_logger.go:107 +0x1001 github.com/tendermint/tmlibs/log.(*tmfmtLogger).Log() <autogenerated>:1 +0x93 github.com/go-kit/kit/log.(*context).Log() /home/ubuntu/go/src/github.com/go-kit/kit/log/log.go:124 +0x248 github.com/tendermint/tmlibs/log.(*tmLogger).Debug() /home/ubuntu/go/src/github.com/tendermint/tmlibs/log/tm_logger.go:64 +0x1d0 github.com/tendermint/tendermint/consensus.(*PeerState).PickSendVote() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:1059 +0x242 github.com/tendermint/tendermint/consensus.(*ConsensusReactor).gossipVotesForHeight() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:789 +0x6ef github.com/tendermint/tendermint/consensus.(*ConsensusReactor).gossipVotesRoutine() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:723 +0x1039 Previous write at 0x00c4214ee940 by goroutine 21: github.com/tendermint/tendermint/consensus.(*PeerState).RecordVote() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:1242 +0x15a github.com/tendermint/tendermint/consensus.(*ConsensusReactor).Receive() github.com/tendermint/tendermint/consensus/_test/_obj_test/reactor.go:309 +0x32e6 github.com/tendermint/tendermint/p2p.createMConnection.func1() /home/ubuntu/go/src/github.com/tendermint/tendermint/p2p/peer.go:365 +0xea github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine() /home/ubuntu/go/src/github.com/tendermint/tendermint/p2p/conn/connection.go:531 +0x779 ```
7 years ago
10 years ago
10 years ago
10 years ago
10 years ago
  1. package consensus
  2. import (
  3. "fmt"
  4. "time"
  5. cstypes "github.com/tendermint/tendermint/consensus/types"
  6. "github.com/tendermint/tendermint/libs/bits"
  7. tmevents "github.com/tendermint/tendermint/libs/events"
  8. "github.com/tendermint/tendermint/libs/log"
  9. "github.com/tendermint/tendermint/libs/service"
  10. tmsync "github.com/tendermint/tendermint/libs/sync"
  11. "github.com/tendermint/tendermint/p2p"
  12. tmcons "github.com/tendermint/tendermint/proto/tendermint/consensus"
  13. tmproto "github.com/tendermint/tendermint/proto/tendermint/types"
  14. sm "github.com/tendermint/tendermint/state"
  15. "github.com/tendermint/tendermint/types"
  16. )
  17. var (
  18. _ service.Service = (*Reactor)(nil)
  19. _ p2p.Wrapper = (*tmcons.Message)(nil)
  20. // ChannelShims contains a map of ChannelDescriptorShim objects, where each
  21. // object wraps a reference to a legacy p2p ChannelDescriptor and the corresponding
  22. // p2p proto.Message the new p2p Channel is responsible for handling.
  23. //
  24. //
  25. // TODO: Remove once p2p refactor is complete.
  26. // ref: https://github.com/tendermint/tendermint/issues/5670
  27. ChannelShims = map[p2p.ChannelID]*p2p.ChannelDescriptorShim{
  28. StateChannel: {
  29. MsgType: new(tmcons.Message),
  30. Descriptor: &p2p.ChannelDescriptor{
  31. ID: byte(StateChannel),
  32. Priority: 6,
  33. SendQueueCapacity: 100,
  34. RecvMessageCapacity: maxMsgSize,
  35. MaxSendBytes: 12000,
  36. },
  37. },
  38. DataChannel: {
  39. MsgType: new(tmcons.Message),
  40. Descriptor: &p2p.ChannelDescriptor{
  41. // TODO: Consider a split between gossiping current block and catchup
  42. // stuff. Once we gossip the whole block there is nothing left to send
  43. // until next height or round.
  44. ID: byte(DataChannel),
  45. Priority: 10,
  46. SendQueueCapacity: 100,
  47. RecvBufferCapacity: 50 * 4096,
  48. RecvMessageCapacity: maxMsgSize,
  49. MaxSendBytes: 40000,
  50. },
  51. },
  52. VoteChannel: {
  53. MsgType: new(tmcons.Message),
  54. Descriptor: &p2p.ChannelDescriptor{
  55. ID: byte(VoteChannel),
  56. Priority: 7,
  57. SendQueueCapacity: 100,
  58. RecvBufferCapacity: 100 * 100,
  59. RecvMessageCapacity: maxMsgSize,
  60. MaxSendBytes: 150,
  61. },
  62. },
  63. VoteSetBitsChannel: {
  64. MsgType: new(tmcons.Message),
  65. Descriptor: &p2p.ChannelDescriptor{
  66. ID: byte(VoteSetBitsChannel),
  67. Priority: 1,
  68. SendQueueCapacity: 2,
  69. RecvBufferCapacity: 1024,
  70. RecvMessageCapacity: maxMsgSize,
  71. MaxSendBytes: 50,
  72. },
  73. },
  74. }
  75. )
  76. const (
  77. StateChannel = p2p.ChannelID(0x20)
  78. DataChannel = p2p.ChannelID(0x21)
  79. VoteChannel = p2p.ChannelID(0x22)
  80. VoteSetBitsChannel = p2p.ChannelID(0x23)
  81. maxMsgSize = 1048576 // 1MB; NOTE: keep in sync with types.PartSet sizes.
  82. blocksToContributeToBecomeGoodPeer = 10000
  83. votesToContributeToBecomeGoodPeer = 10000
  84. listenerIDConsensus = "consensus-reactor"
  85. )
  86. type ReactorOption func(*Reactor)
  87. // Reactor defines a reactor for the consensus service.
  88. type Reactor struct {
  89. service.BaseService
  90. state *State
  91. eventBus *types.EventBus
  92. Metrics *Metrics
  93. mtx tmsync.RWMutex
  94. peers map[p2p.NodeID]*PeerState
  95. waitSync bool
  96. stateCh *p2p.Channel
  97. dataCh *p2p.Channel
  98. voteCh *p2p.Channel
  99. voteSetBitsCh *p2p.Channel
  100. peerUpdates *p2p.PeerUpdates
  101. // NOTE: We need a dedicated stateCloseCh channel for signaling closure of
  102. // the StateChannel due to the fact that the StateChannel message handler
  103. // performs a send on the VoteSetBitsChannel. This is an antipattern, so having
  104. // this dedicated channel,stateCloseCh, is necessary in order to avoid data races.
  105. stateCloseCh chan struct{}
  106. closeCh chan struct{}
  107. }
  108. // NewReactor returns a reference to a new consensus reactor, which implements
  109. // the service.Service interface. It accepts a logger, consensus state, references
  110. // to relevant p2p Channels and a channel to listen for peer updates on. The
  111. // reactor will close all p2p Channels when stopping.
  112. func NewReactor(
  113. logger log.Logger,
  114. cs *State,
  115. stateCh *p2p.Channel,
  116. dataCh *p2p.Channel,
  117. voteCh *p2p.Channel,
  118. voteSetBitsCh *p2p.Channel,
  119. peerUpdates *p2p.PeerUpdates,
  120. waitSync bool,
  121. options ...ReactorOption,
  122. ) *Reactor {
  123. r := &Reactor{
  124. state: cs,
  125. waitSync: waitSync,
  126. peers: make(map[p2p.NodeID]*PeerState),
  127. Metrics: NopMetrics(),
  128. stateCh: stateCh,
  129. dataCh: dataCh,
  130. voteCh: voteCh,
  131. voteSetBitsCh: voteSetBitsCh,
  132. peerUpdates: peerUpdates,
  133. stateCloseCh: make(chan struct{}),
  134. closeCh: make(chan struct{}),
  135. }
  136. r.BaseService = *service.NewBaseService(logger, "Consensus", r)
  137. for _, opt := range options {
  138. opt(r)
  139. }
  140. return r
  141. }
  142. // OnStart starts separate go routines for each p2p Channel and listens for
  143. // envelopes on each. In addition, it also listens for peer updates and handles
  144. // messages on that p2p channel accordingly. The caller must be sure to execute
  145. // OnStop to ensure the outbound p2p Channels are closed.
  146. func (r *Reactor) OnStart() error {
  147. r.Logger.Debug("consensus wait sync", "wait_sync", r.WaitSync())
  148. // start routine that computes peer statistics for evaluating peer quality
  149. //
  150. // TODO: Evaluate if we need this to be synchronized via WaitGroup as to not
  151. // leak the goroutine when stopping the reactor.
  152. go r.peerStatsRoutine()
  153. r.subscribeToBroadcastEvents()
  154. if !r.WaitSync() {
  155. if err := r.state.Start(); err != nil {
  156. return err
  157. }
  158. }
  159. go r.processStateCh()
  160. go r.processDataCh()
  161. go r.processVoteCh()
  162. go r.processVoteSetBitsCh()
  163. go r.processPeerUpdates()
  164. return nil
  165. }
  166. // OnStop stops the reactor by signaling to all spawned goroutines to exit and
  167. // blocking until they all exit, as well as unsubscribing from events and stopping
  168. // state.
  169. func (r *Reactor) OnStop() {
  170. r.unsubscribeFromBroadcastEvents()
  171. if err := r.state.Stop(); err != nil {
  172. r.Logger.Error("failed to stop consensus state", "err", err)
  173. }
  174. if !r.WaitSync() {
  175. r.state.Wait()
  176. }
  177. r.mtx.Lock()
  178. peers := r.peers
  179. r.mtx.Unlock()
  180. // wait for all spawned peer goroutines to gracefully exit
  181. for _, ps := range peers {
  182. ps.closer.Close()
  183. }
  184. for _, ps := range peers {
  185. ps.broadcastWG.Wait()
  186. }
  187. // Close the StateChannel goroutine separately since it uses its own channel
  188. // to signal closure.
  189. close(r.stateCloseCh)
  190. <-r.stateCh.Done()
  191. // Close closeCh to signal to all spawned goroutines to gracefully exit. All
  192. // p2p Channels should execute Close().
  193. close(r.closeCh)
  194. // Wait for all p2p Channels to be closed before returning. This ensures we
  195. // can easily reason about synchronization of all p2p Channels and ensure no
  196. // panics will occur.
  197. <-r.voteSetBitsCh.Done()
  198. <-r.dataCh.Done()
  199. <-r.voteCh.Done()
  200. <-r.peerUpdates.Done()
  201. }
  202. // SetEventBus sets the reactor's event bus.
  203. func (r *Reactor) SetEventBus(b *types.EventBus) {
  204. r.eventBus = b
  205. r.state.SetEventBus(b)
  206. }
  207. // WaitSync returns whether the consensus reactor is waiting for state/fast sync.
  208. func (r *Reactor) WaitSync() bool {
  209. r.mtx.RLock()
  210. defer r.mtx.RUnlock()
  211. return r.waitSync
  212. }
  213. // ReactorMetrics sets the reactor's metrics as an option function.
  214. func ReactorMetrics(metrics *Metrics) ReactorOption {
  215. return func(r *Reactor) { r.Metrics = metrics }
  216. }
  217. // SwitchToConsensus switches from fast-sync mode to consensus mode. It resets
  218. // the state, turns off fast-sync, and starts the consensus state-machine.
  219. func (r *Reactor) SwitchToConsensus(state sm.State, skipWAL bool) {
  220. r.Logger.Info("switching to consensus")
  221. // we have no votes, so reconstruct LastCommit from SeenCommit
  222. if state.LastBlockHeight > 0 {
  223. r.state.reconstructLastCommit(state)
  224. }
  225. // NOTE: The line below causes broadcastNewRoundStepRoutine() to broadcast a
  226. // NewRoundStepMessage.
  227. r.state.updateToState(state)
  228. r.mtx.Lock()
  229. r.waitSync = false
  230. r.mtx.Unlock()
  231. r.Metrics.FastSyncing.Set(0)
  232. r.Metrics.StateSyncing.Set(0)
  233. if skipWAL {
  234. r.state.doWALCatchup = false
  235. }
  236. if err := r.state.Start(); err != nil {
  237. panic(fmt.Sprintf(`failed to start consensus state: %v
  238. conS:
  239. %+v
  240. conR:
  241. %+v`, err, r.state, r))
  242. }
  243. }
  244. // String returns a string representation of the Reactor.
  245. //
  246. // NOTE: For now, it is just a hard-coded string to avoid accessing unprotected
  247. // shared variables.
  248. //
  249. // TODO: improve!
  250. func (r *Reactor) String() string {
  251. return "ConsensusReactor"
  252. }
  253. // StringIndented returns an indented string representation of the Reactor.
  254. func (r *Reactor) StringIndented(indent string) string {
  255. r.mtx.RLock()
  256. defer r.mtx.RUnlock()
  257. s := "ConsensusReactor{\n"
  258. s += indent + " " + r.state.StringIndented(indent+" ") + "\n"
  259. for _, ps := range r.peers {
  260. s += indent + " " + ps.StringIndented(indent+" ") + "\n"
  261. }
  262. s += indent + "}"
  263. return s
  264. }
  265. // GetPeerState returns PeerState for a given NodeID.
  266. func (r *Reactor) GetPeerState(peerID p2p.NodeID) (*PeerState, bool) {
  267. r.mtx.RLock()
  268. defer r.mtx.RUnlock()
  269. ps, ok := r.peers[peerID]
  270. return ps, ok
  271. }
  272. func (r *Reactor) broadcastNewRoundStepMessage(rs *cstypes.RoundState) {
  273. r.stateCh.Out <- p2p.Envelope{
  274. Broadcast: true,
  275. Message: makeRoundStepMessage(rs),
  276. }
  277. }
  278. func (r *Reactor) broadcastNewValidBlockMessage(rs *cstypes.RoundState) {
  279. psHeader := rs.ProposalBlockParts.Header()
  280. r.stateCh.Out <- p2p.Envelope{
  281. Broadcast: true,
  282. Message: &tmcons.NewValidBlock{
  283. Height: rs.Height,
  284. Round: rs.Round,
  285. BlockPartSetHeader: psHeader.ToProto(),
  286. BlockParts: rs.ProposalBlockParts.BitArray().ToProto(),
  287. IsCommit: rs.Step == cstypes.RoundStepCommit,
  288. },
  289. }
  290. }
  291. func (r *Reactor) broadcastHasVoteMessage(vote *types.Vote) {
  292. r.stateCh.Out <- p2p.Envelope{
  293. Broadcast: true,
  294. Message: &tmcons.HasVote{
  295. Height: vote.Height,
  296. Round: vote.Round,
  297. Type: vote.Type,
  298. Index: vote.ValidatorIndex,
  299. },
  300. }
  301. }
  302. // subscribeToBroadcastEvents subscribes for new round steps and votes using the
  303. // internal pubsub defined in the consensus state to broadcast them to peers
  304. // upon receiving.
  305. func (r *Reactor) subscribeToBroadcastEvents() {
  306. err := r.state.evsw.AddListenerForEvent(
  307. listenerIDConsensus,
  308. types.EventNewRoundStep,
  309. func(data tmevents.EventData) {
  310. r.broadcastNewRoundStepMessage(data.(*cstypes.RoundState))
  311. },
  312. )
  313. if err != nil {
  314. r.Logger.Error("failed to add listener for events", "err", err)
  315. }
  316. err = r.state.evsw.AddListenerForEvent(
  317. listenerIDConsensus,
  318. types.EventValidBlock,
  319. func(data tmevents.EventData) {
  320. r.broadcastNewValidBlockMessage(data.(*cstypes.RoundState))
  321. },
  322. )
  323. if err != nil {
  324. r.Logger.Error("failed to add listener for events", "err", err)
  325. }
  326. err = r.state.evsw.AddListenerForEvent(
  327. listenerIDConsensus,
  328. types.EventVote,
  329. func(data tmevents.EventData) {
  330. r.broadcastHasVoteMessage(data.(*types.Vote))
  331. },
  332. )
  333. if err != nil {
  334. r.Logger.Error("failed to add listener for events", "err", err)
  335. }
  336. }
  337. func (r *Reactor) unsubscribeFromBroadcastEvents() {
  338. r.state.evsw.RemoveListener(listenerIDConsensus)
  339. }
  340. func makeRoundStepMessage(rs *cstypes.RoundState) *tmcons.NewRoundStep {
  341. return &tmcons.NewRoundStep{
  342. Height: rs.Height,
  343. Round: rs.Round,
  344. Step: uint32(rs.Step),
  345. SecondsSinceStartTime: int64(time.Since(rs.StartTime).Seconds()),
  346. LastCommitRound: rs.LastCommit.GetRound(),
  347. }
  348. }
  349. func (r *Reactor) sendNewRoundStepMessage(peerID p2p.NodeID) {
  350. rs := r.state.GetRoundState()
  351. msg := makeRoundStepMessage(rs)
  352. r.stateCh.Out <- p2p.Envelope{
  353. To: peerID,
  354. Message: msg,
  355. }
  356. }
  357. func (r *Reactor) gossipDataForCatchup(rs *cstypes.RoundState, prs *cstypes.PeerRoundState, ps *PeerState) {
  358. logger := r.Logger.With("height", prs.Height).With("peer", ps.peerID)
  359. if index, ok := prs.ProposalBlockParts.Not().PickRandom(); ok {
  360. // ensure that the peer's PartSetHeader is correct
  361. blockMeta := r.state.blockStore.LoadBlockMeta(prs.Height)
  362. if blockMeta == nil {
  363. logger.Error(
  364. "failed to load block meta",
  365. "our_height", rs.Height,
  366. "blockstore_base", r.state.blockStore.Base(),
  367. "blockstore_height", r.state.blockStore.Height(),
  368. )
  369. time.Sleep(r.state.config.PeerGossipSleepDuration)
  370. return
  371. } else if !blockMeta.BlockID.PartSetHeader.Equals(prs.ProposalBlockPartSetHeader) {
  372. logger.Info(
  373. "peer ProposalBlockPartSetHeader mismatch; sleeping",
  374. "block_part_set_header", blockMeta.BlockID.PartSetHeader,
  375. "peer_block_part_set_header", prs.ProposalBlockPartSetHeader,
  376. )
  377. time.Sleep(r.state.config.PeerGossipSleepDuration)
  378. return
  379. }
  380. part := r.state.blockStore.LoadBlockPart(prs.Height, index)
  381. if part == nil {
  382. logger.Error(
  383. "failed to load block part",
  384. "index", index,
  385. "block_part_set_header", blockMeta.BlockID.PartSetHeader,
  386. "peer_block_part_set_header", prs.ProposalBlockPartSetHeader,
  387. )
  388. time.Sleep(r.state.config.PeerGossipSleepDuration)
  389. return
  390. }
  391. partProto, err := part.ToProto()
  392. if err != nil {
  393. logger.Error("failed to convert block part to proto", "err", err)
  394. time.Sleep(r.state.config.PeerGossipSleepDuration)
  395. return
  396. }
  397. logger.Debug("sending block part for catchup", "round", prs.Round, "index", index)
  398. r.dataCh.Out <- p2p.Envelope{
  399. To: ps.peerID,
  400. Message: &tmcons.BlockPart{
  401. Height: prs.Height, // not our height, so it does not matter.
  402. Round: prs.Round, // not our height, so it does not matter
  403. Part: *partProto,
  404. },
  405. }
  406. return
  407. }
  408. time.Sleep(r.state.config.PeerGossipSleepDuration)
  409. }
  410. func (r *Reactor) gossipDataRoutine(ps *PeerState) {
  411. logger := r.Logger.With("peer", ps.peerID)
  412. defer ps.broadcastWG.Done()
  413. OUTER_LOOP:
  414. for {
  415. if !r.IsRunning() {
  416. return
  417. }
  418. select {
  419. case <-ps.closer.Done():
  420. // The peer is marked for removal via a PeerUpdate as the doneCh was
  421. // explicitly closed to signal we should exit.
  422. return
  423. default:
  424. }
  425. rs := r.state.GetRoundState()
  426. prs := ps.GetRoundState()
  427. // Send proposal Block parts?
  428. if rs.ProposalBlockParts.HasHeader(prs.ProposalBlockPartSetHeader) {
  429. if index, ok := rs.ProposalBlockParts.BitArray().Sub(prs.ProposalBlockParts.Copy()).PickRandom(); ok {
  430. part := rs.ProposalBlockParts.GetPart(index)
  431. partProto, err := part.ToProto()
  432. if err != nil {
  433. logger.Error("failed to convert block part to proto", "err", err)
  434. return
  435. }
  436. logger.Debug("sending block part", "height", prs.Height, "round", prs.Round)
  437. r.dataCh.Out <- p2p.Envelope{
  438. To: ps.peerID,
  439. Message: &tmcons.BlockPart{
  440. Height: rs.Height, // this tells peer that this part applies to us
  441. Round: rs.Round, // this tells peer that this part applies to us
  442. Part: *partProto,
  443. },
  444. }
  445. ps.SetHasProposalBlockPart(prs.Height, prs.Round, index)
  446. continue OUTER_LOOP
  447. }
  448. }
  449. // if the peer is on a previous height that we have, help catch up
  450. blockStoreBase := r.state.blockStore.Base()
  451. if blockStoreBase > 0 && 0 < prs.Height && prs.Height < rs.Height && prs.Height >= blockStoreBase {
  452. heightLogger := logger.With("height", prs.Height)
  453. // If we never received the commit message from the peer, the block parts
  454. // will not be initialized.
  455. if prs.ProposalBlockParts == nil {
  456. blockMeta := r.state.blockStore.LoadBlockMeta(prs.Height)
  457. if blockMeta == nil {
  458. heightLogger.Error(
  459. "failed to load block meta",
  460. "blockstoreBase", blockStoreBase,
  461. "blockstoreHeight", r.state.blockStore.Height(),
  462. )
  463. time.Sleep(r.state.config.PeerGossipSleepDuration)
  464. } else {
  465. ps.InitProposalBlockParts(blockMeta.BlockID.PartSetHeader)
  466. }
  467. // Continue the loop since prs is a copy and not effected by this
  468. // initialization.
  469. continue OUTER_LOOP
  470. }
  471. r.gossipDataForCatchup(rs, prs, ps)
  472. continue OUTER_LOOP
  473. }
  474. // if height and round don't match, sleep
  475. if (rs.Height != prs.Height) || (rs.Round != prs.Round) {
  476. time.Sleep(r.state.config.PeerGossipSleepDuration)
  477. continue OUTER_LOOP
  478. }
  479. // By here, height and round match.
  480. // Proposal block parts were already matched and sent if any were wanted.
  481. // (These can match on hash so the round doesn't matter)
  482. // Now consider sending other things, like the Proposal itself.
  483. // Send Proposal && ProposalPOL BitArray?
  484. if rs.Proposal != nil && !prs.Proposal {
  485. // Proposal: share the proposal metadata with peer.
  486. {
  487. propProto := rs.Proposal.ToProto()
  488. logger.Debug("sending proposal", "height", prs.Height, "round", prs.Round)
  489. r.dataCh.Out <- p2p.Envelope{
  490. To: ps.peerID,
  491. Message: &tmcons.Proposal{
  492. Proposal: *propProto,
  493. },
  494. }
  495. // NOTE: A peer might have received a different proposal message, so
  496. // this Proposal msg will be rejected!
  497. ps.SetHasProposal(rs.Proposal)
  498. }
  499. // ProposalPOL: lets peer know which POL votes we have so far. The peer
  500. // must receive ProposalMessage first. Note, rs.Proposal was validated,
  501. // so rs.Proposal.POLRound <= rs.Round, so we definitely have
  502. // rs.Votes.Prevotes(rs.Proposal.POLRound).
  503. if 0 <= rs.Proposal.POLRound {
  504. pPol := rs.Votes.Prevotes(rs.Proposal.POLRound).BitArray()
  505. pPolProto := pPol.ToProto()
  506. logger.Debug("sending POL", "height", prs.Height, "round", prs.Round)
  507. r.dataCh.Out <- p2p.Envelope{
  508. To: ps.peerID,
  509. Message: &tmcons.ProposalPOL{
  510. Height: rs.Height,
  511. ProposalPolRound: rs.Proposal.POLRound,
  512. ProposalPol: *pPolProto,
  513. },
  514. }
  515. }
  516. continue OUTER_LOOP
  517. }
  518. // nothing to do -- sleep
  519. time.Sleep(r.state.config.PeerGossipSleepDuration)
  520. continue OUTER_LOOP
  521. }
  522. }
  523. // pickSendVote picks a vote and sends it to the peer. It will return true if
  524. // there is a vote to send and false otherwise.
  525. func (r *Reactor) pickSendVote(ps *PeerState, votes types.VoteSetReader) bool {
  526. if vote, ok := ps.PickVoteToSend(votes); ok {
  527. r.Logger.Debug("sending vote message", "ps", ps, "vote", vote)
  528. r.voteCh.Out <- p2p.Envelope{
  529. To: ps.peerID,
  530. Message: &tmcons.Vote{
  531. Vote: vote.ToProto(),
  532. },
  533. }
  534. ps.SetHasVote(vote)
  535. return true
  536. }
  537. return false
  538. }
  539. func (r *Reactor) gossipVotesForHeight(rs *cstypes.RoundState, prs *cstypes.PeerRoundState, ps *PeerState) bool {
  540. logger := r.Logger.With("height", prs.Height).With("peer", ps.peerID)
  541. // if there are lastCommits to send...
  542. if prs.Step == cstypes.RoundStepNewHeight {
  543. if r.pickSendVote(ps, rs.LastCommit) {
  544. logger.Debug("picked rs.LastCommit to send")
  545. return true
  546. }
  547. }
  548. // if there are POL prevotes to send...
  549. if prs.Step <= cstypes.RoundStepPropose && prs.Round != -1 && prs.Round <= rs.Round && prs.ProposalPOLRound != -1 {
  550. if polPrevotes := rs.Votes.Prevotes(prs.ProposalPOLRound); polPrevotes != nil {
  551. if r.pickSendVote(ps, polPrevotes) {
  552. logger.Debug("picked rs.Prevotes(prs.ProposalPOLRound) to send", "round", prs.ProposalPOLRound)
  553. return true
  554. }
  555. }
  556. }
  557. // if there are prevotes to send...
  558. if prs.Step <= cstypes.RoundStepPrevoteWait && prs.Round != -1 && prs.Round <= rs.Round {
  559. if r.pickSendVote(ps, rs.Votes.Prevotes(prs.Round)) {
  560. logger.Debug("picked rs.Prevotes(prs.Round) to send", "round", prs.Round)
  561. return true
  562. }
  563. }
  564. // if there are precommits to send...
  565. if prs.Step <= cstypes.RoundStepPrecommitWait && prs.Round != -1 && prs.Round <= rs.Round {
  566. if r.pickSendVote(ps, rs.Votes.Precommits(prs.Round)) {
  567. logger.Debug("picked rs.Precommits(prs.Round) to send", "round", prs.Round)
  568. return true
  569. }
  570. }
  571. // if there are prevotes to send...(which are needed because of validBlock mechanism)
  572. if prs.Round != -1 && prs.Round <= rs.Round {
  573. if r.pickSendVote(ps, rs.Votes.Prevotes(prs.Round)) {
  574. logger.Debug("picked rs.Prevotes(prs.Round) to send", "round", prs.Round)
  575. return true
  576. }
  577. }
  578. // if there are POLPrevotes to send...
  579. if prs.ProposalPOLRound != -1 {
  580. if polPrevotes := rs.Votes.Prevotes(prs.ProposalPOLRound); polPrevotes != nil {
  581. if r.pickSendVote(ps, polPrevotes) {
  582. logger.Debug("picked rs.Prevotes(prs.ProposalPOLRound) to send", "round", prs.ProposalPOLRound)
  583. return true
  584. }
  585. }
  586. }
  587. return false
  588. }
  589. func (r *Reactor) gossipVotesRoutine(ps *PeerState) {
  590. logger := r.Logger.With("peer", ps.peerID)
  591. defer ps.broadcastWG.Done()
  592. // XXX: simple hack to throttle logs upon sleep
  593. logThrottle := 0
  594. OUTER_LOOP:
  595. for {
  596. if !r.IsRunning() {
  597. return
  598. }
  599. select {
  600. case <-ps.closer.Done():
  601. // The peer is marked for removal via a PeerUpdate as the doneCh was
  602. // explicitly closed to signal we should exit.
  603. return
  604. default:
  605. }
  606. rs := r.state.GetRoundState()
  607. prs := ps.GetRoundState()
  608. switch logThrottle {
  609. case 1: // first sleep
  610. logThrottle = 2
  611. case 2: // no more sleep
  612. logThrottle = 0
  613. }
  614. // if height matches, then send LastCommit, Prevotes, and Precommits
  615. if rs.Height == prs.Height {
  616. if r.gossipVotesForHeight(rs, prs, ps) {
  617. continue OUTER_LOOP
  618. }
  619. }
  620. // special catchup logic -- if peer is lagging by height 1, send LastCommit
  621. if prs.Height != 0 && rs.Height == prs.Height+1 {
  622. if r.pickSendVote(ps, rs.LastCommit) {
  623. logger.Debug("picked rs.LastCommit to send", "height", prs.Height)
  624. continue OUTER_LOOP
  625. }
  626. }
  627. // catchup logic -- if peer is lagging by more than 1, send Commit
  628. blockStoreBase := r.state.blockStore.Base()
  629. if blockStoreBase > 0 && prs.Height != 0 && rs.Height >= prs.Height+2 && prs.Height >= blockStoreBase {
  630. // Load the block commit for prs.Height, which contains precommit
  631. // signatures for prs.Height.
  632. if commit := r.state.blockStore.LoadBlockCommit(prs.Height); commit != nil {
  633. if r.pickSendVote(ps, commit) {
  634. logger.Debug("picked Catchup commit to send", "height", prs.Height)
  635. continue OUTER_LOOP
  636. }
  637. }
  638. }
  639. if logThrottle == 0 {
  640. // we sent nothing -- sleep
  641. logThrottle = 1
  642. logger.Debug(
  643. "no votes to send; sleeping",
  644. "rs.Height", rs.Height,
  645. "prs.Height", prs.Height,
  646. "localPV", rs.Votes.Prevotes(rs.Round).BitArray(), "peerPV", prs.Prevotes,
  647. "localPC", rs.Votes.Precommits(rs.Round).BitArray(), "peerPC", prs.Precommits,
  648. )
  649. } else if logThrottle == 2 {
  650. logThrottle = 1
  651. }
  652. time.Sleep(r.state.config.PeerGossipSleepDuration)
  653. continue OUTER_LOOP
  654. }
  655. }
  656. // NOTE: `queryMaj23Routine` has a simple crude design since it only comes
  657. // into play for liveness when there's a signature DDoS attack happening.
  658. func (r *Reactor) queryMaj23Routine(ps *PeerState) {
  659. defer ps.broadcastWG.Done()
  660. OUTER_LOOP:
  661. for {
  662. if !r.IsRunning() {
  663. return
  664. }
  665. select {
  666. case <-ps.closer.Done():
  667. // The peer is marked for removal via a PeerUpdate as the doneCh was
  668. // explicitly closed to signal we should exit.
  669. return
  670. default:
  671. }
  672. // maybe send Height/Round/Prevotes
  673. {
  674. rs := r.state.GetRoundState()
  675. prs := ps.GetRoundState()
  676. if rs.Height == prs.Height {
  677. if maj23, ok := rs.Votes.Prevotes(prs.Round).TwoThirdsMajority(); ok {
  678. r.stateCh.Out <- p2p.Envelope{
  679. To: ps.peerID,
  680. Message: &tmcons.VoteSetMaj23{
  681. Height: prs.Height,
  682. Round: prs.Round,
  683. Type: tmproto.PrevoteType,
  684. BlockID: maj23.ToProto(),
  685. },
  686. }
  687. time.Sleep(r.state.config.PeerQueryMaj23SleepDuration)
  688. }
  689. }
  690. }
  691. // maybe send Height/Round/Precommits
  692. {
  693. rs := r.state.GetRoundState()
  694. prs := ps.GetRoundState()
  695. if rs.Height == prs.Height {
  696. if maj23, ok := rs.Votes.Precommits(prs.Round).TwoThirdsMajority(); ok {
  697. r.stateCh.Out <- p2p.Envelope{
  698. To: ps.peerID,
  699. Message: &tmcons.VoteSetMaj23{
  700. Height: prs.Height,
  701. Round: prs.Round,
  702. Type: tmproto.PrecommitType,
  703. BlockID: maj23.ToProto(),
  704. },
  705. }
  706. time.Sleep(r.state.config.PeerQueryMaj23SleepDuration)
  707. }
  708. }
  709. }
  710. // maybe send Height/Round/ProposalPOL
  711. {
  712. rs := r.state.GetRoundState()
  713. prs := ps.GetRoundState()
  714. if rs.Height == prs.Height && prs.ProposalPOLRound >= 0 {
  715. if maj23, ok := rs.Votes.Prevotes(prs.ProposalPOLRound).TwoThirdsMajority(); ok {
  716. r.stateCh.Out <- p2p.Envelope{
  717. To: ps.peerID,
  718. Message: &tmcons.VoteSetMaj23{
  719. Height: prs.Height,
  720. Round: prs.ProposalPOLRound,
  721. Type: tmproto.PrevoteType,
  722. BlockID: maj23.ToProto(),
  723. },
  724. }
  725. time.Sleep(r.state.config.PeerQueryMaj23SleepDuration)
  726. }
  727. }
  728. }
  729. // Little point sending LastCommitRound/LastCommit, these are fleeting and
  730. // non-blocking.
  731. // maybe send Height/CatchupCommitRound/CatchupCommit
  732. {
  733. prs := ps.GetRoundState()
  734. if prs.CatchupCommitRound != -1 && prs.Height > 0 && prs.Height <= r.state.blockStore.Height() &&
  735. prs.Height >= r.state.blockStore.Base() {
  736. if commit := r.state.LoadCommit(prs.Height); commit != nil {
  737. r.stateCh.Out <- p2p.Envelope{
  738. To: ps.peerID,
  739. Message: &tmcons.VoteSetMaj23{
  740. Height: prs.Height,
  741. Round: commit.Round,
  742. Type: tmproto.PrecommitType,
  743. BlockID: commit.BlockID.ToProto(),
  744. },
  745. }
  746. time.Sleep(r.state.config.PeerQueryMaj23SleepDuration)
  747. }
  748. }
  749. }
  750. time.Sleep(r.state.config.PeerQueryMaj23SleepDuration)
  751. continue OUTER_LOOP
  752. }
  753. }
  754. // processPeerUpdate process a peer update message. For new or reconnected peers,
  755. // we create a peer state if one does not exist for the peer, which should always
  756. // be the case, and we spawn all the relevant goroutine to broadcast messages to
  757. // the peer. During peer removal, we remove the peer for our set of peers and
  758. // signal to all spawned goroutines to gracefully exit in a non-blocking manner.
  759. func (r *Reactor) processPeerUpdate(peerUpdate p2p.PeerUpdate) {
  760. r.Logger.Debug("received peer update", "peer", peerUpdate.NodeID, "status", peerUpdate.Status)
  761. r.mtx.Lock()
  762. defer r.mtx.Unlock()
  763. switch peerUpdate.Status {
  764. case p2p.PeerStatusUp:
  765. // Do not allow starting new broadcasting goroutines after reactor shutdown
  766. // has been initiated. This can happen after we've manually closed all
  767. // peer goroutines and closed r.closeCh, but the router still sends in-flight
  768. // peer updates.
  769. if !r.IsRunning() {
  770. return
  771. }
  772. var (
  773. ps *PeerState
  774. ok bool
  775. )
  776. ps, ok = r.peers[peerUpdate.NodeID]
  777. if !ok {
  778. ps = NewPeerState(r.Logger, peerUpdate.NodeID)
  779. r.peers[peerUpdate.NodeID] = ps
  780. }
  781. if !ps.IsRunning() {
  782. // Set the peer state's closer to signal to all spawned goroutines to exit
  783. // when the peer is removed. We also set the running state to ensure we
  784. // do not spawn multiple instances of the same goroutines and finally we
  785. // set the waitgroup counter so we know when all goroutines have exited.
  786. ps.broadcastWG.Add(3)
  787. ps.SetRunning(true)
  788. // start goroutines for this peer
  789. go r.gossipDataRoutine(ps)
  790. go r.gossipVotesRoutine(ps)
  791. go r.queryMaj23Routine(ps)
  792. // Send our state to the peer. If we're fast-syncing, broadcast a
  793. // RoundStepMessage later upon SwitchToConsensus().
  794. if !r.waitSync {
  795. go r.sendNewRoundStepMessage(ps.peerID)
  796. }
  797. }
  798. case p2p.PeerStatusDown:
  799. ps, ok := r.peers[peerUpdate.NodeID]
  800. if ok && ps.IsRunning() {
  801. // signal to all spawned goroutines for the peer to gracefully exit
  802. ps.closer.Close()
  803. go func() {
  804. // Wait for all spawned broadcast goroutines to exit before marking the
  805. // peer state as no longer running and removal from the peers map.
  806. ps.broadcastWG.Wait()
  807. r.mtx.Lock()
  808. delete(r.peers, peerUpdate.NodeID)
  809. r.mtx.Unlock()
  810. ps.SetRunning(false)
  811. }()
  812. }
  813. }
  814. }
  815. // handleStateMessage handles envelopes sent from peers on the StateChannel.
  816. // An error is returned if the message is unrecognized or if validation fails.
  817. // If we fail to find the peer state for the envelope sender, we perform a no-op
  818. // and return. This can happen when we process the envelope after the peer is
  819. // removed.
  820. func (r *Reactor) handleStateMessage(envelope p2p.Envelope, msgI Message) error {
  821. ps, ok := r.GetPeerState(envelope.From)
  822. if !ok || ps == nil {
  823. r.Logger.Debug("failed to find peer state", "peer", envelope.From, "ch_id", "StateChannel")
  824. return nil
  825. }
  826. switch msg := envelope.Message.(type) {
  827. case *tmcons.NewRoundStep:
  828. r.state.mtx.RLock()
  829. initialHeight := r.state.state.InitialHeight
  830. r.state.mtx.RUnlock()
  831. if err := msgI.(*NewRoundStepMessage).ValidateHeight(initialHeight); err != nil {
  832. r.Logger.Error("peer sent us an invalid msg", "msg", msg, "err", err)
  833. return err
  834. }
  835. ps.ApplyNewRoundStepMessage(msgI.(*NewRoundStepMessage))
  836. case *tmcons.NewValidBlock:
  837. ps.ApplyNewValidBlockMessage(msgI.(*NewValidBlockMessage))
  838. case *tmcons.HasVote:
  839. ps.ApplyHasVoteMessage(msgI.(*HasVoteMessage))
  840. case *tmcons.VoteSetMaj23:
  841. r.state.mtx.RLock()
  842. height, votes := r.state.Height, r.state.Votes
  843. r.state.mtx.RUnlock()
  844. if height != msg.Height {
  845. return nil
  846. }
  847. vsmMsg := msgI.(*VoteSetMaj23Message)
  848. // peer claims to have a maj23 for some BlockID at <H,R,S>
  849. err := votes.SetPeerMaj23(msg.Round, msg.Type, ps.peerID, vsmMsg.BlockID)
  850. if err != nil {
  851. return err
  852. }
  853. // Respond with a VoteSetBitsMessage showing which votes we have and
  854. // consequently shows which we don't have.
  855. var ourVotes *bits.BitArray
  856. switch vsmMsg.Type {
  857. case tmproto.PrevoteType:
  858. ourVotes = votes.Prevotes(msg.Round).BitArrayByBlockID(vsmMsg.BlockID)
  859. case tmproto.PrecommitType:
  860. ourVotes = votes.Precommits(msg.Round).BitArrayByBlockID(vsmMsg.BlockID)
  861. default:
  862. panic("bad VoteSetBitsMessage field type; forgot to add a check in ValidateBasic?")
  863. }
  864. eMsg := &tmcons.VoteSetBits{
  865. Height: msg.Height,
  866. Round: msg.Round,
  867. Type: msg.Type,
  868. BlockID: msg.BlockID,
  869. }
  870. if votesProto := ourVotes.ToProto(); votesProto != nil {
  871. eMsg.Votes = *votesProto
  872. }
  873. r.voteSetBitsCh.Out <- p2p.Envelope{
  874. To: envelope.From,
  875. Message: eMsg,
  876. }
  877. default:
  878. return fmt.Errorf("received unknown message on StateChannel: %T", msg)
  879. }
  880. return nil
  881. }
  882. // handleDataMessage handles envelopes sent from peers on the DataChannel. If we
  883. // fail to find the peer state for the envelope sender, we perform a no-op and
  884. // return. This can happen when we process the envelope after the peer is
  885. // removed.
  886. func (r *Reactor) handleDataMessage(envelope p2p.Envelope, msgI Message) error {
  887. logger := r.Logger.With("peer", envelope.From, "ch_id", "DataChannel")
  888. ps, ok := r.GetPeerState(envelope.From)
  889. if !ok || ps == nil {
  890. r.Logger.Debug("failed to find peer state")
  891. return nil
  892. }
  893. if r.WaitSync() {
  894. logger.Info("ignoring message received during sync", "msg", msgI)
  895. return nil
  896. }
  897. switch msg := envelope.Message.(type) {
  898. case *tmcons.Proposal:
  899. pMsg := msgI.(*ProposalMessage)
  900. ps.SetHasProposal(pMsg.Proposal)
  901. r.state.peerMsgQueue <- msgInfo{pMsg, envelope.From}
  902. case *tmcons.ProposalPOL:
  903. ps.ApplyProposalPOLMessage(msgI.(*ProposalPOLMessage))
  904. case *tmcons.BlockPart:
  905. bpMsg := msgI.(*BlockPartMessage)
  906. ps.SetHasProposalBlockPart(bpMsg.Height, bpMsg.Round, int(bpMsg.Part.Index))
  907. r.Metrics.BlockParts.With("peer_id", string(envelope.From)).Add(1)
  908. r.state.peerMsgQueue <- msgInfo{bpMsg, envelope.From}
  909. default:
  910. return fmt.Errorf("received unknown message on DataChannel: %T", msg)
  911. }
  912. return nil
  913. }
  914. // handleVoteMessage handles envelopes sent from peers on the VoteChannel. If we
  915. // fail to find the peer state for the envelope sender, we perform a no-op and
  916. // return. This can happen when we process the envelope after the peer is
  917. // removed.
  918. func (r *Reactor) handleVoteMessage(envelope p2p.Envelope, msgI Message) error {
  919. logger := r.Logger.With("peer", envelope.From, "ch_id", "VoteChannel")
  920. ps, ok := r.GetPeerState(envelope.From)
  921. if !ok || ps == nil {
  922. r.Logger.Debug("failed to find peer state")
  923. return nil
  924. }
  925. if r.WaitSync() {
  926. logger.Info("ignoring message received during sync", "msg", msgI)
  927. return nil
  928. }
  929. switch msg := envelope.Message.(type) {
  930. case *tmcons.Vote:
  931. r.state.mtx.RLock()
  932. height, valSize, lastCommitSize := r.state.Height, r.state.Validators.Size(), r.state.LastCommit.Size()
  933. r.state.mtx.RUnlock()
  934. vMsg := msgI.(*VoteMessage)
  935. ps.EnsureVoteBitArrays(height, valSize)
  936. ps.EnsureVoteBitArrays(height-1, lastCommitSize)
  937. ps.SetHasVote(vMsg.Vote)
  938. r.state.peerMsgQueue <- msgInfo{vMsg, envelope.From}
  939. default:
  940. return fmt.Errorf("received unknown message on VoteChannel: %T", msg)
  941. }
  942. return nil
  943. }
  944. // handleVoteSetBitsMessage handles envelopes sent from peers on the
  945. // VoteSetBitsChannel. If we fail to find the peer state for the envelope sender,
  946. // we perform a no-op and return. This can happen when we process the envelope
  947. // after the peer is removed.
  948. func (r *Reactor) handleVoteSetBitsMessage(envelope p2p.Envelope, msgI Message) error {
  949. logger := r.Logger.With("peer", envelope.From, "ch_id", "VoteSetBitsChannel")
  950. ps, ok := r.GetPeerState(envelope.From)
  951. if !ok || ps == nil {
  952. r.Logger.Debug("failed to find peer state")
  953. return nil
  954. }
  955. if r.WaitSync() {
  956. logger.Info("ignoring message received during sync", "msg", msgI)
  957. return nil
  958. }
  959. switch msg := envelope.Message.(type) {
  960. case *tmcons.VoteSetBits:
  961. r.state.mtx.RLock()
  962. height, votes := r.state.Height, r.state.Votes
  963. r.state.mtx.RUnlock()
  964. vsbMsg := msgI.(*VoteSetBitsMessage)
  965. if height == msg.Height {
  966. var ourVotes *bits.BitArray
  967. switch msg.Type {
  968. case tmproto.PrevoteType:
  969. ourVotes = votes.Prevotes(msg.Round).BitArrayByBlockID(vsbMsg.BlockID)
  970. case tmproto.PrecommitType:
  971. ourVotes = votes.Precommits(msg.Round).BitArrayByBlockID(vsbMsg.BlockID)
  972. default:
  973. panic("bad VoteSetBitsMessage field type; forgot to add a check in ValidateBasic?")
  974. }
  975. ps.ApplyVoteSetBitsMessage(vsbMsg, ourVotes)
  976. } else {
  977. ps.ApplyVoteSetBitsMessage(vsbMsg, nil)
  978. }
  979. default:
  980. return fmt.Errorf("received unknown message on VoteSetBitsChannel: %T", msg)
  981. }
  982. return nil
  983. }
  984. // handleMessage handles an Envelope sent from a peer on a specific p2p Channel.
  985. // It will handle errors and any possible panics gracefully. A caller can handle
  986. // any error returned by sending a PeerError on the respective channel.
  987. //
  988. // NOTE: We process these messages even when we're fast_syncing. Messages affect
  989. // either a peer state or the consensus state. Peer state updates can happen in
  990. // parallel, but processing of proposals, block parts, and votes are ordered by
  991. // the p2p channel.
  992. //
  993. // NOTE: We block on consensus state for proposals, block parts, and votes.
  994. func (r *Reactor) handleMessage(chID p2p.ChannelID, envelope p2p.Envelope) (err error) {
  995. defer func() {
  996. if e := recover(); e != nil {
  997. err = fmt.Errorf("panic in processing message: %v", e)
  998. r.Logger.Error("recovering from processing message panic", "err", err)
  999. }
  1000. }()
  1001. // We wrap the envelope's message in a Proto wire type so we can convert back
  1002. // the domain type that individual channel message handlers can work with. We
  1003. // do this here once to avoid having to do it for each individual message type.
  1004. // and because a large part of the core business logic depends on these
  1005. // domain types opposed to simply working with the Proto types.
  1006. protoMsg := new(tmcons.Message)
  1007. if err := protoMsg.Wrap(envelope.Message); err != nil {
  1008. return err
  1009. }
  1010. msgI, err := MsgFromProto(protoMsg)
  1011. if err != nil {
  1012. return err
  1013. }
  1014. r.Logger.Debug("received message", "ch_id", chID, "message", msgI, "peer", envelope.From)
  1015. switch chID {
  1016. case StateChannel:
  1017. err = r.handleStateMessage(envelope, msgI)
  1018. case DataChannel:
  1019. err = r.handleDataMessage(envelope, msgI)
  1020. case VoteChannel:
  1021. err = r.handleVoteMessage(envelope, msgI)
  1022. case VoteSetBitsChannel:
  1023. err = r.handleVoteSetBitsMessage(envelope, msgI)
  1024. default:
  1025. err = fmt.Errorf("unknown channel ID (%d) for envelope (%v)", chID, envelope)
  1026. }
  1027. return err
  1028. }
  1029. // processStateCh initiates a blocking process where we listen for and handle
  1030. // envelopes on the StateChannel. Any error encountered during message
  1031. // execution will result in a PeerError being sent on the StateChannel. When
  1032. // the reactor is stopped, we will catch the signal and close the p2p Channel
  1033. // gracefully.
  1034. func (r *Reactor) processStateCh() {
  1035. defer r.stateCh.Close()
  1036. for {
  1037. select {
  1038. case envelope := <-r.stateCh.In:
  1039. if err := r.handleMessage(r.stateCh.ID, envelope); err != nil {
  1040. r.Logger.Error("failed to process message", "ch_id", r.stateCh.ID, "envelope", envelope, "err", err)
  1041. r.stateCh.Error <- p2p.PeerError{
  1042. NodeID: envelope.From,
  1043. Err: err,
  1044. }
  1045. }
  1046. case <-r.stateCloseCh:
  1047. r.Logger.Debug("stopped listening on StateChannel; closing...")
  1048. return
  1049. }
  1050. }
  1051. }
  1052. // processDataCh initiates a blocking process where we listen for and handle
  1053. // envelopes on the DataChannel. Any error encountered during message
  1054. // execution will result in a PeerError being sent on the DataChannel. When
  1055. // the reactor is stopped, we will catch the signal and close the p2p Channel
  1056. // gracefully.
  1057. func (r *Reactor) processDataCh() {
  1058. defer r.dataCh.Close()
  1059. for {
  1060. select {
  1061. case envelope := <-r.dataCh.In:
  1062. if err := r.handleMessage(r.dataCh.ID, envelope); err != nil {
  1063. r.Logger.Error("failed to process message", "ch_id", r.dataCh.ID, "envelope", envelope, "err", err)
  1064. r.dataCh.Error <- p2p.PeerError{
  1065. NodeID: envelope.From,
  1066. Err: err,
  1067. }
  1068. }
  1069. case <-r.closeCh:
  1070. r.Logger.Debug("stopped listening on DataChannel; closing...")
  1071. return
  1072. }
  1073. }
  1074. }
  1075. // processVoteCh initiates a blocking process where we listen for and handle
  1076. // envelopes on the VoteChannel. Any error encountered during message
  1077. // execution will result in a PeerError being sent on the VoteChannel. When
  1078. // the reactor is stopped, we will catch the signal and close the p2p Channel
  1079. // gracefully.
  1080. func (r *Reactor) processVoteCh() {
  1081. defer r.voteCh.Close()
  1082. for {
  1083. select {
  1084. case envelope := <-r.voteCh.In:
  1085. if err := r.handleMessage(r.voteCh.ID, envelope); err != nil {
  1086. r.Logger.Error("failed to process message", "ch_id", r.voteCh.ID, "envelope", envelope, "err", err)
  1087. r.voteCh.Error <- p2p.PeerError{
  1088. NodeID: envelope.From,
  1089. Err: err,
  1090. }
  1091. }
  1092. case <-r.closeCh:
  1093. r.Logger.Debug("stopped listening on VoteChannel; closing...")
  1094. return
  1095. }
  1096. }
  1097. }
  1098. // processVoteCh initiates a blocking process where we listen for and handle
  1099. // envelopes on the VoteSetBitsChannel. Any error encountered during message
  1100. // execution will result in a PeerError being sent on the VoteSetBitsChannel.
  1101. // When the reactor is stopped, we will catch the signal and close the p2p
  1102. // Channel gracefully.
  1103. func (r *Reactor) processVoteSetBitsCh() {
  1104. defer r.voteSetBitsCh.Close()
  1105. for {
  1106. select {
  1107. case envelope := <-r.voteSetBitsCh.In:
  1108. if err := r.handleMessage(r.voteSetBitsCh.ID, envelope); err != nil {
  1109. r.Logger.Error("failed to process message", "ch_id", r.voteSetBitsCh.ID, "envelope", envelope, "err", err)
  1110. r.voteSetBitsCh.Error <- p2p.PeerError{
  1111. NodeID: envelope.From,
  1112. Err: err,
  1113. }
  1114. }
  1115. case <-r.closeCh:
  1116. r.Logger.Debug("stopped listening on VoteSetBitsChannel; closing...")
  1117. return
  1118. }
  1119. }
  1120. }
  1121. // processPeerUpdates initiates a blocking process where we listen for and handle
  1122. // PeerUpdate messages. When the reactor is stopped, we will catch the signal and
  1123. // close the p2p PeerUpdatesCh gracefully.
  1124. func (r *Reactor) processPeerUpdates() {
  1125. defer r.peerUpdates.Close()
  1126. for {
  1127. select {
  1128. case peerUpdate := <-r.peerUpdates.Updates():
  1129. r.processPeerUpdate(peerUpdate)
  1130. case <-r.closeCh:
  1131. r.Logger.Debug("stopped listening on peer updates channel; closing...")
  1132. return
  1133. }
  1134. }
  1135. }
  1136. func (r *Reactor) peerStatsRoutine() {
  1137. for {
  1138. if !r.IsRunning() {
  1139. r.Logger.Info("stopping peerStatsRoutine")
  1140. return
  1141. }
  1142. select {
  1143. case msg := <-r.state.statsMsgQueue:
  1144. ps, ok := r.GetPeerState(msg.PeerID)
  1145. if !ok || ps == nil {
  1146. r.Logger.Debug("attempt to update stats for non-existent peer", "peer", msg.PeerID)
  1147. continue
  1148. }
  1149. switch msg.Msg.(type) {
  1150. case *VoteMessage:
  1151. if numVotes := ps.RecordVote(); numVotes%votesToContributeToBecomeGoodPeer == 0 {
  1152. r.peerUpdates.SendUpdate(p2p.PeerUpdate{
  1153. NodeID: msg.PeerID,
  1154. Status: p2p.PeerStatusGood,
  1155. })
  1156. }
  1157. case *BlockPartMessage:
  1158. if numParts := ps.RecordBlockPart(); numParts%blocksToContributeToBecomeGoodPeer == 0 {
  1159. r.peerUpdates.SendUpdate(p2p.PeerUpdate{
  1160. NodeID: msg.PeerID,
  1161. Status: p2p.PeerStatusGood,
  1162. })
  1163. }
  1164. }
  1165. case <-r.closeCh:
  1166. return
  1167. }
  1168. }
  1169. }