* mempool: add a safety check, write tests for mempoolIDs
and document 65536 limit in the mempool reactor spec
follow-up to https://github.com/tendermint/tendermint/pull/2778
* rename the test
* fixes after Ismail's review
Closes#1798
This is done by making every mempool tx maintain a list of peers who its received the tx from. Instead of using the 20byte peer ID, it instead uses a local map from peerID to uint16 counter, so every peer adds 2 bytes. (Word aligned to probably make it 8 bytes)
This also required resetting the callback function on every CheckTx. This likely has performance ramifications for instruction caching. The actual setting operation isn't costly with the removal of defers in this PR.
* Make the mempool not gossip txs back to peers its received it from
* Fix adversarial memleak
* Don't break interface
* Update changelog
* Forgot to add a mtx
* forgot a mutex
* Update mempool/reactor.go
Co-Authored-By: ValarDragon <ValarDragon@users.noreply.github.com>
* Update mempool/mempool.go
Co-Authored-By: ValarDragon <ValarDragon@users.noreply.github.com>
* Use unknown peer ID
Co-Authored-By: ValarDragon <ValarDragon@users.noreply.github.com>
* fix compilation
* use next wait chan logic when skipping
* Minor fixes
* Add TxInfo
* Add reverse map
* Make activeID's auto-reserve 0
* 0 -> UnknownPeerID
Co-Authored-By: ValarDragon <ValarDragon@users.noreply.github.com>
* Switch to making the normal case set a callback on the reqres object
The recheck case is still done via the global callback, and stats
are also set via global callback
* fix merge conflict
* Addres comments
* Add cache tests
* add cache tests
* minor fixes
* update metrics in reqResCb and reformat code
* goimport -w mempool/reactor.go
* mempool: update memTx senders
I had to introduce txsMap for quick mempoolTx lookups.
* change senders type from []uint16 to sync.Map
Fixes DATA RACE:
```
Read at 0x00c0013fcd3a by goroutine 183:
github.com/tendermint/tendermint/mempool.(*MempoolReactor).broadcastTxRoutine()
/go/src/github.com/tendermint/tendermint/mempool/reactor.go:195 +0x3c7
Previous write at 0x00c0013fcd3a by D[2019-02-27|10:10:49.058] Read PacketMsg switch=3 peer=35bc1e3558c182927b31987eeff3feb3d58a0fc5@127.0.0.1
:46552 conn=MConn{pipe} packet="PacketMsg{30:2B06579D0A143EB78F3D3299DE8213A51D4E11FB05ACE4D6A14F T:1}"
goroutine 190:
github.com/tendermint/tendermint/mempool.(*Mempool).CheckTxWithInfo()
/go/src/github.com/tendermint/tendermint/mempool/mempool.go:387 +0xdc1
github.com/tendermint/tendermint/mempool.(*MempoolReactor).Receive()
/go/src/github.com/tendermint/tendermint/mempool/reactor.go:134 +0xb04
github.com/tendermint/tendermint/p2p.createMConnection.func1()
/go/src/github.com/tendermint/tendermint/p2p/peer.go:374 +0x25b
github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine()
/go/src/github.com/tendermint/tendermint/p2p/conn/connection.go:599 +0xcce
Goroutine 183 (running) created at:
D[2019-02-27|10:10:49.058] Send switch=2 peer=1efafad5443abeea4b7a8155218e4369525d987e@127.0.0.1:46193 channel=48 conn=MConn{pipe} m
sgBytes=2B06579D0A146194480ADAE00C2836ED7125FEE65C1D9DD51049
github.com/tendermint/tendermint/mempool.(*MempoolReactor).AddPeer()
/go/src/github.com/tendermint/tendermint/mempool/reactor.go:105 +0x1b1
github.com/tendermint/tendermint/p2p.(*Switch).startInitPeer()
/go/src/github.com/tendermint/tendermint/p2p/switch.go:683 +0x13b
github.com/tendermint/tendermint/p2p.(*Switch).addPeer()
/go/src/github.com/tendermint/tendermint/p2p/switch.go:650 +0x585
github.com/tendermint/tendermint/p2p.(*Switch).addPeerWithConnection()
/go/src/github.com/tendermint/tendermint/p2p/test_util.go:145 +0x939
github.com/tendermint/tendermint/p2p.Connect2Switches.func2()
/go/src/github.com/tendermint/tendermint/p2p/test_util.go:109 +0x50
I[2019-02-27|10:10:49.058] Added good transaction validator=0 tx=43B4D1F0F03460BD262835C4AA560DB860CFBBE85BD02386D83DAC38C67B3AD7 res="&{CheckTx:gas_w
anted:1 }" height=0 total=375
Goroutine 190 (running) created at:
github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart()
/go/src/github.com/tendermint/tendermint/p2p/conn/connection.go:210 +0x313
github.com/tendermint/tendermint/libs/common.(*BaseService).Start()
/go/src/github.com/tendermint/tendermint/libs/common/service.go:139 +0x4df
github.com/tendermint/tendermint/p2p.(*peer).OnStart()
/go/src/github.com/tendermint/tendermint/p2p/peer.go:179 +0x56
github.com/tendermint/tendermint/libs/common.(*BaseService).Start()
/go/src/github.com/tendermint/tendermint/libs/common/service.go:139 +0x4df
github.com/tendermint/tendermint/p2p.(*peer).Start()
<autogenerated>:1 +0x43
github.com/tendermint/tendermint/p2p.(*Switch).startInitPeer()
```
* explain the choice of a map DS for senders
* extract ids pool/mapper to a separate struct
* fix literal copies lock value from senders: sync.Map contains sync.Mutex
* use sync.Map#LoadOrStore instead of Load
* fixes after Ismail's review
* rename resCbNormal to resCbFirstTime
* Update proposer-selection.md
* Fixed typos
* fixed typos
* Attempt to address some comments
* Update proposer-selection.md
* Update proposer-selection.md
* Update proposer-selection.md
Added the normalization step.
* Addressed review comments
* New example for normalization section
Added a new example to better show the need for normalization
Added requirement for changing validator set
Addressed review comments
* Fixed problem with R2
* fixed the math for new validator
* test
* more small updates
* Moved the centering above the round-robin election
- the centering is now done before the actual round-robin block
- updated examples
- cleanup
* change to reflect new implementation for new validator
Also
- init substructures to avoid panic in pb2tm.ConsensusParams
Before: if csp.Block is nil and we later try to access/write to it,
we'll panic.
After: if csp.Block is nil and we later try to access/write to it,
there'll be no panic.
https://tools.ietf.org/html/rfc6962#section-2.1
"The largest power of two less than the number of items" is actually correct!
For n > 1, let k be the largest power of two smaller than n (i.e.,
k < n <= 2k).
* make BlockTimeIota a consensus parameter, not a locally configurable option
Refs #2920
* make TimeIota int64 ms
Refs #2920
* update Gopkg.toml
* fixes after Ethan's review
* fix TestRemoteSignerProposalSigningFailed
* update changelog
* docs: explain create_empty_blocks configurations
Closes#3307
* Vagrantfile: install nodejs for docs
* update docs instructions
npm install does not make sense since there's no packages.json file
* explain broadcast_tx_* tx format
Closes#536
* docs: explain how transaction ordering works
Closes#2904
* bring in consensus parameters explained
* example for create_empty_blocks_interval
* bring in explanation from https://github.com/tendermint/tendermint/issues/2487#issuecomment-424899799
* link to formatting instead of duplicating info
* Begin simple merkle compatibility PR
* Fix query_test
* Use trillian test vectors
* Change the split point per RFC 6962
* update spec
* refactor innerhash to match spec
* Update changelog
* Address @liamsi's comments
* Write the comment requested by @liamsi
* Consistent order fields of Timestamp/BlockID fields in CanonicalVote and
CanonicalProposal
* update spec too
* Introduce and use IsZero & IsComplete:
- update IsZero method according to spec and introduce IsComplete
- use methods in validate basic to validate: proposals come with a
"complete" blockId and votes are either complete or empty
- update spec: BlockID.IsNil() -> BlockID.IsZero() and fix typo
* BlockID comes first
* fix tests
* Enforce validators can only use the correct pubkey type
* adapt to variable renames
* Address comments from #2636
* separate updating and validation logic
* update spec
* Add test case for TestStringSliceEqual, clarify slice copying code
* Address @ebuchman's comments
* Split up testing validator update execution, and its validation
* Vagrantfile: install dev_tools
Follow-up on https://github.com/tendermint/tendermint/pull/2824
* update consensus params spec
* fix test name
* rpc_test: panic if failed to start listener
also
- remove http_server#MustListen
- align StartHTTPServer and StartHTTPAndTLSServer functions
* dep: allow minor releases for grpc
* dedup with spec/abci/client-server
* fixup abci/readme
* link to getting started in abci/README
* https
* spec/abci: some deduplication
* docs: remove extraneous comment
* Introduce EventValidBlock for informing peer about wanted block
* Merge with develop
* Add isCommit flag to NewValidBlock message
- Add test for the case of +2/3 Precommit from the previous round
* p2p: add protocol Version to NodeInfo
* update node pkg. remove extraneous version files
* update changelog and docs
* fix test
* p2p: Version -> ProtocolVersion; more ValidateBasic and tests
* types: add Version to Header
* abci: add Version to Header
* state: add Version to State
* node: check software and state protocol versions match
* update changelog
* docs/spec: update for versions
* state: more tests
* remove TODOs
* remove empty test
* WIP: switching to fixed offsets for SignBytes
* add version field to sign bytes and update order
* more comments on test-cases and add a tc with a chainID
* remove amino:"write_empty" tag
- it doesn't affect if default fixed size fields ((u)int64) are
written or not
- add comment about int->int64 casting
* update CHANGELOG_PENDING
* update documentation
* add back link to issue #1622 in documentation
* remove JSON tags and add (failing test-case)
* fix failing test
* update test-vectors due to added `Type` field
* change Type field from string to byte and add new type alias
- SignedMsgType replaces VoteTypePrevote, VoteTypePrecommit and adds new
ProposalType to separate votes from proposal when signed
- update test-vectors
* fix remains from rebasing
* use SignMessageType instead of byte everywhere
* fixes from review
* require block.Time of the fist block to be genesis time
Refs #2587:
```
We only start validating block.Time when Height > 1, because there is no
commit to compute the median timestamp from for the first block. This
means a faulty proposer could make the first block with whatever time
they want.
Instead, we should require the timestamp of block 1 to match the genesis
time.
I discovered this while refactoring the ValidateBlock tests to be
table-driven while working on tests for #2560.
```
* do not accept blocks with negative height
* update changelog and spec
* nanos precision for test genesis time
* Fix failing test (#2607)
* switch to amino for SignBytes and add Reply with error message
- currently only Vote is done
* switch Reply type in socket for other messages
- add error description on error
* add TODOs regarding error handling
* address comments from peer review session (thx @xla)
- contains all changes besides the test-coverage / error'ing branches
* increase test coverage:
- add tests for each newly introduced error'ing code path
* return error if received wrong response
* add test for wrong response branches (ErrUnexpectedResponse)
* update CHANGELOG_PENDING and related documentation (spec)
* fix typo: s/CanonicallockID/CanonicalBlockID
* fixes from review
* docs: link consensus to blockchain spec. closes#2422
* docs: deprecate research section. closes#2401
* docs: fix some links
* docs: fix some markdown lists
* docs: fix more links
Anton Kaliaev
Dev Ojha
Ethan Buchman
Anca Zamfir
YOSHIDA Masanori
Ismail Khoffi
srmo
Zach
Jernej Kos
Zarko Milosevic
Joon