* WIP: Starts adding remote signer test harness
This commit adds a new command to Tendermint to allow for us to build a
standalone binary to test remote signers such as KMS
(https://github.com/tendermint/kms).
Right now, all it does is test that the local public key matches the
public key reported by the client, and fails at the point where it
attempts to get the client to sign a proposal.
* Fixes typo
* Fixes proposal validation test
This commit fixes the proposal validation test as per #3149. It also
moves the test harness into its own internal package to isolate its
exports from the `privval` package.
* Adds vote signing validation
* Applying recommendations from #3149
* Adds function descriptions for test harness
* Adds ability to ask remote signer to shut down
Prior to this commit, the remote signer needs to manually be shut down,
which is not ideal for automated testing. This commit allows us to send
a poison pill message to the KMS to let it shut down gracefully once
testing is done (whether the tests pass or fail).
* Adds tests for remote signer test harness
This commit makes some minor modifications to a few files to allow for
testing of the remote signer test harness. Two tests are added here:
checking for a fully successful (the ideal) case, and for the case where
the maximum number of retries has been reached when attempting to accept
incoming connections from the remote signer.
* Condenses serialization of proposals and votes using existing Tendermint functions
* Removes now-unnecessary amino import and codec
* Adds error message for vote signing failure
* Adds key extraction command for integration test
Took the code from here:
https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to
create a simple utility command to extract a key from a local Tendermint
validator for use in KMS integration testing.
* Makes path expansion success non-compulsory
* Fixes segfault on SIGTERM
We need an additional variable to keep track of whether we're
successfully connected, otherwise hitting Ctrl+Break during execution
causes a segmentation fault. This now allows for a clean shutdown.
* Consolidates shutdown checks
* Adds comments indicating codes for easy lookup
* Adds Docker build for remote signer harness
Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to
override the image name and Dockerfile using environment variables.
Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow
for building the `remote_val_harness` Docker image.
* Adds build_remote_val_harness_docker_image to .PHONY
* Removes remote signer poison pill messaging functionality
* Reduces fluff code in command line parsing
As per
https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788,
this reduces the amount of fluff code in the PR down to the bare
minimum.
* Fixes ordering of error check and info log
* Moves remove_val_harness cmd into tools folder
It seems to make sense to rather keep the remote signer test harness in
its own tool folder (now rather named `tm-signer-harness` to keep with
the tool naming convention). It is actually a separate tool, not meant
to be one of the core binaries, but supplementary and supportive.
* Updates documentation for tm-signer-harness
* Refactors flag parsing to be more compact and less redundant
* Adds version sub-command help
* Removes extraneous flags parsing
* Adds CHANGELOG_PENDING entry for tm-signer-harness
* Improves test coverage
Adds a few extra parameters to the `MockPV` type to fake broken vote and
proposal signing. Also adds some more tests for the test harness so as
to increase coverage for failed cases.
* Fixes formatting for CHANGELOG_PENDING.md
* Fix formatting for documentation config
* Point users towards official Tendermint docs for tools documentation
* Point users towards official Tendermint docs for tm-signer-harness
* Remove extraneous constant
* Rename TestHarness.sc to TestHarness.spv for naming consistency
* Refactor to remove redundant goroutine
* Refactor conditional to cleaner switch statement and better error handling for listener protocol
* Remove extraneous goroutine
* Add note about installing tmkms via Cargo
* Fix typo in naming of output signing key
* Add note about where to find chain ID
* Replace /home/user with ~/ for brevity
* Fixes "signer.key" typo
* Minor edits for clarification for tm-signer-harness bulid/setup process
* types: memoize height/round in commit instead of first vote
* types: commit.ValidateBasic in VerifyCommit
* types: new CommitSig alias for Vote
In preparation for reducing the redundancy in Commits, we introduce the
CommitSig as an alias for Vote. This is non-breaking on the protocol,
and minor breaking on the Go API, as Commit now contains a list of
CommitSig instead of Vote.
* remove dependence on ToVote
* update some comments
* fix tests
* fix tests
* fixes from review
* base verifier: bc->bv and check chainid
* improve some comments
* comments in dynamic verifier
* fix comment in doc about BaseVerifier
It requires the validator set to perfectly match.
* failing test for #2862
* move errTooMuchChange to types. fixes#2862
* changelog, comments
* ic -> dv
* update comment, link to issue
* more proposer priority tests
- test that we don't reset to zero when updating / adding
- test that same power validators alternate
* add another test to track / simulate similar behaviour as in #2960
* address some of Chris' review comments
* address some more of Chris' review comments
* temporarily pushing branch with the following changes:
The total power might change if:
- a validator is added
- a validator is removed
- a validator is updated
Decrement the accums (of all validators) directly after any of these events
(by the inverse of the change)
* Fix 2960 by re-normalizing / scaling priorities to be in bounds of total
power, additionally:
- remove heap where it doesn't make sense
- avg. only at the end of IncrementProposerPriority instead of each
iteration
- update (and slightly improve)
TestAveragingInIncrementProposerPriorityWithVotingPower to reflect
above changes
* Fix 2960 by re-normalizing / scaling priorities to be in bounds of total
power, additionally:
- remove heap where it doesn't make sense
- avg. only at the end of IncrementProposerPriority instead of each
iteration
- update (and slightly improve)
TestAveragingInIncrementProposerPriorityWithVotingPower to reflect
above changes
* fix tests
* add comment
* update changelog pending & some minor changes
* comment about division will floor the result & fix typo
* Update TestLargeGenesisValidator:
- remove TODO and increase large genesis validator's voting power
accordingly
* move changelog entry to P2P Protocol
* Ceil instead of flooring when dividing & update test
* quickly fix failing TestProposerPriorityDoesNotGetResetToZero:
- divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower)
* fix typo: rename getValWitMostPriority -> getValWithMostPriority
* test proposer frequencies
* return absolute value for diff. keep testing
* use for loop for div
* cleanup, more tests
* spellcheck
* get rid of using floats: manually ceil where necessary
* Remove float, simplify, fix tests to match chris's proof (#3157)
* consensus: createProposalBlock function
* blockExecutor.CreateProposalBlock
- factored out of consensus pkg into a method on blockExec
- new private interfaces for mempool ("txNotifier") and evpool with one function each
- consensus tests still require more mempool methods
* failing test for CreateProposalBlock
* Fix bug in include evidece into block
* evidence: change maxBytes to maxSize
* MaxEvidencePerBlock
- changed to return both the max number and the max bytes
- preparation for #2590
* changelog
* fix linter
* Fix from review
Co-Authored-By: ebuchman <ethan@coinculture.info>
* Begin simple merkle compatibility PR
* Fix query_test
* Use trillian test vectors
* Change the split point per RFC 6962
* update spec
* refactor innerhash to match spec
* Update changelog
* Address @liamsi's comments
* Write the comment requested by @liamsi
* Consistent order fields of Timestamp/BlockID fields in CanonicalVote and
CanonicalProposal
* update spec too
* Introduce and use IsZero & IsComplete:
- update IsZero method according to spec and introduce IsComplete
- use methods in validate basic to validate: proposals come with a
"complete" blockId and votes are either complete or empty
- update spec: BlockID.IsNil() -> BlockID.IsZero() and fix typo
* BlockID comes first
* fix tests
* Don't use pointer receivers for PubKeyMultisigThreshold
* test that showcases panic when PubKeyMultisigThreshold are used in sdk:
- deserialization will fail in `readInfo` which tries to read a
`crypto.PubKey` into a `localInfo` (called by
cosmos-sdk/client/keys.GetKeyInfo)
* Update changelog
* Rename routeTable to nameTable, multisig key is no longer a pointer
* sed -i 's/PubKeyAminoRoute/PubKeyAminoName/g' `grep -lrw PubKeyAminoRoute .`
upon Jae's request
* AminoRoutes -> AminoNames
* sed -e 's/PrivKeyAminoRoute/PrivKeyAminoName/g'
* Update crypto/encoding/amino/amino.go
Co-Authored-By: alessio <quadrispro@ubuntu.com>
* WIP: tests for #2785
* rebase onto develop
* add Bucky's test without changing ValidatorSet.Update
* make TestValidatorSetBasic fail
* add ProposerPriority preserving fix to ValidatorSet.Update to fix
TestValidatorSetBasic
* fix randValidator_ to stay in bounds of MaxTotalVotingPower
* check for expected proposer and remove some duplicate code
* actually limit the voting power of random validator ...
* fix test
* types: ValidatorSet.Update preserves ProposerPriority
This solves the other issue discovered as part of #2718,
where Accum (now called ProposerPriority) is reset to
0 every time a validator is updated.
* update changelog
* add test
* update comment
* Update types/validator_set_test.go
Co-Authored-By: ebuchman <ethan@coinculture.info>
* set the accum of a new validator to (-total voting power):
- disincentivize validators to unbond, then rebon to reset their
negative Accum to zero
additional unrelated changes:
- do not capitalize error msgs
- fix typo
* review comments: (re)capitalize errors & delete obsolete comments
* More changes suggested by @melekes
* WIP: do not batch clip (#2809)
* substract avgAccum on each iteration
- temporarily skip test
* remove unused method safeMulClip / safeMul
* always substract the avg accum
- temp. skip another test
* remove overflow / underflow tests & add tests for avgAccum:
- add test for computeAvgAccum
- as we substract the avgAccum now we will not trivially over/underflow
* address @cwgoes' comments
* shift by avg at the end of IncrementAccum
* Add comment to MaxTotalVotingPower
* Guard inputs to not exceed MaxTotalVotingPower
* Address review comments:
- do not fetch current validator from set again
- update error message
* Address a few review comments:
- fix typo
- extract variable
* address more review comments:
- clarify 1.125*totalVotingPower == totalVotingPower + (totalVotingPower >> 3)
* review comments: panic instead of "clipping":
- total voting power is guarded to not exceed MaxTotalVotingPower ->
panic if this invariant is violated
* fix failing test
* Enforce validators can only use the correct pubkey type
* adapt to variable renames
* Address comments from #2636
* separate updating and validation logic
* update spec
* Add test case for TestStringSliceEqual, clarify slice copying code
* Address @ebuchman's comments
* Split up testing validator update execution, and its validation
* Vagrantfile: install dev_tools
Follow-up on https://github.com/tendermint/tendermint/pull/2824
* update consensus params spec
* fix test name
* rpc_test: panic if failed to start listener
also
- remove http_server#MustListen
- align StartHTTPServer and StartHTTPAndTLSServer functions
* dep: allow minor releases for grpc
* add proposer info to EventCompleteProposal
* create separate EventData structure for CompleteProposal
* cant us rs.Proposal to get BlockID because it is not guaranteed to be set yet
* copying RoundState isnt helping us here
* add Step back to make compatible with original RoundState event. update changelog
* add NewRound event
* fix test
* remove unneeded RoundState
* put height round step into a struct
* pull out ValidatorInfo struct. add ensureProposal assert
* remove height-round-state sub-struct refactor
* minor fixes from review
* fix amino overhead computation for Tx:
- also count the fieldnum / typ3
- add method to compute overhead per Tx
- slightly clarify comment on MaxAminoOverheadForBlock
- add tests
* fix TestReapMaxBytesMaxGas according to amino overhead
* fix TestMempoolFilters according to amino overhead
* address review comments:
- add a note about fieldNum = 1
- add forgotten godoc comment
* fix and use sm.TxPreCheck
* fix test
* remove print statement
* validate reactor messages
Refs #2683
* validate blockchain messages
Refs #2683
* validate evidence messages
Refs #2683
* todo
* check ProposalPOL and signature sizes
* add a changelog entry
* check addr is valid when we add it to the addrbook
* validate incoming netAddr (not just nil check!)
* fixes after Bucky's review
* check timestamps
* beef up block#ValidateBasic
* move some checks into bcBlockResponseMessage
* update Gopkg.lock
Fix
```
grouped write of manifest, lock and vendor: failed to export github.com/tendermint/go-amino: fatal: failed to unpack tree object 6dcc6ddc14
```
by running `dep ensure -update`
* bump year since now we check it
* generate test/p2p/data on the fly using tendermint testnet
* allow sync chains older than 1 year
* use full path when creating a testnet
* move testnet gen to test/docker/Dockerfile
* relax LastCommitRound check
Refs #2737
* fix conflicts after merge
* add small comment
* some ValidateBasic updates
* fixes
* AppHash length is not fixed
* Introduce EventValidBlock for informing peer about wanted block
* Merge with develop
* Add isCommit flag to NewValidBlock message
- Add test for the case of +2/3 Precommit from the previous round
* types: add Version to Header
* abci: add Version to Header
* state: add Version to State
* node: check software and state protocol versions match
* update changelog
* docs/spec: update for versions
* state: more tests
* remove TODOs
* remove empty test
Thane Thomson
Ethan Buchman
Anton Kaliaev
Ismail Khoffi
Dev Ojha
Zarko Milosevic
Alessio Treglia
Zach
srmo
nagarajmanjunath
Jernej Kos
JamesRay
Jae Kwon
kevlubkcm
Catalin Pirvu
Zarko Milosevic