* prometheus/metrics: two new metrics for consensus
- add consensus_validator_power metric so if a node is a validator it can see its own power in prometheus
- add last_signed_height metric so if a node is a validator it can be aware of at which height the most recent time it signed.
- closes: #3773
- closes: #3083
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* check if signature is present
* minor change and check if sig is not absent
* add changelog entry
* Update consensus/state.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update CHANGELOG_PENDING.md
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update CHANGELOG_PENDING.md
* add label with validator address
* change address to vali_address
* add metric missed blocks
* add changelog entry for missed blocks metric
* address naming & add docs
also
- do not send any LastCommitInfo.Votes for 1st block.
- use cs.LastValidators (not cs.Validators) in recordMetrics when counting missing validators
note:
remember that the first LastCommit is intentionally empty, so it makes
sense for LastCommitInfo.Votes to also be empty. Because we can't really
tell if validator signed or not. Similar for ^ metrics, we can't say if
particular validator was missing or not.
Closes#4192
* lint: golint issue fixes
- on my local machine golint is a lot stricter than the bot so slowly going through and fixing things.
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* more fixes from golint
* remove isPeerPersistentFn
* add changelog entry
## Issue
Implement a new subcommand: tendermint debug. This subcommand itself has two subcommands:
$ tendermint debug kill <pid> </path/to/out.zip> --home=</path/to/app.d>
Writes debug info into a compressed archive. The archive will contain the following:
├── config.toml
├── consensus_state.json
├── net_info.json
├── stacktrace.out
├── status.json
└── wal
The Tendermint process will be killed.
$ tendermint debug dump </path/to/out> --home=</path/to/app.d>
This command will perform similar to kill except it only polls the node and dumps debugging data every frequency seconds to a compressed archive under a given destination directory. Each archive will contain:
├── consensus_state.json
├── goroutine.out
├── heap.out
├── net_info.json
├── status.json
└── wal
Note:
goroutine.out and heap.out will only be written if a profile address is provided and is operational.
This command is blocking and will log any error.
replaces: #3327closes: #3249
## Commits:
* Implement debug tool command stubs
* Implement net getters and zip logic
* Update zip dir API and add home flag
* Add simple godocs for kill aux functions
* Move IO util to new file and implement copy WAL func
* Implement copy config function
* Implement killProc
* Remove debug fmt
* Validate output file input
* Direct STDERR to file
* Godoc updates
* Sleep prior to killing tail proc
* Minor cleanup of godocs
* Move debug command and add make target
* Rename command handler function
* Add example to command long descr
* Move current implementation to cmd/tendermint/commands/debug
* Update kill cmd long description
* Implement dump command
* Add pending log entry
* Add gosec nolint
* Add error check for Mkdir
* Add os.IsNotExist(err)
* Add to debugging section in running-in-prod doc
* libs/common: Refactor libs/common 5
- move mathematical functions and types out of `libs/common` to math pkg
- move net functions out of `libs/common` to net pkg
- move string functions out of `libs/common` to strings pkg
- move async functions out of `libs/common` to async pkg
- move bit functions out of `libs/common` to bits pkg
- move cmap functions out of `libs/common` to cmap pkg
- move os functions out of `libs/common` to os pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix testing issues
* fix tests
closes#41417
woooooooooooooooooo kill the cmn pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* fix goimport issues
* run gofmt
Closes#4202
- nobody's supporting them
- users who want to try TM should use pre-build binaries
- devs should be able to install using git clone (or use Vagrant)
- validators don't use them (my guess)
* libs/common: Refactor libs/common 4
- move byte function out of cmn to its own pkg
- move tempfile out of cmn to its own pkg
- move throttletimer to its own pkg
ref #4147
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* fix linting issues
Fixes#4014
The reason being when you call any endpoint supporting optional height
and use `height=0`, it will return an error. For example:
```
$ curl localhost:2667/consensus_params?height=0
{
"jsonrpc": "2.0",
"id": -1,
"error": {
"code": -32603,
"message": "Internal error",
"data": "height must be greater than 0"
}
}
```
* libs/common: refactor libs common 3
- move nil.go into types folder and make private
- move service & baseservice out of common into service pkg
ref #4147
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* libs/common: refactor libs/common 2
- move random function to there own pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* change imports and usage throughout repo
* fix goimports
* add changelog entry
* libs/common: Refactor libs/common 01
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* regenerate proto files, move intslice to where its used
* update kv.KVPair(s) to kv.Pair(s)
* add changelog entry
* make intInSlice private
I put it as the awesome repo ecosytem section as this is currently the only place where there is tendermint only ecosystem information.
- closes#4198
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* Rename Tag(s) to Event(s)
- tag was replaced with event, but in some places it still mentions tag, would be easier to understand if we tried to replace it with event to not confuse people.
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* more changes from tag -> event
* rename events to compositeKeys and keys
* Apply suggestions from code review
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* add minor documentation on how composite keys are constructed
* rename eventkey to compositekey
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* add changelog entry & add info to regenerate confid to changelog entry
To execute the integration.sh, you need to run "source ./integration.sh". just execute command "./integration.sh" will not work.
It is because "source ~/.profile" executes in a separate shell than the integration.sh script
# **turn on the go module, default is auto. The value is off, if tendermint source code
#is downloaded under $GOPATH/src directory
echo "export GO111MODULE=on" >> ~/.profile
# ** use git clone instead of go get.
# once go module is on, go get will download source code to
# specific version directory under $GOPATH/pkg/mod the make
# script will not work
git clone https://github.com/tendermint/tendermint.git
#** need to install the package, otherwise tendermint testnet will not execute
make install
implementation spec of Improved Trusted Peering ADR-050 by B-Harvest
- add unconditional_peer_ids and persistent_peers_max_dial_period to config
- add unconditionalPeerIDs map to Switch struct
default config value of persistent_peers_max_dial_period is 0s(disabled)
Refs #4072, #4053
* Add processor prototype
* Change processor API
+ expose a simple `handle` function which mutates internal state
* schedule event handling
* rename schedule -> scheduler
* fill in handle function
* processor tests
* fix gofmt and ohter golangci issues
* scopelint var on range scope
* add check for short block received
* small test reorg
* ci fix changes
* go.mod revert
* some cleanup and review comments
* scheduler fixes and unit tests, also small processor changes.
changed scPeerPruned to include a list of pruned peers
touchPeer to check peer state and remove the blocks from blockStates if the peer removal causes the max peer height to be lower.
remove the block at sc.initHeight
changed peersInactiveSince, peersSlowerThan, getPeersAtHeight check peer state
prunablePeers to return a sorted list of peers
lastRate in markReceived() attempted to divide by 0, temp fix.
fixed allBlocksProcessed conditions
maxHeight() and minHeight() to return sc.initHeight if no ready peers present
make selectPeer() deterministic.
added handleBlockProcessError()
added termination cond. (sc.allBlocksProcessed()) to handleTryPrunePeer() and others.
changed pcBlockVerificationFailure to include peer of H+2 block along with the one for H+1
changed the processor to call purgePeer on block verification failure.
fixed processor tests
added scheduler tests.
* typo and ci fixes
* remove height from scBlockRequest, golangci fixes
* limit on blockState map, updated tests
* remove unused
* separate test for maxHeight(), used for sched. validation
* use Math.Min
* fix golangci
* Document the semantics of blockStates in the scheduler
* better docs
* distinguish between unknown and invalid blockstate
* Standardize peer filtering methods
* feedback
* s/getPeersAtHeight/getPeersAtHeightOrAbove
* small notes
* Update blockchain/v2/scheduler.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update comments based on feedback
* Add enum offset
* panic on nil block in processor
* remove unused max height calculation
* format shorter line
Fix for #4164
The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed.
Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower.
Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied.
Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update.
## Commits:
* tests for overflow detection and prevention
* test fix
* more tests
* fix the false overflow warnings and golint
* scopelint warning fix
* review comments
* variant with using sort by amount of change in power
* compute separately number new validators in update
* types: use a switch in processChanges
* more review comments
* types: use HasAddress in numNewValidators
* types: refactor verifyUpdates
copy updates, sort them by delta and use resulting slice to calculate
tvpAfterUpdatesBeforeRemovals.
* remove unused structs
* review comments
* update changelog
* p2p/conn: simplify secret connection handshake malleability fix with merlin
Introduces new dependencies on github.com/gtank/merlin and sha3 as a cryptographic primitive
This also only uses the transcript hash as a MAC.
* p2p/conn: avoid string to byte conversion
https://github.com/uber-go/guide/blob/master/style.md#avoid-string-to-byte-conversion
* types: change `Commit` to consist of just signatures
These are final changes towards removing votes from commit and leaving
only signatures (see ADR-25)
Fixes#1648
* bring back TestCommitToVoteSetWithVotesForAnotherBlockOrNilBlock
+ add absent flag to Vote to indicate that it's for another block
* encode nil votes as CommitSig with BlockIDFlagAbsent
+ make Commit#Precommits array of non-pointers
because precommit will never be nil
* add NewCommitSigAbsent and Absent() funcs
* uncomment validation in CommitSig#ValidateBasic
* add comments to ValidatorSet funcs
* add a changelog entry
* break instead of continue
continue does not make sense in these cases
* types: rename Commit#Precommits to Signatures
* swagger: fix /commit response
* swagger: change block_id_flag type
* fix merge conflicts
Refs #1771
ADR: https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-044-lite-client-with-weak-subjectivity.md
## Commits:
* add Verifier and VerifyCommitTrusting
* add two more checks
make trustLevel an option
* float32 for trustLevel
* check newHeader time
* started writing lite Client
* unify Verify methods
* ensure h2.Header.bfttime < h1.Header.bfttime + tp
* move trust checks into Verify function
* add more comments
* more docs
* started writing tests
* unbonding period failures
* tests are green
* export ErrNewHeaderTooFarIntoFuture
* make golangci happy
* test for non-adjusted headers
* more precision
* providers and stores
* VerifyHeader and VerifyHeaderAtHeight funcs
* fix compile errors
* remove lastVerifiedHeight, persist new trusted header
* sequential verification
* remove TrustedStore option
* started writing tests for light client
* cover basic cases for linear verification
* bisection tests PASS
* rename BisectingVerification to SkippingVerification
* refactor the code
* add TrustedHeader method
* consolidate sequential verification tests
* consolidate skipping verification tests
* rename trustedVals to trustedNextVals
* start writing docs
* ValidateTrustLevel func and ErrOldHeaderExpired error
* AutoClient and example tests
* fix errors
* update doc
* remove ErrNewHeaderTooFarIntoFuture
This check is unnecessary given existing a) ErrOldHeaderExpired b)
h2.Time > now checks.
* return an error if we're at more recent height
* add comments
* add LastSignedHeaderHeight method to Store
I think it's fine if Store tracks last height
* copy over proxy from old lite package
* make TrustedHeader return latest if height=0
* modify LastSignedHeaderHeight to return an error if no headers exist
* copy over proxy impl
* refactor proxy and start http lite client
* Tx and BlockchainInfo methods
* Block method
* commit method
* code compiles again
* lite client compiles
* extract updateLiteClientIfNeededTo func
* move final parts
* add placeholder for tests
* force usage of lite http client in proxy
* comment out query tests for now
* explicitly mention tp: trusting period
* verify nextVals in VerifyHeader
* refactor bisection
* move the NextValidatorsHash check into updateTrustedHeaderAndVals
+ update the comment
* add ConsensusParams method to RPC client
* add ConsensusParams to rpc/mock/client
* change trustLevel type to a new cmn.Fraction type
+ update SkippingVerification comment
* stress out trustLevel is only used for non-adjusted headers
* fixes after Fede's review
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
* compare newHeader with a header from an alternative provider
* save pivot header
Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349122824
* check header can still be trusted in TrustedHeader
Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349101424
* lite: update Validators and Block endpoints
- Block no longer contains BlockMeta
- Validators now accept two additional params: page and perPage
* make linter happy
* docs: remove specs, they live in spec repo (#4172)
* docs: remove specs, they live in spec repo
- moving specs to spec repo
- https://github.com/tendermint/spec/pull/62 PR for updating them
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add makefile command to copy in sepcs from specREPO
- move cloning of spec repo to pre and post scripts
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
Marko
dependabot-preview[bot]
Anton Kaliaev
Ethan Buchman
Ayush Kaul
Prince Sinha
Alexander Bezobchuk
Cuong Manh Le
dongsamb
Sean Braithwaite
Tess Rinearson
Sunny Aggarwal
Jonathan Gimeno
w1t
Black3HDF
Anca Zamfir
Zaki Manian
Leslie Cheung