This pull request merges in the changes for implementing Proposer-based timestamps into `master`. The power was primarily being done in the `wb/proposer-based-timestamps` branch, with changes being merged into that branch during development. This pull request represents an amalgamation of the changes made into that development branch. All of the changes that were placed into that branch have been cleanly rebased on top of the latest `master`. The changes compile and the tests pass insofar as our tests in general pass.
### Note To Reviewers
These changes have been extensively reviewed during development. There is not much new here. In the interest of making effective use of time, I would recommend against trying to perform a complete audit of the changes presented and instead examine for mistakes that may have occurred during the process of rebasing the changes. I gave the complete change set a first pass for any issues, but additional eyes would be very appreciated.
In sum, this change set does the following:
closes#6942
merges in #6849
This is a very small change, but removes a method from the
`service.Service` interface (a win!) and forces callers to explicitly
pass loggers in to objects during construction rather than (later)
injecting them. There's not a real need for this kind of lazy
construction of loggers, and I think a decent potential for confusion
for mutable loggers.
The main concern I have is that this changes the constructor API for
ABCI clients. I think this is fine, and I suspect that as we plumb
contexts through, and make changes to the RPC services there'll be a
number of similar sorts of changes to various (quasi) public
interfaces, which I think we should welcome.
This PR adds an initial set of metrics for use ABCI. The initial metrics enable the calculation of timing histograms and call counts for each of the ABCI methods. The metrics are also labeled as either 'sync' or 'async' to determine if the method call was performed using ABCI's `*Async` methods.
An example of these metrics is included here for reference:
```
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.0001"} 0
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.0004"} 5
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.002"} 12
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.009"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.02"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.1"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.65"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="2"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="6"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="25"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="+Inf"} 13
tendermint_abci_connection_method_timing_sum{chain_id="ci",method="commit",type="sync"} 0.007802058000000001
tendermint_abci_connection_method_timing_count{chain_id="ci",method="commit",type="sync"} 13
```
These metrics can easily be graphed using prometheus's `histogram_quantile(...)` method to pick out a particular quantile to graph or examine. I chose buckets that were somewhat of an estimate of expected range of times for ABCI operations. They start at .0001 seconds and range to 25 seconds. The hope is that this range captures enough possible times to be useful for us and operators.
The code in the Tendermint repository makes heavy use of import aliasing.
This is made necessary by our extensive reuse of common base package names, and
by repetition of similar names across different subdirectories.
Unfortunately we have not been very consistent about which packages we alias in
various circumstances, and the aliases we use vary. In the spirit of the advice
in the style guide and https://github.com/golang/go/wiki/CodeReviewComments#imports,
his change makes an effort to clean up and normalize import aliasing.
This change makes no API or behavioral changes. It is a pure cleanup intended
o help make the code more readable to developers (including myself) trying to
understand what is being imported where.
Only unexported names have been modified, and the changes were generated and
applied mechanically with gofmt -r and comby, respecting the lexical and
syntactic rules of Go. Even so, I did not fix every inconsistency. Where the
changes would be too disruptive, I left it alone.
The principles I followed in this cleanup are:
- Remove aliases that restate the package name.
- Remove aliases where the base package name is unambiguous.
- Move overly-terse abbreviations from the import to the usage site.
- Fix lexical issues (remove underscores, remove capitalization).
- Fix import groupings to more closely match the style guide.
- Group blank (side-effecting) imports and ensure they are commented.
- Add aliases to multiple imports with the same base package name.
* Added BlockStore.DeleteBlock()
* Added initial block pruner prototype
* wip
* Added BlockStore.PruneBlocks()
* Added consensus setting for block pruning
* Added BlockStore base
* Error on replay if base does not have blocks
* Handle missing blocks when sending VoteSetMaj23Message
* Error message tweak
* Properly update blockstore state
* Error message fix again
* blockchain: ignore peer missing blocks
* Added FIXME
* Added test for block replay with truncated history
* Handle peer base in blockchain reactor
* Improved replay error handling
* Added tests for Store.PruneBlocks()
* Fix non-RPC handling of truncated block history
* Panic on missing block meta in needProofBlock()
* Updated changelog
* Handle truncated block history in RPC layer
* Added info about earliest block in /status RPC
* Reorder height and base in blockchain reactor messages
* Updated changelog
* Fix tests
* Appease linter
* Minor review fixes
* Non-empty BlockStores should always have base > 0
* Update code to assume base > 0 invariant
* Added blockstore tests for pruning to 0
* Make sure we don't prune below the current base
* Added BlockStore.Size()
* config: added retain_blocks recommendations
* Update v1 blockchain reactor to handle blockstore base
* Added state database pruning
* Propagate errors on missing validator sets
* Comment tweaks
* Improved error message
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* use ABCI field ResponseCommit.retain_height instead of retain-blocks config option
* remove State.RetainHeight, return value instead
* fix minor issues
* rename pruneHeights() to pruneBlocks()
* noop to fix GitHub borkage
Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
* test functions take time.Now and other minor changes
* updated remaining test files
* Update validation_test.go
* fix typo
* go fmt
* import time
Co-authored-by: Marko <marbar3778@yahoo.com>
* evidence: introduce time.Duration to evidence params
- add time.duration to evidence
- this pr is taking pr #2606 and updating it to use both time and height
- closes#2565
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix testing and genesis cfg in signer harness
* remove debugging fmt
* change maxageheight to maxagenumblocks, rename other things to block instead of height
* further check of duration
* check duration to not send peers outdated evidence
* change some lines, onward and upward
* refactor evidence package
* add a changelog pending entry
* make mockbadevidence have time and use it
* add what could possibly be called a test case
* remove mockbadevidence and mockgoodevidence in favor of mockevidence
* add a comment for err that is returned
* add a changelog for removal of good & bad evidence
* add a test for adding evidence
* fix test
* add ev to types in testcase
* Update evidence/pool_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update evidence/pool_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* fix tests
* fix linting
Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
* lint: golint issue fixes
- on my local machine golint is a lot stricter than the bot so slowly going through and fixing things.
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* more fixes from golint
* remove isPeerPersistentFn
* add changelog entry
* libs/common: refactor libs/common 2
- move random function to there own pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* change imports and usage throughout repo
* fix goimports
* add changelog entry
* types: change `Commit` to consist of just signatures
These are final changes towards removing votes from commit and leaving
only signatures (see ADR-25)
Fixes#1648
* bring back TestCommitToVoteSetWithVotesForAnotherBlockOrNilBlock
+ add absent flag to Vote to indicate that it's for another block
* encode nil votes as CommitSig with BlockIDFlagAbsent
+ make Commit#Precommits array of non-pointers
because precommit will never be nil
* add NewCommitSigAbsent and Absent() funcs
* uncomment validation in CommitSig#ValidateBasic
* add comments to ValidatorSet funcs
* add a changelog entry
* break instead of continue
continue does not make sense in these cases
* types: rename Commit#Precommits to Signatures
* swagger: fix /commit response
* swagger: change block_id_flag type
* fix merge conflicts
* Fix long line errors in abci, crypto, and libs packages
* Fix long lines in p2p and rpc packages
* Fix long lines in abci, state, and tools packages
* Fix long lines in behaviour and blockchain packages
* Fix long lines in cmd and config packages
* Begin fixing long lines in consensus package
* Finish fixing long lines in consensus package
* Add lll exclusion for lines containing URLs
* Fix long lines in crypto package
* Fix long lines in evidence package
* Fix long lines in mempool and node packages
* Fix long lines in libs package
* Fix long lines in lite package
* Fix new long line in node package
* Fix long lines in p2p package
* Ignore gocritic warning
* Fix long lines in privval package
* Fix long lines in rpc package
* Fix long lines in scripts package
* Fix long lines in state package
* Fix long lines in tools package
* Fix long lines in types package
* Enable lll linter
* Remove db from tendemrint in favor of tendermint/tm-cmn
- remove db from `libs`
- update dependancy, there have been no breaking changes in the updated deps
- https://github.com/grpc/grpc-go/releases
- https://github.com/golang/protobuf/releases
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* changelog add
* gofmt
* more gofmt
* Expose priv validators for use in testing
* Generalize block header validation test past height 1
* Remove ineffectual assignment
* Remove redundant SaveState call
* Reorder comment for clarity
* Use the block executor ApplyBlock function instead of implementing a stripped-down version of it
* Remove commented-out code
* Remove unnecessary test
The required tests already appear to be implemented (implicitly) through
the TestValidateBlockHeader test.
* Allow for catching of specific error types during TestValidateBlockCommit
* Make return error testable
* Clean up and add TestValidateBlockCommit code
* Fix formatting
* Extract function to create a new mock test app
* Update comment for clarity
* Fix comment
* Add skeleton code for evidence-related test
* Allow for addressing priv val by address
* Generalize test beyond a single validator
* Generalize TestValidateBlockEvidence past first height
* Reorder code to clearly separate tests and utility code
* Use a common constant for stop height for testing in state/validation_test.go
* Refactor errors to resemble existing conventions
* Fix formatting
* Extract common helper functions
Having the tests littered with helper functions makes them less easily
readable imho, so I've pulled them out into a separate file. This also
makes it easier to see what helper functions are available during
testing, so we minimize the chance of duplication when writing new
tests.
* Remove unused parameter
* Remove unused parameters
* Add field keys
* Remove unused height constant
* Fix typo
* Fix incorrect return error
* Add field keys
* Use separate package for tests
This refactors all of the state package's tests into a state_test
package, so as to keep any usage of the state package's internal methods
explicit.
Any internal methods/constants used by tests are now explicitly exported
in state/export_test.go
* Refactor: extract helper function to make, validate, execute and commit a block
* Rename state function to makeState
* Remove redundant constant for number of validators
* Refactor mock evidence registration into TestMain
* Remove extraneous nVals variable
* Replace function-level TODOs with file-level TODO and explanation
* Remove extraneous comment
* Fix linting issues brought up by GolangCI (pulled in from latest merge from develop)
William Banfield
Sam Kleinman
M. J. Fromberger
JayT106
Callum Waters
Marko
Anton Kaliaev
Erik Grinaker
Shivani Joshi
Phil Salant
Thane Thomson