Closes#4603
Commands used (VIM):
```
:args `rg -l errors.Wrap`
:argdo normal @q | update
```
where q is a macros rewriting the `errors.Wrap` to `fmt.Errorf`.
creates a proof of lock change which is an array of votes that proves the validator was permitted to change locked block and vote again. This proof is stored in the evidence pool and is used as a part of amnesia evidence
* Added BlockStore.DeleteBlock()
* Added initial block pruner prototype
* wip
* Added BlockStore.PruneBlocks()
* Added consensus setting for block pruning
* Added BlockStore base
* Error on replay if base does not have blocks
* Handle missing blocks when sending VoteSetMaj23Message
* Error message tweak
* Properly update blockstore state
* Error message fix again
* blockchain: ignore peer missing blocks
* Added FIXME
* Added test for block replay with truncated history
* Handle peer base in blockchain reactor
* Improved replay error handling
* Added tests for Store.PruneBlocks()
* Fix non-RPC handling of truncated block history
* Panic on missing block meta in needProofBlock()
* Updated changelog
* Handle truncated block history in RPC layer
* Added info about earliest block in /status RPC
* Reorder height and base in blockchain reactor messages
* Updated changelog
* Fix tests
* Appease linter
* Minor review fixes
* Non-empty BlockStores should always have base > 0
* Update code to assume base > 0 invariant
* Added blockstore tests for pruning to 0
* Make sure we don't prune below the current base
* Added BlockStore.Size()
* config: added retain_blocks recommendations
* Update v1 blockchain reactor to handle blockstore base
* Added state database pruning
* Propagate errors on missing validator sets
* Comment tweaks
* Improved error message
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* use ABCI field ResponseCommit.retain_height instead of retain-blocks config option
* remove State.RetainHeight, return value instead
* fix minor issues
* rename pruneHeights() to pruneBlocks()
* noop to fix GitHub borkage
Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
* prometheus/metrics: two new metrics for consensus
- add consensus_validator_power metric so if a node is a validator it can see its own power in prometheus
- add last_signed_height metric so if a node is a validator it can be aware of at which height the most recent time it signed.
- closes: #3773
- closes: #3083
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* check if signature is present
* minor change and check if sig is not absent
* add changelog entry
* Update consensus/state.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update CHANGELOG_PENDING.md
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update CHANGELOG_PENDING.md
* add label with validator address
* change address to vali_address
* add metric missed blocks
* add changelog entry for missed blocks metric
* address naming & add docs
also
- do not send any LastCommitInfo.Votes for 1st block.
- use cs.LastValidators (not cs.Validators) in recordMetrics when counting missing validators
note:
remember that the first LastCommit is intentionally empty, so it makes
sense for LastCommitInfo.Votes to also be empty. Because we can't really
tell if validator signed or not. Similar for ^ metrics, we can't say if
particular validator was missing or not.
Closes#4192
* libs/common: Refactor libs/common 5
- move mathematical functions and types out of `libs/common` to math pkg
- move net functions out of `libs/common` to net pkg
- move string functions out of `libs/common` to strings pkg
- move async functions out of `libs/common` to async pkg
- move bit functions out of `libs/common` to bits pkg
- move cmap functions out of `libs/common` to cmap pkg
- move os functions out of `libs/common` to os pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix testing issues
* fix tests
closes#41417
woooooooooooooooooo kill the cmn pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* fix goimport issues
* run gofmt
* libs/common: refactor libs common 3
- move nil.go into types folder and make private
- move service & baseservice out of common into service pkg
ref #4147
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* types: change `Commit` to consist of just signatures
These are final changes towards removing votes from commit and leaving
only signatures (see ADR-25)
Fixes#1648
* bring back TestCommitToVoteSetWithVotesForAnotherBlockOrNilBlock
+ add absent flag to Vote to indicate that it's for another block
* encode nil votes as CommitSig with BlockIDFlagAbsent
+ make Commit#Precommits array of non-pointers
because precommit will never be nil
* add NewCommitSigAbsent and Absent() funcs
* uncomment validation in CommitSig#ValidateBasic
* add comments to ValidatorSet funcs
* add a changelog entry
* break instead of continue
continue does not make sense in these cases
* types: rename Commit#Precommits to Signatures
* swagger: fix /commit response
* swagger: change block_id_flag type
* fix merge conflicts
* Removal of TotalTx & NumTx
- Removed totalTx and numTx
closes#2521
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* abci proto changes
* proto number fix
* txfilter_test fix
* comments on PR
* further changes
* bring back metrics
* fix indexer
* fix TestBlockMaxDataBytes and TestBlockMaxDataBytesUnknownEvidence
* indexer service back to header
* statistics.go fix
* fix ci
* listen for blocks, not headers
to be able to record txs throughput
* fix TestNetworkNewBlock
* fix tests
* fix tests in types package
* fixes after Anton's review
* fix tests
* bring back `consensus_total_txs` metric
I mistakenly thought it was removed.
* improve changelog
* remove LastBlockTotalTx from state
* docs: remove getNumTxs from BeginBlock Java example
* cs: panic only when WAL#WriteSync fails
- modify WAL#Write and WAL#WriteSync to return an error
* fix test
* types: validate Part#Proof
add ValidateBasic to crypto/merkle/SimpleProof
* cs: limit max bit array size and block parts count
* cs: test new limits
* cs: only assert important stuff
* update changelog and bump version to 0.32.7
* fixes after Ethan's review
* align max wal msg and max consensus msg sizes
* fix tests
* fix test
* add change log for 31.11
* Fix long line errors in abci, crypto, and libs packages
* Fix long lines in p2p and rpc packages
* Fix long lines in abci, state, and tools packages
* Fix long lines in behaviour and blockchain packages
* Fix long lines in cmd and config packages
* Begin fixing long lines in consensus package
* Finish fixing long lines in consensus package
* Add lll exclusion for lines containing URLs
* Fix long lines in crypto package
* Fix long lines in evidence package
* Fix long lines in mempool and node packages
* Fix long lines in libs package
* Fix long lines in lite package
* Fix new long line in node package
* Fix long lines in p2p package
* Ignore gocritic warning
* Fix long lines in privval package
* Fix long lines in rpc package
* Fix long lines in scripts package
* Fix long lines in state package
* Fix long lines in tools package
* Fix long lines in types package
* Enable lll linter
* cs: check for SkipTimeoutCommit or wait timeout in handleTxsAvailable
Previously, if create_empty_blocks was set to false, TM sometimes could
create 2 consecutive blocks without actually waiting for timeoutCommit
(1 sec. by default).
```
I[2019-08-29|07:32:43.874] Executed block module=state height=47 validTxs=10 invalidTxs=0
I[2019-08-29|07:32:43.876] Committed state module=state height=47 txs=10 appHash=F88C010000000000
I[2019-08-29|07:32:44.885] Executed block module=state height=48 validTxs=2 invalidTxs=0
I[2019-08-29|07:32:44.887] Committed state module=state height=48 txs=2 appHash=FC8C010000000000
I[2019-08-29|07:32:44.908] Executed block module=state height=49 validTxs=8 invalidTxs=0
I[2019-08-29|07:32:44.909] Committed state module=state height=49 txs=8 appHash=8C8D010000000000
I[2019-08-29|07:32:45.886] Executed block module=state height=50 validTxs=2 invalidTxs=0
I[2019-08-29|07:32:45.895] Committed state module=state height=50 txs=2 appHash=908D010000000000
I[2019-08-29|07:32:45.908] Executed block module=state height=51 validTxs=8 invalidTxs=0
I[2019-08-29|07:32:45.909] Committed state module=state height=51 txs=8 appHash=A08D010000000000
```
This commit fixes that by adding a check to handleTxsAvailable.
Fixes#3908
* update changelog
* schedule timeoutCommit if StartTime is in the future
or SkipTimeoutCommit=true && we DON'T have all the votes
* fix TestReactorCreatesBlockWhenEmptyBlocksFalse
by checking if we have LastCommit or not
* address Ethan's comments
* Remove unnecessary type conversions
* Consolidate repeated strings into consts
* Clothe return statements
* Update blockchain/v1/reactor_fsm_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
This PR repairs linter errors seen when running the following commands:
golangci-lint run --no-config --disable-all=true --enable=unconvert
golangci-lint run --no-config --disable-all=true --enable=goconst
golangci-lint run --no-config --disable-all=true --enable=nakedret
Contributes to #3262
Add gocritic as a linter
The linting is not complete, but should i complete in this PR or in a following.
23 files have been touched so it may be better to do in a following PR
Commits:
* Add gocritic to linting
- Added gocritic to linting
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* gocritic
* pr comments
* remove switch in cmdBatch
cleanup to add linter
grpc change:
https://godoc.org/google.golang.org/grpc#WithContextDialerhttps://godoc.org/google.golang.org/grpc#WithDialer
grpc/grpc-go#2627
prometheous change:
due to UninstrumentedHandler, being deprecated in the future
empty branch = empty if or else statement
didn't delete them entirely but commented
couldn't find a reason to have them
could not replicate the issue #3406
but if want to keep it commented then we should comment out the if statement as well
* VoteSignBytes builds CanonicalVote
* CommitVotes implements VoteSetReader
- new CommitVotes struct holds both the Commit and the ValidatorSet and
implements VoteSetReader
- ToVote takes a ValidatorSet
* fix TestCommit
* use CommitSig.BlockID
Commits may include votes for a different BlockID, could be nil,
or different altogether. This means we can't use `commit.BlockID`
for reconstructing the sign bytes, since up to -1/3 of the commits
might be for independent BlockIDs. This means CommitSig will need to
include an indicator for what BlockID it signed - if it's not the
committed one or nil, it will need to include it fully in order to be
verified. This is unfortunate but unavoidable so long as we include
votes for non-committed BlockIDs (which we do to track validator
liveness)
* fixes from review
* remove CommitVotes. CommitSig contains address
* remove commit.canonicalVote method
* toVote -> getVote, takes valIdx
* update adr-025
* commit.ToVoteSet -> CommitToVoteSet
* add test
* fix from review
* Remove deprecated PanicXXX functions from codebase
As per discussion over
[here](https://github.com/tendermint/tendermint/pull/3456#discussion_r278423492),
we need to remove these `PanicXXX` functions and eliminate our
dependence on them. In this PR, each and every `PanicXXX` function call
is replaced with a simple `panic` call.
* add a changelog entry
Refs #3262
This fixes two small bugs:
1) lite/dbprovider: return `ok` instead of true in parse* functions. It's weird that we're ignoring `ok` value before.
2) consensus/state: previously because of the shadowing we almost never output "Error with msg". Now we declare both `added` and `err` in the beginning of the function, so there's no shadowing.
* make BlockTimeIota a consensus parameter, not a locally configurable option
Refs #2920
* make TimeIota int64 ms
Refs #2920
* update Gopkg.toml
* fixes after Ethan's review
* fix TestRemoteSignerProposalSigningFailed
* update changelog
As per #3043, this adds a ticker to sync the WAL every 2s while the WAL is running.
* Flush WAL every 2s
This adds a ticker that flushes the WAL every 2s while the WAL is
running. This is related to #3043.
* Fix spelling
* Increase timeout to 2mins for slower build environments
* Make WAL sync interval configurable
* Add TODO to replace testChan with more comprehensive testBus
* Remove extraneous debug statement
* Remove testChan in favour of using system time
As per
https://github.com/tendermint/tendermint/pull/3300#discussion_r255886586,
this removes the `testChan` WAL member and replaces the approach with a
system time-oriented one. In this new approach, we keep track of the
system time at which each flush and periodic flush successfully
occurred.
The naming of the various functions is also updated here to be more
consistent with "flushing" as opposed to "sync'ing".
* Update naming convention and ensure lock for timestamp update
* Add Flush method as part of WAL interface
Adds a `Flush` method as part of the WAL interface to enforce the idea
that we can manually trigger a WAL flush from outside of the WAL. This
is employed in the consensus state management to flush the WAL prior to
signing votes/proposals, as per https://github.com/tendermint/tendermint/issues/3043#issuecomment-453853630
* Update CHANGELOG_PENDING
* Remove mutex approach and replace with DI
The dependency injection approach to dealing with testing concerns could
allow similar effects to some kind of "testing bus"-based approach. This
commit introduces an example of this, where instead of relying on
(potentially fragile) timing of things between the code and the test, we
inject code into the function under test that can signal the test
through a channel.
This allows us to avoid the `time.Sleep()`-based approach previously
employed.
* Update comment on WAL flushing during vote signing
Co-Authored-By: thanethomson <connect@thanethomson.com>
* Simplify flush interval definition
Co-Authored-By: thanethomson <connect@thanethomson.com>
* Expand commentary on WAL disk flushing
Co-Authored-By: thanethomson <connect@thanethomson.com>
* Add broken test to illustrate WAL sync test problem
Removes test-related state (dependency injection code) from the WAL data
structure and adds test code to illustrate the problem with using
`WALGenerateNBlocks` and `wal.SearchForEndHeight` to test periodic
sync'ing.
* Fix test error messages
* Use WAL group buffer size to check for flush
A function is added to `libs/autofile/group.go#Group` in order to return
the size of the buffered data (i.e. data that has not yet been flushed
to disk). The test now checks that, prior to a `time.Sleep`, the group
buffer has data in it. After the `time.Sleep` (during which time the
periodic flush should have been called), the buffer should be empty.
* Remove config root dir removal from #3291
* Add godoc for NewWAL mentioning periodic sync
* types.NewCommit
* use types.NewCommit everywhere
* fix log in unsafe_reset
* memoize height and round in constructor
* notes about deprecating toVote
* bring back memoizeHeightRound
* types: memoize height/round in commit instead of first vote
* types: commit.ValidateBasic in VerifyCommit
* types: new CommitSig alias for Vote
In preparation for reducing the redundancy in Commits, we introduce the
CommitSig as an alias for Vote. This is non-breaking on the protocol,
and minor breaking on the Go API, as Commit now contains a list of
CommitSig instead of Vote.
* remove dependence on ToVote
* update some comments
* fix tests
* fix tests
* fixes from review
* consensus: createProposalBlock function
* blockExecutor.CreateProposalBlock
- factored out of consensus pkg into a method on blockExec
- new private interfaces for mempool ("txNotifier") and evpool with one function each
- consensus tests still require more mempool methods
* failing test for CreateProposalBlock
* Fix bug in include evidece into block
* evidence: change maxBytes to maxSize
* MaxEvidencePerBlock
- changed to return both the max number and the max bytes
- preparation for #2590
* changelog
* fix linter
* Fix from review
Co-Authored-By: ebuchman <ethan@coinculture.info>
* #2815 do not broadcast heartbeat proposal when we are non-validator
* #2815 adding preliminary changelog entry
* #2815 cosmetics and added test
* #2815 missed a little detail
- it's enough to call getAddress() once here
* #2815 remove debug logging from tests
* #2815 OK. I seem to be doing something fundamentally wrong here
* #2815 next iteration of proposalHeartbeat tests
- try and use "ensure" pattern in common_test
* 2815 incorporate review comments
* add proposer info to EventCompleteProposal
* create separate EventData structure for CompleteProposal
* cant us rs.Proposal to get BlockID because it is not guaranteed to be set yet
* copying RoundState isnt helping us here
* add Step back to make compatible with original RoundState event. update changelog
* add NewRound event
* fix test
* remove unneeded RoundState
* put height round step into a struct
* pull out ValidatorInfo struct. add ensureProposal assert
* remove height-round-state sub-struct refactor
* minor fixes from review
Anton Kaliaev
Callum Waters
Erik Grinaker
Marko
Erik Grinaker
tau3
Phil Salant
Ethan Buchman
Thane Thomson
Zarko Milosevic
srmo
Ismail Khoffi
kevlubkcm