* Added BlockStore.DeleteBlock()

* Added initial block pruner prototype

* wip

* Added BlockStore.PruneBlocks()

* Added consensus setting for block pruning

* Added BlockStore base

* Error on replay if base does not have blocks

* Handle missing blocks when sending VoteSetMaj23Message

* Error message tweak

* Properly update blockstore state

* Error message fix again

* blockchain: ignore peer missing blocks

* Added FIXME

* Added test for block replay with truncated history

* Handle peer base in blockchain reactor

* Improved replay error handling

* Added tests for Store.PruneBlocks()

* Fix non-RPC handling of truncated block history

* Panic on missing block meta in needProofBlock()

* Updated changelog

* Handle truncated block history in RPC layer

* Added info about earliest block in /status RPC

* Reorder height and base in blockchain reactor messages

* Updated changelog

* Fix tests

* Appease linter

* Minor review fixes

* Non-empty BlockStores should always have base > 0

* Update code to assume base > 0 invariant

* Added blockstore tests for pruning to 0

* Make sure we don't prune below the current base

* Added BlockStore.Size()

* config: added retain_blocks recommendations

* Update v1 blockchain reactor to handle blockstore base

* Added state database pruning

* Propagate errors on missing validator sets

* Comment tweaks

* Improved error message

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* use ABCI field ResponseCommit.retain_height instead of retain-blocks config option

* remove State.RetainHeight, return value instead

* fix minor issues

* rename pruneHeights() to pruneBlocks()

* noop to fix GitHub borkage

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

* dep: update tm-db to 0.4.0

- update 0.4.0 as it is a breaking change and cannot be handled by depndabot

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

*  more work towards error handling

* error and emtpy bytes handling

* work on tests

* add changelog entry, change some error handling

* address some pr comments

* panic in a few more places

* move error higher up

* redo some error handling

* fix some bz == nil to len(bz) == 0

* change statebytes

* types: change `Commit` to consist of just signatures

These are final changes towards removing votes from commit and leaving
only signatures (see ADR-25)

Fixes #1648

* bring back TestCommitToVoteSetWithVotesForAnotherBlockOrNilBlock

+ add absent flag to Vote to indicate that it's for another block

* encode nil votes as CommitSig with BlockIDFlagAbsent

+ make Commit#Precommits array of non-pointers
because precommit will never be nil

* add NewCommitSigAbsent and Absent() funcs

* uncomment validation in CommitSig#ValidateBasic

* add comments to ValidatorSet funcs

* add a changelog entry

* break instead of continue

continue does not make sense in these cases

* types: rename Commit#Precommits to Signatures

* swagger: fix /commit response

* swagger: change block_id_flag type

* fix merge conflicts

* Removal of TotalTx & NumTx

- Removed totalTx and numTx

closes #2521

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* abci proto changes

* proto number fix

* txfilter_test fix

* comments on PR

* further changes

* bring back metrics

* fix indexer

* fix TestBlockMaxDataBytes and TestBlockMaxDataBytesUnknownEvidence

* indexer service back to header

* statistics.go fix

* fix ci

* listen for blocks, not headers

to be able to record txs throughput

* fix TestNetworkNewBlock

* fix tests

* fix tests in types package

* fixes after Anton's review

* fix tests

* bring back `consensus_total_txs` metric

I mistakenly thought it was removed.

* improve changelog

* remove LastBlockTotalTx from state

* docs: remove getNumTxs from BeginBlock Java example

* Fix maligned structs

* Fix interfacer errors

* Revert accidental go.mod and go.sum changes

* Revert P2PConfig struct maligned reorder

* Revert PeerRoundState struct maligned reordering

* Revert RoundState struct maligned reordering

* Reorder WSClient struct

* Revert accidental type change

* Clean up type change

* Clean up type changes

* Revert to types.ABCIApplicationServer in GRPCServer struct

* Revert maligned changes to BaseConfig struct

* Fix tests in io_test.go

* Fix client_test package tests

* Fix reactor tests in consensus package

* Fix new interfacer errors

* Pin range scope vars

* Don't disable scopelint

This PR repairs linter errors seen when running the following commands:
golangci-lint run --no-config --disable-all=true --enable=scopelint

Contributes to #3262

* (1/2) of replace errors.go with github.com/pkg/errors

ref #3862

- step one in removing instances of errors.go in favor of github.com/pkg/errors

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* gofmt

* add in /store

* tm-cmn to tm-db

* go.mod changes

* go.mod changes

* more go.mod

* fix tm-db

* ci fix, pending change

* go routines in blockchain reactor

* Added reference to the go routine diagram

* Initial commit

* cleanup

* Undo testing_logger change, committed by mistake

* Fix the test loggers

* pulled some fsm code into pool.go

* added pool tests

* changes to the design

added block requests under peer

moved the request trigger in the reactor poolRoutine, triggered now by a ticker

in general moved everything required for making block requests smarter in the poolRoutine

added a simple map of heights to keep track of what will need to be requested next

added a few more tests

* send errors to FSM in a different channel than blocks

send errors (RemovePeer) from switch on a different channel than the
one receiving blocks
renamed channels
added more pool tests

* more pool tests

* lint errors

* more tests

* more tests

* switch fast sync to new implementation

* fixed data race in tests

* cleanup

* finished fsm tests

* address golangci comments :)

* address golangci comments :)

* Added timeout on next block needed to advance

* updating docs and cleanup

* fix issue in test from previous cleanup

* cleanup

* Added termination scenarios, tests and more cleanup

* small fixes to adr, comments and cleanup

* Fix bug in sendRequest()

If we tried to send a request to a peer not present in the switch, a
missing continue statement caused the request to be blackholed in a peer
that was removed and never retried.

While this bug was manifesting, the reactor kept asking for other
blocks that would be stored and never consumed. Added the number of
unconsumed blocks in the math for requesting blocks ahead of current
processing height so eventually there will be no more blocks requested
until the already received ones are consumed.

* remove bpPeer's didTimeout field

* Use distinct err codes for peer timeout and FSM timeouts

* Don't allow peers to update with lower height

* review comments from Ethan and Zarko

* some cleanup, renaming, comments

* Move block execution in separate goroutine

* Remove pool's numPending

* review comments

* fix lint, remove old blockchain reactor and duplicates in fsm tests

* small reorg around peer after review comments

* add the reactor spec

* verify block only once

* review comments

* change to int for max number of pending requests

* cleanup and godoc

* Add configuration flag fast sync version

* golangci fixes

* fix config template

* move both reactor versions under blockchain

* cleanup, golint, renaming stuff

* updated documentation, fixed more golint warnings

* integrate with behavior package

* sync with master

* gofmt

* add changelog_pending entry

* move to improvments

* suggestion to changelog entry

* Remove db from tendemrint in favor of tendermint/tm-cmn

- remove db from `libs`
- update dependancy, there have been no breaking changes in the updated deps
	- https://github.com/grpc/grpc-go/releases
	- https://github.com/golang/protobuf/releases

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* changelog add

* gofmt

* more gofmt

* changelog: use issue number instead of PR number

* follow up to #3291

- rpc/test/helpers.go add StopTendermint(node) func
- remove ensureDir(filepath.Dir(walFile), 0700)
- mempool/mempool_test.go add type cleanupFunc func()

* cmd/show_validator: wrap err to make it more clear

* improve ResetTestRootWithChainID() concurrency safety

Rely on ioutil.TempDir() to create test root directories and ensure
multiple same-chain id test cases can run in parallel.

* Update config/toml.go

Co-Authored-By: alessio <quadrispro@ubuntu.com>

* clean up test directories after completion

Closes: #1034

* Remove redundant EnsureDir call

* s/PanicSafety()/panic()/s

* Put create dir functionality back in ResetTestRootWithChainID

* Place test directories in OS's tempdir

In modern UNIX and UNIX-like systems /tmp is very often
mounted as tmpfs. This might speed test execution a bit.

* Set 0700 to a const

* rootsDirs -> configRootDirs

* Don't double remove directories

* Avoid global variables

* Fix consensus tests

* Reduce defer stack

* Address review comments

* Try to fix tests

* Update CHANGELOG_PENDING.md

Co-Authored-By: alessio <quadrispro@ubuntu.com>

* Update consensus/common_test.go

Co-Authored-By: alessio <quadrispro@ubuntu.com>

* Update consensus/common_test.go

Co-Authored-By: alessio <quadrispro@ubuntu.com>

* types.NewCommit

* use types.NewCommit everywhere

* fix log in unsafe_reset

* memoize height and round in constructor

* notes about deprecating toVote

* bring back memoizeHeightRound

* types: memoize height/round in commit instead of first vote

* types: commit.ValidateBasic in VerifyCommit

* types: new CommitSig alias for Vote

In preparation for reducing the redundancy in Commits, we introduce the
CommitSig as an alias for Vote. This is non-breaking on the protocol,
and minor breaking on the Go API, as Commit now contains a list of
CommitSig instead of Vote.

* remove dependence on ToVote

* update some comments

* fix tests

* fix tests

* fixes from review

* Begin simple merkle compatibility PR

* Fix query_test

* Use trillian test vectors

* Change the split point per RFC 6962

* update spec

* refactor innerhash to match spec

* Update changelog

* Address @liamsi's comments

* Write the comment requested by @liamsi

* fix fastsync may stuck by a wrong block

* fixes from updates

* fixes from review

* Align spec with the changes

* fmt

* Implement BFT time

* set LastValidators when creating state in state helper

for heights >= 2

make BlockMeta Header a non-pointer

Refs #693

This reverts commit d02c5d1e30.

Also, fix consensus liveness issue.

This reverts commit 96a3502

NOTE we only updating for tmlibs and abci

Fixes:

```
panic: Panicked on a Sanity Check: can't encode times below 1970 [recovered]
        panic: Panicked on a Sanity Check: can't encode times below 1970

goroutine 2042 [running]:
testing.tRunner.func1(0xc420e8c0f0)
        /usr/local/go/src/testing/testing.go:711 +0x5d9
panic(0xcd9e20, 0xc420c8c270)
        /usr/local/go/src/runtime/panic.go:491 +0x2a2
github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common.PanicSanity(0xcd9e20, 0xf8ddd0)
        /go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/tmlibs/common/errors.go:26 +0x120
github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire.WriteTime(0x0, 0x0, 0x0, 0x1306440, 0xc4201607e0, 0xc420e31658, 0xc420e31680)
        /go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire/time.go:19 +0x11e
github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire.writeReflectBinary(0xdc9e40, 0xc4201fcf30, 0x199, 0x1317b80, 0xdc9e40, 0xc98451, 0x9, 0x0, 0xdc9e40, 0xc420ead9c0, ...)
        /go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire/reflect.go:525 +0x22f0
github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire.writeReflectBinary(0xd25400, 0xc42000e2e8, 0x196, 0x1317b80, 0xd92d60, 0xc9a195, 0xa, 0x0, 0xcc8ce0, 0xc420164920, ...)
        /go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire/reflect.go:530 +0x21e7
github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire.writeReflectBinary(0xcc8ce0, 0xc4209f95b8, 0x197, 0x1317b80, 0xcc8ce0, 0xc9a195, 0xa, 0x0, 0xcc8ce0, 0xc420164920, ...)
        /go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire/reflect.go:518 +0x2509
github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire.writeReflectBinary(0xd873e0, 0xc4209f9580, 0x16, 0x1317b80, 0xd79400, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire/reflect.go:530 +0x21e7
github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire.WriteBinary(0xd873e0, 0xc4209f9580, 0x1306440, 0xc4201607e0, 0xc420e31658, 0xc420e31680)
        /go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire/wire.go:80 +0x15f
github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire.BinaryBytes(0xd873e0, 0xc4209f9580, 0x3, 0x8, 0xc420160798)
        /go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-wire/util.go:15 +0xb8
github.com/tendermint/tendermint/blockchain.(*BlockStore).SaveBlock(0xc4201342a0, 0xc420eac180, 0xc420130640, 0xc4209f9580)
        github.com/tendermint/tendermint/blockchain/_test/_obj_test/store.go:192 +0x439
github.com/tendermint/tendermint/blockchain.TestBlockStoreSaveLoadBlock(0xc420e8c0f0)
        /go/src/github.com/tendermint/tendermint/blockchain/store_test.go:128 +0x609
testing.tRunner(0xc420e8c0f0, 0xf20830)
        /usr/local/go/src/testing/testing.go:746 +0x16d
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:789 +0x569
exit status 2
FAIL
```

With feedback from @ebuchman, to make the tests nicer
and less fragile.

Add tests to test store, to the fullest reasonable extent for
paths that can taken by input arguments altering internal behavior,
as well as by mutating content in the DB.