after they were pruned by the scheduler

Closes #5513

also, since multiple StopPeerForError calls may be executed in parallel,
only execute StopPeerForError once

Closes #5541

Before: scheduler receives psBlockProcessed event, but does not mark block as processed because peer timed out (or was removed for other reasons) and all associated blocks were rescheduled.

After: scheduler receives psBlockProcessed event and marks block as processed in any case (even if peer who provided this block errors).

Closes #5387

When a peer is stopped due to some network issue, the Reactor calls scheduler#handleRemovePeer, which removes the peer from the scheduler. BUT the peer stays in the processor, which sometimes could lead to "duplicate block enqueued by processor" panic WHEN the same block is requested by the scheduler again from a different peer. The solution is to return scPeerError, which will be propagated to the processor. The processor will clean up the blocks associated with the peer in purgePeer.

Closes #5513, #5517

Blockchain reactors were not updated to handle arbitrary initial height after #5191.

Integrates the blockchain v2 reactor with state sync, fixes #4765. This mostly involves deferring fast syncing until after state sync completes. I tried a few different approaches, this was the least effort:

* `Reactor.events` is `nil` if no fast sync is in progress, in which case events are not dispatched - most importantly `AddPeer`.

* Accept status messages from unknown peers in the scheduler and register them as ready. On fast sync startup, broadcast status requests to all existing peers.

* When switching from state sync, first send a `bcResetState` message to the processor and scheduler to update their states - most importantly the initial block height.

* When fast sync completes, shut down event loop, scheduler and processor, and set `events` channel to `nil`.

* Added BlockStore.DeleteBlock()

* Added initial block pruner prototype

* wip

* Added BlockStore.PruneBlocks()

* Added consensus setting for block pruning

* Added BlockStore base

* Error on replay if base does not have blocks

* Handle missing blocks when sending VoteSetMaj23Message

* Error message tweak

* Properly update blockstore state

* Error message fix again

* blockchain: ignore peer missing blocks

* Added FIXME

* Added test for block replay with truncated history

* Handle peer base in blockchain reactor

* Improved replay error handling

* Added tests for Store.PruneBlocks()

* Fix non-RPC handling of truncated block history

* Panic on missing block meta in needProofBlock()

* Updated changelog

* Handle truncated block history in RPC layer

* Added info about earliest block in /status RPC

* Reorder height and base in blockchain reactor messages

* Updated changelog

* Fix tests

* Appease linter

* Minor review fixes

* Non-empty BlockStores should always have base > 0

* Update code to assume base > 0 invariant

* Added blockstore tests for pruning to 0

* Make sure we don't prune below the current base

* Added BlockStore.Size()

* config: added retain_blocks recommendations

* Update v1 blockchain reactor to handle blockstore base

* Added state database pruning

* Propagate errors on missing validator sets

* Comment tweaks

* Improved error message

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* use ABCI field ResponseCommit.retain_height instead of retain-blocks config option

* remove State.RetainHeight, return value instead

* fix minor issues

* rename pruneHeights() to pruneBlocks()

* noop to fix GitHub borkage

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

* format: add format cmd & goimport repo

- replaced format command
- added goimports to format command
- ran goimports

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* fix outliers & undo proto file changes

The work includes the reactor which ties together all the seperate routines involved in the design of the blockchain v2 refactor. This PR replaces #4067 which got far too large and messy after a failed attempt to rebase.

## Commits:

* Blockchainv 2 reactor:

	+ I cleaner copy of the work done in #4067 which fell too far behind and was a nightmare to rebase.
	+ The work includes the reactor which ties together all the seperate routines involved in the design of the blockchain v2 refactor.

* fixes after merge

* reorder iIO interface methodset

* change iO -> IO

* panic before send nil block

* rename switchToConsensus -> trySwitchToConsensus

* rename tdState -> tmState

* Update blockchain/v2/reactor.go

Co-Authored-By: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com>

* remove peer when it sends a block unsolicited

* check for not ready in markReceived

* fix error

* fix the pcFinished event

* typo fix

* add documentation for processor fields

* simplify time.Since

* try and make the linter happy

* some doc updates

* fix channel diagram

* Update adr-043-blockchain-riri-org.md

* panic on nil switch

* liting fixes

* account for nil block in bBlockResponseMessage

* panic on duplicate block enqueued by processor

* linting

* goimport reactor_test.go

Co-authored-by: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com>
Co-authored-by: Anca Zamfir <ancazamfir@users.noreply.github.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

* Add processor prototype

* Change processor API

    + expose a simple `handle` function which mutates internal state

* schedule event handling

* rename schedule -> scheduler

* fill in handle function

* processor tests

* fix gofmt and ohter golangci issues

* scopelint var on range scope

* add check for short block received

* small test reorg

* ci fix changes

* go.mod revert

* some cleanup and review comments

* scheduler fixes and unit tests, also small processor changes.

changed scPeerPruned to include a list of pruned peers

touchPeer to check peer state and remove the blocks from blockStates if the peer removal causes the max peer height to be lower.

remove the block at sc.initHeight

changed peersInactiveSince, peersSlowerThan, getPeersAtHeight check peer state

prunablePeers to return a sorted list of peers

lastRate in markReceived() attempted to divide by 0, temp fix.

fixed allBlocksProcessed conditions

maxHeight() and minHeight() to return sc.initHeight if no ready peers present

make selectPeer() deterministic.

added handleBlockProcessError()

added termination cond. (sc.allBlocksProcessed()) to handleTryPrunePeer() and others.

changed pcBlockVerificationFailure to include peer of H+2 block along with the one for H+1

changed the processor to call purgePeer on block verification failure.

fixed processor tests

added scheduler tests.

* typo and ci fixes

* remove height from scBlockRequest, golangci fixes

* limit on blockState map, updated tests

* remove unused

* separate test for maxHeight(), used for sched. validation

* use Math.Min

* fix golangci

* Document the semantics of blockStates in the scheduler

* better docs

* distinguish between unknown and invalid blockstate

* Standardize peer filtering methods

* feedback

* s/getPeersAtHeight/getPeersAtHeightOrAbove

* small notes

* Update blockchain/v2/scheduler.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update comments based on feedback

* Add enum offset

* panic on nil block in processor

* remove unused max height calculation

* format shorter line