Since the light client work introduced in v0.33 it appears full nodes
are no longer fully verifying commit signatures during block execution -
they stop after +2/3. See in VerifyCommit:
0c7fd316eb/types/validator_set.go (L700-L703)

This means proposers can propose blocks that contain valid +2/3
signatures and then the rest of the signatures can be whatever they
want. They can claim that all the other validators signed just by
including a CommitSig with arbitrary signature data. While this doesn't
seem to impact safety of Tendermint per se, it means that Commits may
contain a lot of invalid data. This is already true of blocks, since
they can include invalid txs filled with garbage, but in that case the
application knows they they are invalid and can punish the proposer. But
since applications dont verify commit signatures directly (they trust
tendermint to do that), they won't be able to detect it.

This can impact incentivization logic in the application that depends on
the LastCommitInfo sent in BeginBlock, which includes which validators
signed. For instance, Gaia incentivizes proposers with a bonus for
including more than +2/3 of the signatures. But a proposer can now claim
that bonus just by including arbitrary data for the final -1/3 of
validators without actually waiting for their signatures. There may be
other tricks that can be played because of this.

In general, the full node should be a fully verifying machine. While
it's true that the light client can avoid verifying all signatures by
stopping after +2/3, the full node can not. Thus the light client and
full node should use distinct VerifyCommit functions if one is going to
stop after +2/3 or otherwise perform less validation (for instance light
clients can also skip verifying votes for nil while full nodes can not).

See a commit with a bad signature that verifies here: 56367fd. From what
I can tell, Tendermint will go on to think this commit is valid and
forward this data to the app, so the app will think the second validator
actually signed when it clearly did not.

Closes #4607

Closes #4392
Refs #4504

Closes: #4420 

Created a new error ErrInvalidHeaderwhich can be formed during the verification process verifier.go and will result in the replacement of the primary provider with a witness by executing: replacePrimaryProvider()

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

Closes: #4488

## Commits:

* run example tests

* introduce max clock drift

clockDrift variable from the spec.
10s should cover most of the clients.

References:

- http://vancouver-webpages.com/time/web.html
- https://blog.codinghorror.com/keeping-time-on-the-pc/

* fix ExampleClient_Update

* add test

* increase clock drift

* fix linter warning

Before we were storing trustedHeader (height=1) and trustedNextVals
(height=2).

After this change, we will be storing trustedHeader (height=1) and
trustedVals (height=1). This a) simplifies the code b) fixes #4399
inconsistent pairing issue c) gives a relayer access to the current
validator set #4470.

The only downside is more jumps during bisection. If validator set
changes between trustedHeader and the next header (by 2/3 or more), the
light client will be forced to download the next header and check that
2/3+ signed the transition. But we don't expect validator set change too
much and too often, so it's an acceptable compromise.

Closes #4470 and #4399

Closes #4398

* divided verify functions

* extacted method

* renamed functions. Created standard Verify function

* checked non-adjacency. separated VerifyCommit

* lint fixes

* fix godoc documentation for VerifyAdjacent and VerifyNonAdjacent

* add a comment about VerifyCommit being the last check

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

* cmd/lite: switch to new lite2 package

* update changelog

* shorten the trusting period explanation

* rename adjusted to adjacent

Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r352140829

* rename ErrTooMuchChange to ErrNotEnoughVotingPowerSigned

Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r352142785

* verify commit is properly signed

* remove no longer trusted headers

* restore trustedHeader and trustedNextVals

* check trustedHeader using options

Refs https://github.com/tendermint/tendermint/pull/4209#issuecomment-562462165

* use correct var when checking if headers are adjacent

in bisection func
+ replace TODO with a comment

https://github.com/tendermint/tendermint/pull/3989#discussion_r352125455

* return header in VerifyHeaderAtHeight

because that way we avoid DB call

+ add godoc comments
+ check if there are no headers yet in AutoClient

https://github.com/tendermint/tendermint/pull/3989#pullrequestreview-315454506

* TestVerifyAdjacentHeaders: add 2 more test-cases

+ add TestVerifyReturnsErrorIfTrustLevelIsInvalid

* lite: avoid overflow when parsing key in db store!

* lite: rename AutoClient#Err to Errs

* lite: add a test for AutoClient

* lite: fix keyPattern and call itr.Next in db store

* lite: add two tests for db store

* lite: add TestClientRemovesNoLongerTrustedHeaders

* lite: test Client#Cleanup

* lite: test restoring trustedHeader

https://github.com/tendermint/tendermint/pull/4209#issuecomment-562462165

* lite: comment out unused code in test_helpers

* fix TestVerifyReturnsErrorIfTrustLevelIsInvalid after merge

* change defaultRemoveNoLongerTrustedHeadersPeriod

and add docs

* write more doc

* lite: uncomment testable examples

* use stdlog.Fatal to stop AutoClient tests

* make lll linter happy

* separate errors for 2 cases

- the validator set of a skipped header cannot be trusted, i.e. <1/3rd
  of h1 validator set has signed (new error, something like
  ErrNewValSetCantBeTrusted)
- the validator set is trusted but < 2/3rds has signed
  (ErrNewHeaderCantBeTrusted)

https://github.com/tendermint/tendermint/pull/4209#discussion_r360331253

* remove all headers (even the last one) that are outside

of the trusting period. By doing this, we avoid checking the
trustedHeader's hash in checkTrustedHeaderUsingOptions (case #1).

https://github.com/tendermint/tendermint/pull/4209#discussion_r360332460

* explain restoreTrustedHeaderAndNextVals better

https://github.com/tendermint/tendermint/pull/4209#discussion_r360602328

* add ConfirmationFunction option

for optionally prompting for user input Y/n before removing headers

Refs https://github.com/tendermint/tendermint/pull/4209#discussion_r360602945

* make cleaning optional

https://github.com/tendermint/tendermint/pull/4209#discussion_r364838189

* return error when user refused to remove headers

* check for double votes in VerifyCommitTrusting

* leave only ErrNewValSetCantBeTrusted error

to differenciate between h2Vals.VerifyCommit and
h1NextVals.VerifyCommitTrusting

* fix example tests

* remove unnecessary if condition

https://github.com/tendermint/tendermint/pull/4209#discussion_r365171847

It will be handled by the above switch.

* verifyCommitBasic does not depend on vals

Co-authored-by: Marko <marbar3778@yahoo.com>

* libs/common: Refactor libs/common 5

- move mathematical functions and types out of `libs/common` to math pkg
- move net functions out of `libs/common` to net pkg
- move string functions out of `libs/common` to strings pkg
- move async functions out of `libs/common` to async pkg
- move bit functions out of `libs/common` to bits pkg
- move cmap functions out of `libs/common` to cmap pkg
- move os functions out of `libs/common` to os pkg

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* fix testing issues

* fix tests

closes #41417

woooooooooooooooooo kill the cmn pkg

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* add changelog entry

* fix goimport issues

* run gofmt

Refs #1771

ADR: https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-044-lite-client-with-weak-subjectivity.md

## Commits:

* add Verifier and VerifyCommitTrusting

* add two more checks

make trustLevel an option

* float32 for trustLevel

* check newHeader time

* started writing lite Client

* unify Verify methods

* ensure h2.Header.bfttime < h1.Header.bfttime + tp

* move trust checks into Verify function

* add more comments

* more docs

* started writing tests

* unbonding period failures

* tests are green

* export ErrNewHeaderTooFarIntoFuture

* make golangci happy

* test for non-adjusted headers

* more precision

* providers and stores

* VerifyHeader and VerifyHeaderAtHeight funcs

* fix compile errors

* remove lastVerifiedHeight, persist new trusted header

* sequential verification

* remove TrustedStore option

* started writing tests for light client

* cover basic cases for linear verification

* bisection tests PASS

* rename BisectingVerification to SkippingVerification

* refactor the code

* add TrustedHeader method

* consolidate sequential verification tests

* consolidate skipping verification tests

* rename trustedVals to trustedNextVals

* start writing docs

* ValidateTrustLevel func and ErrOldHeaderExpired error

* AutoClient and example tests

* fix errors

* update doc

* remove ErrNewHeaderTooFarIntoFuture

This check is unnecessary given existing a) ErrOldHeaderExpired b)
h2.Time > now checks.

* return an error if we're at more recent height

* add comments

* add LastSignedHeaderHeight method to Store

I think it's fine if Store tracks last height

* copy over proxy from old lite package

* make TrustedHeader return latest if height=0

* modify LastSignedHeaderHeight to return an error if no headers exist

* copy over proxy impl

* refactor proxy and start http lite client

* Tx and BlockchainInfo methods

* Block method

* commit method

* code compiles again

* lite client compiles

* extract updateLiteClientIfNeededTo func

* move final parts

* add placeholder for tests

* force usage of lite http client in proxy

* comment out query tests for now

* explicitly mention tp: trusting period

* verify nextVals in VerifyHeader

* refactor bisection

* move the NextValidatorsHash check into updateTrustedHeaderAndVals

+ update the comment

* add ConsensusParams method to RPC client

* add ConsensusParams to rpc/mock/client

* change trustLevel type to a new cmn.Fraction type

+ update SkippingVerification comment

* stress out trustLevel is only used for non-adjusted headers

* fixes after Fede's review

Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>

* compare newHeader with a header from an alternative provider

* save pivot header

Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349122824

* check header can still be trusted in TrustedHeader

Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349101424

* lite: update Validators and Block endpoints

- Block no longer contains BlockMeta
- Validators now accept two additional params: page and perPage

* make linter happy