implementation spec of Improved Trusted Peering ADR-050 by B-Harvest

- add unconditional_peer_ids and persistent_peers_max_dial_period to config
- add unconditionalPeerIDs map to Switch struct

default config value of persistent_peers_max_dial_period is 0s(disabled)

Refs #4072, #4053

* Fix long line errors in abci, crypto, and libs packages

* Fix long lines in p2p and rpc packages

* Fix long lines in abci, state, and tools packages

* Fix long lines in behaviour and blockchain packages

* Fix long lines in cmd and config packages

* Begin fixing long lines in consensus package

* Finish fixing long lines in consensus package

* Add lll exclusion for lines containing URLs

* Fix long lines in crypto package

* Fix long lines in evidence package

* Fix long lines in mempool and node packages

* Fix long lines in libs package

* Fix long lines in lite package

* Fix new long line in node package

* Fix long lines in p2p package

* Ignore gocritic warning

* Fix long lines in privval package

* Fix long lines in rpc package

* Fix long lines in scripts package

* Fix long lines in state package

* Fix long lines in tools package

* Fix long lines in types package

* Enable lll linter

* New lint version upgrade

- linter was upgraded

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* enable-a;; is deprecated

* minor change

* another try

* some more changes

* some more changes

* reenable prealloc

* add version till bot is fixed

Fixes #4026

* update gogo/protobuf version from v1.2.1 to v1.3.0

Also errcheck from v1.1.0 to v1.2.0

See full diff in
- https://github.com/gogo/protobuf/compare/v1.2.1...v1.3.0
- https://github.com/kisielk/errcheck/compare/v1.1.0...v1.2.0

Changelog:

Tested versions:

go 1.12.9
protoc 3.7.1
Improvements:

    plugin/stringer - Handle repeated and/or nullable types a bit better now.
    plugin/size - Remove the loop in sovXXX by using bit twiddling.
        Thanks: https://github.com/apelisse
    plugin/marshalto - Implemented a reverse marshal strategy which allows for faster marshalling. This now avoids a recursive (and repeated) call to Size().
        Thanks: https://github.com/apelisse
    plugin/compare - Added support for for oneof types.

Bug fixes:

    protoc-gen-gogo/generator - Fix assignment to entry in nil map.
        Thanks: https://github.com/tgulacsi
    protoc-gen-gogo/generator - Allows plugins to call RecordTypeUse without panicking.
        Thanks: https://github.com/fedenusy
    proto/extensions - Fixed set extension regression. We did not clear the extensions before setting.
    io/uint32 - fix uint32reader bug that causes ReadMsg to recreate buffer when lengths are the same.
        Thanks: https://github.com/SebiSujar
    proto/table_merge: Fix merge of non-nullable slices.
        Thanks: https://github.com/euroelessar

Upstream commits:

    merged in golang/protobuf commit 318d17de72 - Add UnimplementedServer for server interface
    merged in golang/protobuf commit b85cd75de7 - protoc-gen-go/grpc: inline errUnimplemented function
    merged in golang/protobuf commit d3c38a4eb4 - protoc-gen-go/grpc: use status and code packages only if needed
    merged in golang/protobuf commit e91709a02e - fix indentation in jsonpb with Any messages
    merged in golang/protobuf commit 8d0c54c124 - protoc-gen-go: generate XXX_OneofWrappers instead of XXX_OneofFuncs

Misc:

    extensions.md - Markdown update.
        Thanks: https://github.com/TennyZhuang
    Readme.md - Added user.
    go/protoc update - Updated to go1.12.x and protoc 3.7.1
    Makefile update - fix go vet shadow tool reference
    test/mixbench - Update mixbench tool. Expose runnable benchmarks via flags.

* update certstrap

* update golangci-lint from v1.13.2 to v1.17.1

* update gox as well

* test

* comment out golangci temporary

* comment out go-deadlock deps

* Add test for bcBlockRequestMessage ValidateBasic method

* Add test for bcNoBlockResponseMessage ValidateBasic method

* Add test for bcStatusRequestMessage ValidateBasic method

* Add test for bcStatusResponseMessage ValidateBasic method

* Add blockchain v1 reactor ValidateBasic tests

* Add test for NewRoundStepMessage ValidateBasic method

* Add test for NewValidBlockMessage ValidateBasic method

* Test BlockParts Size

* Import cmn package

* Add test for ProposalPOLMessage ValidateBasic method

* Add test for BlockPartMessage ValidateBasic method

* Add test for HasVoteMessage ValidateBasic method

* Add test for VoteSetMaj23Message ValidateBasic method

* Add test for VoteSetBitsMessage ValidateBasic method

* Fix linter errors

* Improve readability

* Add test for BaseConfig ValidateBasic method

* Add test for RPCConfig ValidateBasic method

* Add test for P2PConfig ValidateBasic method

* Add test for MempoolConfig ValidateBasic method

* Add test for FastSyncConfig ValidateBasic method

* Add test for ConsensusConfig ValidateBasic method

* Add test for InstrumentationConfig ValidateBasic method

* Add test for BlockID ValidateBasic method

* Add test for SignedHeader ValidateBasic method

* Add test for MockGoodEvidence and MockBadEvidence ValidateBasic methods

* Remove debug logging

Co-Authored-By: Marko <marbar3778@yahoo.com>

* Update MempoolConfig field

* Test a single struct field at a time, for maintainability

Fixes #2740

Fix #3868 (comment)

Commits:

* mempool: make `max_tx_bytes` configurable instead of `max_msg_bytes`

* update CHANGELOG_PENDING

* apply suggestions from code review

* config: move max_msg_bytes into mempool section

It was incorrectly placed in fastsync section during
4d7cd8055b (diff-092cdc48047eeb4c0bca311a2e1b8ae6)
merge.

Fixes #3868

* add changelog entry

* go routines in blockchain reactor

* Added reference to the go routine diagram

* Initial commit

* cleanup

* Undo testing_logger change, committed by mistake

* Fix the test loggers

* pulled some fsm code into pool.go

* added pool tests

* changes to the design

added block requests under peer

moved the request trigger in the reactor poolRoutine, triggered now by a ticker

in general moved everything required for making block requests smarter in the poolRoutine

added a simple map of heights to keep track of what will need to be requested next

added a few more tests

* send errors to FSM in a different channel than blocks

send errors (RemovePeer) from switch on a different channel than the
one receiving blocks
renamed channels
added more pool tests

* more pool tests

* lint errors

* more tests

* more tests

* switch fast sync to new implementation

* fixed data race in tests

* cleanup

* finished fsm tests

* address golangci comments :)

* address golangci comments :)

* Added timeout on next block needed to advance

* updating docs and cleanup

* fix issue in test from previous cleanup

* cleanup

* Added termination scenarios, tests and more cleanup

* small fixes to adr, comments and cleanup

* Fix bug in sendRequest()

If we tried to send a request to a peer not present in the switch, a
missing continue statement caused the request to be blackholed in a peer
that was removed and never retried.

While this bug was manifesting, the reactor kept asking for other
blocks that would be stored and never consumed. Added the number of
unconsumed blocks in the math for requesting blocks ahead of current
processing height so eventually there will be no more blocks requested
until the already received ones are consumed.

* remove bpPeer's didTimeout field

* Use distinct err codes for peer timeout and FSM timeouts

* Don't allow peers to update with lower height

* review comments from Ethan and Zarko

* some cleanup, renaming, comments

* Move block execution in separate goroutine

* Remove pool's numPending

* review comments

* fix lint, remove old blockchain reactor and duplicates in fsm tests

* small reorg around peer after review comments

* add the reactor spec

* verify block only once

* review comments

* change to int for max number of pending requests

* cleanup and godoc

* Add configuration flag fast sync version

* golangci fixes

* fix config template

* move both reactor versions under blockchain

* cleanup, golint, renaming stuff

* updated documentation, fixed more golint warnings

* integrate with behavior package

* sync with master

* gofmt

* add changelog_pending entry

* move to improvments

* suggestion to changelog entry

* mempool: make max_msg_bytes configurable

* apply suggestions from code review

* update changelog pending

* apply suggestions from code review again

* rpc: make max_body_bytes and max_header_bytes configurable

* update changelog pending

* Make RPC bind to localhost by default

* Add CHANGELOG_PENDING entry

* Allow testnet command to override RPC listen address

* Update localnet test to bind RPC to 0.0.0.0

* Update p2p test to bind RPC to 0.0.0.0

* Remove rpc-laddr parameter

* Update localnet to use config template with RPC listen address override

* Use config template override method for RPC listen address

* Build config template into localnode image

* Build localnode image locally before starting localnet

* Move testnet config overrides into templates

* Revert deletion of config overrides

* Remove extraneous config parameter overrides

* libs/db: conditional compilation

For cleveldb: go build -tags cleveldb
For boltdb: go build -tags boltdb

Fixes #3611

* document db_backend param better

* remove deprecated LevelDBBackend

* update changelog

* add missing lines

* add new line

* fix TestRemoteDB

* add a line about boltdb tag

* Revert "remove deprecated LevelDBBackend"

This reverts commit 1aa85453f7.

* make PR non breaking

* change DEPRECATED label format

https://stackoverflow.com/a/36360323/820520

* Remove deprecated PanicXXX functions from codebase

As per discussion over
[here](https://github.com/tendermint/tendermint/pull/3456#discussion_r278423492),
we need to remove these `PanicXXX` functions and eliminate our
dependence on them. In this PR, each and every `PanicXXX` function call
is replaced with a simple `panic` call.

* add a changelog entry

Refs #3419

* Make sure config.TimeoutBroadcastTxCommit < rpcserver.WriteTimeout()

* remove redundant comment

* libs/rpc/http_server: move Read/WriteTimeout into Config

* increase defaults for read/write timeouts

Based on this article
https://www.digitalocean.com/community/tutorials/how-to-optimize-nginx-configuration

* WriteTimeout should be larger than TimeoutBroadcastTxCommit

* set a deadline for subscribing to txs

* extract duration into const

* add two changelog entries

* Update CHANGELOG_PENDING.md

Co-Authored-By: melekes <anton.kalyaev@gmail.com>

* Update CHANGELOG_PENDING.md

Co-Authored-By: melekes <anton.kalyaev@gmail.com>

* 12 -> 10

* changelog

* changelog

* limit number of /subscribe clients and queries per client

Add the following config variables (under [rpc] section):
  * max_subscription_clients
  * max_subscriptions_per_client
  * timeout_broadcast_tx_commit

Fixes #2826

new HTTPClient interface for subscriptions

finalize HTTPClient events interface

remove EventSubscriber

fix data race

```
WARNING: DATA RACE
Read at 0x00c000a36060 by goroutine 129:
  github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe.func1()
      /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:168 +0x1f0

Previous write at 0x00c000a36060 by goroutine 132:
  github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe()
      /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:191 +0x4e0
  github.com/tendermint/tendermint/rpc/client.WaitForOneEvent()
      /go/src/github.com/tendermint/tendermint/rpc/client/helpers.go:64 +0x178
  github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync.func1()
      /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:139 +0x298
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 129 (running) created at:
  github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe()
      /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:164 +0x4b7
  github.com/tendermint/tendermint/rpc/client.WaitForOneEvent()
      /go/src/github.com/tendermint/tendermint/rpc/client/helpers.go:64 +0x178
  github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync.func1()
      /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:139 +0x298
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 132 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:878 +0x659
  github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync()
      /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:119 +0x186
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162
==================
```

lite client works (tested manually)

godoc comments

httpclient: do not close the out channel

use TimeoutBroadcastTxCommit

no timeout for unsubscribe

but 1s Local (5s HTTP) timeout for resubscribe

format code

change Subscribe#out cap to 1

and replace config vars with RPCConfig

TimeoutBroadcastTxCommit can't be greater than rpcserver.WriteTimeout

rpc: Context as first parameter to all functions

reformat code

fixes after my own review

fixes after Ethan's review

add test stubs

fix config.toml

* fixes after manual testing

- rpc: do not recommend to use BroadcastTxCommit because it's slow and wastes
Tendermint resources (pubsub)
- rpc: better error in Subscribe and BroadcastTxCommit
- HTTPClient: do not resubscribe if err = ErrAlreadySubscribed

* fixes after Ismail's review

* Update rpc/grpc/grpc_test.go

Co-Authored-By: melekes <anton.kalyaev@gmail.com>

* make BlockTimeIota a consensus parameter, not a locally configurable option

Refs #2920

* make TimeIota int64 ms

Refs #2920

* update Gopkg.toml

* fixes after Ethan's review

* fix TestRemoteSignerProposalSigningFailed

* update changelog

* bound mempool memory usage

Closes #3079

* rename SizeBytes to TxsTotalBytes

and other small fixes after Zarko's review

* rename MaxBytes to MaxTxsTotalBytes

* make ErrMempoolIsFull more informative

* expose mempool's txs_total_bytes via RPC

* test full response

* fixes after Ethan's review

* config: rename mempool.size to mempool.max_txs

https://github.com/tendermint/tendermint/pull/3248#discussion_r254034004

* test more cases

https://github.com/tendermint/tendermint/pull/3248#discussion_r254036532

* simplify test

* Revert "config: rename mempool.size to mempool.max_txs"

This reverts commit 39bfa36961.

* rename count back to n_txs

to make a change non-breaking

* rename max_txs_total_bytes to max_txs_bytes

* format code

* fix TestWALPeriodicSync

The test was sometimes failing due to processFlushTicks being called too
early. The solution is to call wal#Start later in the test.

* Apply suggestions from code review

* improve ResetTestRootWithChainID() concurrency safety

Rely on ioutil.TempDir() to create test root directories and ensure
multiple same-chain id test cases can run in parallel.

* Update config/toml.go

Co-Authored-By: alessio <quadrispro@ubuntu.com>

* clean up test directories after completion

Closes: #1034

* Remove redundant EnsureDir call

* s/PanicSafety()/panic()/s

* Put create dir functionality back in ResetTestRootWithChainID

* Place test directories in OS's tempdir

In modern UNIX and UNIX-like systems /tmp is very often
mounted as tmpfs. This might speed test execution a bit.

* Set 0700 to a const

* rootsDirs -> configRootDirs

* Don't double remove directories

* Avoid global variables

* Fix consensus tests

* Reduce defer stack

* Address review comments

* Try to fix tests

* Update CHANGELOG_PENDING.md

Co-Authored-By: alessio <quadrispro@ubuntu.com>

* Update consensus/common_test.go

Co-Authored-By: alessio <quadrispro@ubuntu.com>

* Update consensus/common_test.go

Co-Authored-By: alessio <quadrispro@ubuntu.com>

* add ResetTestRootWithChainID

* modify chainid

* split immutable and mutable parts of priv_validator.json

* fix bugs

* minor changes

* retrig test

* delete scripts/wire2amino.go

* fix test

* fixes from review

* privval: remove mtx

* rearrange priv_validator.go

* upgrade path

* write tests for the upgrade

* fix for unsafe_reset_all

* add test

* add reset test

* config: cors options are arrays of strings, not strings

Fixes #2980

* docs: update tendermint-core/configuration.html page

* set allow_duplicate_ip to false

* in `tendermint testnet`, set allow_duplicate_ip to true

Refs #2712

* fixes after Ismail's review

* config: cors options are arrays of strings, not strings

Fixes #2980

* docs: update tendermint-core/configuration.html page

* set allow_duplicate_ip to false

* in `tendermint testnet`, set allow_duplicate_ip to true

Refs #2712

* fixes after Ismail's review

* Revert "set allow_duplicate_ip to false"

This reverts commit 24c1094ebc.

Refs #2877

* Introduce a structured logging option

* rename StructuredLog to LogFormat

* add changelog entry

* move log_format under log_level

* only invoke InitWAL/CloseWAL if WalPath is not empty

Closes #2717

* panic if WAL is not initialized when calling CloseWAL

* add a changelog entry

* validate reactor messages

Refs #2683

* validate blockchain messages

Refs #2683

* validate evidence messages

Refs #2683

* todo

* check ProposalPOL and signature sizes

* add a changelog entry

* check addr is valid when we add it to the addrbook

* validate incoming netAddr (not just nil check!)

* fixes after Bucky's review

* check timestamps

* beef up block#ValidateBasic

* move some checks into bcBlockResponseMessage

* update Gopkg.lock

Fix

```
grouped write of manifest, lock and vendor: failed to export github.com/tendermint/go-amino: fatal: failed to unpack tree object 6dcc6ddc14
```

by running `dep ensure -update`

* bump year since now we check it

* generate test/p2p/data on the fly using tendermint testnet

* allow sync chains older than 1 year

* use full path when creating a testnet

* move testnet gen to test/docker/Dockerfile

* relax LastCommitRound check

Refs #2737

* fix conflicts after merge

* add small comment

* some ValidateBasic updates

* fixes

* AppHash length is not fixed

* Fix termination issues and improve tests

* Improve formatting and tests based on reviewer feedback

* require block.Time of the fist block to be genesis time

Refs #2587:

```
We only start validating block.Time when Height > 1, because there is no
commit to compute the median timestamp from for the first block. This
means a faulty proposer could make the first block with whatever time
they want.

Instead, we should require the timestamp of block 1 to match the genesis
time.

I discovered this while refactoring the ValidateBlock tests to be
table-driven while working on tests for #2560.
```

* do not accept blocks with negative height

* update changelog and spec

* nanos precision for test genesis time

* Fix failing test (#2607)

* specify time unit for FlushThrottleTimeout in TestP2PConfig
Refs #2555
* [tm-bench] refactor code
https://github.com/tendermint/tendermint/pull/2405#pullrequestreview-157166387

* Disable transitioning to new round upon 2/3+ of Precommit nils

Pull in ensureVote test function from https://github.com/tendermint/tendermint/pull/2132

* Add several ensureX test methods to wrap channel read with timeout

* Revert panic in tests

* timeouts as time.Duration are also breaking for old configs
* split BaseConfig#ValidateBasic into smaller methods

* remove some XXX

* config: remove Mempool.RecheckEmpty

* docs: remove recheck_empty

* tmlibs -> libs

* update changelog

* address some comments from review of #2298

* add missing options to config.toml template and docs
Refs #2232
* config#ValidateBasic
Refs #2232
* [config] timeouts as time.Duration, not ints
Why:
- native type provides better guarantees than ", in ms" comment (harder
to shoot yourself in the leg)
- flexibility: you can change units

* Add additional metrics to p2p and consensus
Partially addresses https://github.com/cosmos/cosmos-sdk/issues/2169.
* WIP
* Updates from code review
* Updates from code review
* Add instrumentation namespace to configuration
* Fix test failure
* Updates from code review
* Add quotes
* Add atomic load
* Use storeint64
* Use addInt64 in writePacketMsgTo

Why:
original fork is abandoned and not supported anymore.

Changes:
- LevelDB 1.19 (LevelDB and Snappy are both compiled and linked statically, so while you will not need them installed on your target machine, you should have a roughly compatible version of libstdc++.)
- snappy and lz4 libs included by default

* addr_book_strick=false on local nets

* link to spec

* spec: remove TODO, see #1749 instead

* spec: make issues from TODOs

* update docs on addr_book_strict option

* Implement BFT time

* set LastValidators when creating state in state helper

for heights >= 2

* config: reduce default mempool size

This reduces the mempool size from 100k to 5k. Note that each secp256k1 sig
takes .5ms to compute. Therefore an adversary could previously delay every
node on the network's computation time upon receiving a block by 50 seconds.

This now reduces that ability to being able to only delay each node by 2.5
seconds. This change should be reverted once ABCI recheck is implemented.

* (squash this) fix test

* db_path is not being parsed

Fix default configuration, db_path is now db_dir.

Closes: cosmos/cosmos-sdk#1712

* Update CHANGELOG_PENDING.md

* [p2p/pex] connect to more than 10 peers

also, remove DefaultMinNumOutboundPeers because a) I am not sure it's
needed b) it's super confusing

look closely

```
maxPeers := sw.config.MaxNumPeers - DefaultMinNumOutboundPeers
if maxPeers <= sw.peers.Size() {
  sw.Logger.Info("Ignoring inbound connection: already have enough peers", "address", inConn.RemoteAddr().String(), "numPeers", sw.peers.Size(), "max", maxPeers)
```

we print maxPeers = config.MaxPeers - DefaultMinNumOutboundPeers. So we
may not have enough peers even though we say we have enough.

Refs #2130

* update spec

* replace MaxNumPeers with MaxNumInboundPeers/MaxNumOutboundPeers

Refs #2130

* update changelog

* make max rpc conns formula visible to users

* update spec

* docs: note max outbound peers excludes persistent

also, add `priv_validator_laddr` to the template

Currently the top level directory contains basically all of the code
for the crypto package. This PR moves the crypto code into submodules
in a similar manner to what `golang/x/crypto` does. This improves code
organization.

Ref discussion: https://github.com/tendermint/tendermint/pull/1966

Closes #1956

* Add private peer ID tracking to AddrBook

* Remove private peer tracking/blocking from pex

* debug level msg when we fail to add private address