* libs/common: refactor libs/common 2

- move random function to there own pkg

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* change imports and usage throughout repo

* fix goimports

* add changelog entry

* libs/common: Refactor libs/common 01

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* regenerate proto files, move intslice to where its used

* update kv.KVPair(s) to kv.Pair(s)

* add changelog entry

* make intInSlice private

* sr25519

* added amino encoding

* fixed dependencies

* Apply suggestions from code review

Co-Authored-By: Tess Rinearson <tess.rinearson@gmail.com>
Co-Authored-By: Marko <marbar3778@yahoo.com>

* file structure

* Apply suggestions from code review

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* address @melekes and @marbar3778 review

* removed nolint

* CHANGELOG and go-schnorrkel mod update

* Rename Tag(s) to Event(s)

- tag was replaced with event, but in some places it still mentions tag, would be easier to understand if we tried to replace it with event to not confuse people.

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* more  changes from tag -> event

* rename events to compositeKeys and keys

* Apply suggestions from code review

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* add minor documentation on how composite keys are constructed

* rename eventkey to compositekey

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* add changelog entry & add info to regenerate confid to changelog entry

implementation spec of Improved Trusted Peering ADR-050 by B-Harvest

- add unconditional_peer_ids and persistent_peers_max_dial_period to config
- add unconditionalPeerIDs map to Switch struct

default config value of persistent_peers_max_dial_period is 0s(disabled)

Refs #4072, #4053

Closes #4065

Fix for #4164
The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed.
Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower.

Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied.

Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update.

## Commits:

* tests for overflow detection and prevention

* test fix

* more tests

* fix the false overflow warnings and golint

* scopelint warning fix

* review comments

* variant with using sort by amount of change in power

* compute separately number new validators in update

* types: use a switch in processChanges

* more review comments

* types: use HasAddress in numNewValidators

* types: refactor verifyUpdates

copy updates, sort them by delta and use resulting slice to calculate
tvpAfterUpdatesBeforeRemovals.

* remove unused structs

* review comments

* update changelog

* p2p/conn: simplify secret connection handshake malleability fix with merlin

Introduces new dependencies on github.com/gtank/merlin and sha3 as a cryptographic primitive

This also only uses the transcript hash as a MAC.

* p2p/conn: avoid string to byte conversion

https://github.com/uber-go/guide/blob/master/style.md#avoid-string-to-byte-conversion

* types: change `Commit` to consist of just signatures

These are final changes towards removing votes from commit and leaving
only signatures (see ADR-25)

Fixes #1648

* bring back TestCommitToVoteSetWithVotesForAnotherBlockOrNilBlock

+ add absent flag to Vote to indicate that it's for another block

* encode nil votes as CommitSig with BlockIDFlagAbsent

+ make Commit#Precommits array of non-pointers
because precommit will never be nil

* add NewCommitSigAbsent and Absent() funcs

* uncomment validation in CommitSig#ValidateBasic

* add comments to ValidatorSet funcs

* add a changelog entry

* break instead of continue

continue does not make sense in these cases

* types: rename Commit#Precommits to Signatures

* swagger: fix /commit response

* swagger: change block_id_flag type

* fix merge conflicts

https://github.com/tendermint/tendermint/pull/4153#discussion_r348868269

Fixes #3986

This pull request is prefixing all the types in proto to avoid conflict.

When a go application is using Tendermint as a library and also define similar types in gogo proto some conflicts might occur (as types is a common package in go).

By prefixing the types with tendermint, this highly reduces the risk of conflicts.

BREAKING CHANGE.

This modification breaks the ABCI Application endpoint.

What was accessible before with `/types.ABCIApplication/Flush` is now accessible with `/tendermint.abci.types.ABCIApplication/Flush`.

Fixes #4069

Closes #4164

- tm-bench has a deprecation warning for 5 releases now, with the major release coming I removed the file and updated the docs to point to `tm-load-test` located in the interchainio repo

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

- update master with changelog from v0.32.8
- bump `tm-version` in `version.go`

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

*libs/common/errors: remove package

- remove errors file from cmn pkg

- use errorf instead of wrap in async function

- add changelog entry

- closes #3862

Fixes #4143

Co-Authored-By: Marko <marbar3778@yahoo.com>

Refs #3262

https://www.jsonrpc.org/specification

What is done in this PR:

    JSONRPCClient: validate that Response.ID matches Request.ID I wanted
    to do the same for the WSClient, but since we're sending events as
    responses, not notifications, checking IDs would require storing
    them in memory indefinitely (and we won't be able to remove them
    upon client unsubscribing because ID is different then).

    Request.ID is now optional. Notification is a Request without an ID.
    Previously "" or 0 were considered as notifications

    Remove #event suffix from ID from an event response (partially fixes
    #2949) ID must be either string, int or null AND must be equal to
    request's ID. Now, because we've implemented events as responses, WS
    clients are tripping when they see Response.ID("0#event") !=
    Request.ID("0"). Implementing events as requests would require a lot
    of time (~ 2 days to completely rewrite WS client and server)

    generate unique ID for each request

    switch to integer IDs instead of "json-client-XYZ"
    id=0 method=/subscribe
    id=0 result=...
    id=1 method=/abci_query
    id=1 result=...

> send events (resulting from /subscribe) as requests+notifications (not
responses)

this will require a lot of work. probably not worth it

* rpc: generate an unique ID for each request
in conformance with JSON-RPC spec

* WSClient: check for unsolicited responses

* fix golangci warnings

* save commit

* fix errors

* remove ID from responses from subscribe
Refs #2949

* clients are safe for concurrent access

* tm-bench: switch to int ID

* fixes after my own review

* comment out sentIDs in WSClient
see commit body for the reason

* remove body.Close
it will be closed automatically

* stop ws connection outside of write/read routines
also, use t.Rate in tm-bench indexer when calculating ID

fix gocritic issues

* update swagger.yaml

* Apply suggestions from code review

* fix stylecheck and golint linter warnings

* update changelog

* update changelog2

* Add pagination to /validators

- closes #3472

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* add swagger params, default returns all

* address pr comments

* golint fix

* swagger default change, change to default in comment

* swagger.yaml: replace x-example with example

https://swagger.io/docs/specification/adding-examples/

* Revert "swagger.yaml: replace x-example with example"

This reverts commit 9df1b006de.

* update changelog and remove extra body close

## Issue:

    Removed BlockMeta in ResultBlock in favor of BlockId for /block
    Added block_size to BlockMeta this is reflected in /blockchain

fixes #3188

added breaking as some clients may be using header from blockmeta instead of block in /block

## Commits:

* cleanup block path

Remove duplication of data in `/block`

fixes #3188

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* Remove duplication of data in `/block`

- Created a secondary type to be used for /block

fixes #3188

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* remove commented out code

* fix ci

* add changelog_pending entry

* remove extra variable

* update swagger

* change int to uint for blocksize

* fix swagger

* remove extensive comments

* update changelog

* fix conflicts after merge

* use int for BlockSize and NumTxs in BlockMeta

- with 99.9% guarantee, the size of either will never reach int32
- most of the Go "Size" stdlib functions return int

* Removal of TotalTx & NumTx

- Removed totalTx and numTx

closes #2521

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* abci proto changes

* proto number fix

* txfilter_test fix

* comments on PR

* further changes

* bring back metrics

* fix indexer

* fix TestBlockMaxDataBytes and TestBlockMaxDataBytesUnknownEvidence

* indexer service back to header

* statistics.go fix

* fix ci

* listen for blocks, not headers

to be able to record txs throughput

* fix TestNetworkNewBlock

* fix tests

* fix tests in types package

* fixes after Anton's review

* fix tests

* bring back `consensus_total_txs` metric

I mistakenly thought it was removed.

* improve changelog

* remove LastBlockTotalTx from state

* docs: remove getNumTxs from BeginBlock Java example

## Issue:

This is an approach to fixing secret connection that is more noise-ish than actually noise.

but it essentially fixes the problem that #3315 is trying to solve by making the secret connection handshake non-malleable. It's easy to understand and I think will be acceptable to @jaekwon

.. the formal reasoning is basically, if the "view" of the transcript between diverges between the sender and the receiver at any point in the protocol, the handshake would terminate.

The base protocol of Station to Station mistakenly assumes that if the sender and receiver arrive at shared secret they have the same view. This is only true for a DH on prime order groups.

This robustly solves the problem by having each cryptographic operation commit to operators view of the protocol.

Another nice thing about a transcript is it provides the basis for "secure" (barring cryptographic breakages, horrible design flaws, or implementation bugs) downgrades, where a backwards compatible handshake can be used to offer newer protocol features/extensions, peers agree to the common subset of what they support, and both sides have to agree on what the other offered for the transcript MAC to verify.

With something like Protos/Amino you already get "extensions" for free (TLS uses a simple TLV format https://tools.ietf.org/html/rfc8446#section-4.2 for extensions not too far off from Protos/Amino), so as long as you cryptographically commit to what they contain in the transcript, it should be possible to extend the protocol in a backwards-compatible manner.

## Commits:

* Minimal changes to remove malleability of secret connection removes the need to check for lower order points.

Breaks compatibility. Secret connections that have no been updated will fail

* Remove the redundant blacklist

* remove remainders of blacklist in tests to make the code compile again

Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* Apply suggestions from code review

Apply Ismail's error handling

Co-Authored-By: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* fix error check for io.ReadFull

Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* Update p2p/conn/secret_connection.go

Co-Authored-By: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* Update p2p/conn/secret_connection.go

Co-Authored-By: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com>

* update changelog and format the code

* move hkdfInit closer to where it's used

BREAKING

Example response:

```json
{
  "jsonrpc": "2.0",
  "id": "",
  "result": {
    "height": "2109",
    "txs_results": null,
    "begin_block_events": null,
    "end_block_events": null,
    "validator_updates": null,
    "consensus_param_updates": null
  }
}
```

Old result consisted of ABCIResponses struct and height. Exposing
internal ABCI structures (which we store in state package) in RPC seems
bad to me for the following reasons:

1) high risk of breaking the API when somebody changes internal structs
(HAPPENED HERE!)
2) RPC is aware of ABCI, which I'm not sure we want

Fixes #4051 

Function `parseRemoteAddr` is forcing protocol HTTP and protocol HTTPs to tcp. This causes the bug in the issue #4051.

I find that the tcp is only needed where `net.Dial`. So I moved the switch to makeHTTPDialer.

## Issue:

Hey, not sure if this is disallowed for any reason specifically, but it would be very beneficial to define additional types to decode tendermint key implementations from bytes, since it uses a static codec. If this is okay, let me know and I will add documentation.

Context: For Ethermint to switch to using Cosmos' keybase, decoding the keys requires this codec to be updated

Just to document, I did experiment with creating a mapping from string to objects to be able to keep track of the key types added to be able to be used in the RegisterAmino(..) call, but because of how go is compiled, cosmos would just use the base types. This may be a useful feature for someone just building on top of Tendermint and not going through Cosmos, but to not add confusion or unnecessary complexity, I left it out.

## Commits:

* Exposes amino codec to be able to decode pk bytes in application

* Change how codec is modified

* Remove unneeded comment

* Fix comment

* Fix comment

* Add registered type to nametable

* Add pending changelog entry

* Reorder change

* Added check if type is registered and added test

* Make test type private

* Remove unnecessary duplicate exists check

Added a small function to be able to change the default retry interval for the privval. The default is 100ms, this function allows to change to any time.Duration.

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

## Issue:

This PR adds an "EXISTS" condition to the event query grammar. It enables querying for the occurrence of an event without having to provide a condition for one of its attributes.

As an example, someone interested in all slashing events might currently catch them with a query such as slash.power > 0.

With this PR the event can be captured with slash.power EXISTS or just slash EXISTS to catch by event type.

## Examples:

`slash EXISTS`

## Commits:

* Add EXISTS condition to query grammar

* Gofmt files

* Move PEG instructions out of auto-generated file to prevent overwrite

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update changelog and add test case

* Merge with other changes in PR #4070

* Add EXISTS to Conditions() func

* Apply gofmt

* Addressing PR comments

when the user searches for a tx (hash=X)

This PR fixes error handling for performing a txindex search.

TxIndex.Get returns

    (txresult, nil) if the transaction is found.
    (nil, nil) if the transaction is not found.
    (nil, error) if error is occurred.

Therefore, if res is not nil, I think TxIndex.Search should return (txresult, nil).

Previously, however, this was not a problem because errors.Wrap returns nil if its first argument err is nil.

* mempool: moved TxInfo parameter into Mempool.CheckTx().

* Updated CHANGELOG_PENDING.md

* Added PR issue to CHANGELOG_PENDING

Fixes #3590

## Issue

Why this pr:
When restarting chain node, sometimes we lost tx index about recent(around 80)blocks, and some client complains that they can't find the tx by RPC call(tx_search) when the tx do exist in the block.

I try to partially fix this issue in a simple way by writing the index data in a sync way.
There is no performance difference under 1K TPS according to our test.

It is still possible that lost index data after restarting the node, but only 2 block data will lost at most.
I try to totally fix this in https://github.com/tendermint/tendermint/pull/3847/files, but this one is simple and can solve most part of the issue. Please review first, thks.

## Comments

Anton:

BEFORE:

BenchmarkTxIndex1-2               100000             12434 ns/op
BenchmarkTxIndex500-2                300           5151564 ns/op
BenchmarkTxIndex1000-2               100          15053910 ns/op
BenchmarkTxIndex2000-2               100          18238892 ns/op
BenchmarkTxIndex10000-2               20         124287930 ns/op

AFTER:

BenchmarkTxIndex1-2                 2000            795431 ns/op
BenchmarkTxIndex500-2                200           6385124 ns/op
BenchmarkTxIndex1000-2               100          11388219 ns/op
BenchmarkTxIndex2000-2               100          20514873 ns/op
BenchmarkTxIndex10000-2               20         107456004 ns/op

Performance drop is pretty steep, but I think it's the right thing to do UNTIL we have a WAL.

* cs: panic only when WAL#WriteSync fails

- modify WAL#Write and WAL#WriteSync to return an error

* fix test

* types: validate Part#Proof

add ValidateBasic to crypto/merkle/SimpleProof

* cs: limit max bit array size and block parts count

* cs: test new limits

* cs: only assert important stuff

* update changelog and bump version to 0.32.7

* fixes after Ethan's review

* align max wal msg and max consensus msg sizes

* fix tests

* fix test

* add change log for 31.11

Some linting/cleanup missed from the initial events refactor
Don't panic; instead, return false, error when matching breaks unexpectedly
Strip non-numeric chars from values when attempting to match against query values
Have the server log during send upon error

* cleanup/lint Query#Conditions and do not panic

* cleanup/lint Query#Matches and do not panic

* cleanup/lint matchValue and do not panic

* rever to panic in Query#Conditions

* linting

* strip alpha chars when attempting to match

* add pending log entries

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* build: update variable names

* update matchValue to return an error

* update Query#Matches to return an error

* update TestMatches

* log error in send

* Fix tests

* Fix TestEmptyQueryMatchesAnything

* fix linting

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/pubsub.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* add missing errors pkg import

* update Query#Conditions to return an error

* update query pkg unit tests

* update TxIndex#Search

* update pending changelog

Fixes #3991

* Include sender when logging rejected txns

* Log as peerID to be consistent with other log messages

* Updated CHANGELOG_PENDING

* Handle nil source

* Updated PR link in CHANGELOG_PENDING

* Renamed TxInfo.SenderAddress and peerAddress til PeerFullID

* Renamed PeerFullID to PeerP2PID

* Forgot to rename a couple of references

also replace TODO placeholder with the actual issue in v0.32.5

* tm-monitor: tweaked formatting of start time and avg tx throughput.

* tm-monitor: update health when validator number is updated.

* Updated CHANGELOG_PENDING

* Added PR number to CHANGELOG_PENDING.

Improves `tm-monitor` formatting of start time (RFC1123 without unnecessary precision) and avg tx throughput (three decimal places). The old tx throughput display was confusing during local testing where the tx rate is low and displayed as 0.

Also updates the monitor health whenever the validator number changes. It otherwise starts with moderate health and fails to update this once it discovers the validators, leading to incorrect health reporting and invalid uptime statistics. Let me know if you would like me to submit this as a separate PR.

### Before:

```
2019-09-29 20:40:00.992834 +0200 CEST m=+0.024057059 up -92030989600.42%

Height: 2518
Avg block time: 1275.496 ms
Avg tx throughput: 0 per sec
Avg block latency: 2.464 ms
Active nodes: 4/4 (health: moderate) Validators: 4

NAME                HEIGHT     BLOCK LATENCY     ONLINE     VALIDATOR     
localhost:26657     2518       0.935 ms          true       true          
localhost:26660     2518       0.710 ms          true       true          
localhost:26662     2518       0.708 ms          true       true          
localhost:26664     2518       0.717 ms          true       true          
```

### After:

```
Sun, 29 Sep 2019 20:21:59 +0200 up 100.00%

Height: 2480
Avg block time: 1361.445 ms
Avg tx throughput: 0.735 per sec
Avg block latency: 4.232 ms
Active nodes: 4/4 (health: full) Validators: 4

NAME                HEIGHT     BLOCK LATENCY     ONLINE     VALIDATOR     
localhost:26657     2480       1.174 ms          true       true          
localhost:26660     2480       1.037 ms          true       true          
localhost:26662     2480       0.981 ms          true       true          
localhost:26664     2480       0.995 ms          true       true          
```

- Release candidate for v0.32.4

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* Remove omitempty from *pb.go

- remove omitempty from *pb.go files
- added command to makefile for everytime `make protoc_all` is run

- open question:
  - Do we want to further remove omitempty from other places
	- https://github.com/tendermint/tendermint/blob/master/rpc/lib/types/types.go#L151
	- and other places
ref #3882

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* bring back omitempty to *pb.go

* Update types/tx.go

* custom marshlers

* undo benchmark `omitepmty`

* golangci lint fix

* cleanup comments

* changelog_pending entry

* Add MempoolClient to Client Interface

closes #3984

- add mempoolclient interface to client interface

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* Update CHANGELOG_PENDING.md

* add golang-ci version

* Correct memory alignment for 32-bit machine. 

Switching the `txsBytes` and `rechecking` fields of `CListMempool` to ensure the correct memory alignment for `atomic.LoadInt64` on 32-bit machine.

* Update CHANGELOG_PENDING.md for #3968

Fixed #3968 `mempool` Memory Loading Error on 32-bit Ubuntu 16.04 Machine.

* Update CHANGELOG_PENDING.md

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* cs: check for SkipTimeoutCommit or wait timeout in handleTxsAvailable

Previously, if create_empty_blocks was set to false, TM sometimes could
create 2 consecutive blocks without actually waiting for timeoutCommit
(1 sec. by default).

```
I[2019-08-29|07:32:43.874] Executed block                               module=state height=47 validTxs=10 invalidTxs=0
I[2019-08-29|07:32:43.876] Committed state                              module=state height=47 txs=10 appHash=F88C010000000000
I[2019-08-29|07:32:44.885] Executed block                               module=state height=48 validTxs=2 invalidTxs=0
I[2019-08-29|07:32:44.887] Committed state                              module=state height=48 txs=2 appHash=FC8C010000000000
I[2019-08-29|07:32:44.908] Executed block                               module=state height=49 validTxs=8 invalidTxs=0
I[2019-08-29|07:32:44.909] Committed state                              module=state height=49 txs=8 appHash=8C8D010000000000
I[2019-08-29|07:32:45.886] Executed block                               module=state height=50 validTxs=2 invalidTxs=0
I[2019-08-29|07:32:45.895] Committed state                              module=state height=50 txs=2 appHash=908D010000000000
I[2019-08-29|07:32:45.908] Executed block                               module=state height=51 validTxs=8 invalidTxs=0
I[2019-08-29|07:32:45.909] Committed state                              module=state height=51 txs=8 appHash=A08D010000000000
```

This commit fixes that by adding a check to handleTxsAvailable.

Fixes #3908

* update changelog

* schedule timeoutCommit if StartTime is in the future

or SkipTimeoutCommit=true && we DON'T have all the votes

* fix TestReactorCreatesBlockWhenEmptyBlocksFalse

by checking if we have LastCommit or not

* address Ethan's comments

fixes #2010

* allow using a custom http client in rpc client

* add tests and fix bug

* fix confusion between remote and address

* parse remote inside NewJSONRPCClientWithHTTPClient

* cleanups

* add warnings

* add changelog entry

* Update CHANGELOG_PENDING.md

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

When using the RPC client in my test suite (with -race enabled), I do a lot of Subscribe/Unsubscribe operations, at some point (randomly) the race detector returns the following warning:

WARNING: DATA RACE
Read at 0x00c0009dbe30 by goroutine 31:
runtime.mapiterinit()
/usr/local/go/src/runtime/map.go:804 +0x0
github.com/tendermint/tendermint/rpc/client.(*WSEvents).redoSubscriptionsAfter()
/go/pkg/mod/github.com/tendermint/tendermint@v0.31.5/rpc/client/httpclient.go:364 +0xc0
github.com/tendermint/tendermint/rpc/client.(*WSEvents).eventListener()
/go/pkg/mod/github.com/tendermint/tendermint@v0.31.5/rpc/client/httpclient.go:393 +0x3c6 

Turns out that the redoSubscriptionAfter is not protecting the access to subscriptions.

The following change protects the read access to the subscription map behind the mutex