* node: fix a bug where `nil` is recorded as node's address

Solution

  AddOurAddress when we know it
  sw.NetAddress is nil in createAddrBookAndSetOnSwitch
  it's set by n.transport.Listen function, which is called during start

Fixes #3716

* use addr instead of n.sw.NetAddress

* add both ExternalAddress and ListenAddress as our addresses

In Python 3 the command outlined in this doc `import codecs; codecs.decode("YWJjZA==", 'base64').decode('ascii')` throws an error:
TypeError: decoding with 'base64' codec failed (TypeError: expected bytes-like object, not str), needs to add 'b' before the encoded string
`import codecs; codecs.decode(b"YWJjZA==", 'base64').decode('ascii')` to make it work

Follow up to #3643

* update changelog

* do not ignore errors returned by PublishWithEvents

## PR

This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events.

    Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like:

"rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}]
"sender": [{"address": "...", "balance": "100uatom"}]

The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...'

    ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair.
    PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings.

closes: #1859
closes: #2905

## Commits:

* Implement Event ABCI type and updates responses to use events

* Update messages_test.go

* Update kvstore.go

* Update event_bus.go

* Update subscription.go

* Update pubsub.go

* Update kvstore.go

* Update query logic to handle slice of strings in events

* Update Empty#Matches and unit tests

* Update pubsub logic

* Update EventBus#Publish

* Update kv tx indexer

* Update godocs

* Update ResultEvent to use slice of strings; update RPC

* Update more tests

* Update abci.md

* Check for key in validateAndStringifyEvents

* Fix KV indexer to skip empty keys

* Fix linting errors

* Update CHANGELOG_PENDING.md

* Update docs/spec/abci/abci.md

Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com>

* Update abci/types/types.proto

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* Update docs/spec/abci/abci.md

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* Update libs/pubsub/query/query.go

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* Update match function to match if ANY value matches

* Implement TestSubscribeDuplicateKeys

* Update TestMatches to include multi-key test cases

* Update events.go

* Update Query interface godoc

* Update match godoc

* Add godoc for matchValue

* DRY-up tx indexing

* Return error from PublishWithEvents in EventBus#Publish

* Update PublishEventNewBlockHeader to return an error

* Fix build

* Update events doc in ABCI

* Update ABCI events godoc

* Implement TestEventBusPublishEventTxDuplicateKeys

* Update TestSubscribeDuplicateKeys to be table-driven

* Remove mod file

* Remove markdown from events godoc

* Implement TestTxSearchDeprecatedIndexing test

* Update to using go mod from dep

* Remove references to make get_vendor_deps

* Specify go version

* Set GO111MODULE=on and add -mod=readonly

* Fix exported env

* switch to using go1.12 everywhere

* Fix test scripts

* Typo:

* Prepend GO111MODULE=on

* remove dep cache

* Revert "remove dep cache"

This reverts commit 45117bda

Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* bring back the dependency cache and change it to cache modules instead
of vendored deps; also:

 - bump version for dependency cache
 - bump version on pkg-cache (includes modules directory)

Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* remove some more traces of dep:
 - remove Gopkg.(toml | lock)
 - update contributing guidlines
 - set global default in circleci (GO111MODULE=on)

Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* global var failed for `test_cover` with
`go: unknown environment setting GO111MODULE=true`
although the var was `GO111MODULE: on`

Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* Changelog pending entry

Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* Add bbolt dependency to go.mod

Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>

* move -mod=readonly to build flags

Fix fundemental BoltDB backend correctness issues.

BoltDB's accessors will return slices that are only valid for the
lifetime of the transaction.  This adds copies where required to prevent
hard to debug crashes (among other things).

`Delete` should queue a delete operation that targets the entire
database, and not just keys that happen to be in the current batch.

* Move peer behaviour into it's own package
* refactor wip
* Adjust API and fix tests
* remove unused test struct
* Better error message
* Restructure:
    + Now behaviour is it's own package, we don't need to include
    PeerBehaviour in every type.
    + Split up behaviours and reporters into seperate files

* doc string fixes
* Fix minor typos
* Update behaviour/reporter.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update behaviour/reporter.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

Fixes #3521

The function NewNetAddressStringWithOptionalID is from a time when peer
IDs were optional. They're not anymore. So this should be renamed to
NewNetAddressString and should ensure the ID is provided.

* update changelog
* use NewNetAddress in transport tests
* use NewNetAddress in TestTransportMultiplexAcceptMultiple

So we can have an error code when rpc fails.

* Update CHANGELOG_PENDING.md
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* wrap correct error
* format code

* Add `chID` label to sent/receive byte mtrics
* add changelog pending entry

Fixes #3616

V0.31

Release/v0.31.7

Fixes #3678

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

Otherwise, we'll be trying to include them in each consecutive block.

The downside is that evil proposers will be able to drop valid
transactions (see #3322).

Reverts https://github.com/tendermint/tendermint/pull/3625

Fixes #3699

Release/v0.31.6

* Move GenesisDocProvider and DefaultGenesisDocProviderFunc

GenesisDocProvider, being a provider of *types.GenesisDoc, makes sense
to be part of the types package.

DefaultGenesisDocProviderFunc, which relies on *config.Config to produce
a types.GenesisDocProvider, makes sense being part of the config
package.

* Add aliases to avoid breaking node package API

* Revert to original structure

After discussion, it appears as though the best place for the relocated
structures is still in the node package. This means that for the v0.31.6
release and into the future, there will be no changes two these two
entities' APIs.

* update changelog

* Update changelog

- less detail about internal stuff like the `mempool`
- focus BUG FIX entries more on what was fixed rather than how

* minor cleanup.

- remove entry for GenesisDocProvider, it's being undone

* minor fix

* Peer behaviour test tweaks:

    Address the remaining test issues mentioned in ##3552 notably:
    + switch to p2p_test package
    + Use `Error` instead of `Errorf` when not using formatting
    + Add expected/got errors to
    `TestMockPeerBehaviourReporterConcurrency` test

* Peer behaviour equal behaviours test

    + slices of PeerBehaviours should be compared as histograms to
    ensure they have the same set of PeerBehaviours at the same
    freequncy.

* TestEqualPeerBehaviours:

    + Add tests for the equivalence between sets of PeerBehaviours

* Allow testnet hostnames to be overridden

This allows one to specify the `--hostname` flag multiple times, each
time providing an additional custom hostname for a respective peer
(validator or non-validator). This overrides any of the
`--hostname-prefix` or `--starting-ip-address` flags.

The string array approach is taken instead of the string slice approach
(see the pflag docs:
https://godoc.org/github.com/spf13/pflag#StringArray) because the string
slice approach (a comma-separated string) doesn't allow for cleaner
multi-line BASH scripts - where this feature is intended to be used.

* Reorder conditional for clarity with simpler earlier return

* Allow for specifying peer hostname suffix

* Quote values in help strings for greater clarity

* Fix command switch

* Add CHANGELOG_PENDING entry for PR

* Allow for unique monikers

The current approach to generating monikers for testnet nodes assigns
the local hostname of the machine on which the testnet config was
generated to all nodes. This results in the same moniker for each and
every node.

This commit makes use of the supplied `--hostname-prefix` and
`--hostname-suffix`, or `--hostname` parameters to generate unique
monikers for each node. Alternatively, another parameter
(`--random-monikers`) allows one to forcibly override all of the other
options with random hexadecimal strings.

* Update CHANGELOG_PENDING entry for new command line switch

* update PULL_REQUEST_TEMPLATE and CONTRIBUTING

* Update .github/PULL_REQUEST_TEMPLATE.md

Co-Authored-By: Thane Thomson <connect@thanethomson.com>

* note ADRs

* fix peer state init to late

Peer does not have a state yet. We set it in AddPeer.
We need an new interface before mconnection is started

* pex message to soon

fix reconnection pex send too fast,
error is caused lastReceivedRequests is still
not deleted when a peer reconnected

* add test case for initpeer

* add prove case

* remove potentially infinite loop

* Update consensus/reactor.go

Co-Authored-By: guagualvcha <baifudong@lancai.cn>

* Update consensus/reactor_test.go

Co-Authored-By: guagualvcha <baifudong@lancai.cn>

* document Reactor interface better

* refactor TestReactorReceiveDoesNotPanicIfAddPeerHasntBeenCalledYet

* fix merge conflicts

* blockchain: remove peer's ID from the pool in InitPeer

Refs #3338

* pex: resetPeersRequestsInfo both upon InitPeer and RemovePeer

* ensure RemovePeer is always called before InitPeer

by removing the peer from the switch last (after we've stopped it and
removed from all reactors)

* add some comments for ConsensusReactor#InitPeer

* fix pex reactor

* format code

* fix spelling

* update changelog

* remove unused methods

* do not clear lastReceivedRequests upon error

only in RemovePeer

* call InitPeer before we start the peer!

* add a comment to InitPeer

* write a test

* use waitUntilSwitchHasAtLeastNPeers func

* bring back timeouts

* Test to ensure Receive panics if InitPeer has not been called

[R4R] Fix missing context parameter in proxy

Improve error message returned from AddSignatureFromPubKey

* p2p/pex: consult seeds in crawlPeersRoutine

This changeset alters the startup behavior for crawlPeersRoutine. Previously
the routine would crawl a random selection of peers on startup. For a
new seed node, there are no peers. As a result, new seed nodes are unable
to bootstrap themselves with a list of peers until another node with a list
of peers connects to the seed. If this node relies on the seed node for peers,
then the two will not discover more peers.

This changeset makes the startup behavior for crawlPeersRoutine connect to
any seed nodes. Upon connecting, a request for peers will be sent to the seed node
thus helping bootstrap our seed node.

* p2p/pex: Adjust error message for no peers

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* p2p: initial implementation of peer behaviour

* [p2p] re-use newMockPeer

* p2p: add inline docs for peer_behaviour interface

* [p2p] Align PeerBehaviour interface (#3558)

* [p2p] make switchedPeerHebaviour private

* [p2p] make storePeerHebaviour private

* [p2p] Add CHANGELOG_PENDING entry for PeerBehaviour

* [p2p] Adjustment naming for PeerBehaviour

* [p2p] Add coarse lock around storedPeerBehaviour

* [p2p]: Fix non-pointer methods in storedPeerBehaviour

    + Structs with embeded locks must specify all methods with pointer
     receivers to avoid creating a copy of the embeded lock.

* [p2p] Thorough refactoring based on comments in #3552

    + Decouple PeerBehaviour interface from Peer by parametrizing
    methods with `p2p.ID` instead of `p2p.Peer`
    + Setter methods wrapped in a write lock
    + Getter methods wrapped in a read lock
    + Getter methods on storedPeerBehaviour now take a `p2p.ID`
    + Getter methods return a copy of underlying stored behaviours
    + Add doc strings to public types and methods

* [p2p] make structs public

* [p2p] Test empty StoredPeerBehaviour

* [p2p] typo fix

* [p2p] add TestStoredPeerBehaviourConcurrency

    + Add a test which uses StoredPeerBehaviour in multiple goroutines
      to ensure thread-safety.

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour_test.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* Update p2p/peer_behaviour.go

Co-Authored-By: brapse <brapse@gmail.com>

* [p2p] field ordering convention

* p2p: peer behaviour refactor

    + Change naming of reporting behaviour to `Report`
    + Remove the responsibility of distinguishing between the categories
    of good and bad behaviour and instead focus on reporting behaviour.

* p2p: rename PeerReporter -> PeerBehaviourReporter

* Add blockchain reactor refactor ADR

* docs: Correct markdown and go syntax in adr-036

* Update docs/architecture/adr-036-blockchain-reactor-refactor.md

Co-Authored-By: milosevic <zarko@interchain.io>

* Add comments and address reviewer comments

* Improve formating

* Small improvements

* adr-036 -> adr-040

for storing batch keys and values in boltDBBatch.

NOTE: batch does not have to be safe for concurrent access. Delete may
be slow, but given it should not be used often, it's ok.

Fixes #3631

* libs/db: conditional compilation

For cleveldb: go build -tags cleveldb
For boltdb: go build -tags boltdb

Fixes #3611

* document db_backend param better

* remove deprecated LevelDBBackend

* update changelog

* add missing lines

* add new line

* fix TestRemoteDB

* add a line about boltdb tag

* Revert "remove deprecated LevelDBBackend"

This reverts commit 1aa85453f7.

* make PR non breaking

* change DEPRECATED label format

https://stackoverflow.com/a/36360323/820520

* mempool: remove only valid (Code==0) txs on Update

so evil proposers can't drop valid txs in Commit stage.

Also remove invalid (Code!=0) txs from the cache so they can be
resubmitted.

Fixes #3322

@rickyyangz:

In the end of commit stage, we will update mempool to remove all the txs
in current block.

// Update mempool.
err = blockExec.mempool.Update(
	block.Height,
	block.Txs,
	TxPreCheck(state),
	TxPostCheck(state),
)

Assum an account has 3 transactions in the mempool, the sequences are
100, 101 and 102 separately, So an evil proposal can only package the
101 and 102 transactions into its proposal block, and leave 100 still in
mempool, then the two txs will be removed from all validators' mempool
when commit. So the account lost the two valid txs.

@ebuchman:

In the longer term we may want to do something like #2639 so we can
validate txs before we commit the block. But even in this case we'd only
want to run the equivalent of CheckTx, which means the DeliverTx could
still fail even if the CheckTx passes depending on how the app handles
the ABCI Code semantics. So more work will be required around the ABCI
code. See also #2185

* add changelog entry and tests

* improve changelog message

* reformat code

## Description

also

    handle errors from DialPeersAsync
    remove nil addr from log msg
    fix TestPEXReactorDoesNotDisconnectFromPersistentPeerInSeedMode

This is a follow-up from
#3593 (review)

Fixes most of the #3617, except #3593 (comment)

## Commits

* rpc: /dial_peers: only mark peers as persistent if flag is on

also

- handle errors from DialPeersAsync
- remove nil addr from log msg
- fix TestPEXReactorDoesNotDisconnectFromPersistentPeerInSeedMode

This is a follow-up from
https://github.com/tendermint/tendermint/pull/3593#pullrequestreview-233556909

* remove a call to AddPersistentPeers

TestDialFail will trigger a reconnect

libs/db: fix boltdb batching

`[]byte` is unhashable and needs to be casted to a string.

See https://github.com/tendermint/tendermint/pull/3610#discussion_r280995538

Ref https://github.com/tendermint/tendermint/issues/3626

Refs https://github.com/tendermint/tendermint/pull/3610#discussion_r281201274

If we do not close, other txs will be stuck forever.

Description:

    add boltdb implement for db interface.

    The boltdb is safe and high performence embedded KV database. It is based on B+tree and Mmap.
    Now it is maintained by etcd-io org. https://github.com/etcd-io/bbolt
    It is much better than LSM tree (levelDB) when RANDOM and SCAN read.

Replaced #3604
Refs #1835

Commits:

* add boltdb for storage

* update boltdb in gopkg.toml

* update bbolt in Gopkg.lock

* dep update  Gopkg.lock

* add boltdb'bucket when create Boltdb

* some modifies

* address my own comments

* fix most of the Iterator tests for boltdb

* bolt does not support concurrent writes while iterating

* add WARNING word

* note that ReadOnly option is not supported

* fixes after Ismail's review

Co-Authored-By: melekes <anton.kalyaev@gmail.com>

* panic if View returns error

plus specify bucket in the top comment