## Issue

Why this pr:
When restarting chain node, sometimes we lost tx index about recent(around 80)blocks, and some client complains that they can't find the tx by RPC call(tx_search) when the tx do exist in the block.

I try to partially fix this issue in a simple way by writing the index data in a sync way.
There is no performance difference under 1K TPS according to our test.

It is still possible that lost index data after restarting the node, but only 2 block data will lost at most.
I try to totally fix this in https://github.com/tendermint/tendermint/pull/3847/files, but this one is simple and can solve most part of the issue. Please review first, thks.

## Comments

Anton:

BEFORE:

BenchmarkTxIndex1-2               100000             12434 ns/op
BenchmarkTxIndex500-2                300           5151564 ns/op
BenchmarkTxIndex1000-2               100          15053910 ns/op
BenchmarkTxIndex2000-2               100          18238892 ns/op
BenchmarkTxIndex10000-2               20         124287930 ns/op

AFTER:

BenchmarkTxIndex1-2                 2000            795431 ns/op
BenchmarkTxIndex500-2                200           6385124 ns/op
BenchmarkTxIndex1000-2               100          11388219 ns/op
BenchmarkTxIndex2000-2               100          20514873 ns/op
BenchmarkTxIndex10000-2               20         107456004 ns/op

Performance drop is pretty steep, but I think it's the right thing to do UNTIL we have a WAL.

Some linting/cleanup missed from the initial events refactor
Don't panic; instead, return false, error when matching breaks unexpectedly
Strip non-numeric chars from values when attempting to match against query values
Have the server log during send upon error

* cleanup/lint Query#Conditions and do not panic

* cleanup/lint Query#Matches and do not panic

* cleanup/lint matchValue and do not panic

* rever to panic in Query#Conditions

* linting

* strip alpha chars when attempting to match

* add pending log entries

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* build: update variable names

* update matchValue to return an error

* update Query#Matches to return an error

* update TestMatches

* log error in send

* Fix tests

* Fix TestEmptyQueryMatchesAnything

* fix linting

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/query/query.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update libs/pubsub/pubsub.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* add missing errors pkg import

* update Query#Conditions to return an error

* update query pkg unit tests

* update TxIndex#Search

* update pending changelog

* Fix long line errors in abci, crypto, and libs packages

* Fix long lines in p2p and rpc packages

* Fix long lines in abci, state, and tools packages

* Fix long lines in behaviour and blockchain packages

* Fix long lines in cmd and config packages

* Begin fixing long lines in consensus package

* Finish fixing long lines in consensus package

* Add lll exclusion for lines containing URLs

* Fix long lines in crypto package

* Fix long lines in evidence package

* Fix long lines in mempool and node packages

* Fix long lines in libs package

* Fix long lines in lite package

* Fix new long line in node package

* Fix long lines in p2p package

* Ignore gocritic warning

* Fix long lines in privval package

* Fix long lines in rpc package

* Fix long lines in scripts package

* Fix long lines in state package

* Fix long lines in tools package

* Fix long lines in types package

* Enable lll linter

* Fix maligned structs

* Fix interfacer errors

* Revert accidental go.mod and go.sum changes

* Revert P2PConfig struct maligned reorder

* Revert PeerRoundState struct maligned reordering

* Revert RoundState struct maligned reordering

* Reorder WSClient struct

* Revert accidental type change

* Clean up type change

* Clean up type changes

* Revert to types.ABCIApplicationServer in GRPCServer struct

* Revert maligned changes to BaseConfig struct

* Fix tests in io_test.go

* Fix client_test package tests

* Fix reactor tests in consensus package

* Fix new interfacer errors

Refs #3262

- Replace the previous intersect call, which was called at each query condition, with a map intersection.
- Replace fmt.Sprintf with string()

closes: #3076

Benchmarks

```
Old
goos: darwin
goarch: amd64
pkg: github.com/tendermint/tendermint/state/txindex/kv
BenchmarkTxSearch-4   	     200	 103641206 ns/op	 7998416 B/op	   71171 allocs/op
PASS
ok  	github.com/tendermint/tendermint/state/txindex/kv	26.019s

New
goos: darwin
goarch: amd64
pkg: github.com/tendermint/tendermint/state/txindex/kv
BenchmarkTxSearch-4   	    1000	  38615024 ns/op	13515226 B/op	  166460 allocs/op
PASS
ok  	github.com/tendermint/tendermint/state/txindex/kv	53.618s
```

~62% performance improvement

Commits:

* Refactor tx search

* Add pending changelog entry

* Add tx search benchmarking

* remove intermediate hashes list

also reset timer in BenchmarkTxSearch
and fix other benchmark

* fix import

* Add test cases

* Fix searching

* Replace fmt.Sprintf with string

* Update state/txindex/kv/kv.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Rename params

* Cleanup

* Check error in benchmarks

* tm-cmn to tm-db

* go.mod changes

* go.mod changes

* more go.mod

* fix tm-db

* ci fix, pending change

* Remove db from tendemrint in favor of tendermint/tm-cmn

- remove db from `libs`
- update dependancy, there have been no breaking changes in the updated deps
	- https://github.com/grpc/grpc-go/releases
	- https://github.com/golang/protobuf/releases

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* changelog add

* gofmt

* more gofmt

## PR

This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events.

    Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like:

"rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}]
"sender": [{"address": "...", "balance": "100uatom"}]

The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...'

    ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair.
    PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings.

closes: #1859
closes: #2905

## Commits:

* Implement Event ABCI type and updates responses to use events

* Update messages_test.go

* Update kvstore.go

* Update event_bus.go

* Update subscription.go

* Update pubsub.go

* Update kvstore.go

* Update query logic to handle slice of strings in events

* Update Empty#Matches and unit tests

* Update pubsub logic

* Update EventBus#Publish

* Update kv tx indexer

* Update godocs

* Update ResultEvent to use slice of strings; update RPC

* Update more tests

* Update abci.md

* Check for key in validateAndStringifyEvents

* Fix KV indexer to skip empty keys

* Fix linting errors

* Update CHANGELOG_PENDING.md

* Update docs/spec/abci/abci.md

Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com>

* Update abci/types/types.proto

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* Update docs/spec/abci/abci.md

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* Update libs/pubsub/query/query.go

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* Update match function to match if ANY value matches

* Implement TestSubscribeDuplicateKeys

* Update TestMatches to include multi-key test cases

* Update events.go

* Update Query interface godoc

* Update match godoc

* Add godoc for matchValue

* DRY-up tx indexing

* Return error from PublishWithEvents in EventBus#Publish

* Update PublishEventNewBlockHeader to return an error

* Fix build

* Update events doc in ABCI

* Update ABCI events godoc

* Implement TestEventBusPublishEventTxDuplicateKeys

* Update TestSubscribeDuplicateKeys to be table-driven

* Remove mod file

* Remove markdown from events godoc

* Implement TestTxSearchDeprecatedIndexing test

ClevelDB requires closing when WriteBatch is no longer needed, https://godoc.org/github.com/jmhodges/levigo#WriteBatch.Close

Fixes the memory leak in https://github.com/cosmos/cosmos-sdk/issues/3842

ClevelDB requires closing when WriteBatch is no longer needed, https://godoc.org/github.com/jmhodges/levigo#WriteBatch.Close

Fixes the memory leak in https://github.com/cosmos/cosmos-sdk/issues/3842

Fixes #2912

* kv indexer: add separator to start key when matching ranges

to avoid including false positives

Refs #2908

* refactor code

* add a test case

Fixes #2908

Fixes #2775

Modify lookForHeight to return a height only there's a equal operator.
Previously, it was returning a height even for range conditions: "height
< 10000".

Fixes #2759

Refs #2051

This reverts commit d02c5d1e30.

This reverts commit 96a3502

Refs #847

* use increment and decrement operators.

* remove unnecessary else branches.

* fix package comment with leading space.

* fix receiver names.

* fix error strings.

* remove omittable code.

* remove redundant return statement.

* Revert changes (code is generated.)

* use cfg as receiver name for all config-related types.

* use lsi as the receiver name for the LastSignedInfo type.

before we're using IteratePrefix, which is wrong because we want full
range, not just "account.number=1".

Refs #1366

encoded as %s. not sure this will work with raw bytes

NOTE we only updating for tmlibs and abci

* linter: address gosimple lints

* linter: make deterministic & a rebase fix

* lint/rpc: fix a gosimple lint

* run linter in CI

* fix rebase mistake

* fix makefile

* ugh

* revert Makefile

* add metalinter to CI

* try this

* linter: last little fix

* need glide

* better

* okayy circle, have it your way

* lints: gosimple

* pr comments

This cleans up some of the code in the state package