## Description

migration of blockchain reactors to protobuf

Closes: #XXX

This PR removes lite & renames lite2 to light throughout the repo

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

Closes: #4944

See e.g. https://github.com/tendermint/tendermint/issues/4925#issuecomment-637929251 for people getting hit by this. Should we document it elsewhere as well?

on 32 bit architectures or ARM

Closes #2103

Reduce the number of targets and make the buildsystem more
flexible by parsing the TENDERMINT_BUILD_OPTIONS command
line variable (a-la Debian, inspired by dpkg-buildpackage's
DEB_BUILD_OPTIONS), e.g:

 $ make install TENDERMINT_BUILD_OPTIONS='cleveldb'

replaces the old:

 $ make install_c

Options can be mix&match'd, e.g.:

 $ make install TENDERMINT_BUILD_OPTIONS='cleveldb race nostrip'

Three options are available:
- nostrip: don't strip debugging symbols nor DWARF tables.
- cleveldb: use cleveldb as db backend instead of goleveldb;
  it switches on the CGO_ENABLED Go environment variale.
- race: pass -race to go build and enable data race detection.

This changeset is a port of gaia pull request: cosmos/gaia#363.

Closes #4670

- defined what variables needed to be changed in the `config.toml` in order to run a validator.
- Briefly explained how a sentry node archtecture should look
- add section explaing importance of key secruity

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* docs: fix links

- fix broken links

closes #4522

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* fix more links

* add enable and disable

* remvoe deadlink

* docs: write about debug kill and dump

Closes #4325

* wrap file tree in code blocks

Closes #4211

* lite2: fix tendermint lite sub command

- better logging
- chainID as an argument
- more examples

* one more log msg

* lite2: fire update right away after start

* turn off auto update in verification tests

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* docs: update Light Client Protocol page

Closes #4331

* one way to get hash & height

* docs: update links to rpc

- links to rpc have not been updated. thank you @okwme

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* Update docs/app-dev/indexing-transactions.md

* docs: minor doc fixes

- minor doc fixes that i ran into while reading things
- test if we have github actions

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* no github actions yet

* add with

* revert and change wording

* evidence: introduce time.Duration to evidence params

- add time.duration to evidence
- this pr is taking pr #2606 and updating it to use both time and height

- closes #2565

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* fix testing and genesis cfg in signer harness

* remove debugging fmt

* change maxageheight to maxagenumblocks, rename other things to block instead of height

* further check of duration

* check duration to not send peers outdated evidence

* change some lines, onward and upward

* refactor evidence package

* add a changelog pending entry

* make mockbadevidence have time and use it

* add what could possibly be called a test case

* remove mockbadevidence and mockgoodevidence in favor of mockevidence

* add a comment for err that is returned

* add a changelog for removal of good & bad evidence

* add a test for adding evidence

* fix test

* add ev to types in testcase

* Update evidence/pool_test.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update evidence/pool_test.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* fix tests

* fix linting

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>

* prometheus/metrics: two new metrics for consensus

- add consensus_validator_power metric so if a node is a validator it can see its own power in prometheus
- add last_signed_height metric so if a node is a validator it can be aware of at which height the most recent time it signed.

- closes: #3773
- closes: #3083

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* check if signature is present

* minor change and check if sig is not absent

* add changelog entry

* Update consensus/state.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update CHANGELOG_PENDING.md

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update CHANGELOG_PENDING.md

* add label with validator address

* change address to vali_address

* add metric missed blocks

* add changelog entry for missed blocks metric

* address naming & add docs

* delete tm-monitor tool #4247

* removed docs

* remove gox

## Issue

Implement a new subcommand: tendermint debug. This subcommand itself has two subcommands:

    $ tendermint debug kill <pid> </path/to/out.zip> --home=</path/to/app.d>

Writes debug info into a compressed archive. The archive will contain the following:

├── config.toml
├── consensus_state.json
├── net_info.json
├── stacktrace.out
├── status.json
└── wal

The Tendermint process will be killed.

    $ tendermint debug dump </path/to/out> --home=</path/to/app.d>

This command will perform similar to kill except it only polls the node and dumps debugging data every frequency seconds to a compressed archive under a given destination directory. Each archive will contain:

├── consensus_state.json
├── goroutine.out
├── heap.out
├── net_info.json
├── status.json
└── wal

Note:

    goroutine.out and heap.out will only be written if a profile address is provided and is operational.
    This command is blocking and will log any error.

replaces: #3327
closes: #3249

## Commits:

* Implement debug tool command stubs

* Implement net getters and zip logic

* Update zip dir API and add home flag

* Add simple godocs for kill aux functions

* Move IO util to new file and implement copy WAL func

* Implement copy config function

* Implement killProc

* Remove debug fmt

* Validate output file input

* Direct STDERR to file

* Godoc updates

* Sleep prior to killing tail proc

* Minor cleanup of godocs

* Move debug command and add make target

* Rename command handler function

* Add example to command long descr

* Move current implementation to cmd/tendermint/commands/debug

* Update kill cmd long description

* Implement dump command

* Add pending log entry

* Add gosec nolint

* Add error check for Mkdir

* Add os.IsNotExist(err)

* Add to debugging section in running-in-prod doc

Closes #3831

The support for rocksdb was added a while back in
https://github.com/tendermint/tm-db/pull/12. This commit merely updates
the documentation.

- chane rpc link to master branch rpc docs, current link leads to 404
- closes #4199

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* Rename Tag(s) to Event(s)

- tag was replaced with event, but in some places it still mentions tag, would be easier to understand if we tried to replace it with event to not confuse people.

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* more  changes from tag -> event

* rename events to compositeKeys and keys

* Apply suggestions from code review

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* add minor documentation on how composite keys are constructed

* rename eventkey to compositekey

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* add changelog entry & add info to regenerate confid to changelog entry

* Removal of TotalTx & NumTx

- Removed totalTx and numTx

closes #2521

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* abci proto changes

* proto number fix

* txfilter_test fix

* comments on PR

* further changes

* bring back metrics

* fix indexer

* fix TestBlockMaxDataBytes and TestBlockMaxDataBytesUnknownEvidence

* indexer service back to header

* statistics.go fix

* fix ci

* listen for blocks, not headers

to be able to record txs throughput

* fix TestNetworkNewBlock

* fix tests

* fix tests in types package

* fixes after Anton's review

* fix tests

* bring back `consensus_total_txs` metric

I mistakenly thought it was removed.

* improve changelog

* remove LastBlockTotalTx from state

* docs: remove getNumTxs from BeginBlock Java example

* docs theme

* vuepress-theme-cosmos

* version bump

* changes to docs

* more code changes

* sidebar order fix

* moar changes

* fixed dev sessions title

* fixed dev sessions title, again

* specs should show up in sidebar

* contents cards

* version bump

* sidebar, rpc

* version bump

* custom footer and super naive search

* version

* minor change to vuepress

* move swagger file

* pre and post scripts

* build

* changed docs build process

* added deployment config

* updated versions file and added deployment filters

Fixes #4026

Fix #3868 (comment)

Commits:

* mempool: make `max_tx_bytes` configurable instead of `max_msg_bytes`

* update CHANGELOG_PENDING

* apply suggestions from code review

* config: move max_msg_bytes into mempool section

It was incorrectly placed in fastsync section during
4d7cd8055b (diff-092cdc48047eeb4c0bca311a2e1b8ae6)
merge.

Fixes #3868

* add changelog entry

* go routines in blockchain reactor

* Added reference to the go routine diagram

* Initial commit

* cleanup

* Undo testing_logger change, committed by mistake

* Fix the test loggers

* pulled some fsm code into pool.go

* added pool tests

* changes to the design

added block requests under peer

moved the request trigger in the reactor poolRoutine, triggered now by a ticker

in general moved everything required for making block requests smarter in the poolRoutine

added a simple map of heights to keep track of what will need to be requested next

added a few more tests

* send errors to FSM in a different channel than blocks

send errors (RemovePeer) from switch on a different channel than the
one receiving blocks
renamed channels
added more pool tests

* more pool tests

* lint errors

* more tests

* more tests

* switch fast sync to new implementation

* fixed data race in tests

* cleanup

* finished fsm tests

* address golangci comments :)

* address golangci comments :)

* Added timeout on next block needed to advance

* updating docs and cleanup

* fix issue in test from previous cleanup

* cleanup

* Added termination scenarios, tests and more cleanup

* small fixes to adr, comments and cleanup

* Fix bug in sendRequest()

If we tried to send a request to a peer not present in the switch, a
missing continue statement caused the request to be blackholed in a peer
that was removed and never retried.

While this bug was manifesting, the reactor kept asking for other
blocks that would be stored and never consumed. Added the number of
unconsumed blocks in the math for requesting blocks ahead of current
processing height so eventually there will be no more blocks requested
until the already received ones are consumed.

* remove bpPeer's didTimeout field

* Use distinct err codes for peer timeout and FSM timeouts

* Don't allow peers to update with lower height

* review comments from Ethan and Zarko

* some cleanup, renaming, comments

* Move block execution in separate goroutine

* Remove pool's numPending

* review comments

* fix lint, remove old blockchain reactor and duplicates in fsm tests

* small reorg around peer after review comments

* add the reactor spec

* verify block only once

* review comments

* change to int for max number of pending requests

* cleanup and godoc

* Add configuration flag fast sync version

* golangci fixes

* fix config template

* move both reactor versions under blockchain

* cleanup, golint, renaming stuff

* updated documentation, fixed more golint warnings

* integrate with behavior package

* sync with master

* gofmt

* add changelog_pending entry

* move to improvments

* suggestion to changelog entry

* mempool: make max_msg_bytes configurable

* apply suggestions from code review

* update changelog pending

* apply suggestions from code review again

* rpc: make max_body_bytes and max_header_bytes configurable

* update changelog pending

Also, document time_iota_ms.

Closes #3723

Commits:

* config: Add ConsensusParams when generating testnet config (#3723)

* docs: Add explanation of time_iota_ms (#3723)

* Update changelog_pending (#3723)

* Quick link fixes throughout docs and repo

- used markdown link tester to find broken links

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* minor change remove slash

* pr comments

* minor fix

* remove docker.develop

* remove master tag

* Add `chID` label to sent/receive byte mtrics
* add changelog pending entry

* Add blockchain reactor refactor ADR

* docs: Correct markdown and go syntax in adr-036

* Update docs/architecture/adr-036-blockchain-reactor-refactor.md

Co-Authored-By: milosevic <zarko@interchain.io>

* Add comments and address reviewer comments

* Improve formating

* Small improvements

* adr-036 -> adr-040

* libs/db: conditional compilation

For cleveldb: go build -tags cleveldb
For boltdb: go build -tags boltdb

Fixes #3611

* document db_backend param better

* remove deprecated LevelDBBackend

* update changelog

* add missing lines

* add new line

* fix TestRemoteDB

* add a line about boltdb tag

* Revert "remove deprecated LevelDBBackend"

This reverts commit 1aa85453f7.

* make PR non breaking

* change DEPRECATED label format

https://stackoverflow.com/a/36360323/820520

Refs #3419

* Make sure config.TimeoutBroadcastTxCommit < rpcserver.WriteTimeout()

* remove redundant comment

* libs/rpc/http_server: move Read/WriteTimeout into Config

* increase defaults for read/write timeouts

Based on this article
https://www.digitalocean.com/community/tutorials/how-to-optimize-nginx-configuration

* WriteTimeout should be larger than TimeoutBroadcastTxCommit

* set a deadline for subscribing to txs

* extract duration into const

* add two changelog entries

* Update CHANGELOG_PENDING.md

Co-Authored-By: melekes <anton.kalyaev@gmail.com>

* Update CHANGELOG_PENDING.md

Co-Authored-By: melekes <anton.kalyaev@gmail.com>

* 12 -> 10

* changelog

* changelog

* limit number of /subscribe clients and queries per client

Add the following config variables (under [rpc] section):
  * max_subscription_clients
  * max_subscriptions_per_client
  * timeout_broadcast_tx_commit

Fixes #2826

new HTTPClient interface for subscriptions

finalize HTTPClient events interface

remove EventSubscriber

fix data race

```
WARNING: DATA RACE
Read at 0x00c000a36060 by goroutine 129:
  github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe.func1()
      /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:168 +0x1f0

Previous write at 0x00c000a36060 by goroutine 132:
  github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe()
      /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:191 +0x4e0
  github.com/tendermint/tendermint/rpc/client.WaitForOneEvent()
      /go/src/github.com/tendermint/tendermint/rpc/client/helpers.go:64 +0x178
  github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync.func1()
      /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:139 +0x298
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 129 (running) created at:
  github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe()
      /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:164 +0x4b7
  github.com/tendermint/tendermint/rpc/client.WaitForOneEvent()
      /go/src/github.com/tendermint/tendermint/rpc/client/helpers.go:64 +0x178
  github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync.func1()
      /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:139 +0x298
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 132 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:878 +0x659
  github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync()
      /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:119 +0x186
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162
==================
```

lite client works (tested manually)

godoc comments

httpclient: do not close the out channel

use TimeoutBroadcastTxCommit

no timeout for unsubscribe

but 1s Local (5s HTTP) timeout for resubscribe

format code

change Subscribe#out cap to 1

and replace config vars with RPCConfig

TimeoutBroadcastTxCommit can't be greater than rpcserver.WriteTimeout

rpc: Context as first parameter to all functions

reformat code

fixes after my own review

fixes after Ethan's review

add test stubs

fix config.toml

* fixes after manual testing

- rpc: do not recommend to use BroadcastTxCommit because it's slow and wastes
Tendermint resources (pubsub)
- rpc: better error in Subscribe and BroadcastTxCommit
- HTTPClient: do not resubscribe if err = ErrAlreadySubscribed

* fixes after Ismail's review

* Update rpc/grpc/grpc_test.go

Co-Authored-By: melekes <anton.kalyaev@gmail.com>

* docs: explain create_empty_blocks configurations

Closes #3307

* Vagrantfile: install nodejs for docs

* update docs instructions

npm install does not make sense since there's no packages.json file

* explain broadcast_tx_* tx format

Closes #536

* docs: explain how transaction ordering works

Closes #2904

* bring in consensus parameters explained

* example for create_empty_blocks_interval

* bring in explanation from https://github.com/tendermint/tendermint/issues/2487#issuecomment-424899799

* link to formatting instead of duplicating info

* bound mempool memory usage

Closes #3079

* rename SizeBytes to TxsTotalBytes

and other small fixes after Zarko's review

* rename MaxBytes to MaxTxsTotalBytes

* make ErrMempoolIsFull more informative

* expose mempool's txs_total_bytes via RPC

* test full response

* fixes after Ethan's review

* config: rename mempool.size to mempool.max_txs

https://github.com/tendermint/tendermint/pull/3248#discussion_r254034004

* test more cases

https://github.com/tendermint/tendermint/pull/3248#discussion_r254036532

* simplify test

* Revert "config: rename mempool.size to mempool.max_txs"

This reverts commit 39bfa36961.

* rename count back to n_txs

to make a change non-breaking

* rename max_txs_total_bytes to max_txs_bytes

* format code

* fix TestWALPeriodicSync

The test was sometimes failing due to processFlushTicks being called too
early. The solution is to call wal#Start later in the test.

* Apply suggestions from code review

Because there's nothing worse than having to copy/paste a link from a
web page to navigate to it 😁

* config: cors options are arrays of strings, not strings

Fixes #2980

* docs: update tendermint-core/configuration.html page

* set allow_duplicate_ip to false

* in `tendermint testnet`, set allow_duplicate_ip to true

Refs #2712

* fixes after Ismail's review

* fix docs / proxy app, closes #2986

* counter_serial

* review comments

* list all possible options

* add changelog entries

* config: cors options are arrays of strings, not strings

Fixes #2980

* docs: update tendermint-core/configuration.html page

* set allow_duplicate_ip to false

* in `tendermint testnet`, set allow_duplicate_ip to true

Refs #2712

* fixes after Ismail's review

* Revert "set allow_duplicate_ip to false"

This reverts commit 24c1094ebc.