This PR make some tweaks to backfill after running e2e tests:
- Separates sync and backfill as two distinct processes that the node calls. The reason is because if sync fails then the node should fail but if backfill fails it is still possible to proceed.
- Removes peers who don't have the block at a height from the local peer list. As the process goes backwards if a node doesn't have a block at a height they're likely pruning blocks and thus they won't have any prior ones either. 
- Sleep when we've run out of peers, then try again.

Closes #3584

state: move pruneBlocks function from consensus/state to state/execution

Closes #5414

There is a possible data race/panic between processBlockchainCh and
processPeerUpdates, since when we send to blockchainCh.Out in one
goroutine and close the channel in the other. The race is seen in some
Github Action runs.

This commit fix the race, by adding a peerUpdatesCh as a bridge between
processPeerUpdates and processBlockchainCh, so the former will send to
this channel, the later will listen and forward the message to
blockchainCh.Out channel.

Updates #6516

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1.5.0 to 1.5.2.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1.5.0...v1.5.2)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

Bumps [github.com/btcsuite/btcd](https://github.com/btcsuite/btcd) from 0.21.0-beta to 0.22.0-beta.
- [Release notes](https://github.com/btcsuite/btcd/releases)
- [Changelog](https://github.com/btcsuite/btcd/blob/master/CHANGES)
- [Commits](https://github.com/btcsuite/btcd/compare/v0.21.0-beta...v0.22.0-beta)

---
updated-dependencies:
- dependency-name: github.com/btcsuite/btcd
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.10.0 to 1.11.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/master/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.10.0...v1.11.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sam Kleinman <garen@tychoish.com>

## Description

Change block_size gauge to a histogram to observe block size overtime

This will help will see which chains have full blocks vs empty. 

closes #5752

## Description

remove IAVL as a Tendermint dependency.

closes #5694 

I dont think this needs a changelog entry.

## Description

Update for logging changes

Bumps [rtCamp/action-slack-notify](https://github.com/rtCamp/action-slack-notify) from 2.1.3 to 2.2.0.
- [Release notes](https://github.com/rtCamp/action-slack-notify/releases)
- [Commits](f565a63638...12e36fc18b0689399306c2e0b3e0f2978b7f1ee7)

---
updated-dependencies:
- dependency-name: rtCamp/action-slack-notify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

## Description

revert #5523 to avoid breaking changes

cc @greg-szabo

## Description

Trying to debug a possible hashing issue, writing test vectors on 0.34 and then porting them to master to double-check it's not a hashing issue.

## Decription

Tag teaming with callum on debugging, wrote this to test headerHash changes from 0.34 to master. Its useful going forward as well.

Remove `Context` from the `TxInfo` type and instead require the caller to pass a `Context` to `CheckTx` which is idiomatic.

closes: #6497

Per tendermint spec, each Channel has a globally unique byte id, which
is mapped to uint8 in Go. However, the proto PacketMsg.ChannelID field
is declared as int32, and when receive the packet, we cast it to a byte
without checking for possible overflow. That leads to a malform packet
with invalid channel id is sent successfully.

To fix it, we just add a check for possible overflow, and return invalid
channel id error.

Fixed #6521

By pre-creating the hasher, instead of creating new one everytime
addrbook.hash is called.

```
name             old time/op    new time/op    delta
AddrBook_hash-8     181ns ±13%      80ns ± 1%  -56.08%  (p=0.000 n=10+10)

name             old alloc/op   new alloc/op   delta
AddrBook_hash-8      216B ± 0%        8B ± 0%  -96.30%  (p=0.000 n=10+10)

name             old allocs/op  new allocs/op  delta
AddrBook_hash-8      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
```

Fixed #6508

## Description 

Upstream https://github.com/lazyledger/lazyledger-core/pull/351 to optimize merkle tree hashing 

### Benchmarking:

```
benchmark                                 old ns/op     new ns/op     delta
BenchmarkHashAlternatives/recursive-8     22914         21949         -4.21%
BenchmarkHashAlternatives/iterative-8     21634         21939         +1.41%

benchmark                                 old allocs     new allocs     delta
BenchmarkHashAlternatives/recursive-8     398            200            -49.75%
BenchmarkHashAlternatives/iterative-8     399            301            -24.56%

benchmark                                 old bytes     new bytes     delta
BenchmarkHashAlternatives/recursive-8     19088         6496          -65.97%
BenchmarkHashAlternatives/iterative-8     21776         13984         -35.78%
```

cc @odeke-em @cuonglm

Addresses a beginning component of #6255

Noticed from profiling that the loggers are memory hungry and alas,
we were passing fmt.Sprintf strings to (*bytes.Buffer).WriteString
which defeats the purpose of using fmt.* This change fixes that and
instead directly invokes fmt.Fprintf.
The benchmarks show the improvement:

```shell
$ benchstat before.txt after.txt
name                     old time/op    new time/op    delta
TMLoggerSimple-8           1.67µs ± 4%    1.69µs ±13%     ~     (p=0.118 n=20+19)
TMLoggerContextual-8       2.01µs ± 8%    1.94µs ± 1%   -3.79%  (p=0.000 n=18+20)
TMFmtLoggerSimple-8        1.20µs ± 3%    1.16µs ± 2%   -3.39%  (p=0.000 n=20+16)
TMFmtLoggerContextual-8    1.53µs ±19%    1.43µs ±13%   -6.10%  (p=0.001 n=20+20)

name                     old alloc/op   new alloc/op   delta
TMLoggerSimple-8             696B ± 0%      616B ± 0%  -11.49%  (p=0.000 n=20+20)
TMLoggerContextual-8       1.02kB ± 0%    0.94kB ± 0%   -7.87%  (p=0.000 n=20+20)
TMFmtLoggerSimple-8          240B ± 0%      160B ± 0%  -33.33%  (p=0.000 n=20+20)
TMFmtLoggerContextual-8      416B ± 0%      336B ± 0%  -19.23%  (p=0.000 n=20+20)

name                     old allocs/op  new allocs/op  delta
TMLoggerSimple-8             13.0 ± 0%      12.0 ± 0%   -7.69%  (p=0.000 n=20+20)
TMLoggerContextual-8         17.0 ± 0%      16.0 ± 0%   -5.88%  (p=0.000 n=20+20)
TMFmtLoggerSimple-8          6.00 ± 0%      5.00 ± 0%  -16.67%  (p=0.000 n=20+20)
TMFmtLoggerContextual-8      8.00 ± 0%      7.00 ± 0%  -12.50%  (p=0.000 n=20+20)
```

Fixes #6502

## Description

Add version back to versions, but allow it to be overridden via a ldflag.

Reason:

Many users are not setting the ldflag causing issues with tooling that relies on it (cosmjs)

closes #6488

cc @webmaster128

Bumps [github.com/lib/pq](https://github.com/lib/pq) from 1.10.1 to 1.10.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/lib/pq/releases">github.com/lib/pq's releases</a>.</em></p>
<blockquote>
<h2>v1.10.2</h2>
<ul>
<li>fix TimeTZ with second offsets</li>
<li>fix GOOS compilation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="2da6713d67"><code>2da6713</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/lib/pq/issues/1039">#1039</a> from otan-cockroach/timetz_fix</li>
<li><a href="ad47bab1aa"><code>ad47bab</code></a> encode: fix TimeTZ with second offsets</li>
<li><a href="99af95f861"><code>99af95f</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/lib/pq/issues/1041">#1041</a> from otan-cockroach/libpq</li>
<li><a href="62fa4b32ec"><code>62fa4b3</code></a> .travis.yml: fix CI</li>
<li><a href="d2b13db12b"><code>d2b13db</code></a> Delete test.yml</li>
<li><a href="a1b1a43f73"><code>a1b1a43</code></a> Create test.yml</li>
<li><a href="b2cfb1abfd"><code>b2cfb1a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/lib/pq/issues/1036">#1036</a> from bukforks/master</li>
<li><a href="6ed3b8ac03"><code>6ed3b8a</code></a> rm unused imports</li>
<li><a href="feb727accb"><code>feb727a</code></a> userCurrent for unsupported GOOS</li>
<li>See full diff in <a href="https://github.com/lib/pq/compare/v1.10.1...v1.10.2">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/lib/pq&package-manager=go_modules&previous-version=1.10.1&new-version=1.10.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

Bumps [actions/cache](https://github.com/actions/cache) from 2.1.5 to 2.1.6.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](https://github.com/actions/cache/compare/v2.1.5...v2.1.6)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

## Description

Cleanup ADR readme and update changelogs and status of ADRs

Bumps [dns-packet](https://github.com/mafintosh/dns-packet) from 1.3.1 to 1.3.4.
- [Release notes](https://github.com/mafintosh/dns-packet/releases)
- [Changelog](https://github.com/mafintosh/dns-packet/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mafintosh/dns-packet/compare/v1.3.1...v1.3.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2.4.0 to 2.5.0.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v2.4.0...v2.5.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Adds `block_id` to the `newblock` websocket event

Closes #6028

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.37.1 to 1.38.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/grpc/grpc-go/releases">google.golang.org/grpc's releases</a>.</em></p>
<blockquote>
<h2>Release 1.38.0</h2>
<h1>API Changes</h1>
<ul>
<li>reflection: accept interface instead of grpc.Server struct in Register() (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4340">#4340</a>)</li>
<li>resolver: add error return value from ClientConn.UpdateState (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4270">#4270</a>)</li>
</ul>
<h1>Behavior Changes</h1>
<ul>
<li>client: do not poll name resolver when errors or bad updates are reported (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4270">#4270</a>)</li>
<li>transport: InTapHandle may return RPC status errors; no longer RST_STREAMs (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4365">#4365</a>)</li>
</ul>
<h1>New Features</h1>
<ul>
<li>client: propagate connection error causes to RPC status (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4311">#4311</a>, <a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4316">#4316</a>)</li>
<li>xds: support inline RDS resource from LDS response (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4299">#4299</a>)</li>
<li>xds: server side support is now experimentally available</li>
<li>server: add ForceServerCodec() to set a custom encoding.Codec on the server (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4205">#4205</a>)
<ul>
<li>Special Thanks: <a href="https://github.com/ash2k"><code>@​ash2k</code></a></li>
</ul>
</li>
</ul>
<h1>Performance Improvements</h1>
<ul>
<li>metadata: reduce memory footprint in FromOutgoingContext (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4360">#4360</a>)
<ul>
<li>Special Thanks: <a href="https://github.com/irfansharif"><code>@​irfansharif</code></a></li>
</ul>
</li>
</ul>
<h1>Bug Fixes</h1>
<ul>
<li>xds/balancergroup: fix rare memory leak after closing ClientConn (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4308">#4308</a>)</li>
</ul>
<h1>Documentation</h1>
<ul>
<li>examples: update xds examples for PSM security (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4256">#4256</a>)</li>
<li>grpc: improve docs on StreamDesc (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4397">#4397</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="0257c86573"><code>0257c86</code></a> Change version to 1.38.0 (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4418">#4418</a>)</li>
<li><a href="ce3e5ec0d8"><code>ce3e5ec</code></a> v1.38.x: backport (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4453">#4453</a>)</li>
<li><a href="5f95ad6233"><code>5f95ad6</code></a> xds: workaround to deflake xds e2e tests (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4413">#4413</a>)</li>
<li><a href="b1940e15f6"><code>b1940e1</code></a> xds: register resources at the mgmt server before requesting them (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4406">#4406</a>)</li>
<li><a href="98c895f7e0"><code>98c895f</code></a> cleanup: use testutils.MarshalAny in more places (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4404">#4404</a>)</li>
<li><a href="12a377b1e4"><code>12a377b</code></a> xds: nack route configuration with regexes that don't compile (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4388">#4388</a>)</li>
<li><a href="c15291b0f5"><code>c15291b</code></a> client: initialize safe config selector when creating ClientConn (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4398">#4398</a>)</li>
<li><a href="328b1d171a"><code>328b1d1</code></a> transport: allow InTapHandle to return status errors (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4365">#4365</a>)</li>
<li><a href="aff517ba8a"><code>aff517b</code></a> xds: make e2e tests use a single management server instance (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4399">#4399</a>)</li>
<li><a href="0439465fe2"><code>0439465</code></a> xds_resolver: fix flaky Test/XDSResolverDelayedOnCommitted (<a href="https://github-redirect.dependabot.com/grpc/grpc-go/issues/4393">#4393</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/grpc/grpc-go/compare/v1.37.1...v1.38.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=google.golang.org/grpc&package-manager=go_modules&previous-version=1.37.1&new-version=1.38.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

Should give a warning in config until this is below issue is resolved
Related to https://github.com/tendermint/tendermint/issues/3055