* build: Make sure to test packages with external tests (#7608)
The test filter was looking for "TestGoFiles", which does not include tests in
a separate package (e.g., "package foo_test" for "package foo").
This caused several packages not to be tested in CI, including:
github.com/tendermint/tendermint/abci/client
github.com/tendermint/tendermint/crypto
github.com/tendermint/tendermint/crypto/tmhash
github.com/tendermint/tendermint/internal/eventbus
github.com/tendermint/tendermint/internal/evidence
github.com/tendermint/tendermint/internal/inspect
github.com/tendermint/tendermint/internal/jsontypes
github.com/tendermint/tendermint/internal/libs/protoio
github.com/tendermint/tendermint/internal/libs/sync
github.com/tendermint/tendermint/internal/p2p/pex
github.com/tendermint/tendermint/internal/pubsub
github.com/tendermint/tendermint/internal/pubsub/query
github.com/tendermint/tendermint/internal/pubsub/query/syntax
github.com/tendermint/tendermint/internal/state/indexer
github.com/tendermint/tendermint/internal/state/indexer/block/kv
github.com/tendermint/tendermint/libs/json
github.com/tendermint/tendermint/libs/log
github.com/tendermint/tendermint/libs/os
github.com/tendermint/tendermint/light
github.com/tendermint/tendermint/light/provider/http
github.com/tendermint/tendermint/privval/grpc
github.com/tendermint/tendermint/proto/tendermint/blocksync
github.com/tendermint/tendermint/proto/tendermint/consensus
github.com/tendermint/tendermint/proto/tendermint/statesync
github.com/tendermint/tendermint/rpc/client
github.com/tendermint/tendermint/rpc/client/mock
github.com/tendermint/tendermint/test/e2e/tests
github.com/tendermint/tendermint/test/fuzz/mempool
github.com/tendermint/tendermint/test/fuzz/p2p/secretconnection
github.com/tendermint/tendermint/test/fuzz/rpc/jsonrpc/server
Updates #7626 and #7634.
(cherry picked from commit aea428d322)
* Fix build tags to satisfy Go 1.16.
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* consensus: calculate prevote message delay metric (#7551)
## What does this pull request do?
This pull requests adds two metrics intended for use in calculating an experimental value for `MessageDelay`.
The metrics are as follows:
```
# HELP tendermint_consensus_complete_prevote_message_delay Difference in seconds between the proposal timestamp and the timestamp of the prevote that achieved 100% of the voting power in the prevote step.
# TYPE tendermint_consensus_complete_prevote_message_delay gauge
tendermint_consensus_complete_prevote_message_delay{chain_id="test-chain-aZbwF1"} 0.013025505
# HELP tendermint_consensus_quorum_prevote_message_delay Difference in seconds between the proposal timestamp and the timestamp of the prevote that achieved a quorum in the prevote step.
# TYPE tendermint_consensus_quorum_prevote_message_delay gauge
tendermint_consensus_quorum_prevote_message_delay{chain_id="test-chain-aZbwF1"} 0.013025505
```
## Why this change?
For more information on what these metrics are calculating, see #7202. The aim is to merge to backport these metrics to v0.34 and run nodes on a few popular chains with these metrics to determine the experimental values for `MessageDelay` on these popular chains and use these to select our default `SynchronyParams.MessageDelay` value.
## Why Gauges for the metrics?
Gauges allow us to overwrite the metric on each successive observation. We can then capture these metrics over time to track the highest and lowest observed value.
(cherry picked from commit 0c82ceaa5f)
# Conflicts:
# internal/consensus/state.go
* fix merge conflicts
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
Co-authored-by: William Banfield <wbanfield@gmail.com>
These tests use a deterministic and unseeded random source to generate
non-colliding filenames for testing. When testing locally, this means tests are
not hermetic from one run to the next.
Use proper temp directories, and clean up after they're done.
(cherry picked from commit 5f85c8f536)
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
* doc: fix typos in /tx_search and /tx.
* docs: update of go snippets for subscribe and unsubscribe operations
Co-authored-by: Carlos Rodriguez <crodveg@gmail.com>
(cherry picked from commit 2f5ad5f8cc)
Co-authored-by: Carlos Rodriguez <carlos@interchain.io>
* build(deps): Bump github.com/adlio/schema from 1.1.15 to 1.2.2
Bumps [github.com/adlio/schema](https://github.com/adlio/schema) from 1.1.15 to 1.2.2.
- [Release notes](https://github.com/adlio/schema/releases)
- [Commits](https://github.com/adlio/schema/compare/v1.1.15...v1.2.2)
---
updated-dependencies:
- dependency-name: github.com/adlio/schema
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Work around API changes in the migrator package.
A recent update inadvertently broke the API by changing the receiver types of
the methods without updating the constructor.
See: https://github.com/adlio/schema/issues/13
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
Noticed in profiles that invoking *VoteSignBytes always created a
bytes.Buffer, then discarded it inside protoio.MarshalDelimited.
I dug further and examined the call paths and noticed that we
unconditionally create the bytes.Buffer, even though we might
have proto messages (in the common case) that implement
MarshalTo([]byte), and invoked varintWriter. Instead by inlining
this case, we skip a bunch of allocations and CPU cycles,
which then reflects properly on all calling functions. Here
are the benchmark results:
```shell
$ benchstat before.txt after.txt
name old time/op new time/op delta
types.VoteSignBytes-8 705ns ± 3% 573ns ± 6% -18.74% (p=0.000 n=18+20)
types.CommitVoteSignBytes-8 8.15µs ± 9% 6.81µs ± 4% -16.51% (p=0.000 n=20+19)
protoio.MarshalDelimitedWithMarshalTo-8 788ns ± 8% 772ns ± 3% -2.01% (p=0.050 n=20+20)
protoio.MarshalDelimitedNoMarshalTo-8 989ns ± 4% 845ns ± 2% -14.51% (p=0.000 n=20+18)
name old alloc/op new alloc/op delta
types.VoteSignBytes-8 792B ± 0% 600B ± 0% -24.24% (p=0.000 n=20+20)
types.CommitVoteSignBytes-8 9.52kB ± 0% 7.60kB ± 0% -20.17% (p=0.000 n=20+20)
protoio.MarshalDelimitedNoMarshalTo-8 808B ± 0% 440B ± 0% -45.54% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
types.VoteSignBytes-8 13.0 ± 0% 10.0 ± 0% -23.08% (p=0.000 n=20+20)
types.CommitVoteSignBytes-8 140 ± 0% 110 ± 0% -21.43% (p=0.000 n=20+20)
protoio.MarshalDelimitedNoMarshalTo-8 10.0 ± 0% 7.0 ± 0% -30.00% (p=0.000 n=20+20)
```
Thanks to Tharsis who tasked me to help them increase TPS and who
are keen on improving Tendermint and efficiency.
(cherry picked from commit 3e92899bd9)
Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com>
Per https://blog.mergify.com/strict-mode-deprecation/, the strict mode
has been deprecated and will be turned off on 10-Jan-2022. This updates
the config to use the new, approved thing instead of the old thing.
(cherry picked from commit 2d4844f97f)
Some of our tests were creating a psql event sink and expecting
it to report (or not report) certain kinds of errors. These tests
were ill-founded in a couple of ways:
1. Tests that required the Postgres driver were not loading it.
This led to spurious successes on tests that wanted "some error"
from the sink constructor, but didn't exercise the right path.
2. Tests that wanted a Postgres sink to succeed without a database.
These tests "passed" because they weren't actually establishing a
connection to the database, but if they had would have failed for
the lack of one.
To fix this:
- Load the postgres driver in tests that need it.
- Verify connectivity before reporting successful creation of a PSQL event sink.
- Remove tests that wanted a psql sink without a database, since that case
is already tested elsewhere.
(cherry picked from commit ab1788b922)
* internal/proxy: add initial set of abci metrics (#7115)
This PR adds an initial set of metrics for use ABCI. The initial metrics enable the calculation of timing histograms and call counts for each of the ABCI methods. The metrics are also labeled as either 'sync' or 'async' to determine if the method call was performed using ABCI's `*Async` methods.
An example of these metrics is included here for reference:
```
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.0001"} 0
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.0004"} 5
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.002"} 12
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.009"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.02"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.1"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="0.65"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="2"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="6"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="25"} 13
tendermint_abci_connection_method_timing_bucket{chain_id="ci",method="commit",type="sync",le="+Inf"} 13
tendermint_abci_connection_method_timing_sum{chain_id="ci",method="commit",type="sync"} 0.007802058000000001
tendermint_abci_connection_method_timing_count{chain_id="ci",method="commit",type="sync"} 13
```
These metrics can easily be graphed using prometheus's `histogram_quantile(...)` method to pick out a particular quantile to graph or examine. I chose buckets that were somewhat of an estimate of expected range of times for ABCI operations. They start at .0001 seconds and range to 25 seconds. The hope is that this range captures enough possible times to be useful for us and operators.
* lint++
* docs: add abci timing metrics to the metrics docs (#7311)
* cherry-pick fixup
The event examples for the query filter language were not updated after the
change of key and value types from []byte to string. Also, the attributes need
to be a slice not a bare value.
(cherry picked from commit da3449599f)
Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
If a subscriber arrives while the pubsub service is shutting down, the existing
code will return a nil subscription without error. With unlucky timing, this
may lead to a nil indirection panic in the RPC service.
To avoid that problem, make sure that when a subscription fails for this
reason, we report a non-nil error so that the client will detect it and give up
gracefully.
This is a manual backport of the changes to how we build and run the protobuf
toolchain images in Docker. The main effect here is to point to the new image
from ghcr.io/tendermint/docker-proto-builder, but to make that work it is also
necessary to update some of the branch pointers.
This change does NOT include the changes from #7269 and #7291 to point to the
proto files in the spec repo. To do that, we will need to create a branch or
tag on the spec that has the released version, which does not exist in the spec
history as it currently stands.
* p2p: reduce peer score for dial failures (#7265)
When dialing fails to succeed we should reduce the score of the peer,
which puts the peer at (potentially) greater chances of being removed
from the peer manager, and reduces the chance of the peer being
gossiped by the PEX reactor.
(cherry picked from commit 27560cf7a4)
Co-authored-by: Sam Kleinman <garen@tychoish.com>
The evidence test produces a set of mock evidence in the evidence pool of the 'Primary' node. The test then fills the evidence pools of secondaries with half of this mock evidence. Finally, the test waits until the secondary has an evidence pool as full as the primary.
The assertions that are removed here were checking that the primary and secondaries' evidence channels were empty. However, nothing in the test actually ensures that the channels are empty. The test only waits for the secondaries to have received the complete set of evidence, and the secondaries already received half of the evidence at the beginning. It's more than possible that the secondaries can receive the complete set of evidence and not finish reading the duplicate evidence off the channels.
(cherry picked from commit 4acd117b5e)
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
As a safety measure, don't allow a query string to be unreasonably
long. The query filter is not especially efficient, so a query that
needs more than basic detail should filter coarsely in the subscriber
and refine on the client side.
This affects Subscribe and TxSearch queries.
(cherry picked from commit 9dc3d7f9a2)
mergify[bot]
dependabot[bot]
William Banfield
Sam Kleinman
M. J. Fromberger
Thane Thomson