You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

216 lines
9.3 KiB

  1. ---
  2. order: 3
  3. ---
  4. # Proposer-Based Timestamps Runbook
  5. Version v0.36 of Tendermint added new constraints for the timestamps included in
  6. each block created by Tendermint. The new constraints mean that validators may
  7. fail to produce valid blocks or may issue `nil` `prevotes` for proposed blocks
  8. depending on the configuration of the validator's local clock.
  9. ## What is this document for?
  10. This document provides a set of actionable steps for application developers and
  11. node operators to diagnose and fix issues related to clock synchronization and
  12. configuration of the Proposer-Based Timestamps [SynchronyParams](https://github.com/tendermint/tendermint/blob/master/spec/core/data_structures.md#synchronyparams).
  13. Use this runbook if you observe that validators are frequently voting `nil` for a block that the rest
  14. of the network votes for or if validators are frequently producing block proposals
  15. that are not voted for by the rest of the network.
  16. ## Requirements
  17. To use this runbook, you must be running a node that has the [Prometheus metrics endpoint enabled](https://github.com/tendermint/tendermint/blob/master/docs/nodes/metrics.md)
  18. and the Tendermint RPC endpoint enabled and accessible.
  19. It is strongly recommended to also run a Prometheus metrics collector to gather and
  20. analyze metrics from the Tendermint node.
  21. ## Debugging a Single Node
  22. If you observe that a single validator is frequently failing to produce blocks or
  23. voting nil for proposals that other validators vote for and suspect it may be
  24. related to clock synchronization, use the following steps to debug and correct the issue.
  25. ### Check Timely Metric
  26. Tendermint exposes a histogram metric for the difference between the timestamp in the proposal
  27. the and the time read from the node's local clock when the proposal is received.
  28. The histogram exposes multiple metrics on the Prometheus `/metrics` endpoint called
  29. * `tendermint_consensus_proposal_timestamp_difference_bucket`.
  30. * `tendermint_consensus_proposal_timestamp_difference_sum`.
  31. * `tendermint_consensus_proposal_timestamp_difference_count`.
  32. Each metric is also labeled with the key `is_timely`, which can have a value of
  33. `true` or `false`.
  34. #### From the Prometheus Collector UI
  35. If you are running a Prometheus collector, navigate to the query web interface and select the 'Graph' tab.
  36. Issue a query for the following:
  37. ```
  38. tendermint_consensus_proposal_timestamp_difference_count{is_timely="false"} /
  39. tendermint_consensus_proposal_timestamp_difference_count{is_timely="true"}
  40. ```
  41. This query will graph the ratio of proposals the node considered timely to those it
  42. considered untimely. If the ratio is increasing, it means that your node is consistently
  43. seeing more proposals that are far from its local clock. If this is the case, you should
  44. check to make sure your local clock is properly synchronized to NTP.
  45. #### From the `/metrics` url
  46. If you are not running a Prometheus collector, navigate to the `/metrics` endpoint
  47. exposed on the Prometheus metrics port with `curl` or a browser.
  48. Search for the `tendermint_consensus_proposal_timestamp_difference_count` metrics.
  49. This metric is labeled with `is_timely`. Investigate the value of
  50. `tendermint_consensus_proposal_timestamp_difference_count` where `is_timely="false"`
  51. and where `is_timely="true"`. Refresh the endpoint and observe if the value of `is_timely="false"`
  52. is growing.
  53. If you observe that `is_timely="false"` is growing, it means that your node is consistently
  54. seeing proposals that are far from its local clock. If this is the case, you should check
  55. to make sure your local clock is properly synchronized to NTP.
  56. ### Checking Clock Sync
  57. NTP configuration and tooling is very specific to the operating system and distribution
  58. that your validator node is running. This guide assumes you have `timedatectl` installed with
  59. [chrony](https://chrony.tuxfamily.org/), a popular tool for interacting with time
  60. synchronization on Linux distributions. If you are using an operating system or
  61. distribution with a different time synchronization mechanism, please consult the
  62. documentation for your operating system to check the status and re-synchronize the daemon.
  63. #### Check if NTP is Enabled
  64. ```shell
  65. $ timedatectl
  66. ```
  67. From the output, ensure that `NTP service` is `active`. If `NTP service` is `inactive`, run:
  68. ```shell
  69. $ timedatectl set-ntp true
  70. ```
  71. Re-run the `timedatectl` command and verify that the change has taken effect.
  72. #### Check if Your NTP Daemon is Synchronized
  73. Check the status of your local `chrony` NTP daemon using by running the following:
  74. ```shell
  75. $ chronyc tracking
  76. ```
  77. If the `chrony` daemon is running, you will see output that indicates its current status.
  78. If the `chrony` daemon is not running, restart it and re-run `chronyc tracking`.
  79. The `System time` field of the response should show a value that is much smaller than 100
  80. milliseconds.
  81. If the value is very large, restart the `chronyd` daemon.
  82. ## Debugging a Network
  83. If you observe that a network is frequently failing to produce blocks and suspect
  84. it may be related to clock synchronization, use the following steps to debug and correct the issue.
  85. ### Check Prevote Message Delay
  86. Tendermint exposes metrics that help determine how synchronized the clocks on a network are.
  87. These metrics are visible on the Prometheus `/metrics` endpoint and are called:
  88. * `tendermint_consensus_quorum_prevote_delay`
  89. * `tendermint_consensus_full_prevote_delay`
  90. These metrics calculate the difference between the timestamp in the proposal message and
  91. the timestamp of a prevote that was issued during consensus.
  92. The `tendermint_consensus_quorum_prevote_delay` metric is the interval in seconds
  93. between the proposal timestamp and the timestamp of the earliest prevote that
  94. achieved a quorum during the prevote step.
  95. The `tendermint_consensus_full_prevote_delay` metric is the interval in seconds
  96. between the proposal timestamp and the timestamp of the latest prevote in a round
  97. where 100% of the validators voted.
  98. #### From the Prometheus Collector UI
  99. If you are running a Prometheus collector, navigate to the query web interface and select the 'Graph' tab.
  100. Issue a query for the following:
  101. ```
  102. sum(tendermint_consensus_quorum_prevote_delay) by (proposer_address)
  103. ```
  104. This query will graph the difference in seconds for each proposer on the network.
  105. If the value is much larger for some proposers, then the issue is likely related to the clock
  106. synchronization of their nodes. Contact those proposers and ensure that their nodes
  107. are properly connected to NTP using the steps for [Debugging a Single Node](#debugging-a-single-node).
  108. If the value is relatively similar for all proposers you should next compare this
  109. value to the `SynchronyParams` values for the network. Continue to the [Checking
  110. Sychrony](#checking-synchrony) steps.
  111. #### From the `/metrics` url
  112. If you are not running a Prometheus collector, navigate to the `/metrics` endpoint
  113. exposed on the Prometheus metrics port.
  114. Search for the `tendermint_consensus_quorum_prevote_delay` metric. There will be one
  115. entry of this metric for each `proposer_address`. If the value of this metric is
  116. much larger for some proposers, then the issue is likely related to synchronization of their
  117. nodes with NTP. Contact those proposers and ensure that their nodes are properly connected
  118. to NTP using the steps for [Debugging a Single Node](#debugging-a-single-node).
  119. If the values are relatively similar for all proposers you should next compare,
  120. you'll need to compare this value to the `SynchronyParams` for the network. Continue
  121. to the [Checking Sychrony](#checking-synchrony) steps.
  122. ### Checking Synchrony
  123. To determine the currently configured `SynchronyParams` for your network, issue a
  124. request to your node's RPC endpoint. For a node running locally with the RPC server
  125. exposed on port `26657`, run the following command:
  126. ```shell
  127. $ curl localhost:26657/consensus_params
  128. ```
  129. The json output will contain a field named `synchrony`, with the following structure:
  130. ```json
  131. {
  132. "precision": "500000000",
  133. "message_delay": "3000000000"
  134. }
  135. ```
  136. The `precision` and `message_delay` values returned are listed in nanoseconds:
  137. In the examples above, the precision is 500ms and the message delay is 3s.
  138. Remember, `tendermint_consensus_quorum_prevote_delay` is listed in seconds.
  139. If the `tendermint_consensus_quorum_prevote_delay` value approaches the sum of `precision` and `message_delay`,
  140. then the value selected for these parameters is too small. Your application will
  141. need to be modified to update the `SynchronyParams` to have larger values.
  142. ### Updating SynchronyParams
  143. The `SynchronyParams` are `ConsensusParameters` which means they are set and updated
  144. by the application running alongside Tendermint. Updates to these parameters must
  145. be passed to the application during the `FinalizeBlock` ABCI method call.
  146. If the application was built using the CosmosSDK, then these parameters can be updated
  147. programatically using a governance proposal. For more information, see the [CosmosSDK
  148. documentation](https://hub.cosmos.network/main/governance/submitting.html#sending-the-transaction-that-submits-your-governance-proposal).
  149. If the application does not implement a way to update the consensus parameters
  150. programatically, then the application itself must be updated to do so. More information on updating
  151. the consensus parameters via ABCI can be found in the [FinalizeBlock documentation](https://github.com/tendermint/tendermint/blob/master/spec/abci++/abci++_methods_002_draft.md#finalizeblock).