You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

307 lines
10 KiB

6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
  1. # Running in production
  2. ## Logging
  3. Default logging level (`main:info,state:info,*:`) should suffice for
  4. normal operation mode. Read [this
  5. post](https://blog.cosmos.network/one-of-the-exciting-new-features-in-0-10-0-release-is-smart-log-level-flag-e2506b4ab756)
  6. for details on how to configure `log_level` config variable. Some of the
  7. modules can be found [here](./how-to-read-logs.md#list-of-modules). If
  8. you're trying to debug Tendermint or asked to provide logs with debug
  9. logging level, you can do so by running tendermint with
  10. `--log_level="*:debug"`.
  11. ## DOS Exposure and Mitigation
  12. Validators are supposed to setup [Sentry Node
  13. Architecture](https://blog.cosmos.network/tendermint-explained-bringing-bft-based-pos-to-the-public-blockchain-domain-f22e274a0fdb)
  14. to prevent Denial-of-service attacks. You can read more about it
  15. [here](../interviews/tendermint-bft.md).
  16. ### P2P
  17. The core of the Tendermint peer-to-peer system is `MConnection`. Each
  18. connection has `MaxPacketMsgPayloadSize`, which is the maximum packet
  19. size and bounded send & receive queues. One can impose restrictions on
  20. send & receive rate per connection (`SendRate`, `RecvRate`).
  21. ### RPC
  22. Endpoints returning multiple entries are limited by default to return 30
  23. elements (100 max).
  24. Rate-limiting and authentication are another key aspects to help protect
  25. against DOS attacks. While in the future we may implement these
  26. features, for now, validators are supposed to use external tools like
  27. [NGINX](https://www.nginx.com/blog/rate-limiting-nginx/) or
  28. [traefik](https://docs.traefik.io/configuration/commons/#rate-limiting)
  29. to achieve the same things.
  30. ## Debugging Tendermint
  31. If you ever have to debug Tendermint, the first thing you should
  32. probably do is to check out the logs. See ["How to read
  33. logs"](./how-to-read-logs.md), where we explain what certain log
  34. statements mean.
  35. If, after skimming through the logs, things are not clear still, the
  36. second TODO is to query the /status RPC endpoint. It provides the
  37. necessary info: whenever the node is syncing or not, what height it is
  38. on, etc.
  39. ```
  40. curl http(s)://{ip}:{rpcPort}/status
  41. ```
  42. `dump_consensus_state` will give you a detailed overview of the
  43. consensus state (proposer, lastest validators, peers states). From it,
  44. you should be able to figure out why, for example, the network had
  45. halted.
  46. ```
  47. curl http(s)://{ip}:{rpcPort}/dump_consensus_state
  48. ```
  49. There is a reduced version of this endpoint - `consensus_state`, which
  50. returns just the votes seen at the current height.
  51. - [Github Issues](https://github.com/tendermint/tendermint/issues)
  52. - [StackOverflow
  53. questions](https://stackoverflow.com/questions/tagged/tendermint)
  54. ## Monitoring Tendermint
  55. Each Tendermint instance has a standard `/health` RPC endpoint, which
  56. responds with 200 (OK) if everything is fine and 500 (or no response) -
  57. if something is wrong.
  58. Other useful endpoints include mentioned earlier `/status`, `/net_info` and
  59. `/validators`.
  60. We have a small tool, called `tm-monitor`, which outputs information from
  61. the endpoints above plus some statistics. The tool can be found
  62. [here](https://github.com/tendermint/tools/tree/master/tm-monitor).
  63. Tendermint also can report and serve Prometheus metrics. See
  64. [Metrics](./metrics.md).
  65. ## What happens when my app dies?
  66. You are supposed to run Tendermint under a [process
  67. supervisor](https://en.wikipedia.org/wiki/Process_supervision) (like
  68. systemd or runit). It will ensure Tendermint is always running (despite
  69. possible errors).
  70. Getting back to the original question, if your application dies,
  71. Tendermint will panic. After a process supervisor restarts your
  72. application, Tendermint should be able to reconnect successfully. The
  73. order of restart does not matter for it.
  74. ## Signal handling
  75. We catch SIGINT and SIGTERM and try to clean up nicely. For other
  76. signals we use the default behaviour in Go: [Default behavior of signals
  77. in Go
  78. programs](https://golang.org/pkg/os/signal/#hdr-Default_behavior_of_signals_in_Go_programs).
  79. ## Corruption
  80. **NOTE:** Make sure you have a backup of the Tendermint data directory.
  81. ### Possible causes
  82. Remember that most corruption is caused by hardware issues:
  83. - RAID controllers with faulty / worn out battery backup, and an unexpected power loss
  84. - Hard disk drives with write-back cache enabled, and an unexpected power loss
  85. - Cheap SSDs with insufficient power-loss protection, and an unexpected power-loss
  86. - Defective RAM
  87. - Defective or overheating CPU(s)
  88. Other causes can be:
  89. - Database systems configured with fsync=off and an OS crash or power loss
  90. - Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit.
  91. - Tendermint bugs
  92. - Operating system bugs
  93. - Admin error (e.g., directly modifying Tendermint data-directory contents)
  94. (Source: https://wiki.postgresql.org/wiki/Corruption)
  95. ### WAL Corruption
  96. If consensus WAL is corrupted at the lastest height and you are trying to start
  97. Tendermint, replay will fail with panic.
  98. Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take:
  99. 1) Delete the WAL file and restart Tendermint. It will attempt to sync with other peers.
  100. 2) Try to repair the WAL file manually:
  101. 1. Create a backup of the corrupted WAL file:
  102. ```
  103. cp "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup
  104. ```
  105. 2. Use `./scripts/wal2json` to create a human-readable version
  106. ```
  107. ./scripts/wal2json/wal2json "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal
  108. ```
  109. 3. Search for a "CORRUPTED MESSAGE" line.
  110. 4. By looking at the previous message and the message after the corrupted one
  111. and looking at the logs, try to rebuild the message. If the consequent
  112. messages are marked as corrupted too (this may happen if length header
  113. got corrupted or some writes did not make it to the WAL ~ truncation),
  114. then remove all the lines starting from the corrupted one and restart
  115. Tendermint.
  116. ```
  117. $EDITOR /tmp/corrupted_wal
  118. ```
  119. 5. After editing, convert this file back into binary form by running:
  120. ```
  121. ./scripts/json2wal/json2wal /tmp/corrupted_wal > "$TMHOME/data/cs.wal/wal"
  122. ```
  123. ## Hardware
  124. ### Processor and Memory
  125. While actual specs vary depending on the load and validators count,
  126. minimal requirements are:
  127. - 1GB RAM
  128. - 25GB of disk space
  129. - 1.4 GHz CPU
  130. SSD disks are preferable for applications with high transaction
  131. throughput.
  132. Recommended:
  133. - 2GB RAM
  134. - 100GB SSD
  135. - x64 2.0 GHz 2v CPU
  136. While for now, Tendermint stores all the history and it may require
  137. significant disk space over time, we are planning to implement state
  138. syncing (See
  139. [this issue](https://github.com/tendermint/tendermint/issues/828)). So,
  140. storing all the past blocks will not be necessary.
  141. ### Operating Systems
  142. Tendermint can be compiled for a wide range of operating systems thanks
  143. to Go language (the list of \$OS/\$ARCH pairs can be found
  144. [here](https://golang.org/doc/install/source#environment)).
  145. While we do not favor any operation system, more secure and stable Linux
  146. server distributions (like Centos) should be preferred over desktop
  147. operation systems (like Mac OS).
  148. ### Miscellaneous
  149. NOTE: if you are going to use Tendermint in a public domain, make sure
  150. you read [hardware recommendations (see "4.
  151. Hardware")](https://cosmos.network/validators) for a validator in the
  152. Cosmos network.
  153. ## Configuration parameters
  154. - `p2p.flush_throttle_timeout` `p2p.max_packet_msg_payload_size`
  155. `p2p.send_rate` `p2p.recv_rate`
  156. If you are going to use Tendermint in a private domain and you have a
  157. private high-speed network among your peers, it makes sense to lower
  158. flush throttle timeout and increase other params.
  159. ```
  160. [p2p]
  161. send_rate=20000000 # 2MB/s
  162. recv_rate=20000000 # 2MB/s
  163. flush_throttle_timeout=10
  164. max_packet_msg_payload_size=10240 # 10KB
  165. ```
  166. - `mempool.recheck`
  167. After every block, Tendermint rechecks every transaction left in the
  168. mempool to see if transactions committed in that block affected the
  169. application state, so some of the transactions left may become invalid.
  170. If that does not apply to your application, you can disable it by
  171. setting `mempool.recheck=false`.
  172. - `mempool.broadcast`
  173. Setting this to false will stop the mempool from relaying transactions
  174. to other peers until they are included in a block. It means only the
  175. peer you send the tx to will see it until it is included in a block.
  176. - `consensus.skip_timeout_commit`
  177. We want `skip_timeout_commit=false` when there is economics on the line
  178. because proposers should wait to hear for more votes. But if you don't
  179. care about that and want the fastest consensus, you can skip it. It will
  180. be kept false by default for public deployments (e.g. [Cosmos
  181. Hub](https://cosmos.network/intro/hub)) while for enterprise
  182. applications, setting it to true is not a problem.
  183. - `consensus.peer_gossip_sleep_duration`
  184. You can try to reduce the time your node sleeps before checking if
  185. theres something to send its peers.
  186. - `consensus.timeout_commit`
  187. You can also try lowering `timeout_commit` (time we sleep before
  188. proposing the next block).
  189. - `p2p.addr_book_strict`
  190. By default, Tendermint checks whenever a peer's address is routable before
  191. saving it to the address book. The address is considered as routable if the IP
  192. is [valid and within allowed
  193. ranges](https://github.com/tendermint/tendermint/blob/27bd1deabe4ba6a2d9b463b8f3e3f1e31b993e61/p2p/netaddress.go#L209).
  194. This may not be the case for private networks, where your IP range is usually
  195. strictly limited and private. If that case, you need to set `addr_book_strict`
  196. to `false` (turn off).
  197. - `rpc.max_open_connections`
  198. By default, the number of simultaneous connections is limited because most OS
  199. give you limited number of file descriptors.
  200. If you want to accept greater number of connections, you will need to increase
  201. these limits.
  202. [Sysctls to tune the system to be able to open more connections](https://github.com/satori-com/tcpkali/blob/master/doc/tcpkali.man.md#sysctls-to-tune-the-system-to-be-able-to-open-more-connections)
  203. ...for N connections, such as 50k:
  204. ```
  205. kern.maxfiles=10000+2*N # BSD
  206. kern.maxfilesperproc=100+2*N # BSD
  207. kern.ipc.maxsockets=10000+2*N # BSD
  208. fs.file-max=10000+2*N # Linux
  209. net.ipv4.tcp_max_orphans=N # Linux
  210. # For load-generating clients.
  211. net.ipv4.ip_local_port_range="10000 65535" # Linux.
  212. net.inet.ip.portrange.first=10000 # BSD/Mac.
  213. net.inet.ip.portrange.last=65535 # (Enough for N < 55535)
  214. net.ipv4.tcp_tw_reuse=1 # Linux
  215. net.inet.tcp.maxtcptw=2*N # BSD
  216. # If using netfilter on Linux:
  217. net.netfilter.nf_conntrack_max=N
  218. echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize
  219. ```
  220. The similar option exists for limiting the number of gRPC connections -
  221. `rpc.grpc_max_open_connections`.