You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

377 lines
14 KiB

6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
  1. # Running in production
  2. ## Database
  3. By default, Tendermint uses the `syndtr/goleveldb` package for its in-process
  4. key-value database. Unfortunately, this implementation of LevelDB seems to suffer under heavy load (see
  5. [#226](https://github.com/syndtr/goleveldb/issues/226)). It may be best to
  6. install the real C-implementation of LevelDB and compile Tendermint to use
  7. that using `make build_c`. See the [install instructions](../introduction/install.md) for details.
  8. Tendermint keeps multiple distinct databases in the `$TMROOT/data`:
  9. - `blockstore.db`: Keeps the entire blockchain - stores blocks,
  10. block commits, and block meta data, each indexed by height. Used to sync new
  11. peers.
  12. - `evidence.db`: Stores all verified evidence of misbehaviour.
  13. - `state.db`: Stores the current blockchain state (ie. height, validators,
  14. consensus params). Only grows if consensus params or validators change. Also
  15. used to temporarily store intermediate results during block processing.
  16. - `tx_index.db`: Indexes txs (and their results) by tx hash and by DeliverTx result tags.
  17. By default, Tendermint will only index txs by their hash, not by their DeliverTx
  18. result tags. See [indexing transactions](../app-dev/indexing-transactions.md) for
  19. details.
  20. There is no current strategy for pruning the databases. Consider reducing
  21. block production by [controlling empty blocks](../tendermint-core/using-tendermint.md#no-empty-blocks)
  22. or by increasing the `consensus.timeout_commit` param. Note both of these are
  23. local settings and not enforced by the consensus.
  24. We're working on [state
  25. syncing](https://github.com/tendermint/tendermint/issues/828),
  26. which will enable history to be thrown away
  27. and recent application state to be directly synced. We'll need to develop solutions
  28. for archival nodes that allow queries on historical transactions and states.
  29. The Cosmos project has had much success just dumping the latest state of a
  30. blockchain to disk and starting a new chain from that state.
  31. ## Logging
  32. Default logging level (`main:info,state:info,*:`) should suffice for
  33. normal operation mode. Read [this
  34. post](https://blog.cosmos.network/one-of-the-exciting-new-features-in-0-10-0-release-is-smart-log-level-flag-e2506b4ab756)
  35. for details on how to configure `log_level` config variable. Some of the
  36. modules can be found [here](./how-to-read-logs.md#list-of-modules). If
  37. you're trying to debug Tendermint or asked to provide logs with debug
  38. logging level, you can do so by running tendermint with
  39. `--log_level="*:debug"`.
  40. ## Write Ahead Logs (WAL)
  41. Tendermint uses write ahead logs for the consensus (`cs.wal`) and the mempool
  42. (`mempool.wal`). Both WALs have a max size of 1GB and are automatically rotated.
  43. ### Consensus WAL
  44. The `consensus.wal` is used to ensure we can recover from a crash at any point
  45. in the consensus state machine.
  46. It writes all consensus messages (timeouts, proposals, block part, or vote)
  47. to a single file, flushing to disk before processing messages from its own
  48. validator. Since Tendermint validators are expected to never sign a conflicting vote, the
  49. WAL ensures we can always recover deterministically to the latest state of the consensus without
  50. using the network or re-signing any consensus messages.
  51. If your `consensus.wal` is corrupted, see [below](#wal-corruption).
  52. ### Mempool WAL
  53. The `mempool.wal` logs all incoming txs before running CheckTx, but is
  54. otherwise not used in any programmatic way. It's just a kind of manual
  55. safe guard. Note the mempool provides no durability guarantees - a tx sent to one or many nodes
  56. may never make it into the blockchain if those nodes crash before being able to
  57. propose it. Clients must monitor their txs by subscribing over websockets,
  58. polling for them, or using `/broadcast_tx_commit`. In the worst case, txs can be
  59. resent from the mempool WAL manually.
  60. For the above reasons, the `mempool.wal` is disabled by default. To enable, set
  61. `mempool.wal_dir` to where you want the WAL to be located (e.g.
  62. `data/mempool.wal`).
  63. ## DOS Exposure and Mitigation
  64. Validators are supposed to setup [Sentry Node
  65. Architecture](https://blog.cosmos.network/tendermint-explained-bringing-bft-based-pos-to-the-public-blockchain-domain-f22e274a0fdb)
  66. to prevent Denial-of-service attacks. You can read more about it
  67. [here](../interviews/tendermint-bft.md).
  68. ### P2P
  69. The core of the Tendermint peer-to-peer system is `MConnection`. Each
  70. connection has `MaxPacketMsgPayloadSize`, which is the maximum packet
  71. size and bounded send & receive queues. One can impose restrictions on
  72. send & receive rate per connection (`SendRate`, `RecvRate`).
  73. ### RPC
  74. Endpoints returning multiple entries are limited by default to return 30
  75. elements (100 max). See the [RPC Documentation](https://tendermint.com/rpc/)
  76. for more information.
  77. Rate-limiting and authentication are another key aspects to help protect
  78. against DOS attacks. While in the future we may implement these
  79. features, for now, validators are supposed to use external tools like
  80. [NGINX](https://www.nginx.com/blog/rate-limiting-nginx/) or
  81. [traefik](https://docs.traefik.io/configuration/commons/#rate-limiting)
  82. to achieve the same things.
  83. ## Debugging Tendermint
  84. If you ever have to debug Tendermint, the first thing you should
  85. probably do is to check out the logs. See [How to read
  86. logs](./how-to-read-logs.md), where we explain what certain log
  87. statements mean.
  88. If, after skimming through the logs, things are not clear still, the
  89. next thing to try is query the /status RPC endpoint. It provides the
  90. necessary info: whenever the node is syncing or not, what height it is
  91. on, etc.
  92. ```
  93. curl http(s)://{ip}:{rpcPort}/status
  94. ```
  95. `dump_consensus_state` will give you a detailed overview of the
  96. consensus state (proposer, lastest validators, peers states). From it,
  97. you should be able to figure out why, for example, the network had
  98. halted.
  99. ```
  100. curl http(s)://{ip}:{rpcPort}/dump_consensus_state
  101. ```
  102. There is a reduced version of this endpoint - `consensus_state`, which
  103. returns just the votes seen at the current height.
  104. - [Github Issues](https://github.com/tendermint/tendermint/issues)
  105. - [StackOverflow
  106. questions](https://stackoverflow.com/questions/tagged/tendermint)
  107. ## Monitoring Tendermint
  108. Each Tendermint instance has a standard `/health` RPC endpoint, which
  109. responds with 200 (OK) if everything is fine and 500 (or no response) -
  110. if something is wrong.
  111. Other useful endpoints include mentioned earlier `/status`, `/net_info` and
  112. `/validators`.
  113. We have a small tool, called `tm-monitor`, which outputs information from
  114. the endpoints above plus some statistics. The tool can be found
  115. [here](https://github.com/tendermint/tendermint/tree/master/tools/tm-monitor).
  116. Tendermint also can report and serve Prometheus metrics. See
  117. [Metrics](./metrics.md).
  118. ## What happens when my app dies?
  119. You are supposed to run Tendermint under a [process
  120. supervisor](https://en.wikipedia.org/wiki/Process_supervision) (like
  121. systemd or runit). It will ensure Tendermint is always running (despite
  122. possible errors).
  123. Getting back to the original question, if your application dies,
  124. Tendermint will panic. After a process supervisor restarts your
  125. application, Tendermint should be able to reconnect successfully. The
  126. order of restart does not matter for it.
  127. ## Signal handling
  128. We catch SIGINT and SIGTERM and try to clean up nicely. For other
  129. signals we use the default behaviour in Go: [Default behavior of signals
  130. in Go
  131. programs](https://golang.org/pkg/os/signal/#hdr-Default_behavior_of_signals_in_Go_programs).
  132. ## Corruption
  133. **NOTE:** Make sure you have a backup of the Tendermint data directory.
  134. ### Possible causes
  135. Remember that most corruption is caused by hardware issues:
  136. - RAID controllers with faulty / worn out battery backup, and an unexpected power loss
  137. - Hard disk drives with write-back cache enabled, and an unexpected power loss
  138. - Cheap SSDs with insufficient power-loss protection, and an unexpected power-loss
  139. - Defective RAM
  140. - Defective or overheating CPU(s)
  141. Other causes can be:
  142. - Database systems configured with fsync=off and an OS crash or power loss
  143. - Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit.
  144. - Tendermint bugs
  145. - Operating system bugs
  146. - Admin error (e.g., directly modifying Tendermint data-directory contents)
  147. (Source: https://wiki.postgresql.org/wiki/Corruption)
  148. ### WAL Corruption
  149. If consensus WAL is corrupted at the lastest height and you are trying to start
  150. Tendermint, replay will fail with panic.
  151. Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take:
  152. 1. Delete the WAL file and restart Tendermint. It will attempt to sync with other peers.
  153. 2. Try to repair the WAL file manually:
  154. 1) Create a backup of the corrupted WAL file:
  155. ```
  156. cp "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup
  157. ```
  158. 2. Use `./scripts/wal2json` to create a human-readable version
  159. ```
  160. ./scripts/wal2json/wal2json "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal
  161. ```
  162. 3. Search for a "CORRUPTED MESSAGE" line.
  163. 4. By looking at the previous message and the message after the corrupted one
  164. and looking at the logs, try to rebuild the message. If the consequent
  165. messages are marked as corrupted too (this may happen if length header
  166. got corrupted or some writes did not make it to the WAL ~ truncation),
  167. then remove all the lines starting from the corrupted one and restart
  168. Tendermint.
  169. ```
  170. $EDITOR /tmp/corrupted_wal
  171. ```
  172. 5. After editing, convert this file back into binary form by running:
  173. ```
  174. ./scripts/json2wal/json2wal /tmp/corrupted_wal $TMHOME/data/cs.wal/wal
  175. ```
  176. ## Hardware
  177. ### Processor and Memory
  178. While actual specs vary depending on the load and validators count,
  179. minimal requirements are:
  180. - 1GB RAM
  181. - 25GB of disk space
  182. - 1.4 GHz CPU
  183. SSD disks are preferable for applications with high transaction
  184. throughput.
  185. Recommended:
  186. - 2GB RAM
  187. - 100GB SSD
  188. - x64 2.0 GHz 2v CPU
  189. While for now, Tendermint stores all the history and it may require
  190. significant disk space over time, we are planning to implement state
  191. syncing (See
  192. [this issue](https://github.com/tendermint/tendermint/issues/828)). So,
  193. storing all the past blocks will not be necessary.
  194. ### Operating Systems
  195. Tendermint can be compiled for a wide range of operating systems thanks
  196. to Go language (the list of \$OS/\$ARCH pairs can be found
  197. [here](https://golang.org/doc/install/source#environment)).
  198. While we do not favor any operation system, more secure and stable Linux
  199. server distributions (like Centos) should be preferred over desktop
  200. operation systems (like Mac OS).
  201. ### Miscellaneous
  202. NOTE: if you are going to use Tendermint in a public domain, make sure
  203. you read [hardware recommendations](https://cosmos.network/validators) for a validator in the
  204. Cosmos network.
  205. ## Configuration parameters
  206. - `p2p.flush_throttle_timeout`
  207. - `p2p.max_packet_msg_payload_size`
  208. - `p2p.send_rate`
  209. - `p2p.recv_rate`
  210. If you are going to use Tendermint in a private domain and you have a
  211. private high-speed network among your peers, it makes sense to lower
  212. flush throttle timeout and increase other params.
  213. ```
  214. [p2p]
  215. send_rate=20000000 # 2MB/s
  216. recv_rate=20000000 # 2MB/s
  217. flush_throttle_timeout=10
  218. max_packet_msg_payload_size=10240 # 10KB
  219. ```
  220. - `mempool.recheck`
  221. After every block, Tendermint rechecks every transaction left in the
  222. mempool to see if transactions committed in that block affected the
  223. application state, so some of the transactions left may become invalid.
  224. If that does not apply to your application, you can disable it by
  225. setting `mempool.recheck=false`.
  226. - `mempool.broadcast`
  227. Setting this to false will stop the mempool from relaying transactions
  228. to other peers until they are included in a block. It means only the
  229. peer you send the tx to will see it until it is included in a block.
  230. - `consensus.skip_timeout_commit`
  231. We want `skip_timeout_commit=false` when there is economics on the line
  232. because proposers should wait to hear for more votes. But if you don't
  233. care about that and want the fastest consensus, you can skip it. It will
  234. be kept false by default for public deployments (e.g. [Cosmos
  235. Hub](https://cosmos.network/intro/hub)) while for enterprise
  236. applications, setting it to true is not a problem.
  237. - `consensus.peer_gossip_sleep_duration`
  238. You can try to reduce the time your node sleeps before checking if
  239. theres something to send its peers.
  240. - `consensus.timeout_commit`
  241. You can also try lowering `timeout_commit` (time we sleep before
  242. proposing the next block).
  243. - `p2p.addr_book_strict`
  244. By default, Tendermint checks whenever a peer's address is routable before
  245. saving it to the address book. The address is considered as routable if the IP
  246. is [valid and within allowed
  247. ranges](https://github.com/tendermint/tendermint/blob/27bd1deabe4ba6a2d9b463b8f3e3f1e31b993e61/p2p/netaddress.go#L209).
  248. This may not be the case for private or local networks, where your IP range is usually
  249. strictly limited and private. If that case, you need to set `addr_book_strict`
  250. to `false` (turn it off).
  251. - `rpc.max_open_connections`
  252. By default, the number of simultaneous connections is limited because most OS
  253. give you limited number of file descriptors.
  254. If you want to accept greater number of connections, you will need to increase
  255. these limits.
  256. [Sysctls to tune the system to be able to open more connections](https://github.com/satori-com/tcpkali/blob/master/doc/tcpkali.man.md#sysctls-to-tune-the-system-to-be-able-to-open-more-connections)
  257. ...for N connections, such as 50k:
  258. ```
  259. kern.maxfiles=10000+2*N # BSD
  260. kern.maxfilesperproc=100+2*N # BSD
  261. kern.ipc.maxsockets=10000+2*N # BSD
  262. fs.file-max=10000+2*N # Linux
  263. net.ipv4.tcp_max_orphans=N # Linux
  264. # For load-generating clients.
  265. net.ipv4.ip_local_port_range="10000 65535" # Linux.
  266. net.inet.ip.portrange.first=10000 # BSD/Mac.
  267. net.inet.ip.portrange.last=65535 # (Enough for N < 55535)
  268. net.ipv4.tcp_tw_reuse=1 # Linux
  269. net.inet.tcp.maxtcptw=2*N # BSD
  270. # If using netfilter on Linux:
  271. net.netfilter.nf_conntrack_max=N
  272. echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize
  273. ```
  274. The similar option exists for limiting the number of gRPC connections -
  275. `rpc.grpc_max_open_connections`.