You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

376 lines
14 KiB

  1. ---
  2. order: 4
  3. ---
  4. # Running in production
  5. If you are building Tendermint from source for use in production, make sure to check out an appropriate Git tag instead of a branch.
  6. ## Database
  7. By default, Tendermint uses the `syndtr/goleveldb` package for its in-process
  8. key-value database. If you want maximal performance, it may be best to install
  9. the real C-implementation of LevelDB and compile Tendermint to use that using
  10. `make build TENDERMINT_BUILD_OPTIONS=cleveldb`. See the [install
  11. instructions](../introduction/install.md) for details.
  12. Tendermint keeps multiple distinct databases in the `$TMROOT/data`:
  13. - `blockstore.db`: Keeps the entire blockchain - stores blocks,
  14. block commits, and block meta data, each indexed by height. Used to sync new
  15. peers.
  16. - `evidence.db`: Stores all verified evidence of misbehaviour.
  17. - `state.db`: Stores the current blockchain state (ie. height, validators,
  18. consensus params). Only grows if consensus params or validators change. Also
  19. used to temporarily store intermediate results during block processing.
  20. - `tx_index.db`: Indexes txs (and their results) by tx hash and by DeliverTx result events.
  21. By default, Tendermint will only index txs by their hash and height, not by their DeliverTx
  22. result events. See [indexing transactions](../app-dev/indexing-transactions.md) for
  23. details.
  24. Applications can expose block pruning strategies to the node operator. Please read the documentation of your application
  25. to find out more details.
  26. Applications can use [state sync](state-sync.md) to help nodes bootstrap quickly.
  27. ## Logging
  28. Default logging level (`log-level = "info"`) should suffice for
  29. normal operation mode. Read [this
  30. post](https://blog.cosmos.network/one-of-the-exciting-new-features-in-0-10-0-release-is-smart-log-level-flag-e2506b4ab756)
  31. for details on how to configure `log-level` config variable. Some of the
  32. modules can be found [here](../nodes/logging#list-of-modules). If
  33. you're trying to debug Tendermint or asked to provide logs with debug
  34. logging level, you can do so by running Tendermint with
  35. `--log-level="debug"`.
  36. ### Consensus WAL
  37. Tendermint uses a write ahead log (WAL) for consensus. The `consensus.wal` is used to ensure we can recover from a crash at any point
  38. in the consensus state machine. It writes all consensus messages (timeouts, proposals, block part, or vote)
  39. to a single file, flushing to disk before processing messages from its own
  40. validator. Since Tendermint validators are expected to never sign a conflicting vote, the
  41. WAL ensures we can always recover deterministically to the latest state of the consensus without
  42. using the network or re-signing any consensus messages. The consensus WAL max size of 1GB and is automatically rotated.
  43. If your `consensus.wal` is corrupted, see [below](#wal-corruption).
  44. ## DOS Exposure and Mitigation
  45. Validators are supposed to setup [Sentry Node
  46. Architecture](./validators.md)
  47. to prevent Denial-of-service attacks.
  48. ### P2P
  49. The core of the Tendermint peer-to-peer system is `MConnection`. Each
  50. connection has `MaxPacketMsgPayloadSize`, which is the maximum packet
  51. size and bounded send & receive queues. One can impose restrictions on
  52. send & receive rate per connection (`SendRate`, `RecvRate`).
  53. The number of open P2P connections can become quite large, and hit the operating system's open
  54. file limit (since TCP connections are considered files on UNIX-based systems). Nodes should be
  55. given a sizable open file limit, e.g. 8192, via `ulimit -n 8192` or other deployment-specific
  56. mechanisms.
  57. ### RPC
  58. Endpoints returning multiple entries are limited by default to return 30
  59. elements (100 max). See the [RPC Documentation](https://docs.tendermint.com/master/rpc/)
  60. for more information.
  61. Rate-limiting and authentication are another key aspects to help protect
  62. against DOS attacks. Validators are supposed to use external tools like
  63. [NGINX](https://www.nginx.com/blog/rate-limiting-nginx/) or
  64. [traefik](https://docs.traefik.io/middlewares/ratelimit/)
  65. to achieve the same things.
  66. ## Debugging Tendermint
  67. If you ever have to debug Tendermint, the first thing you should probably do is
  68. check out the logs. See [Logging](../nodes/logging.md), where we
  69. explain what certain log statements mean.
  70. If, after skimming through the logs, things are not clear still, the next thing
  71. to try is querying the `/status` RPC endpoint. It provides the necessary info:
  72. whenever the node is syncing or not, what height it is on, etc.
  73. ```bash
  74. curl http(s)://{ip}:{rpcPort}/status
  75. ```
  76. `/dump_consensus_state` will give you a detailed overview of the consensus
  77. state (proposer, latest validators, peers states). From it, you should be able
  78. to figure out why, for example, the network had halted.
  79. ```bash
  80. curl http(s)://{ip}:{rpcPort}/dump_consensus_state
  81. ```
  82. There is a reduced version of this endpoint - `/consensus_state`, which returns
  83. just the votes seen at the current height.
  84. If, after consulting with the logs and above endpoints, you still have no idea
  85. what's happening, consider using `tendermint debug kill` sub-command. This
  86. command will scrap all the available info and kill the process. See
  87. [Debugging](../tools/debugging.md) for the exact format.
  88. You can inspect the resulting archive yourself or create an issue on
  89. [Github](https://github.com/tendermint/tendermint). Before opening an issue
  90. however, be sure to check if there's [no existing
  91. issue](https://github.com/tendermint/tendermint/issues) already.
  92. ## Monitoring Tendermint
  93. Each Tendermint instance has a standard `/health` RPC endpoint, which responds
  94. with 200 (OK) if everything is fine and 500 (or no response) - if something is
  95. wrong.
  96. Other useful endpoints include mentioned earlier `/status`, `/net_info` and
  97. `/validators`.
  98. Tendermint also can report and serve Prometheus metrics. See
  99. [Metrics](./metrics.md).
  100. `tendermint debug dump` sub-command can be used to periodically dump useful
  101. information into an archive. See [Debugging](../tools/debugging.md) for more
  102. information.
  103. ## What happens when my app dies
  104. You are supposed to run Tendermint under a [process
  105. supervisor](https://en.wikipedia.org/wiki/Process_supervision) (like
  106. systemd or runit). It will ensure Tendermint is always running (despite
  107. possible errors).
  108. Getting back to the original question, if your application dies,
  109. Tendermint will panic. After a process supervisor restarts your
  110. application, Tendermint should be able to reconnect successfully. The
  111. order of restart does not matter for it.
  112. ## Signal handling
  113. We catch SIGINT and SIGTERM and try to clean up nicely. For other
  114. signals we use the default behavior in Go: [Default behavior of signals
  115. in Go
  116. programs](https://golang.org/pkg/os/signal/#hdr-Default_behavior_of_signals_in_Go_programs).
  117. ## Corruption
  118. **NOTE:** Make sure you have a backup of the Tendermint data directory.
  119. ### Possible causes
  120. Remember that most corruption is caused by hardware issues:
  121. - RAID controllers with faulty / worn out battery backup, and an unexpected power loss
  122. - Hard disk drives with write-back cache enabled, and an unexpected power loss
  123. - Cheap SSDs with insufficient power-loss protection, and an unexpected power-loss
  124. - Defective RAM
  125. - Defective or overheating CPU(s)
  126. Other causes can be:
  127. - Database systems configured with fsync=off and an OS crash or power loss
  128. - Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit.
  129. - Tendermint bugs
  130. - Operating system bugs
  131. - Admin error (e.g., directly modifying Tendermint data-directory contents)
  132. (Source: <https://wiki.postgresql.org/wiki/Corruption>)
  133. ### WAL Corruption
  134. If consensus WAL is corrupted at the latest height and you are trying to start
  135. Tendermint, replay will fail with panic.
  136. Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take:
  137. 1. Delete the WAL file and restart Tendermint. It will attempt to sync with other peers.
  138. 2. Try to repair the WAL file manually:
  139. 1) Create a backup of the corrupted WAL file:
  140. ```sh
  141. cp "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup
  142. ```
  143. 2) Use `./scripts/wal2json` to create a human-readable version:
  144. ```sh
  145. ./scripts/wal2json/wal2json "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal
  146. ```
  147. 3) Search for a "CORRUPTED MESSAGE" line.
  148. 4) By looking at the previous message and the message after the corrupted one
  149. and looking at the logs, try to rebuild the message. If the consequent
  150. messages are marked as corrupted too (this may happen if length header
  151. got corrupted or some writes did not make it to the WAL ~ truncation),
  152. then remove all the lines starting from the corrupted one and restart
  153. Tendermint.
  154. ```sh
  155. $EDITOR /tmp/corrupted_wal
  156. ```
  157. 5) After editing, convert this file back into binary form by running:
  158. ```sh
  159. ./scripts/json2wal/json2wal /tmp/corrupted_wal $TMHOME/data/cs.wal/wal
  160. ```
  161. ## Hardware
  162. ### Processor and Memory
  163. While actual specs vary depending on the load and validators count, minimal
  164. requirements are:
  165. - 1GB RAM
  166. - 25GB of disk space
  167. - 1.4 GHz CPU
  168. SSD disks are preferable for applications with high transaction throughput.
  169. Recommended:
  170. - 2GB RAM
  171. - 100GB SSD
  172. - x64 2.0 GHz 2v CPU
  173. While for now, Tendermint stores all the history and it may require significant
  174. disk space over time, we are planning to implement state syncing (See [this
  175. issue](https://github.com/tendermint/tendermint/issues/828)). So, storing all
  176. the past blocks will not be necessary.
  177. ### Validator signing on 32 bit architectures (or ARM)
  178. Both our `ed25519` and `secp256k1` implementations require constant time
  179. `uint64` multiplication. Non-constant time crypto can (and has) leaked
  180. private keys on both `ed25519` and `secp256k1`. This doesn't exist in hardware
  181. on 32 bit x86 platforms ([source](https://bearssl.org/ctmul.html)), and it
  182. depends on the compiler to enforce that it is constant time. It's unclear at
  183. this point whenever the Golang compiler does this correctly for all
  184. implementations.
  185. **We do not support nor recommend running a validator on 32 bit architectures OR
  186. the "VIA Nano 2000 Series", and the architectures in the ARM section rated
  187. "S-".**
  188. ### Operating Systems
  189. Tendermint can be compiled for a wide range of operating systems thanks to Go
  190. language (the list of \$OS/\$ARCH pairs can be found
  191. [here](https://golang.org/doc/install/source#environment)).
  192. While we do not favor any operation system, more secure and stable Linux server
  193. distributions (like Centos) should be preferred over desktop operation systems
  194. (like Mac OS).
  195. Native Windows support is not provided. If you are using a windows machine, you can try using the [bash shell](https://docs.microsoft.com/en-us/windows/wsl/install-win10).
  196. ### Miscellaneous
  197. NOTE: if you are going to use Tendermint in a public domain, make sure
  198. you read [hardware recommendations](https://cosmos.network/validators) for a validator in the
  199. Cosmos network.
  200. ## Configuration parameters
  201. - `p2p.flush-throttle-timeout`
  202. - `p2p.max-packet-msg-payload-size`
  203. - `p2p.send-rate`
  204. - `p2p.recv-rate`
  205. If you are going to use Tendermint in a private domain and you have a
  206. private high-speed network among your peers, it makes sense to lower
  207. flush throttle timeout and increase other params.
  208. ```toml
  209. [p2p]
  210. send-rate=20000000 # 2MB/s
  211. recv-rate=20000000 # 2MB/s
  212. flush-throttle-timeout=10
  213. max-packet-msg-payload-size=10240 # 10KB
  214. ```
  215. - `mempool.recheck`
  216. After every block, Tendermint rechecks every transaction left in the
  217. mempool to see if transactions committed in that block affected the
  218. application state, so some of the transactions left may become invalid.
  219. If that does not apply to your application, you can disable it by
  220. setting `mempool.recheck=false`.
  221. - `mempool.broadcast`
  222. Setting this to false will stop the mempool from relaying transactions
  223. to other peers until they are included in a block. It means only the
  224. peer you send the tx to will see it until it is included in a block.
  225. - `consensus.skip-timeout-commit`
  226. We want `skip-timeout-commit=false` when there is economics on the line
  227. because proposers should wait to hear for more votes. But if you don't
  228. care about that and want the fastest consensus, you can skip it. It will
  229. be kept false by default for public deployments (e.g. [Cosmos
  230. Hub](https://cosmos.network/intro/hub)) while for enterprise
  231. applications, setting it to true is not a problem.
  232. - `consensus.peer-gossip-sleep-duration`
  233. You can try to reduce the time your node sleeps before checking if
  234. theres something to send its peers.
  235. - `consensus.timeout-commit`
  236. You can also try lowering `timeout-commit` (time we sleep before
  237. proposing the next block).
  238. - `p2p.addr-book-strict`
  239. By default, Tendermint checks whenever a peer's address is routable before
  240. saving it to the address book. The address is considered as routable if the IP
  241. is [valid and within allowed
  242. ranges](https://github.com/tendermint/tendermint/blob/27bd1deabe4ba6a2d9b463b8f3e3f1e31b993e61/p2p/netaddress.go#L209).
  243. This may not be the case for private or local networks, where your IP range is usually
  244. strictly limited and private. If that case, you need to set `addr-book-strict`
  245. to `false` (turn it off).
  246. - `rpc.max-open-connections`
  247. By default, the number of simultaneous connections is limited because most OS
  248. give you limited number of file descriptors.
  249. If you want to accept greater number of connections, you will need to increase
  250. these limits.
  251. [Sysctls to tune the system to be able to open more connections](https://github.com/satori-com/tcpkali/blob/master/doc/tcpkali.man.md#sysctls-to-tune-the-system-to-be-able-to-open-more-connections)
  252. The process file limits must also be increased, e.g. via `ulimit -n 8192`.
  253. ...for N connections, such as 50k:
  254. ```md
  255. kern.maxfiles=10000+2*N # BSD
  256. kern.maxfilesperproc=100+2*N # BSD
  257. kern.ipc.maxsockets=10000+2*N # BSD
  258. fs.file-max=10000+2*N # Linux
  259. net.ipv4.tcp_max_orphans=N # Linux
  260. # For load-generating clients.
  261. net.ipv4.ip_local_port_range="10000 65535" # Linux.
  262. net.inet.ip.portrange.first=10000 # BSD/Mac.
  263. net.inet.ip.portrange.last=65535 # (Enough for N < 55535)
  264. net.ipv4.tcp_tw_reuse=1 # Linux
  265. net.inet.tcp.maxtcptw=2*N # BSD
  266. # If using netfilter on Linux:
  267. net.netfilter.nf_conntrack_max=N
  268. echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize
  269. ```
  270. The similar option exists for limiting the number of gRPC connections -
  271. `rpc.grpc-max-open-connections`.