You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

403 lines
15 KiB

6 years ago
6 years ago
cli: debug sub-command (#4227) ## Issue Implement a new subcommand: tendermint debug. This subcommand itself has two subcommands: $ tendermint debug kill <pid> </path/to/out.zip> --home=</path/to/app.d> Writes debug info into a compressed archive. The archive will contain the following: ├── config.toml ├── consensus_state.json ├── net_info.json ├── stacktrace.out ├── status.json └── wal The Tendermint process will be killed. $ tendermint debug dump </path/to/out> --home=</path/to/app.d> This command will perform similar to kill except it only polls the node and dumps debugging data every frequency seconds to a compressed archive under a given destination directory. Each archive will contain: ├── consensus_state.json ├── goroutine.out ├── heap.out ├── net_info.json ├── status.json └── wal Note: goroutine.out and heap.out will only be written if a profile address is provided and is operational. This command is blocking and will log any error. replaces: #3327 closes: #3249 ## Commits: * Implement debug tool command stubs * Implement net getters and zip logic * Update zip dir API and add home flag * Add simple godocs for kill aux functions * Move IO util to new file and implement copy WAL func * Implement copy config function * Implement killProc * Remove debug fmt * Validate output file input * Direct STDERR to file * Godoc updates * Sleep prior to killing tail proc * Minor cleanup of godocs * Move debug command and add make target * Rename command handler function * Add example to command long descr * Move current implementation to cmd/tendermint/commands/debug * Update kill cmd long description * Implement dump command * Add pending log entry * Add gosec nolint * Add error check for Mkdir * Add os.IsNotExist(err) * Add to debugging section in running-in-prod doc
5 years ago
cli: debug sub-command (#4227) ## Issue Implement a new subcommand: tendermint debug. This subcommand itself has two subcommands: $ tendermint debug kill <pid> </path/to/out.zip> --home=</path/to/app.d> Writes debug info into a compressed archive. The archive will contain the following: ├── config.toml ├── consensus_state.json ├── net_info.json ├── stacktrace.out ├── status.json └── wal The Tendermint process will be killed. $ tendermint debug dump </path/to/out> --home=</path/to/app.d> This command will perform similar to kill except it only polls the node and dumps debugging data every frequency seconds to a compressed archive under a given destination directory. Each archive will contain: ├── consensus_state.json ├── goroutine.out ├── heap.out ├── net_info.json ├── status.json └── wal Note: goroutine.out and heap.out will only be written if a profile address is provided and is operational. This command is blocking and will log any error. replaces: #3327 closes: #3249 ## Commits: * Implement debug tool command stubs * Implement net getters and zip logic * Update zip dir API and add home flag * Add simple godocs for kill aux functions * Move IO util to new file and implement copy WAL func * Implement copy config function * Implement killProc * Remove debug fmt * Validate output file input * Direct STDERR to file * Godoc updates * Sleep prior to killing tail proc * Minor cleanup of godocs * Move debug command and add make target * Rename command handler function * Add example to command long descr * Move current implementation to cmd/tendermint/commands/debug * Update kill cmd long description * Implement dump command * Add pending log entry * Add gosec nolint * Add error check for Mkdir * Add os.IsNotExist(err) * Add to debugging section in running-in-prod doc
5 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
  1. ---
  2. order: 5
  3. ---
  4. # Running in production
  5. ## Database
  6. By default, Tendermint uses the `syndtr/goleveldb` package for its in-process
  7. key-value database. Unfortunately, this implementation of LevelDB seems to suffer under heavy load (see
  8. [#226](https://github.com/syndtr/goleveldb/issues/226)). It may be best to
  9. install the real C-implementation of LevelDB and compile Tendermint to use
  10. that using `make build TENDERMINT_BUILD_OPTIONS=cleveldb`. See the [install instructions](../introduction/install.md) for details.
  11. Tendermint keeps multiple distinct databases in the `$TMROOT/data`:
  12. - `blockstore.db`: Keeps the entire blockchain - stores blocks,
  13. block commits, and block meta data, each indexed by height. Used to sync new
  14. peers.
  15. - `evidence.db`: Stores all verified evidence of misbehaviour.
  16. - `state.db`: Stores the current blockchain state (ie. height, validators,
  17. consensus params). Only grows if consensus params or validators change. Also
  18. used to temporarily store intermediate results during block processing.
  19. - `tx_index.db`: Indexes txs (and their results) by tx hash and by DeliverTx result events.
  20. By default, Tendermint will only index txs by their hash, not by their DeliverTx
  21. result events. See [indexing transactions](../app-dev/indexing-transactions.md) for
  22. details.
  23. There is no current strategy for pruning the databases. Consider reducing
  24. block production by [controlling empty blocks](../tendermint-core/using-tendermint.md#no-empty-blocks)
  25. or by increasing the `consensus.timeout_commit` param. Note both of these are
  26. local settings and not enforced by the consensus.
  27. We're working on [state
  28. syncing](https://github.com/tendermint/tendermint/issues/828),
  29. which will enable history to be thrown away
  30. and recent application state to be directly synced. We'll need to develop solutions
  31. for archival nodes that allow queries on historical transactions and states.
  32. The Cosmos project has had much success just dumping the latest state of a
  33. blockchain to disk and starting a new chain from that state.
  34. ## Logging
  35. Default logging level (`main:info,state:info,*:`) should suffice for
  36. normal operation mode. Read [this
  37. post](https://blog.cosmos.network/one-of-the-exciting-new-features-in-0-10-0-release-is-smart-log-level-flag-e2506b4ab756)
  38. for details on how to configure `log_level` config variable. Some of the
  39. modules can be found [here](./how-to-read-logs.md#list-of-modules). If
  40. you're trying to debug Tendermint or asked to provide logs with debug
  41. logging level, you can do so by running tendermint with
  42. `--log_level="*:debug"`.
  43. ## Write Ahead Logs (WAL)
  44. Tendermint uses write ahead logs for the consensus (`cs.wal`) and the mempool
  45. (`mempool.wal`). Both WALs have a max size of 1GB and are automatically rotated.
  46. ### Consensus WAL
  47. The `consensus.wal` is used to ensure we can recover from a crash at any point
  48. in the consensus state machine.
  49. It writes all consensus messages (timeouts, proposals, block part, or vote)
  50. to a single file, flushing to disk before processing messages from its own
  51. validator. Since Tendermint validators are expected to never sign a conflicting vote, the
  52. WAL ensures we can always recover deterministically to the latest state of the consensus without
  53. using the network or re-signing any consensus messages.
  54. If your `consensus.wal` is corrupted, see [below](#wal-corruption).
  55. ### Mempool WAL
  56. The `mempool.wal` logs all incoming txs before running CheckTx, but is
  57. otherwise not used in any programmatic way. It's just a kind of manual
  58. safe guard. Note the mempool provides no durability guarantees - a tx sent to one or many nodes
  59. may never make it into the blockchain if those nodes crash before being able to
  60. propose it. Clients must monitor their txs by subscribing over websockets,
  61. polling for them, or using `/broadcast_tx_commit`. In the worst case, txs can be
  62. resent from the mempool WAL manually.
  63. For the above reasons, the `mempool.wal` is disabled by default. To enable, set
  64. `mempool.wal_dir` to where you want the WAL to be located (e.g.
  65. `data/mempool.wal`).
  66. ## DOS Exposure and Mitigation
  67. Validators are supposed to setup [Sentry Node
  68. Architecture](https://blog.cosmos.network/tendermint-explained-bringing-bft-based-pos-to-the-public-blockchain-domain-f22e274a0fdb)
  69. to prevent Denial-of-service attacks. You can read more about it
  70. [here](../interviews/tendermint-bft.md).
  71. ### P2P
  72. The core of the Tendermint peer-to-peer system is `MConnection`. Each
  73. connection has `MaxPacketMsgPayloadSize`, which is the maximum packet
  74. size and bounded send & receive queues. One can impose restrictions on
  75. send & receive rate per connection (`SendRate`, `RecvRate`).
  76. The number of open P2P connections can become quite large, and hit the operating system's open
  77. file limit (since TCP connections are considered files on UNIX-based systems). Nodes should be
  78. given a sizeable open file limit, e.g. 8192, via `ulimit -n 8192` or other deployment-specific
  79. mechanisms.
  80. ### RPC
  81. Endpoints returning multiple entries are limited by default to return 30
  82. elements (100 max). See the [RPC Documentation](https://docs.tendermint.com/master/rpc/)
  83. for more information.
  84. Rate-limiting and authentication are another key aspects to help protect
  85. against DOS attacks. While in the future we may implement these
  86. features, for now, validators are supposed to use external tools like
  87. [NGINX](https://www.nginx.com/blog/rate-limiting-nginx/) or
  88. [traefik](https://docs.traefik.io/middlewares/ratelimit/)
  89. to achieve the same things.
  90. ## Debugging Tendermint
  91. If you ever have to debug Tendermint, the first thing you should probably do is
  92. check out the logs. See [How to read logs](./how-to-read-logs.md), where we
  93. explain what certain log statements mean.
  94. If, after skimming through the logs, things are not clear still, the next thing
  95. to try is querying the `/status` RPC endpoint. It provides the necessary info:
  96. whenever the node is syncing or not, what height it is on, etc.
  97. ```bash
  98. curl http(s)://{ip}:{rpcPort}/status
  99. ```
  100. `/dump_consensus_state` will give you a detailed overview of the consensus
  101. state (proposer, latest validators, peers states). From it, you should be able
  102. to figure out why, for example, the network had halted.
  103. ```bash
  104. curl http(s)://{ip}:{rpcPort}/dump_consensus_state
  105. ```
  106. There is a reduced version of this endpoint - `/consensus_state`, which returns
  107. just the votes seen at the current height.
  108. If, after consulting with the logs and above endpoints, you still have no idea
  109. what's happening, consider using `tendermint debug kill` sub-command. This
  110. command will scrap all the available info and kill the process. See
  111. [Debugging](../tools/debugging.md) for the exact format.
  112. You can inspect the resulting archive yourself or create an issue on
  113. [Github](https://github.com/tendermint/tendermint). Before opening an issue
  114. however, be sure to check if there's [no existing
  115. issue](https://github.com/tendermint/tendermint/issues) already.
  116. ## Monitoring Tendermint
  117. Each Tendermint instance has a standard `/health` RPC endpoint, which responds
  118. with 200 (OK) if everything is fine and 500 (or no response) - if something is
  119. wrong.
  120. Other useful endpoints include mentioned earlier `/status`, `/net_info` and
  121. `/validators`.
  122. Tendermint also can report and serve Prometheus metrics. See
  123. [Metrics](./metrics.md).
  124. `tendermint debug dump` sub-command can be used to periodically dump useful
  125. information into an archive. See [Debugging](../tools/debugging.md) for more
  126. information.
  127. ## What happens when my app dies?
  128. You are supposed to run Tendermint under a [process
  129. supervisor](https://en.wikipedia.org/wiki/Process_supervision) (like
  130. systemd or runit). It will ensure Tendermint is always running (despite
  131. possible errors).
  132. Getting back to the original question, if your application dies,
  133. Tendermint will panic. After a process supervisor restarts your
  134. application, Tendermint should be able to reconnect successfully. The
  135. order of restart does not matter for it.
  136. ## Signal handling
  137. We catch SIGINT and SIGTERM and try to clean up nicely. For other
  138. signals we use the default behaviour in Go: [Default behavior of signals
  139. in Go
  140. programs](https://golang.org/pkg/os/signal/#hdr-Default_behavior_of_signals_in_Go_programs).
  141. ## Corruption
  142. **NOTE:** Make sure you have a backup of the Tendermint data directory.
  143. ### Possible causes
  144. Remember that most corruption is caused by hardware issues:
  145. - RAID controllers with faulty / worn out battery backup, and an unexpected power loss
  146. - Hard disk drives with write-back cache enabled, and an unexpected power loss
  147. - Cheap SSDs with insufficient power-loss protection, and an unexpected power-loss
  148. - Defective RAM
  149. - Defective or overheating CPU(s)
  150. Other causes can be:
  151. - Database systems configured with fsync=off and an OS crash or power loss
  152. - Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit.
  153. - Tendermint bugs
  154. - Operating system bugs
  155. - Admin error (e.g., directly modifying Tendermint data-directory contents)
  156. (Source: https://wiki.postgresql.org/wiki/Corruption)
  157. ### WAL Corruption
  158. If consensus WAL is corrupted at the lastest height and you are trying to start
  159. Tendermint, replay will fail with panic.
  160. Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take:
  161. 1. Delete the WAL file and restart Tendermint. It will attempt to sync with other peers.
  162. 2. Try to repair the WAL file manually:
  163. 1) Create a backup of the corrupted WAL file:
  164. ```
  165. cp "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup
  166. ```
  167. 2. Use `./scripts/wal2json` to create a human-readable version
  168. ```
  169. ./scripts/wal2json/wal2json "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal
  170. ```
  171. 3. Search for a "CORRUPTED MESSAGE" line.
  172. 4. By looking at the previous message and the message after the corrupted one
  173. and looking at the logs, try to rebuild the message. If the consequent
  174. messages are marked as corrupted too (this may happen if length header
  175. got corrupted or some writes did not make it to the WAL ~ truncation),
  176. then remove all the lines starting from the corrupted one and restart
  177. Tendermint.
  178. ```
  179. $EDITOR /tmp/corrupted_wal
  180. ```
  181. 5. After editing, convert this file back into binary form by running:
  182. ```
  183. ./scripts/json2wal/json2wal /tmp/corrupted_wal $TMHOME/data/cs.wal/wal
  184. ```
  185. ## Hardware
  186. ### Processor and Memory
  187. While actual specs vary depending on the load and validators count, minimal
  188. requirements are:
  189. - 1GB RAM
  190. - 25GB of disk space
  191. - 1.4 GHz CPU
  192. SSD disks are preferable for applications with high transaction throughput.
  193. Recommended:
  194. - 2GB RAM
  195. - 100GB SSD
  196. - x64 2.0 GHz 2v CPU
  197. While for now, Tendermint stores all the history and it may require significant
  198. disk space over time, we are planning to implement state syncing (See [this
  199. issue](https://github.com/tendermint/tendermint/issues/828)). So, storing all
  200. the past blocks will not be necessary.
  201. ### Validator signing on 32 bit architectures (or ARM)
  202. Both our `ed25519` and `secp256k1` implementations require constant time
  203. `uint64` multiplication. Non-constant time crypto can (and has) leaked
  204. private keys on both `ed25519` and `secp256k1`. This doesn't exist in hardware
  205. on 32 bit x86 platforms ([source](https://bearssl.org/ctmul.html)), and it
  206. depends on the compiler to enforce that it is constant time. It's unclear at
  207. this point whenever the Golang compiler does this correctly for all
  208. implementations.
  209. **We do not support nor recommend running a validator on 32 bit architectures OR
  210. the "VIA Nano 2000 Series", and the architectures in the ARM section rated
  211. "S-".**
  212. ### Operating Systems
  213. Tendermint can be compiled for a wide range of operating systems thanks to Go
  214. language (the list of \$OS/\$ARCH pairs can be found
  215. [here](https://golang.org/doc/install/source#environment)).
  216. While we do not favor any operation system, more secure and stable Linux server
  217. distributions (like Centos) should be preferred over desktop operation systems
  218. (like Mac OS).
  219. ### Miscellaneous
  220. NOTE: if you are going to use Tendermint in a public domain, make sure
  221. you read [hardware recommendations](https://cosmos.network/validators) for a validator in the
  222. Cosmos network.
  223. ## Configuration parameters
  224. - `p2p.flush_throttle_timeout`
  225. - `p2p.max_packet_msg_payload_size`
  226. - `p2p.send_rate`
  227. - `p2p.recv_rate`
  228. If you are going to use Tendermint in a private domain and you have a
  229. private high-speed network among your peers, it makes sense to lower
  230. flush throttle timeout and increase other params.
  231. ```
  232. [p2p]
  233. send_rate=20000000 # 2MB/s
  234. recv_rate=20000000 # 2MB/s
  235. flush_throttle_timeout=10
  236. max_packet_msg_payload_size=10240 # 10KB
  237. ```
  238. - `mempool.recheck`
  239. After every block, Tendermint rechecks every transaction left in the
  240. mempool to see if transactions committed in that block affected the
  241. application state, so some of the transactions left may become invalid.
  242. If that does not apply to your application, you can disable it by
  243. setting `mempool.recheck=false`.
  244. - `mempool.broadcast`
  245. Setting this to false will stop the mempool from relaying transactions
  246. to other peers until they are included in a block. It means only the
  247. peer you send the tx to will see it until it is included in a block.
  248. - `consensus.skip_timeout_commit`
  249. We want `skip_timeout_commit=false` when there is economics on the line
  250. because proposers should wait to hear for more votes. But if you don't
  251. care about that and want the fastest consensus, you can skip it. It will
  252. be kept false by default for public deployments (e.g. [Cosmos
  253. Hub](https://cosmos.network/intro/hub)) while for enterprise
  254. applications, setting it to true is not a problem.
  255. - `consensus.peer_gossip_sleep_duration`
  256. You can try to reduce the time your node sleeps before checking if
  257. theres something to send its peers.
  258. - `consensus.timeout_commit`
  259. You can also try lowering `timeout_commit` (time we sleep before
  260. proposing the next block).
  261. - `p2p.addr_book_strict`
  262. By default, Tendermint checks whenever a peer's address is routable before
  263. saving it to the address book. The address is considered as routable if the IP
  264. is [valid and within allowed
  265. ranges](https://github.com/tendermint/tendermint/blob/27bd1deabe4ba6a2d9b463b8f3e3f1e31b993e61/p2p/netaddress.go#L209).
  266. This may not be the case for private or local networks, where your IP range is usually
  267. strictly limited and private. If that case, you need to set `addr_book_strict`
  268. to `false` (turn it off).
  269. - `rpc.max_open_connections`
  270. By default, the number of simultaneous connections is limited because most OS
  271. give you limited number of file descriptors.
  272. If you want to accept greater number of connections, you will need to increase
  273. these limits.
  274. [Sysctls to tune the system to be able to open more connections](https://github.com/satori-com/tcpkali/blob/master/doc/tcpkali.man.md#sysctls-to-tune-the-system-to-be-able-to-open-more-connections)
  275. The process file limits must also be increased, e.g. via `ulimit -n 8192`.
  276. ...for N connections, such as 50k:
  277. ```
  278. kern.maxfiles=10000+2*N # BSD
  279. kern.maxfilesperproc=100+2*N # BSD
  280. kern.ipc.maxsockets=10000+2*N # BSD
  281. fs.file-max=10000+2*N # Linux
  282. net.ipv4.tcp_max_orphans=N # Linux
  283. # For load-generating clients.
  284. net.ipv4.ip_local_port_range="10000 65535" # Linux.
  285. net.inet.ip.portrange.first=10000 # BSD/Mac.
  286. net.inet.ip.portrange.last=65535 # (Enough for N < 55535)
  287. net.ipv4.tcp_tw_reuse=1 # Linux
  288. net.inet.tcp.maxtcptw=2*N # BSD
  289. # If using netfilter on Linux:
  290. net.netfilter.nf_conntrack_max=N
  291. echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize
  292. ```
  293. The similar option exists for limiting the number of gRPC connections -
  294. `rpc.grpc_max_open_connections`.