From b542dce2e1cd4e0c98a6f88a66e69acb6d9fcedc Mon Sep 17 00:00:00 2001 From: Anton Kaliaev Date: Mon, 28 May 2018 16:20:15 +0400 Subject: [PATCH] [docs] signal handling Refs #1494 --- docs/running-in-production.rst | 49 +++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/docs/running-in-production.rst b/docs/running-in-production.rst index ec3b46db6..c2af48cad 100644 --- a/docs/running-in-production.rst +++ b/docs/running-in-production.rst @@ -6,13 +6,11 @@ Logging Default logging level (``main:info,state:info,*:``) should suffice for normal operation mode. Read `this post -__` +`__ for details on how to configure ``log_level`` config variable. Some of the -modules can be found `here <./how-to-read-logs.html#list-of-modules>__`. - -If you're trying to debug Tendermint or asked to provide logs with debug -logging level, you can do so by running tendermint with -``--log_level="*:debug"``. +modules can be found `here <./how-to-read-logs.html#list-of-modules>`__. If +you're trying to debug Tendermint or asked to provide logs with debug logging +level, you can do so by running tendermint with ``--log_level="*:debug"``. Consensus WAL ------------- @@ -20,11 +18,11 @@ Consensus WAL Consensus module writes every message to the WAL (write-ahead log). It also issues fsync syscall through `File#Sync -__` for messages signed by this node (to +`__ for messages signed by this node (to prevent double signing). Under the hood, it uses `autofile.Group -__`, which +`__, which rotates files when those get too big (> 10MB). The total maximum size is 1GB. We only need the latest block and the block before it, @@ -47,15 +45,15 @@ private validator contains the ``LastSignBytes`` and then we’ll replay the precommit from the WAL. Make sure to read about `WAL corruption -<./specification/corruption.html#wal-corruption>__` and recovery strategies. +<./specification/corruption.html#wal-corruption>`__ and recovery strategies. DOS Exposure and Mitigation --------------------------- Validators are supposed to setup `Sentry Node Architecture -__` +`__ to prevent Denial-of-service attacks. You can read more about it `here -__`. +`__. Blockchain Reactor ~~~~~~~~~~~~~~~~~~ @@ -80,7 +78,7 @@ Sending incorrectly encoded data will result in stopping the peer. Evidence Reactor ~~~~~~~~~~~~~~~~ -`#1503 __` +`#1503 `__ Sending invalid evidence will result in stopping the peer. @@ -90,7 +88,7 @@ in stopping the peer. PEX Reactor ~~~~~~~~~~~ -Defines only ``SendQueueCapacity``. `#1503 __` +Defines only ``SendQueueCapacity``. `#1503 `__ Implements rate-limiting by enforcing minimal time between two consecutive ``pexRequestMessage`` requests. If the peer sends us addresses we did not ask, @@ -102,12 +100,12 @@ in stopping the peer. Mempool Reactor ~~~~~~~~~~~~~~~ -`#1503 __` +`#1503 `__ Mempool maintains a cache of the last 10000 transactions to prevent replaying old transactions (plus transactions coming from other validators, who are continually exchanging transactions). Read `Replay Protection -<./app-development.html#replay-protection>` for details. +<./app-development.html#replay-protection>`__ for details. Sending incorrectly encoded data or data exceeding ``maxMsgSize`` will result in stopping the peer. @@ -129,15 +127,15 @@ elements (100 max). Rate-limiting and authentication are another key aspects to help protect against DOS attacks. While in the future we may implement these features, for now, validators are supposed to use external tools like `NGINX -__` or `traefik -__` to archive +`__ or `traefik +`__ to archive the same things. Debugging Tendermint -------------------- If you ever have to debug Tendermint, the first thing you should probably do is -to check out the logs. See `"How to read logs" <./how-to-read-logs.html>__`, +to check out the logs. See `"How to read logs" <./how-to-read-logs.html>`__, where we explain what certain log statements mean. If, after skimming through the logs, things are not clear still, the second @@ -159,8 +157,8 @@ $ curl http(s)://{ip}:{rpcPort}/dump_consensus_state There is a reduced version of this endpoint - `/consensus_state`, which returns just the votes seen at the current height. -- `Github Issues __` -- `StackOverflow questions __` +- `Github Issues `__ +- `StackOverflow questions `__ Monitoring Tendermint --------------------- @@ -174,16 +172,23 @@ Other useful endpoints include mentioned earlier `/status`, `/net_info` and We have a small tool, called tm-monitor, which outputs information from the endpoints above plus some statistics. The tool can be found `here -__`. +`__. What happens when my app die? ----------------------------- You are supposed to run Tendermint under a `process supervisor -__` (like systemd or runit). +`__ (like systemd or runit). It will ensure Tendermint is always running (despite possible errors). Getting back to the original question, if your application dies, Tendermint will panic. After a process supervisor restarts your application, Tendermint should be able to reconnect successfully. The order of restart does not matter for it. + +Signal handling +--------------- + +We catch SIGINT and SIGTERM and try to clean up nicely. For other signals we +use the default behaviour in Go: `Default behavior of signals in Go programs +`__.