|
|
@ -6,13 +6,11 @@ Logging |
|
|
|
|
|
|
|
Default logging level (``main:info,state:info,*:``) should suffice for normal |
|
|
|
operation mode. Read `this post |
|
|
|
<https://blog.cosmos.network/one-of-the-exciting-new-features-in-0-10-0-release-is-smart-log-level-flag-e2506b4ab756>__` |
|
|
|
<https://blog.cosmos.network/one-of-the-exciting-new-features-in-0-10-0-release-is-smart-log-level-flag-e2506b4ab756>`__ |
|
|
|
for details on how to configure ``log_level`` config variable. Some of the |
|
|
|
modules can be found `here <./how-to-read-logs.html#list-of-modules>__`. |
|
|
|
|
|
|
|
If you're trying to debug Tendermint or asked to provide logs with debug |
|
|
|
logging level, you can do so by running tendermint with |
|
|
|
``--log_level="*:debug"``. |
|
|
|
modules can be found `here <./how-to-read-logs.html#list-of-modules>`__. If |
|
|
|
you're trying to debug Tendermint or asked to provide logs with debug logging |
|
|
|
level, you can do so by running tendermint with ``--log_level="*:debug"``. |
|
|
|
|
|
|
|
Consensus WAL |
|
|
|
------------- |
|
|
@ -20,11 +18,11 @@ Consensus WAL |
|
|
|
Consensus module writes every message to the WAL (write-ahead log). |
|
|
|
|
|
|
|
It also issues fsync syscall through `File#Sync |
|
|
|
<https://golang.org/pkg/os/#File.Sync>__` for messages signed by this node (to |
|
|
|
<https://golang.org/pkg/os/#File.Sync>`__ for messages signed by this node (to |
|
|
|
prevent double signing). |
|
|
|
|
|
|
|
Under the hood, it uses `autofile.Group |
|
|
|
<https://godoc.org/github.com/tendermint/tmlibs/autofile#Group>__`, which |
|
|
|
<https://godoc.org/github.com/tendermint/tmlibs/autofile#Group>`__, which |
|
|
|
rotates files when those get too big (> 10MB). |
|
|
|
|
|
|
|
The total maximum size is 1GB. We only need the latest block and the block before it, |
|
|
@ -47,15 +45,15 @@ private validator contains the ``LastSignBytes`` and then we’ll replay the |
|
|
|
precommit from the WAL. |
|
|
|
|
|
|
|
Make sure to read about `WAL corruption |
|
|
|
<./specification/corruption.html#wal-corruption>__` and recovery strategies. |
|
|
|
<./specification/corruption.html#wal-corruption>`__ and recovery strategies. |
|
|
|
|
|
|
|
DOS Exposure and Mitigation |
|
|
|
--------------------------- |
|
|
|
|
|
|
|
Validators are supposed to setup `Sentry Node Architecture |
|
|
|
<https://blog.cosmos.network/tendermint-explained-bringing-bft-based-pos-to-the-public-blockchain-domain-f22e274a0fdb>__` |
|
|
|
<https://blog.cosmos.network/tendermint-explained-bringing-bft-based-pos-to-the-public-blockchain-domain-f22e274a0fdb>`__ |
|
|
|
to prevent Denial-of-service attacks. You can read more about it `here |
|
|
|
<https://github.com/tendermint/aib-data/blob/develop/medium/TendermintBFT.md>__`. |
|
|
|
<https://github.com/tendermint/aib-data/blob/develop/medium/TendermintBFT.md>`__. |
|
|
|
|
|
|
|
Blockchain Reactor |
|
|
|
~~~~~~~~~~~~~~~~~~ |
|
|
@ -80,7 +78,7 @@ Sending incorrectly encoded data will result in stopping the peer. |
|
|
|
Evidence Reactor |
|
|
|
~~~~~~~~~~~~~~~~ |
|
|
|
|
|
|
|
`#1503 <https://github.com/tendermint/tendermint/issues/1503>__` |
|
|
|
`#1503 <https://github.com/tendermint/tendermint/issues/1503>`__ |
|
|
|
|
|
|
|
Sending invalid evidence will result in stopping the peer. |
|
|
|
|
|
|
@ -90,7 +88,7 @@ in stopping the peer. |
|
|
|
PEX Reactor |
|
|
|
~~~~~~~~~~~ |
|
|
|
|
|
|
|
Defines only ``SendQueueCapacity``. `#1503 <https://github.com/tendermint/tendermint/issues/1503>__` |
|
|
|
Defines only ``SendQueueCapacity``. `#1503 <https://github.com/tendermint/tendermint/issues/1503>`__ |
|
|
|
|
|
|
|
Implements rate-limiting by enforcing minimal time between two consecutive |
|
|
|
``pexRequestMessage`` requests. If the peer sends us addresses we did not ask, |
|
|
@ -102,12 +100,12 @@ in stopping the peer. |
|
|
|
Mempool Reactor |
|
|
|
~~~~~~~~~~~~~~~ |
|
|
|
|
|
|
|
`#1503 <https://github.com/tendermint/tendermint/issues/1503>__` |
|
|
|
`#1503 <https://github.com/tendermint/tendermint/issues/1503>`__ |
|
|
|
|
|
|
|
Mempool maintains a cache of the last 10000 transactions to prevent replaying |
|
|
|
old transactions (plus transactions coming from other validators, who are |
|
|
|
continually exchanging transactions). Read `Replay Protection |
|
|
|
<./app-development.html#replay-protection>` for details. |
|
|
|
<./app-development.html#replay-protection>`__ for details. |
|
|
|
|
|
|
|
Sending incorrectly encoded data or data exceeding ``maxMsgSize`` will result |
|
|
|
in stopping the peer. |
|
|
@ -129,15 +127,15 @@ elements (100 max). |
|
|
|
Rate-limiting and authentication are another key aspects to help protect |
|
|
|
against DOS attacks. While in the future we may implement these features, for |
|
|
|
now, validators are supposed to use external tools like `NGINX |
|
|
|
<https://www.nginx.com/blog/rate-limiting-nginx/>__` or `traefik |
|
|
|
<https://docs.traefik.io/configuration/commons/#rate-limiting>__` to archive |
|
|
|
<https://www.nginx.com/blog/rate-limiting-nginx/>`__ or `traefik |
|
|
|
<https://docs.traefik.io/configuration/commons/#rate-limiting>`__ to archive |
|
|
|
the same things. |
|
|
|
|
|
|
|
Debugging Tendermint |
|
|
|
-------------------- |
|
|
|
|
|
|
|
If you ever have to debug Tendermint, the first thing you should probably do is |
|
|
|
to check out the logs. See `"How to read logs" <./how-to-read-logs.html>__`, |
|
|
|
to check out the logs. See `"How to read logs" <./how-to-read-logs.html>`__, |
|
|
|
where we explain what certain log statements mean. |
|
|
|
|
|
|
|
If, after skimming through the logs, things are not clear still, the second |
|
|
@ -159,8 +157,8 @@ $ curl http(s)://{ip}:{rpcPort}/dump_consensus_state |
|
|
|
There is a reduced version of this endpoint - `/consensus_state`, which |
|
|
|
returns just the votes seen at the current height. |
|
|
|
|
|
|
|
- `Github Issues <https://github.com/tendermint/tendermint/issues>__` |
|
|
|
- `StackOverflow questions <https://stackoverflow.com/questions/tagged/tendermint>__` |
|
|
|
- `Github Issues <https://github.com/tendermint/tendermint/issues>`__ |
|
|
|
- `StackOverflow questions <https://stackoverflow.com/questions/tagged/tendermint>`__ |
|
|
|
|
|
|
|
Monitoring Tendermint |
|
|
|
--------------------- |
|
|
@ -174,16 +172,23 @@ Other useful endpoints include mentioned earlier `/status`, `/net_info` and |
|
|
|
|
|
|
|
We have a small tool, called tm-monitor, which outputs information from the |
|
|
|
endpoints above plus some statistics. The tool can be found `here |
|
|
|
<https://github.com/tendermint/tools/tree/master/tm-monitor>__`. |
|
|
|
<https://github.com/tendermint/tools/tree/master/tm-monitor>`__. |
|
|
|
|
|
|
|
What happens when my app die? |
|
|
|
----------------------------- |
|
|
|
|
|
|
|
You are supposed to run Tendermint under a `process supervisor |
|
|
|
<https://en.wikipedia.org/wiki/Process_supervision>__` (like systemd or runit). |
|
|
|
<https://en.wikipedia.org/wiki/Process_supervision>`__ (like systemd or runit). |
|
|
|
It will ensure Tendermint is always running (despite possible errors). |
|
|
|
|
|
|
|
Getting back to the original question, if your application dies, Tendermint |
|
|
|
will panic. After a process supervisor restarts your application, Tendermint |
|
|
|
should be able to reconnect successfully. The order of restart does not matter |
|
|
|
for it. |
|
|
|
|
|
|
|
Signal handling |
|
|
|
--------------- |
|
|
|
|
|
|
|
We catch SIGINT and SIGTERM and try to clean up nicely. For other signals we |
|
|
|
use the default behaviour in Go: `Default behavior of signals in Go programs |
|
|
|
<https://golang.org/pkg/os/signal/#hdr-Default_behavior_of_signals_in_Go_programs>`__. |