You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

357 lines
12 KiB

6 years ago
limit number of /subscribe clients and queries per client (#3269) * limit number of /subscribe clients and queries per client Add the following config variables (under [rpc] section): * max_subscription_clients * max_subscriptions_per_client * timeout_broadcast_tx_commit Fixes #2826 new HTTPClient interface for subscriptions finalize HTTPClient events interface remove EventSubscriber fix data race ``` WARNING: DATA RACE Read at 0x00c000a36060 by goroutine 129: github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe.func1() /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:168 +0x1f0 Previous write at 0x00c000a36060 by goroutine 132: github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe() /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:191 +0x4e0 github.com/tendermint/tendermint/rpc/client.WaitForOneEvent() /go/src/github.com/tendermint/tendermint/rpc/client/helpers.go:64 +0x178 github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync.func1() /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:139 +0x298 testing.tRunner() /usr/local/go/src/testing/testing.go:827 +0x162 Goroutine 129 (running) created at: github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe() /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:164 +0x4b7 github.com/tendermint/tendermint/rpc/client.WaitForOneEvent() /go/src/github.com/tendermint/tendermint/rpc/client/helpers.go:64 +0x178 github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync.func1() /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:139 +0x298 testing.tRunner() /usr/local/go/src/testing/testing.go:827 +0x162 Goroutine 132 (running) created at: testing.(*T).Run() /usr/local/go/src/testing/testing.go:878 +0x659 github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync() /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:119 +0x186 testing.tRunner() /usr/local/go/src/testing/testing.go:827 +0x162 ================== ``` lite client works (tested manually) godoc comments httpclient: do not close the out channel use TimeoutBroadcastTxCommit no timeout for unsubscribe but 1s Local (5s HTTP) timeout for resubscribe format code change Subscribe#out cap to 1 and replace config vars with RPCConfig TimeoutBroadcastTxCommit can't be greater than rpcserver.WriteTimeout rpc: Context as first parameter to all functions reformat code fixes after my own review fixes after Ethan's review add test stubs fix config.toml * fixes after manual testing - rpc: do not recommend to use BroadcastTxCommit because it's slow and wastes Tendermint resources (pubsub) - rpc: better error in Subscribe and BroadcastTxCommit - HTTPClient: do not resubscribe if err = ErrAlreadySubscribed * fixes after Ismail's review * Update rpc/grpc/grpc_test.go Co-Authored-By: melekes <anton.kalyaev@gmail.com>
6 years ago
limit number of /subscribe clients and queries per client (#3269) * limit number of /subscribe clients and queries per client Add the following config variables (under [rpc] section): * max_subscription_clients * max_subscriptions_per_client * timeout_broadcast_tx_commit Fixes #2826 new HTTPClient interface for subscriptions finalize HTTPClient events interface remove EventSubscriber fix data race ``` WARNING: DATA RACE Read at 0x00c000a36060 by goroutine 129: github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe.func1() /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:168 +0x1f0 Previous write at 0x00c000a36060 by goroutine 132: github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe() /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:191 +0x4e0 github.com/tendermint/tendermint/rpc/client.WaitForOneEvent() /go/src/github.com/tendermint/tendermint/rpc/client/helpers.go:64 +0x178 github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync.func1() /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:139 +0x298 testing.tRunner() /usr/local/go/src/testing/testing.go:827 +0x162 Goroutine 129 (running) created at: github.com/tendermint/tendermint/rpc/client.(*Local).Subscribe() /go/src/github.com/tendermint/tendermint/rpc/client/localclient.go:164 +0x4b7 github.com/tendermint/tendermint/rpc/client.WaitForOneEvent() /go/src/github.com/tendermint/tendermint/rpc/client/helpers.go:64 +0x178 github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync.func1() /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:139 +0x298 testing.tRunner() /usr/local/go/src/testing/testing.go:827 +0x162 Goroutine 132 (running) created at: testing.(*T).Run() /usr/local/go/src/testing/testing.go:878 +0x659 github.com/tendermint/tendermint/rpc/client_test.TestTxEventsSentWithBroadcastTxSync() /go/src/github.com/tendermint/tendermint/rpc/client/event_test.go:119 +0x186 testing.tRunner() /usr/local/go/src/testing/testing.go:827 +0x162 ================== ``` lite client works (tested manually) godoc comments httpclient: do not close the out channel use TimeoutBroadcastTxCommit no timeout for unsubscribe but 1s Local (5s HTTP) timeout for resubscribe format code change Subscribe#out cap to 1 and replace config vars with RPCConfig TimeoutBroadcastTxCommit can't be greater than rpcserver.WriteTimeout rpc: Context as first parameter to all functions reformat code fixes after my own review fixes after Ethan's review add test stubs fix config.toml * fixes after manual testing - rpc: do not recommend to use BroadcastTxCommit because it's slow and wastes Tendermint resources (pubsub) - rpc: better error in Subscribe and BroadcastTxCommit - HTTPClient: do not resubscribe if err = ErrAlreadySubscribed * fixes after Ismail's review * Update rpc/grpc/grpc_test.go Co-Authored-By: melekes <anton.kalyaev@gmail.com>
6 years ago
  1. # Configuration
  2. Tendermint Core can be configured via a TOML file in
  3. `$TMHOME/config/config.toml`. Some of these parameters can be overridden by
  4. command-line flags. For most users, the options in the `##### main base configuration options #####` are intended to be modified while
  5. config options further below are intended for advance power users.
  6. ## Options
  7. The default configuration file create by `tendermint init` has all
  8. the parameters set with their default values. It will look something
  9. like the file below, however, double check by inspecting the
  10. `config.toml` created with your version of `tendermint` installed:
  11. ```
  12. # This is a TOML config file.
  13. # For more information, see https://github.com/toml-lang/toml
  14. ##### main base config options #####
  15. # TCP or UNIX socket address of the ABCI application,
  16. # or the name of an ABCI application compiled in with the Tendermint binary
  17. proxy_app = "tcp://127.0.0.1:26658"
  18. # A custom human readable name for this node
  19. moniker = "anonymous"
  20. # If this node is many blocks behind the tip of the chain, FastSync
  21. # allows them to catchup quickly by downloading blocks in parallel
  22. # and verifying their commits
  23. fast_sync = true
  24. # Database backend: leveldb | memdb | cleveldb
  25. db_backend = "leveldb"
  26. # Database directory
  27. db_dir = "data"
  28. # Output level for logging, including package level options
  29. log_level = "main:info,state:info,*:error"
  30. # Output format: 'plain' (colored text) or 'json'
  31. log_format = "plain"
  32. ##### additional base config options #####
  33. # Path to the JSON file containing the initial validator set and other meta data
  34. genesis_file = "config/genesis.json"
  35. # Path to the JSON file containing the private key to use as a validator in the consensus protocol
  36. priv_validator_file = "config/priv_validator.json"
  37. # TCP or UNIX socket address for Tendermint to listen on for
  38. # connections from an external PrivValidator process
  39. priv_validator_laddr = ""
  40. # Path to the JSON file containing the private key to use for node authentication in the p2p protocol
  41. node_key_file = "config/node_key.json"
  42. # Mechanism to connect to the ABCI application: socket | grpc
  43. abci = "socket"
  44. # TCP or UNIX socket address for the profiling server to listen on
  45. prof_laddr = ""
  46. # If true, query the ABCI app on connecting to a new peer
  47. # so the app can decide if we should keep the connection or not
  48. filter_peers = false
  49. ##### advanced configuration options #####
  50. ##### rpc server configuration options #####
  51. [rpc]
  52. # TCP or UNIX socket address for the RPC server to listen on
  53. laddr = "tcp://0.0.0.0:26657"
  54. # A list of origins a cross-domain request can be executed from
  55. # Default value '[]' disables cors support
  56. # Use '["*"]' to allow any origin
  57. cors_allowed_origins = []
  58. # A list of methods the client is allowed to use with cross-domain requests
  59. cors_allowed_methods = ["HEAD", "GET", "POST"]
  60. # A list of non simple headers the client is allowed to use with cross-domain requests
  61. cors_allowed_headers = ["Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time"]
  62. # TCP or UNIX socket address for the gRPC server to listen on
  63. # NOTE: This server only supports /broadcast_tx_commit
  64. grpc_laddr = ""
  65. # Maximum number of simultaneous connections.
  66. # Does not include RPC (HTTP&WebSocket) connections. See max_open_connections
  67. # If you want to accept a larger number than the default, make sure
  68. # you increase your OS limits.
  69. # 0 - unlimited.
  70. # Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}
  71. # 1024 - 40 - 10 - 50 = 924 = ~900
  72. grpc_max_open_connections = 900
  73. # Activate unsafe RPC commands like /dial_seeds and /unsafe_flush_mempool
  74. unsafe = false
  75. # Maximum number of simultaneous connections (including WebSocket).
  76. # Does not include gRPC connections. See grpc_max_open_connections
  77. # If you want to accept a larger number than the default, make sure
  78. # you increase your OS limits.
  79. # 0 - unlimited.
  80. # Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}
  81. # 1024 - 40 - 10 - 50 = 924 = ~900
  82. max_open_connections = 900
  83. # Maximum number of unique clientIDs that can /subscribe
  84. # If you're using /broadcast_tx_commit, set to the estimated maximum number
  85. # of broadcast_tx_commit calls per block.
  86. max_subscription_clients = 100
  87. # Maximum number of unique queries a given client can /subscribe to
  88. # If you're using GRPC (or Local RPC client) and /broadcast_tx_commit, set to
  89. # the estimated # maximum number of broadcast_tx_commit calls per block.
  90. max_subscriptions_per_client = 5
  91. # How long to wait for a tx to be committed during /broadcast_tx_commit.
  92. # WARNING: Using a value larger than 10s will result in increasing the
  93. # global HTTP write timeout, which applies to all connections and endpoints.
  94. # See https://github.com/tendermint/tendermint/issues/3435
  95. timeout_broadcast_tx_commit = "10s"
  96. ##### peer to peer configuration options #####
  97. [p2p]
  98. # Address to listen for incoming connections
  99. laddr = "tcp://0.0.0.0:26656"
  100. # Address to advertise to peers for them to dial
  101. # If empty, will use the same port as the laddr,
  102. # and will introspect on the listener or use UPnP
  103. # to figure out the address.
  104. external_address = ""
  105. # Comma separated list of seed nodes to connect to
  106. seeds = ""
  107. # Comma separated list of nodes to keep persistent connections to
  108. persistent_peers = ""
  109. # UPNP port forwarding
  110. upnp = false
  111. # Path to address book
  112. addr_book_file = "config/addrbook.json"
  113. # Set true for strict address routability rules
  114. # Set false for private or local networks
  115. addr_book_strict = true
  116. # Maximum number of inbound peers
  117. max_num_inbound_peers = 40
  118. # Maximum number of outbound peers to connect to, excluding persistent peers
  119. max_num_outbound_peers = 10
  120. # Time to wait before flushing messages out on the connection
  121. flush_throttle_timeout = "100ms"
  122. # Maximum size of a message packet payload, in bytes
  123. max_packet_msg_payload_size = 1024
  124. # Rate at which packets can be sent, in bytes/second
  125. send_rate = 5120000
  126. # Rate at which packets can be received, in bytes/second
  127. recv_rate = 5120000
  128. # Set true to enable the peer-exchange reactor
  129. pex = true
  130. # Seed mode, in which node constantly crawls the network and looks for
  131. # peers. If another node asks it for addresses, it responds and disconnects.
  132. #
  133. # Does not work if the peer-exchange reactor is disabled.
  134. seed_mode = false
  135. # Comma separated list of peer IDs to keep private (will not be gossiped to other peers)
  136. private_peer_ids = ""
  137. # Toggle to disable guard against peers connecting from the same ip.
  138. allow_duplicate_ip = false
  139. # Peer connection configuration.
  140. handshake_timeout = "20s"
  141. dial_timeout = "3s"
  142. ##### mempool configuration options #####
  143. [mempool]
  144. recheck = true
  145. broadcast = true
  146. wal_dir = ""
  147. # Maximum number of transactions in the mempool
  148. size = 5000
  149. # Limit the total size of all txs in the mempool.
  150. # This only accounts for raw transactions (e.g. given 1MB transactions and
  151. # max_txs_bytes=5MB, mempool will only accept 5 transactions).
  152. max_txs_bytes = 1073741824
  153. # Size of the cache (used to filter transactions we saw earlier) in transactions
  154. cache_size = 10000
  155. ##### consensus configuration options #####
  156. [consensus]
  157. wal_file = "data/cs.wal/wal"
  158. timeout_propose = "3s"
  159. timeout_propose_delta = "500ms"
  160. timeout_prevote = "1s"
  161. timeout_prevote_delta = "500ms"
  162. timeout_precommit = "1s"
  163. timeout_precommit_delta = "500ms"
  164. timeout_commit = "1s"
  165. # Make progress as soon as we have all the precommits (as if TimeoutCommit = 0)
  166. skip_timeout_commit = false
  167. # EmptyBlocks mode and possible interval between empty blocks
  168. create_empty_blocks = true
  169. create_empty_blocks_interval = "0s"
  170. # Reactor sleep duration parameters
  171. peer_gossip_sleep_duration = "100ms"
  172. peer_query_maj23_sleep_duration = "2s"
  173. # Block time parameters. Corresponds to the minimum time increment between consecutive blocks.
  174. blocktime_iota = "1s"
  175. ##### transactions indexer configuration options #####
  176. [tx_index]
  177. # What indexer to use for transactions
  178. #
  179. # Options:
  180. # 1) "null"
  181. # 2) "kv" (default) - the simplest possible indexer, backed by key-value storage (defaults to levelDB; see DBBackend).
  182. indexer = "kv"
  183. # Comma-separated list of tags to index (by default the only tag is "tx.hash")
  184. #
  185. # You can also index transactions by height by adding "tx.height" tag here.
  186. #
  187. # It's recommended to index only a subset of tags due to possible memory
  188. # bloat. This is, of course, depends on the indexer's DB and the volume of
  189. # transactions.
  190. index_tags = ""
  191. # When set to true, tells indexer to index all tags (predefined tags:
  192. # "tx.hash", "tx.height" and all tags from DeliverTx responses).
  193. #
  194. # Note this may be not desirable (see the comment above). IndexTags has a
  195. # precedence over IndexAllTags (i.e. when given both, IndexTags will be
  196. # indexed).
  197. index_all_tags = false
  198. ##### instrumentation configuration options #####
  199. [instrumentation]
  200. # When true, Prometheus metrics are served under /metrics on
  201. # PrometheusListenAddr.
  202. # Check out the documentation for the list of available metrics.
  203. prometheus = false
  204. # Address to listen for Prometheus collector(s) connections
  205. prometheus_listen_addr = ":26660"
  206. # Maximum number of simultaneous connections.
  207. # If you want to accept a larger number than the default, make sure
  208. # you increase your OS limits.
  209. # 0 - unlimited.
  210. max_open_connections = 3
  211. # Instrumentation namespace
  212. namespace = "tendermint"
  213. ```
  214. ## Empty blocks VS no empty blocks
  215. **create_empty_blocks = true**
  216. If `create_empty_blocks` is set to `true` in your config, blocks will be
  217. created ~ every second (with default consensus parameters). You can regulate
  218. the delay between blocks by changing the `timeout_commit`. E.g. `timeout_commit
  219. = "10s"` should result in ~ 10 second blocks.
  220. **create_empty_blocks = false**
  221. In this setting, blocks are created when transactions received.
  222. Note after the block H, Tendermint creates something we call a "proof block"
  223. (only if the application hash changed) H+1. The reason for this is to support
  224. proofs. If you have a transaction in block H that changes the state to X, the
  225. new application hash will only be included in block H+1. If after your
  226. transaction is committed, you want to get a lite-client proof for the new state
  227. (X), you need the new block to be committed in order to do that because the new
  228. block has the new application hash for the state X. That's why we make a new
  229. (empty) block if the application hash changes. Otherwise, you won't be able to
  230. make a proof for the new state.
  231. Plus, if you set `create_empty_blocks_interval` to something other than the
  232. default (`0`), Tendermint will be creating empty blocks even in the absence of
  233. transactions every `create_empty_blocks_interval`. For instance, with
  234. `create_empty_blocks = false` and `create_empty_blocks_interval = "30s"`,
  235. Tendermint will only create blocks if there are transactions, or after waiting
  236. 30 seconds without receiving any transactions.
  237. ## Consensus timeouts explained
  238. There's a variety of information about timeouts in [Running in
  239. production](./running-in-production.html)
  240. You can also find more detailed technical explanation in the spec: [The latest
  241. gossip on BFT consensus](https://arxiv.org/abs/1807.04938).
  242. ```
  243. [consensus]
  244. ...
  245. timeout_propose = "3s"
  246. timeout_propose_delta = "500ms"
  247. timeout_prevote = "1s"
  248. timeout_prevote_delta = "500ms"
  249. timeout_precommit = "1s"
  250. timeout_precommit_delta = "500ms"
  251. timeout_commit = "1s"
  252. ```
  253. Note that in a successful round, the only timeout that we absolutely wait no
  254. matter what is `timeout_commit`.
  255. Here's a brief summary of the timeouts:
  256. - `timeout_propose` = how long we wait for a proposal block before prevoting
  257. nil
  258. - `timeout_propose_delta` = how much timeout_propose increases with each round
  259. - `timeout_prevote` = how long we wait after receiving +2/3 prevotes for
  260. anything (ie. not a single block or nil)
  261. - `timeout_prevote_delta` = how much the timeout_prevote increases with each
  262. round
  263. - `timeout_precommit` = how long we wait after receiving +2/3 precommits for
  264. anything (ie. not a single block or nil)
  265. - `timeout_precommit_delta` = how much the timeout_precommit increases with
  266. each round
  267. - `timeout_commit` = how long we wait after committing a block, before starting
  268. on the new height (this gives us a chance to receive some more precommits,
  269. even though we already have +2/3)