You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

145 lines
8.0 KiB

rpc/lib/client & server: try to conform to JSON-RPC 2.0 spec (#4141) https://www.jsonrpc.org/specification What is done in this PR: JSONRPCClient: validate that Response.ID matches Request.ID I wanted to do the same for the WSClient, but since we're sending events as responses, not notifications, checking IDs would require storing them in memory indefinitely (and we won't be able to remove them upon client unsubscribing because ID is different then). Request.ID is now optional. Notification is a Request without an ID. Previously "" or 0 were considered as notifications Remove #event suffix from ID from an event response (partially fixes #2949) ID must be either string, int or null AND must be equal to request's ID. Now, because we've implemented events as responses, WS clients are tripping when they see Response.ID("0#event") != Request.ID("0"). Implementing events as requests would require a lot of time (~ 2 days to completely rewrite WS client and server) generate unique ID for each request switch to integer IDs instead of "json-client-XYZ" id=0 method=/subscribe id=0 result=... id=1 method=/abci_query id=1 result=... > send events (resulting from /subscribe) as requests+notifications (not responses) this will require a lot of work. probably not worth it * rpc: generate an unique ID for each request in conformance with JSON-RPC spec * WSClient: check for unsolicited responses * fix golangci warnings * save commit * fix errors * remove ID from responses from subscribe Refs #2949 * clients are safe for concurrent access * tm-bench: switch to int ID * fixes after my own review * comment out sentIDs in WSClient see commit body for the reason * remove body.Close it will be closed automatically * stop ws connection outside of write/read routines also, use t.Rate in tm-bench indexer when calculating ID fix gocritic issues * update swagger.yaml * Apply suggestions from code review * fix stylecheck and golint linter warnings * update changelog * update changelog2
5 years ago
rpc/lib/client & server: try to conform to JSON-RPC 2.0 spec (#4141) https://www.jsonrpc.org/specification What is done in this PR: JSONRPCClient: validate that Response.ID matches Request.ID I wanted to do the same for the WSClient, but since we're sending events as responses, not notifications, checking IDs would require storing them in memory indefinitely (and we won't be able to remove them upon client unsubscribing because ID is different then). Request.ID is now optional. Notification is a Request without an ID. Previously "" or 0 were considered as notifications Remove #event suffix from ID from an event response (partially fixes #2949) ID must be either string, int or null AND must be equal to request's ID. Now, because we've implemented events as responses, WS clients are tripping when they see Response.ID("0#event") != Request.ID("0"). Implementing events as requests would require a lot of time (~ 2 days to completely rewrite WS client and server) generate unique ID for each request switch to integer IDs instead of "json-client-XYZ" id=0 method=/subscribe id=0 result=... id=1 method=/abci_query id=1 result=... > send events (resulting from /subscribe) as requests+notifications (not responses) this will require a lot of work. probably not worth it * rpc: generate an unique ID for each request in conformance with JSON-RPC spec * WSClient: check for unsolicited responses * fix golangci warnings * save commit * fix errors * remove ID from responses from subscribe Refs #2949 * clients are safe for concurrent access * tm-bench: switch to int ID * fixes after my own review * comment out sentIDs in WSClient see commit body for the reason * remove body.Close it will be closed automatically * stop ws connection outside of write/read routines also, use t.Rate in tm-bench indexer when calculating ID fix gocritic issues * update swagger.yaml * Apply suggestions from code review * fix stylecheck and golint linter warnings * update changelog * update changelog2
5 years ago
p2p: make SecretConnection non-malleable (#3668) ## Issue: This is an approach to fixing secret connection that is more noise-ish than actually noise. but it essentially fixes the problem that #3315 is trying to solve by making the secret connection handshake non-malleable. It's easy to understand and I think will be acceptable to @jaekwon .. the formal reasoning is basically, if the "view" of the transcript between diverges between the sender and the receiver at any point in the protocol, the handshake would terminate. The base protocol of Station to Station mistakenly assumes that if the sender and receiver arrive at shared secret they have the same view. This is only true for a DH on prime order groups. This robustly solves the problem by having each cryptographic operation commit to operators view of the protocol. Another nice thing about a transcript is it provides the basis for "secure" (barring cryptographic breakages, horrible design flaws, or implementation bugs) downgrades, where a backwards compatible handshake can be used to offer newer protocol features/extensions, peers agree to the common subset of what they support, and both sides have to agree on what the other offered for the transcript MAC to verify. With something like Protos/Amino you already get "extensions" for free (TLS uses a simple TLV format https://tools.ietf.org/html/rfc8446#section-4.2 for extensions not too far off from Protos/Amino), so as long as you cryptographically commit to what they contain in the transcript, it should be possible to extend the protocol in a backwards-compatible manner. ## Commits: * Minimal changes to remove malleability of secret connection removes the need to check for lower order points. Breaks compatibility. Secret connections that have no been updated will fail * Remove the redundant blacklist * remove remainders of blacklist in tests to make the code compile again Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com> * Apply suggestions from code review Apply Ismail's error handling Co-Authored-By: Ismail Khoffi <Ismail.Khoffi@gmail.com> * fix error check for io.ReadFull Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com> * Update p2p/conn/secret_connection.go Co-Authored-By: Ismail Khoffi <Ismail.Khoffi@gmail.com> * Update p2p/conn/secret_connection.go Co-Authored-By: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com> * update changelog and format the code * move hkdfInit closer to where it's used
5 years ago
p2p: make SecretConnection non-malleable (#3668) ## Issue: This is an approach to fixing secret connection that is more noise-ish than actually noise. but it essentially fixes the problem that #3315 is trying to solve by making the secret connection handshake non-malleable. It's easy to understand and I think will be acceptable to @jaekwon .. the formal reasoning is basically, if the "view" of the transcript between diverges between the sender and the receiver at any point in the protocol, the handshake would terminate. The base protocol of Station to Station mistakenly assumes that if the sender and receiver arrive at shared secret they have the same view. This is only true for a DH on prime order groups. This robustly solves the problem by having each cryptographic operation commit to operators view of the protocol. Another nice thing about a transcript is it provides the basis for "secure" (barring cryptographic breakages, horrible design flaws, or implementation bugs) downgrades, where a backwards compatible handshake can be used to offer newer protocol features/extensions, peers agree to the common subset of what they support, and both sides have to agree on what the other offered for the transcript MAC to verify. With something like Protos/Amino you already get "extensions" for free (TLS uses a simple TLV format https://tools.ietf.org/html/rfc8446#section-4.2 for extensions not too far off from Protos/Amino), so as long as you cryptographically commit to what they contain in the transcript, it should be possible to extend the protocol in a backwards-compatible manner. ## Commits: * Minimal changes to remove malleability of secret connection removes the need to check for lower order points. Breaks compatibility. Secret connections that have no been updated will fail * Remove the redundant blacklist * remove remainders of blacklist in tests to make the code compile again Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com> * Apply suggestions from code review Apply Ismail's error handling Co-Authored-By: Ismail Khoffi <Ismail.Khoffi@gmail.com> * fix error check for io.ReadFull Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com> * Update p2p/conn/secret_connection.go Co-Authored-By: Ismail Khoffi <Ismail.Khoffi@gmail.com> * Update p2p/conn/secret_connection.go Co-Authored-By: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com> * update changelog and format the code * move hkdfInit closer to where it's used
5 years ago
cli: debug sub-command (#4227) ## Issue Implement a new subcommand: tendermint debug. This subcommand itself has two subcommands: $ tendermint debug kill <pid> </path/to/out.zip> --home=</path/to/app.d> Writes debug info into a compressed archive. The archive will contain the following: ├── config.toml ├── consensus_state.json ├── net_info.json ├── stacktrace.out ├── status.json └── wal The Tendermint process will be killed. $ tendermint debug dump </path/to/out> --home=</path/to/app.d> This command will perform similar to kill except it only polls the node and dumps debugging data every frequency seconds to a compressed archive under a given destination directory. Each archive will contain: ├── consensus_state.json ├── goroutine.out ├── heap.out ├── net_info.json ├── status.json └── wal Note: goroutine.out and heap.out will only be written if a profile address is provided and is operational. This command is blocking and will log any error. replaces: #3327 closes: #3249 ## Commits: * Implement debug tool command stubs * Implement net getters and zip logic * Update zip dir API and add home flag * Add simple godocs for kill aux functions * Move IO util to new file and implement copy WAL func * Implement copy config function * Implement killProc * Remove debug fmt * Validate output file input * Direct STDERR to file * Godoc updates * Sleep prior to killing tail proc * Minor cleanup of godocs * Move debug command and add make target * Rename command handler function * Add example to command long descr * Move current implementation to cmd/tendermint/commands/debug * Update kill cmd long description * Implement dump command * Add pending log entry * Add gosec nolint * Add error check for Mkdir * Add os.IsNotExist(err) * Add to debugging section in running-in-prod doc
5 years ago
Improved tm-monitor formatting (#4023) * tm-monitor: tweaked formatting of start time and avg tx throughput. * tm-monitor: update health when validator number is updated. * Updated CHANGELOG_PENDING * Added PR number to CHANGELOG_PENDING. Improves `tm-monitor` formatting of start time (RFC1123 without unnecessary precision) and avg tx throughput (three decimal places). The old tx throughput display was confusing during local testing where the tx rate is low and displayed as 0. Also updates the monitor health whenever the validator number changes. It otherwise starts with moderate health and fails to update this once it discovers the validators, leading to incorrect health reporting and invalid uptime statistics. Let me know if you would like me to submit this as a separate PR. ### Before: ``` 2019-09-29 20:40:00.992834 +0200 CEST m=+0.024057059 up -92030989600.42% Height: 2518 Avg block time: 1275.496 ms Avg tx throughput: 0 per sec Avg block latency: 2.464 ms Active nodes: 4/4 (health: moderate) Validators: 4 NAME HEIGHT BLOCK LATENCY ONLINE VALIDATOR localhost:26657 2518 0.935 ms true true localhost:26660 2518 0.710 ms true true localhost:26662 2518 0.708 ms true true localhost:26664 2518 0.717 ms true true ``` ### After: ``` Sun, 29 Sep 2019 20:21:59 +0200 up 100.00% Height: 2480 Avg block time: 1361.445 ms Avg tx throughput: 0.735 per sec Avg block latency: 4.232 ms Active nodes: 4/4 (health: full) Validators: 4 NAME HEIGHT BLOCK LATENCY ONLINE VALIDATOR localhost:26657 2480 1.174 ms true true localhost:26660 2480 1.037 ms true true localhost:26662 2480 0.981 ms true true localhost:26664 2480 0.995 ms true true ```
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
  1. ## v0.32.9
  2. \*\*
  3. This release contains breaking changes to the `Block#Header`, specifically
  4. `NumTxs` and `TotalTxs` were removed (\#2521). Here's how this change affects
  5. different modules:
  6. - apps: it breaks the ABCI header field numbering
  7. - state: it breaks the format of `State` on disk
  8. - RPC: all RPC requests which expose the header broke
  9. - Go API: the `Header` broke
  10. - P2P: since blocks go over the wire, technically the P2P protocol broke
  11. Also, blocks are significantly smaller 🔥 because we got rid of the redundant
  12. information in `Block#LastCommit`. `Commit` now mainly consists of a signature
  13. and a validator address plus a timestamp. Note we may remove the validator
  14. address & timestamp fields in the future (see ADR-25).
  15. Special thanks to external contributors on this release:
  16. @erikgrinaker, @PSalant726, @gchaincl, @gregzaitsev, @princesinha19, @Stumble
  17. Friendly reminder, we have a [bug bounty
  18. program](https://hackerone.com/tendermint).
  19. ### BREAKING CHANGES:
  20. - CLI/RPC/Config
  21. - [rpc] \#3471 Paginate `/validators` response (default: 30 vals per page)
  22. - [rpc] \#3188 Remove `BlockMeta` in `ResultBlock` in favor of `BlockId` for `/block`
  23. - [rpc] `/block_results` response format updated (see RPC docs for details)
  24. ```
  25. {
  26. "jsonrpc": "2.0",
  27. "id": "",
  28. "result": {
  29. "height": "2109",
  30. "txs_results": null,
  31. "begin_block_events": null,
  32. "end_block_events": null,
  33. "validator_updates": null,
  34. "consensus_param_updates": null
  35. }
  36. }
  37. ```
  38. - [rpc] [\#4141](https://github.com/tendermint/tendermint/pull/4141) Remove `#event` suffix from the ID in event responses.
  39. `{"jsonrpc": "2.0", "id": 0, "result": ...}`
  40. - [rpc] [\#4141](https://github.com/tendermint/tendermint/pull/4141) Switch to integer IDs instead of `json-client-XYZ`
  41. ```
  42. id=0 method=/subscribe
  43. id=0 result=...
  44. id=1 method=/abci_query
  45. id=1 result=...
  46. ```
  47. - ID is unique for each request;
  48. - Request.ID is now optional. Notification is a Request without an ID. Previously ID="" or ID=0 were considered as notifications.
  49. - [config] \#4046 Rename tag(s) to CompositeKey & places where tag is still present it was renamed to event or events. Find how a compositeKey is constructed [here](https://github.com/tendermint/tendermint/blob/6d05c531f7efef6f0619155cf10ae8557dd7832f/docs/app-dev/indexing-transactions.md)
  50. - You will have to generate a new config for your Tendermint node(s)
  51. - Apps
  52. - [tm-bench] Removed tm-bench in favor of [tm-load-test](https://github.com/interchainio/tm-load-test)
  53. - Go API
  54. - [rpc/client] \#3471 `Validators` now requires two more args: `page` and `perPage`
  55. - [libs/common] \#3262 Make error the last parameter of `Task` (@PSalant726)
  56. - [cs/types] \#3262 Rename `GotVoteFromUnwantedRoundError` to `ErrGotVoteFromUnwantedRound` (@PSalant726)
  57. - [libs/common] \#3862 Remove `errors.go` from `libs/common`
  58. - [libs/common] \#4230 Move `KV` out of common to its own pkg
  59. - [libs/common] \#4230 Rename `cmn.KVPair(s)` to `kv.Pair(s)`s
  60. - [libs/common] \#4232 Move `Service` & `BaseService` from `libs/common` to `libs/service`
  61. - [libs/common] \#4232 Move `common/nil.go` to `types/utils.go` & make the functions private
  62. - [libs/common] \#4231 Move random functions from `libs/common` into pkg `rand`
  63. - [libs/common] \#4237 Move byte functions from `libs/common` into pkg `bytes`
  64. - [libs/common] \#4237 Move throttletimer functions from `libs/common` into pkg `timer`
  65. - [libs/common] \#4237 Move tempfile functions from `libs/common` into pkg `tempfile`
  66. - [libs/common] \#4240 Move os functions from `libs/common` into pkg `os`
  67. - [libs/common] \#4240 Move net functions from `libs/common` into pkg `net`
  68. - [libs/common] \#4240 Move mathematical functions and types out of `libs/common` to `math` pkg
  69. - [libs/common] \#4240 Move string functions out of `libs/common` to `strings` pkg
  70. - [libs/common] \#4240 Move async functions out of `libs/common` to `async` pkg
  71. - [libs/common] \#4240 Move bit functions out of `libs/common` to `bits` pkg
  72. - [libs/common] \#4240 Move cmap functions out of `libs/common` to `cmap` pkg
  73. - [libs/common] \#4258 Remove `Rand` from all `rand` pkg functions
  74. - Blockchain Protocol
  75. - [abci] \#2521 Remove `TotalTxs` and `NumTxs` from `Header`
  76. - [types] [\#4151](https://github.com/tendermint/tendermint/pull/4151) Enforce ordering of votes in DuplicateVoteEvidence to be lexicographically sorted on BlockID
  77. - [types] \#1648 Change `Commit` to consist of just signatures
  78. - P2P Protocol
  79. - [p2p] [\#3668](https://github.com/tendermint/tendermint/pull/3668) Make `SecretConnection` non-malleable
  80. - [proto] [\#3986](https://github.com/tendermint/tendermint/pull/3986) Prefix protobuf types to avoid name conflicts.
  81. - ABCI becomes `tendermint.abci.types` with the new API endpoint `/tendermint.abci.types.ABCIApplication/`
  82. - core_grpc becomes `tendermint.rpc.grpc` with the new API endpoint `/tendermint.rpc.grpc.BroadcastAPI/`
  83. - merkle becomes `tendermint.crypto.merkle`
  84. - libs.common becomes `tendermint.libs.common`
  85. - proto3 becomes `tendermint.types.proto3`
  86. ### FEATURES:
  87. - [p2p] \#4053 Add `unconditional_peer_ids` and `persistent_peers_max_dial_period` config variables (see ADR-050) (@dongsam)
  88. - [tools] [\#4227](https://github.com/tendermint/tendermint/pull/4227) Implement `tendermint debug kill` and
  89. `tendermint debug dump` commands for Tendermint node debugging functionality. See `--help` in both
  90. commands for further documentation and usage.
  91. - [cli] \#4234 Add `--db_backend and --db_dir` flags (@princesinha19)
  92. - [cli] \#4113 Add optional `--genesis_hash` flag to check genesis hash upon startup
  93. - [config] \#3831 Add support for [RocksDB](https://rocksdb.org/) (@Stumble)
  94. - [metrics] \#4263 Add
  95. - `consensus_validator_power`: track your validators power
  96. - `consensus_validator_last_signed_height`: track at which height the validator last signed
  97. - `consensus_validator_missed_blocks`: total amount of missed blocks for a validator
  98. as gauges in prometheus for validator specific metrics
  99. ### IMPROVEMENTS:
  100. - [rpc] \#3188 Added `block_size` to `BlockMeta` this is reflected in `/blockchain`
  101. - [types] \#2521 Add `NumTxs` to `BlockMeta` and `EventDataNewBlockHeader`
  102. - [docs] [\#4111](https://github.com/tendermint/tendermint/issues/4111) Replaced dead whitepaper link in README.md
  103. - [p2p] [\#4185](https://github.com/tendermint/tendermint/pull/4185) Simplify `SecretConnection` handshake with merlin
  104. - [cli] [\#4065](https://github.com/tendermint/tendermint/issues/4065) Add `--consensus.create_empty_blocks_interval` flag (@jgimeno)
  105. - [docs] [\#4065](https://github.com/tendermint/tendermint/issues/4065) Document `--consensus.create_empty_blocks_interval` flag (@jgimeno)
  106. - [crypto] [\#4190](https://github.com/tendermint/tendermint/pull/4190) Added SR25519 signature scheme
  107. - [abci] [\#4177] kvstore: Return `LastBlockHeight` and `LastBlockAppHash` in `Info` (@princesinha19)
  108. - [rpc] [\#2741](https://github.com/tendermint/tendermint/issues/2741) Add `proposer` to `/consensus_state` response (@princesinha19)
  109. ### BUG FIXES:
  110. - [rpc/lib][\#4051](https://github.com/tendermint/tendermint/pull/4131) Fix RPC client, which was previously resolving https protocol to http (@yenkhoon)
  111. - [rpc] [\#4141](https://github.com/tendermint/tendermint/pull/4141) JSONRPCClient: validate that Response.ID matches Request.ID
  112. - [rpc] [\#4141](https://github.com/tendermint/tendermint/pull/4141) WSClient: check for unsolicited responses
  113. - [types] [\4164](https://github.com/tendermint/tendermint/pull/4164) Prevent temporary power overflows on validator updates
  114. - [cs] \#4069 Don't panic when block meta is not found in store (@gregzaitsev)
  115. - [types] \#4164 Prevent temporary power overflows on validator updates (joint
  116. efforts of @gchaincl and @ancazamfir)
  117. - [p2p] \#4140 `SecretConnection`: use the transcript solely for authentication (i.e. MAC)
  118. - [consensus/types] \#4243 fix BenchmarkRoundStateDeepCopy panics (@cuonglm)
  119. - [rpc] \#4256 Pass `outCapacity` to `eventBus#Subscribe` when subscribing using a local client