You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

407 lines
12 KiB

7 years ago
7 years ago
add support for block pruning via ABCI Commit response (#4588) * Added BlockStore.DeleteBlock() * Added initial block pruner prototype * wip * Added BlockStore.PruneBlocks() * Added consensus setting for block pruning * Added BlockStore base * Error on replay if base does not have blocks * Handle missing blocks when sending VoteSetMaj23Message * Error message tweak * Properly update blockstore state * Error message fix again * blockchain: ignore peer missing blocks * Added FIXME * Added test for block replay with truncated history * Handle peer base in blockchain reactor * Improved replay error handling * Added tests for Store.PruneBlocks() * Fix non-RPC handling of truncated block history * Panic on missing block meta in needProofBlock() * Updated changelog * Handle truncated block history in RPC layer * Added info about earliest block in /status RPC * Reorder height and base in blockchain reactor messages * Updated changelog * Fix tests * Appease linter * Minor review fixes * Non-empty BlockStores should always have base > 0 * Update code to assume base > 0 invariant * Added blockstore tests for pruning to 0 * Make sure we don't prune below the current base * Added BlockStore.Size() * config: added retain_blocks recommendations * Update v1 blockchain reactor to handle blockstore base * Added state database pruning * Propagate errors on missing validator sets * Comment tweaks * Improved error message Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * use ABCI field ResponseCommit.retain_height instead of retain-blocks config option * remove State.RetainHeight, return value instead * fix minor issues * rename pruneHeights() to pruneBlocks() * noop to fix GitHub borkage Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
4 years ago
abci: Refactor tagging events using list of lists (#3643) ## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test
5 years ago
abci: Refactor tagging events using list of lists (#3643) ## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test
5 years ago
7 years ago
  1. syntax = "proto3";
  2. package tendermint.abci;
  3. option go_package = "github.com/tendermint/tendermint/abci/types";
  4. // For more information on gogo.proto, see:
  5. // https://github.com/gogo/protobuf/blob/master/extensions.md
  6. import "tendermint/crypto/proof.proto";
  7. import "tendermint/types/types.proto";
  8. import "tendermint/crypto/keys.proto";
  9. import "tendermint/types/params.proto";
  10. import "google/protobuf/timestamp.proto";
  11. import "gogoproto/gogo.proto";
  12. // This file is copied from http://github.com/tendermint/abci
  13. // NOTE: When using custom types, mind the warnings.
  14. // https://github.com/gogo/protobuf/blob/master/custom_types.md#warnings-and-issues
  15. //----------------------------------------
  16. // Request types
  17. message Request {
  18. oneof value {
  19. RequestEcho echo = 1;
  20. RequestFlush flush = 2;
  21. RequestInfo info = 3;
  22. RequestSetOption set_option = 4;
  23. RequestInitChain init_chain = 5;
  24. RequestQuery query = 6;
  25. RequestBeginBlock begin_block = 7;
  26. RequestCheckTx check_tx = 8;
  27. RequestDeliverTx deliver_tx = 9;
  28. RequestEndBlock end_block = 10;
  29. RequestCommit commit = 11;
  30. RequestListSnapshots list_snapshots = 12;
  31. RequestOfferSnapshot offer_snapshot = 13;
  32. RequestLoadSnapshotChunk load_snapshot_chunk = 14;
  33. RequestApplySnapshotChunk apply_snapshot_chunk = 15;
  34. }
  35. }
  36. message RequestEcho {
  37. string message = 1;
  38. }
  39. message RequestFlush {}
  40. message RequestInfo {
  41. string version = 1;
  42. uint64 block_version = 2;
  43. uint64 p2p_version = 3;
  44. }
  45. // nondeterministic
  46. message RequestSetOption {
  47. string key = 1;
  48. string value = 2;
  49. }
  50. message RequestInitChain {
  51. google.protobuf.Timestamp time = 1
  52. [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
  53. string chain_id = 2;
  54. ConsensusParams consensus_params = 3;
  55. repeated ValidatorUpdate validators = 4 [(gogoproto.nullable) = false];
  56. bytes app_state_bytes = 5;
  57. int64 initial_height = 6;
  58. }
  59. message RequestQuery {
  60. bytes data = 1;
  61. string path = 2;
  62. int64 height = 3;
  63. bool prove = 4;
  64. }
  65. message RequestBeginBlock {
  66. bytes hash = 1;
  67. tendermint.types.Header header = 2 [(gogoproto.nullable) = false];
  68. LastCommitInfo last_commit_info = 3 [(gogoproto.nullable) = false];
  69. repeated Evidence byzantine_validators = 4 [(gogoproto.nullable) = false];
  70. }
  71. enum CheckTxType {
  72. NEW = 0 [(gogoproto.enumvalue_customname) = "New"];
  73. RECHECK = 1 [(gogoproto.enumvalue_customname) = "Recheck"];
  74. }
  75. message RequestCheckTx {
  76. bytes tx = 1;
  77. CheckTxType type = 2;
  78. }
  79. message RequestDeliverTx {
  80. bytes tx = 1;
  81. }
  82. message RequestEndBlock {
  83. int64 height = 1;
  84. }
  85. message RequestCommit {}
  86. // lists available snapshots
  87. message RequestListSnapshots {
  88. }
  89. // offers a snapshot to the application
  90. message RequestOfferSnapshot {
  91. Snapshot snapshot = 1; // snapshot offered by peers
  92. bytes app_hash = 2; // light client-verified app hash for snapshot height
  93. }
  94. // loads a snapshot chunk
  95. message RequestLoadSnapshotChunk {
  96. uint64 height = 1;
  97. uint32 format = 2;
  98. uint32 chunk = 3;
  99. }
  100. // Applies a snapshot chunk
  101. message RequestApplySnapshotChunk {
  102. uint32 index = 1;
  103. bytes chunk = 2;
  104. string sender = 3;
  105. }
  106. //----------------------------------------
  107. // Response types
  108. message Response {
  109. oneof value {
  110. ResponseException exception = 1;
  111. ResponseEcho echo = 2;
  112. ResponseFlush flush = 3;
  113. ResponseInfo info = 4;
  114. ResponseSetOption set_option = 5;
  115. ResponseInitChain init_chain = 6;
  116. ResponseQuery query = 7;
  117. ResponseBeginBlock begin_block = 8;
  118. ResponseCheckTx check_tx = 9;
  119. ResponseDeliverTx deliver_tx = 10;
  120. ResponseEndBlock end_block = 11;
  121. ResponseCommit commit = 12;
  122. ResponseListSnapshots list_snapshots = 13;
  123. ResponseOfferSnapshot offer_snapshot = 14;
  124. ResponseLoadSnapshotChunk load_snapshot_chunk = 15;
  125. ResponseApplySnapshotChunk apply_snapshot_chunk = 16;
  126. }
  127. }
  128. // nondeterministic
  129. message ResponseException {
  130. string error = 1;
  131. }
  132. message ResponseEcho {
  133. string message = 1;
  134. }
  135. message ResponseFlush {}
  136. message ResponseInfo {
  137. string data = 1;
  138. string version = 2;
  139. uint64 app_version = 3;
  140. int64 last_block_height = 4;
  141. bytes last_block_app_hash = 5;
  142. }
  143. // nondeterministic
  144. message ResponseSetOption {
  145. uint32 code = 1;
  146. // bytes data = 2;
  147. string log = 3;
  148. string info = 4;
  149. }
  150. message ResponseInitChain {
  151. ConsensusParams consensus_params = 1;
  152. repeated ValidatorUpdate validators = 2 [(gogoproto.nullable) = false];
  153. bytes app_hash = 3;
  154. }
  155. message ResponseQuery {
  156. uint32 code = 1;
  157. // bytes data = 2; // use "value" instead.
  158. string log = 3; // nondeterministic
  159. string info = 4; // nondeterministic
  160. int64 index = 5;
  161. bytes key = 6;
  162. bytes value = 7;
  163. tendermint.crypto.ProofOps proof_ops = 8;
  164. int64 height = 9;
  165. string codespace = 10;
  166. }
  167. message ResponseBeginBlock {
  168. repeated Event events = 1
  169. [(gogoproto.nullable) = false, (gogoproto.jsontag) = "events,omitempty"];
  170. }
  171. message ResponseCheckTx {
  172. uint32 code = 1;
  173. bytes data = 2;
  174. string log = 3; // nondeterministic
  175. string info = 4; // nondeterministic
  176. int64 gas_wanted = 5 [json_name = "gas_wanted"];
  177. int64 gas_used = 6 [json_name = "gas_used"];
  178. repeated Event events = 7
  179. [(gogoproto.nullable) = false, (gogoproto.jsontag) = "events,omitempty"];
  180. string codespace = 8;
  181. }
  182. message ResponseDeliverTx {
  183. uint32 code = 1;
  184. bytes data = 2;
  185. string log = 3; // nondeterministic
  186. string info = 4; // nondeterministic
  187. int64 gas_wanted = 5 [json_name = "gas_wanted"];
  188. int64 gas_used = 6 [json_name = "gas_used"];
  189. repeated Event events = 7
  190. [(gogoproto.nullable) = false, (gogoproto.jsontag) = "events,omitempty"];
  191. string codespace = 8;
  192. }
  193. message ResponseEndBlock {
  194. repeated ValidatorUpdate validator_updates = 1
  195. [(gogoproto.nullable) = false];
  196. ConsensusParams consensus_param_updates = 2;
  197. repeated Event events = 3
  198. [(gogoproto.nullable) = false, (gogoproto.jsontag) = "events,omitempty"];
  199. }
  200. message ResponseCommit {
  201. // reserve 1
  202. bytes data = 2;
  203. int64 retain_height = 3;
  204. }
  205. message ResponseListSnapshots {
  206. repeated Snapshot snapshots = 1;
  207. }
  208. message ResponseOfferSnapshot {
  209. Result result = 1;
  210. enum Result {
  211. UNKNOWN = 0; // Unknown result, abort all snapshot restoration
  212. ACCEPT = 1; // Snapshot accepted, apply chunks
  213. ABORT = 2; // Abort all snapshot restoration
  214. REJECT = 3; // Reject this specific snapshot, try others
  215. REJECT_FORMAT = 4; // Reject all snapshots of this format, try others
  216. REJECT_SENDER = 5; // Reject all snapshots from the sender(s), try others
  217. }
  218. }
  219. message ResponseLoadSnapshotChunk {
  220. bytes chunk = 1;
  221. }
  222. message ResponseApplySnapshotChunk {
  223. Result result = 1;
  224. repeated uint32 refetch_chunks = 2; // Chunks to refetch and reapply
  225. repeated string reject_senders = 3; // Chunk senders to reject and ban
  226. enum Result {
  227. UNKNOWN = 0; // Unknown result, abort all snapshot restoration
  228. ACCEPT = 1; // Chunk successfully accepted
  229. ABORT = 2; // Abort all snapshot restoration
  230. RETRY = 3; // Retry chunk (combine with refetch and reject)
  231. RETRY_SNAPSHOT = 4; // Retry snapshot (combine with refetch and reject)
  232. REJECT_SNAPSHOT = 5; // Reject this snapshot, try others
  233. }
  234. }
  235. //----------------------------------------
  236. // Misc.
  237. // ConsensusParams contains all consensus-relevant parameters
  238. // that can be adjusted by the abci app
  239. message ConsensusParams {
  240. BlockParams block = 1;
  241. tendermint.types.EvidenceParams evidence = 2;
  242. tendermint.types.ValidatorParams validator = 3;
  243. tendermint.types.VersionParams version = 4;
  244. }
  245. // BlockParams contains limits on the block size.
  246. message BlockParams {
  247. // Note: must be greater than 0
  248. int64 max_bytes = 1;
  249. // Note: must be greater or equal to -1
  250. int64 max_gas = 2;
  251. }
  252. message LastCommitInfo {
  253. int32 round = 1;
  254. repeated VoteInfo votes = 2 [(gogoproto.nullable) = false];
  255. }
  256. // Event allows application developers to attach additional information to
  257. // ResponseBeginBlock, ResponseEndBlock, ResponseCheckTx and ResponseDeliverTx.
  258. // Later, transactions may be queried using these events.
  259. message Event {
  260. string type = 1;
  261. repeated EventAttribute attributes = 2 [
  262. (gogoproto.nullable) = false,
  263. (gogoproto.jsontag) = "attributes,omitempty"
  264. ];
  265. }
  266. // EventAttribute is a single key-value pair, associated with an event.
  267. message EventAttribute {
  268. bytes key = 1;
  269. bytes value = 2;
  270. bool index = 3; // nondeterministic
  271. }
  272. // TxResult contains results of executing the transaction.
  273. //
  274. // One usage is indexing transaction results.
  275. message TxResult {
  276. int64 height = 1;
  277. uint32 index = 2;
  278. bytes tx = 3;
  279. ResponseDeliverTx result = 4 [(gogoproto.nullable) = false];
  280. }
  281. //----------------------------------------
  282. // Blockchain Types
  283. // Validator
  284. message Validator {
  285. bytes address = 1; // The first 20 bytes of SHA256(public key)
  286. // PubKey pub_key = 2 [(gogoproto.nullable)=false];
  287. int64 power = 3; // The voting power
  288. }
  289. // ValidatorUpdate
  290. message ValidatorUpdate {
  291. tendermint.crypto.PublicKey pub_key = 1 [(gogoproto.nullable) = false];
  292. int64 power = 2;
  293. }
  294. // VoteInfo
  295. message VoteInfo {
  296. Validator validator = 1 [(gogoproto.nullable) = false];
  297. bool signed_last_block = 2;
  298. }
  299. enum EvidenceType {
  300. UNKNOWN = 0;
  301. DUPLICATE_VOTE = 1;
  302. LIGHT_CLIENT_ATTACK = 2;
  303. }
  304. message Evidence {
  305. EvidenceType type = 1;
  306. // The offending validator
  307. Validator validator = 2 [(gogoproto.nullable) = false];
  308. // The height when the offense occurred
  309. int64 height = 3;
  310. // The corresponding time where the offense occurred
  311. google.protobuf.Timestamp time = 4 [
  312. (gogoproto.nullable) = false,
  313. (gogoproto.stdtime) = true
  314. ];
  315. // Total voting power of the validator set in case the ABCI application does
  316. // not store historical validators.
  317. // https://github.com/tendermint/tendermint/issues/4581
  318. int64 total_voting_power = 5;
  319. }
  320. //----------------------------------------
  321. // State Sync Types
  322. message Snapshot {
  323. uint64 height = 1; // The height at which the snapshot was taken
  324. uint32 format = 2; // The application-specific snapshot format
  325. uint32 chunks = 3; // Number of chunks in the snapshot
  326. bytes hash = 4; // Arbitrary snapshot hash, equal only if identical
  327. bytes metadata = 5; // Arbitrary application metadata
  328. }
  329. //----------------------------------------
  330. // Service Definition
  331. service ABCIApplication {
  332. rpc Echo(RequestEcho) returns (ResponseEcho);
  333. rpc Flush(RequestFlush) returns (ResponseFlush);
  334. rpc Info(RequestInfo) returns (ResponseInfo);
  335. rpc SetOption(RequestSetOption) returns (ResponseSetOption);
  336. rpc DeliverTx(RequestDeliverTx) returns (ResponseDeliverTx);
  337. rpc CheckTx(RequestCheckTx) returns (ResponseCheckTx);
  338. rpc Query(RequestQuery) returns (ResponseQuery);
  339. rpc Commit(RequestCommit) returns (ResponseCommit);
  340. rpc InitChain(RequestInitChain) returns (ResponseInitChain);
  341. rpc BeginBlock(RequestBeginBlock) returns (ResponseBeginBlock);
  342. rpc EndBlock(RequestEndBlock) returns (ResponseEndBlock);
  343. rpc ListSnapshots(RequestListSnapshots) returns (ResponseListSnapshots);
  344. rpc OfferSnapshot(RequestOfferSnapshot) returns (ResponseOfferSnapshot);
  345. rpc LoadSnapshotChunk(RequestLoadSnapshotChunk) returns (ResponseLoadSnapshotChunk);
  346. rpc ApplySnapshotChunk(RequestApplySnapshotChunk) returns (ResponseApplySnapshotChunk);
  347. }