You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

314 lines
15 KiB

  1. # ADR 053: State Sync Prototype
  2. This ADR outlines the plan for an initial state sync prototype, and is subject to change as we gain feedback and experience. It builds on discussions and findings in [ADR-042](./adr-042-state-sync.md), see that for background information.
  3. ## Changelog
  4. * 2020-01-28: Initial draft (Erik Grinaker)
  5. * 2020-02-18: Updates after initial prototype (Erik Grinaker)
  6. * ABCI: added missing `reason` fields.
  7. * ABCI: used 32-bit 1-based chunk indexes (was 64-bit 0-based).
  8. * ABCI: moved `RequestApplySnapshotChunk.chain_hash` to `RequestOfferSnapshot.app_hash`.
  9. * Gaia: snapshots must include node versions as well, both for inner and leaf nodes.
  10. * Added experimental prototype info.
  11. * Added open questions and implementation plan.
  12. * 2020-03-29: Strengthened and simplified ABCI interface (Erik Grinaker)
  13. * ABCI: replaced `chunks` with `chunk_hashes` in `Snapshot`.
  14. * ABCI: removed `SnapshotChunk` message.
  15. * ABCI: renamed `GetSnapshotChunk` to `LoadSnapshotChunk`.
  16. * ABCI: chunks are now exchanged simply as `bytes`.
  17. * ABCI: chunks are now 0-indexed, for parity with `chunk_hashes` array.
  18. * Reduced maximum chunk size to 16 MB, and increased snapshot message size to 4 MB.
  19. ## Context
  20. State sync will allow a new node to receive a snapshot of the application state without downloading blocks or going through consensus. This bootstraps the node significantly faster than the current fast sync system, which replays all historical blocks.
  21. Background discussions and justifications are detailed in [ADR-042](./adr-042-state-sync.md). Its recommendations can be summarized as:
  22. * The application periodically takes full state snapshots (i.e. eager snapshots).
  23. * The application splits snapshots into smaller chunks that can be individually verified against a chain app hash.
  24. * Tendermint uses the light client to obtain a trusted chain app hash for verification.
  25. * Tendermint discovers and downloads snapshot chunks in parallel from multiple peers, and passes them to the application via ABCI to be applied and verified against the chain app hash.
  26. * Historical blocks are not backfilled, so state synced nodes will have a truncated block history.
  27. ## Tendermint Proposal
  28. This describes the snapshot/restore process seen from Tendermint. The interface is kept as small and general as possible to give applications maximum flexibility.
  29. ### Snapshot Data Structure
  30. A node can have multiple snapshots taken at various heights. Snapshots can be taken in different application-specified formats (e.g. MessagePack as format `1` and Protobuf as format `2`, or similarly for schema versioning). Each snapshot consists of multiple chunks containing the actual state data, for parallel downloads and reduced memory usage.
  31. ```proto
  32. message Snapshot {
  33. uint64 height = 1; // The height at which the snapshot was taken
  34. uint32 format = 2; // The application-specific snapshot format
  35. repeated bytes chunk_hashes = 3; // SHA-256 checksums of all chunks, in order
  36. bytes metadata = 4; // Arbitrary application metadata
  37. }
  38. ```
  39. Chunks are exchanged simply as `bytes`, and cannot be larger than 16 MB. `Snapshot` messages should be less than 4 MB.
  40. ### ABCI Interface
  41. ```proto
  42. // Lists available snapshots
  43. message RequestListSnapshots {}
  44. message ResponseListSnapshots {
  45. repeated Snapshot snapshots = 1;
  46. }
  47. // Offers a snapshot to the application
  48. message RequestOfferSnapshot {
  49. Snapshot snapshot = 1;
  50. bytes app_hash = 2;
  51. }
  52. message ResponseOfferSnapshot {
  53. bool accepted = 1;
  54. Reason reason = 2;
  55. enum Reason { // Reason why snapshot was rejected
  56. unknown = 0; // Unknown or generic reason
  57. invalid_height = 1; // Height is rejected: avoid this height
  58. invalid_format = 2; // Format is rejected: avoid this format
  59. }
  60. }
  61. // Loads a snapshot chunk
  62. message RequestLoadSnapshotChunk {
  63. uint64 height = 1;
  64. uint32 format = 2;
  65. uint32 chunk = 3; // Zero-indexed
  66. }
  67. message ResponseLoadSnapshotChunk {
  68. bytes chunk = 1;
  69. }
  70. // Applies a snapshot chunk
  71. message RequestApplySnapshotChunk {
  72. bytes chunk = 1;
  73. }
  74. message ResponseApplySnapshotChunk {
  75. bool applied = 1;
  76. Reason reason = 2; // Reason why chunk failed
  77. enum Reason { // Reason why chunk failed
  78. unknown = 0; // Unknown or generic reason
  79. verify_failed = 1; // Snapshot verification failed
  80. }
  81. }
  82. ```
  83. ### Taking Snapshots
  84. Tendermint is not aware of the snapshotting process at all, it is entirely an application concern. The following guarantees must be provided:
  85. * **Periodic:** snapshots must be taken periodically, not on-demand, for faster restores, lower load, and less DoS risk.
  86. * **Deterministic:** snapshots must be deterministic, and identical across all nodes - typically by taking a snapshot at given height intervals.
  87. * **Consistent:** snapshots must be consistent, i.e. not affected by concurrent writes - typically by using a data store that supports versioning and/or snapshot isolation.
  88. * **Asynchronous:** snapshots must be asynchronous, i.e. not halt block processing and state transitions.
  89. * **Chunked:** snapshots must be split into chunks of reasonable size (on the order of megabytes), and each chunk must be verifiable against the chain app hash.
  90. * **Garbage collected:** snapshots must be garbage collected periodically.
  91. ### Restoring Snapshots
  92. Nodes should have options for enabling state sync and/or fast sync, and be provided a trusted header hash for the light client.
  93. When starting an empty node with state sync and fast sync enabled, snapshots are restored as follows:
  94. 1. The node checks that it is empty, i.e. that it has no state nor blocks.
  95. 2. The node contacts the given seeds to discover peers.
  96. 3. The node contacts a set of full nodes, and verifies the trusted block header using the given hash via the light client.
  97. 4. The node requests available snapshots via P2P from peers, via `RequestListSnapshots`. Peers will return the 10 most recent snapshots, one message per snapshot.
  98. 5. The node aggregates snapshots from multiple peers, ordered by height and format (in reverse). If there are `chunk_hashes` mismatches between different snapshots, the one hosted by the largest amount of peers is chosen. The node iterates over all snapshots in reverse order by height and format until it finds one that satisfies all of the following conditions:
  99. * The snapshot height's block is considered trustworthy by the light client (i.e. snapshot height is greater than trusted header and within unbonding period of the latest trustworthy block).
  100. * The snapshot's height or format hasn't been explicitly rejected by an earlier `RequestOfferSnapshot` call (via `invalid_height` or `invalid_format`).
  101. * The application accepts the `RequestOfferSnapshot` call.
  102. 6. The node downloads chunks in parallel from multiple peers, via `RequestLoadSnapshotChunk`, and both the sender and receiver verifies their checksums. Chunk messages cannot exceed 16 MB.
  103. 7. The node passes chunks sequentially to the app via `RequestApplySnapshotChunk`.
  104. 8. Once all chunks have been applied, the node compares the app hash to the chain app hash, and if they do not match it either errors or discards the state and starts over.
  105. 9. The node switches to fast sync to catch up blocks that were committed while restoring the snapshot.
  106. 10. The node switches to normal consensus mode.
  107. ## Gaia Proposal
  108. This describes the snapshot process seen from Gaia, using format version `1`. The serialization format is unspecified, but likely to be compressed Amino or Protobuf.
  109. ### Snapshot Metadata
  110. In the initial version there is no snapshot metadata, so it is set to an empty byte buffer.
  111. Once all chunks have been successfully built, snapshot metadata should be stored in a database and served via `RequestListSnapshots`.
  112. ### Snapshot Chunk Format
  113. The Gaia data structure consists of a set of named IAVL trees. A root hash is constructed by taking the root hashes of each of the IAVL trees, then constructing a Merkle tree of the sorted name/hash map.
  114. IAVL trees are versioned, but a snapshot only contains the version relevant for the snapshot height. All historical versions are ignored.
  115. IAVL trees are insertion-order dependent, so key/value pairs must be set in an appropriate insertion order to produce the same tree branching structure. This insertion order can be found by doing a breadth-first scan of all nodes (including inner nodes) and collecting unique keys in order. However, the node hash also depends on the node's version, so snapshots must contain the inner nodes' version numbers as well.
  116. For the initial prototype, each chunk consists of a complete dump of all node data for all nodes in an entire IAVL tree. Thus the number of chunks equals the number of persistent stores in Gaia. No incremental verification of chunks is done, only a final app hash comparison at the end of the snapshot restoration.
  117. For a production version, it should be sufficient to store key/value/version for all nodes (leaf and inner) in insertion order, chunked in some appropriate way. If per-chunk verification is required, the chunk must also contain enough information to reconstruct the Merkle proofs all the way up to the root of the multistore, e.g. by storing a complete subtree's key/value/version data plus Merkle hashes of all other branches up to the multistore root. The exact approach will depend on tradeoffs between size, time, and verification. IAVL RangeProofs are not recommended, since these include redundant data such as proofs for intermediate and leaf nodes that can be derived from the above data.
  118. Chunks should be built greedily by collecting node data up to some size limit (e.g. 10 MB) and serializing it. Chunk data is stored in the file system as `snapshots/<height>/<format>/<chunk>`, and a SHA-256 checksum is stored along with the snapshot metadata.
  119. ### Snapshot Scheduling
  120. Snapshots should be taken at some configurable height interval, e.g. every 1000 blocks. All nodes should preferably have the same snapshot schedule, such that all nodes can serve chunks for a given snapshot.
  121. Taking consistent snapshots of IAVL trees is greatly simplified by them being versioned: simply snapshot the version that corresponds to the snapshot height, while concurrent writes create new versions. IAVL pruning must not prune a version that is being snapshotted.
  122. Snapshots must also be garbage collected after some configurable time, e.g. by keeping the latest `n` snapshots.
  123. ## Experimental Prototype
  124. An experimental but functional state sync prototype is available in the `erik/statesync-prototype` branches of the Tendermint, IAVL, Cosmos SDK, and Gaia repositories. To fetch the necessary branches:
  125. ```sh
  126. $ mkdir statesync
  127. $ cd statesync
  128. $ git clone git@github.com:tendermint/tendermint -b erik/statesync-prototype
  129. $ git clone git@github.com:tendermint/iavl -b erik/statesync-prototype
  130. $ git clone git@github.com:cosmos/cosmos-sdk -b erik/statesync-prototype
  131. $ git clone git@github.com:cosmos/gaia -b erik/statesync-prototype
  132. ```
  133. To spin up three nodes of a four-node testnet:
  134. ```sh
  135. $ cd gaia
  136. $ ./tools/start.sh
  137. ```
  138. Wait for the first snapshot to be taken at height 3, then (in a separate terminal) start the fourth node with state sync enabled:
  139. ```sh
  140. $ ./tools/sync.sh
  141. ```
  142. To stop the testnet, run:
  143. ```sh
  144. $ ./tools/stop.sh
  145. ```
  146. ## Resolved Questions
  147. * Is it OK for state-synced nodes to not have historical blocks nor historical IAVL versions?
  148. > Yes, this is as intended. Maybe backfill blocks later.
  149. * Do we need incremental chunk verification for first version?
  150. > No, we'll start simple. Can add chunk verification via a new snapshot format without any breaking changes in Tendermint. For adversarial conditions, maybe consider support for whitelisting peers to download chunks from.
  151. * Should the snapshot ABCI interface be a separate optional ABCI service, or mandatory?
  152. > Mandatory, to keep things simple for now. It will therefore be a breaking change and push the release. For apps using the Cosmos SDK, we can provide a default implementation that does not serve snapshots and errors when trying to apply them.
  153. * How can we make sure `ListSnapshots` data is valid? An adversary can provide fake/invalid snapshots to DoS peers.
  154. > For now, just pick snapshots that are available on a large number of peers. Maybe support whitelisting. We may consider e.g. placing snapshot manifests on the blockchain later.
  155. * Should we punish nodes that provide invalid snapshots? How?
  156. > No, these are full nodes not validators, so we can't punish them. Just disconnect from them and ignore them.
  157. * Should we call these snapshots? The SDK already uses the term "snapshot" for `PruningOptions.SnapshotEvery`, and state sync will introduce additional SDK options for snapshot scheduling and pruning that are not related to IAVL snapshotting or pruning.
  158. > Yes. Hopefully these concepts are distinct enough that we can refer to state sync snapshots and IAVL snapshots without too much confusion.
  159. * Should we store snapshot and chunk metadata in a database? Can we use the database for chunks?
  160. > As a first approach, store metadata in a database and chunks in the filesystem.
  161. * Should a snapshot at height H be taken before or after the block at H is processed? E.g. RPC `/commit` returns app_hash after _previous_ height, i.e. _before_ current height.
  162. > After commit.
  163. * Do we need to support all versions of blockchain reactor (i.e. fast sync)?
  164. > We should remove the v1 reactor completely once v2 has stabilized.
  165. * Should `ListSnapshots` be a streaming API instead of a request/response API?
  166. > No, just use a max message size.
  167. ## Implementation Plan
  168. ### Core Tasks
  169. * **Tendermint:** light client P2P transport [#4456](https://github.com/tendermint/tendermint/issues/4456)
  170. * **IAVL:** export/import API [#210](https://github.com/tendermint/iavl/issues/210)
  171. * **Cosmos SDK:** snapshotting, scheduling, and pruning [#5689](https://github.com/cosmos/cosmos-sdk/issues/5689)
  172. * **Tendermint:** support starting with a truncated block history
  173. * **Tendermint:** state sync reactor and ABCI interface [#828](https://github.com/tendermint/tendermint/issues/828)
  174. * **Cosmos SDK:** snapshot ABCI implementation [#5690](https://github.com/cosmos/cosmos-sdk/issues/5690)
  175. ### Nice-to-Haves
  176. * **Tendermint:** staged reactor startup (state sync → fast sync → block replay → wal replay → consensus)
  177. > Let's do a time-boxed prototype (a few days) and see how much work it will be.
  178. * Notify P2P peers about channel changes [#4394](https://github.com/tendermint/tendermint/issues/4394)
  179. * Check peers have certain channels [#1148](https://github.com/tendermint/tendermint/issues/1148)
  180. * **Tendermint:** prune blockchain history [#3652](https://github.com/tendermint/tendermint/issues/3652)
  181. * **Tendermint:** allow genesis to start from non-zero height [#2543](https://github.com/tendermint/tendermint/issues/2543)
  182. ### Follow-up Tasks
  183. * **Tendermint:** light client verification for fast sync [#4457](https://github.com/tendermint/tendermint/issues/4457)
  184. * **Tendermint:** allow start with only blockstore [#3713](https://github.com/tendermint/tendermint/issues/3713)
  185. * **Tendermint:** node should go back to fast-syncing when lagging significantly [#129](https://github.com/tendermint/tendermint/issues/129)
  186. * **Tendermint:** backfill historical blocks [#4629](https://github.com/tendermint/tendermint/issues/4629)
  187. ## Status
  188. Accepted
  189. ## References
  190. * [ADR-042](./adr-042-state-sync.md) and its references