You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

344 lines
12 KiB

  1. # Encoding
  2. ## Amino
  3. Tendermint uses the proto3 derivative [Amino](https://github.com/tendermint/go-amino) for all data structures.
  4. Think of Amino as an object-oriented proto3 with native JSON support.
  5. The goal of the Amino encoding protocol is to bring parity between application
  6. logic objects and persistence objects.
  7. Please see the [Amino
  8. specification](https://github.com/tendermint/go-amino#amino-encoding-for-go) for
  9. more details.
  10. Notably, every object that satisfies an interface (eg. a particular kind of p2p message,
  11. or a particular kind of pubkey) is registered with a global name, the hash of
  12. which is included in the object's encoding as the so-called "prefix bytes".
  13. We define the `func AminoEncode(obj interface{}) []byte` function to take an
  14. arbitrary object and return the Amino encoded bytes.
  15. ## Byte Arrays
  16. The encoding of a byte array is simply the raw-bytes prefixed with the length of
  17. the array as a `UVarint` (what proto calls a `Varint`).
  18. For details on varints, see the [protobuf
  19. spec](https://developers.google.com/protocol-buffers/docs/encoding#varints).
  20. For example, the byte-array `[0xA, 0xB]` would be encoded as `0x020A0B`,
  21. while a byte-array containing 300 entires beginning with `[0xA, 0xB, ...]` would
  22. be encoded as `0xAC020A0B...` where `0xAC02` is the UVarint encoding of 300.
  23. ## Hashing
  24. Tendermint uses `SHA256` as its hash function.
  25. Objects are always Amino encoded before being hashed.
  26. So `SHA256(obj)` is short for `SHA256(AminoEncode(obj))`.
  27. ## Public Key Cryptography
  28. Tendermint uses Amino to distinguish between different types of private keys,
  29. public keys, and signatures. Additionally, for each public key, Tendermint
  30. defines an Address function that can be used as a more compact identifier in
  31. place of the public key. Here we list the concrete types, their names,
  32. and prefix bytes for public keys and signatures, as well as the address schemes
  33. for each PubKey. Note for brevity we don't
  34. include details of the private keys beyond their type and name, as they can be
  35. derived the same way as the others using Amino.
  36. All registered objects are encoded by Amino using a 4-byte PrefixBytes that
  37. uniquely identifies the object and includes information about its underlying
  38. type. For details on how PrefixBytes are computed, see the [Amino
  39. spec](https://github.com/tendermint/go-amino#computing-the-prefix-and-disambiguation-bytes).
  40. In what follows, we provide the type names and prefix bytes directly.
  41. Notice that when encoding byte-arrays, the length of the byte-array is appended
  42. to the PrefixBytes. Thus the encoding of a byte array becomes `<PrefixBytes> <Length> <ByteArray>`. In other words, to encode any type listed below you do not need to be
  43. familiar with amino encoding.
  44. You can simply use below table and concatenate Prefix || Length (of raw bytes) || raw bytes
  45. ( while || stands for byte concatenation here).
  46. | Type | Name | Prefix | Length | Notes |
  47. | ----------------------- | ---------------------------------- | ---------- | -------- | ----- |
  48. | PubKeyEd25519 | tendermint/PubKeyEd25519 | 0x1624DE64 | 0x20 | |
  49. | PubKeySecp256k1 | tendermint/PubKeySecp256k1 | 0xEB5AE987 | 0x21 | |
  50. | PrivKeyEd25519 | tendermint/PrivKeyEd25519 | 0xA3288910 | 0x40 | |
  51. | PrivKeySecp256k1 | tendermint/PrivKeySecp256k1 | 0xE1B0F79B | 0x20 | |
  52. | PubKeyMultisigThreshold | tendermint/PubKeyMultisigThreshold | 0x22C1F7E2 | variable | |
  53. ### Example
  54. For example, the 33-byte (or 0x21-byte in hex) Secp256k1 pubkey
  55. `020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9`
  56. would be encoded as
  57. `EB5AE98721020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9`
  58. ### Key Types
  59. Each type specifies it's own pubkey, address, and signature format.
  60. #### Ed25519
  61. TODO: pubkey
  62. The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key:
  63. ```
  64. address = SHA256(pubkey)[:20]
  65. ```
  66. The signature is the raw 64-byte ED25519 signature.
  67. #### Secp256k1
  68. TODO: pubkey
  69. The address is the RIPEMD160 hash of the SHA256 hash of the OpenSSL compressed public key:
  70. ```
  71. address = RIPEMD160(SHA256(pubkey))
  72. ```
  73. This is the same as Bitcoin.
  74. The signature is the 64-byte concatenation of ECDSA `r` and `s` (ie. `r || s`),
  75. where `s` is lexicographically less than its inverse, to prevent malleability.
  76. This is like Ethereum, but without the extra byte for pubkey recovery, since
  77. Tendermint assumes the pubkey is always provided anyway.
  78. #### Multisig
  79. TODO
  80. ## Other Common Types
  81. ### BitArray
  82. The BitArray is used in some consensus messages to represent votes received from
  83. validators, or parts received in a block. It is represented
  84. with a struct containing the number of bits (`Bits`) and the bit-array itself
  85. encoded in base64 (`Elems`).
  86. ```go
  87. type BitArray struct {
  88. Bits int
  89. Elems []uint64
  90. }
  91. ```
  92. This type is easily encoded directly by Amino.
  93. Note BitArray receives a special JSON encoding in the form of `x` and `_`
  94. representing `1` and `0`. Ie. the BitArray `10110` would be JSON encoded as
  95. `"x_xx_"`
  96. ### Part
  97. Part is used to break up blocks into pieces that can be gossiped in parallel
  98. and securely verified using a Merkle tree of the parts.
  99. Part contains the index of the part (`Index`), the actual
  100. underlying data of the part (`Bytes`), and a Merkle proof that the part is contained in
  101. the set (`Proof`).
  102. ```go
  103. type Part struct {
  104. Index int
  105. Bytes []byte
  106. Proof SimpleProof
  107. }
  108. ```
  109. See details of SimpleProof, below.
  110. ### MakeParts
  111. Encode an object using Amino and slice it into parts.
  112. Tendermint uses a part size of 65536 bytes.
  113. ```go
  114. func MakeParts(block Block) []Part
  115. ```
  116. ## Merkle Trees
  117. For an overview of Merkle trees, see
  118. [wikipedia](https://en.wikipedia.org/wiki/Merkle_tree)
  119. We use the RFC 6962 specification of a merkle tree, with sha256 as the hash function.
  120. Merkle trees are used throughout Tendermint to compute a cryptographic digest of a data structure.
  121. The differences between RFC 6962 and the simplest form a merkle tree are that:
  122. 1. leaf nodes and inner nodes have different hashes.
  123. This is for "second pre-image resistance", to prevent the proof to an inner node being valid as the proof of a leaf.
  124. The leaf nodes are `SHA256(0x00 || leaf_data)`, and inner nodes are `SHA256(0x01 || left_hash || right_hash)`.
  125. 2. When the number of items isn't a power of two, the left half of the tree is as big as it could be.
  126. (The largest power of two less than the number of items) This allows new leaves to be added with less
  127. recomputation. For example:
  128. ```
  129. Simple Tree with 6 items Simple Tree with 7 items
  130. * *
  131. / \ / \
  132. / \ / \
  133. / \ / \
  134. / \ / \
  135. * * * *
  136. / \ / \ / \ / \
  137. / \ / \ / \ / \
  138. / \ / \ / \ / \
  139. * * h4 h5 * * * h6
  140. / \ / \ / \ / \ / \
  141. h0 h1 h2 h3 h0 h1 h2 h3 h4 h5
  142. ```
  143. ### MerkleRoot
  144. The function `MerkleRoot` is a simple recursive function defined as follows:
  145. ```go
  146. // SHA256(0x00 || leaf)
  147. func leafHash(leaf []byte) []byte {
  148. return tmhash.Sum(append(0x00, leaf...))
  149. }
  150. // SHA256(0x01 || left || right)
  151. func innerHash(left []byte, right []byte) []byte {
  152. return tmhash.Sum(append(0x01, append(left, right...)...))
  153. }
  154. // largest power of 2 less than k
  155. func getSplitPoint(k int) { ... }
  156. func MerkleRoot(items [][]byte) []byte{
  157. switch len(items) {
  158. case 0:
  159. return nil
  160. case 1:
  161. return leafHash(items[0])
  162. default:
  163. k := getSplitPoint(len(items))
  164. left := MerkleRoot(items[:k])
  165. right := MerkleRoot(items[k:])
  166. return innerHash(left, right)
  167. }
  168. }
  169. ```
  170. Note: `MerkleRoot` operates on items which are arbitrary byte arrays, not
  171. necessarily hashes. For items which need to be hashed first, we introduce the
  172. `Hashes` function:
  173. ```
  174. func Hashes(items [][]byte) [][]byte {
  175. return SHA256 of each item
  176. }
  177. ```
  178. Note: we will abuse notion and invoke `MerkleRoot` with arguments of type `struct` or type `[]struct`.
  179. For `struct` arguments, we compute a `[][]byte` containing the amino encoding of each
  180. field in the struct, in the same order the fields appear in the struct.
  181. For `[]struct` arguments, we compute a `[][]byte` by amino encoding the individual `struct` elements.
  182. ### Simple Merkle Proof
  183. Proof that a leaf is in a Merkle tree is composed as follows:
  184. ```golang
  185. type SimpleProof struct {
  186. Total int
  187. Index int
  188. LeafHash []byte
  189. Aunts [][]byte
  190. }
  191. ```
  192. Which is verified as follows:
  193. ```golang
  194. func (proof SimpleProof) Verify(rootHash []byte, leaf []byte) bool {
  195. assert(proof.LeafHash, leafHash(leaf)
  196. computedHash := computeHashFromAunts(proof.Index, proof.Total, proof.LeafHash, proof.Aunts)
  197. return computedHash == rootHash
  198. }
  199. func computeHashFromAunts(index, total int, leafHash []byte, innerHashes [][]byte) []byte{
  200. assert(index < total && index >= 0 && total > 0)
  201. if total == 1{
  202. assert(len(proof.Aunts) == 0)
  203. return leafHash
  204. }
  205. assert(len(innerHashes) > 0)
  206. numLeft := getSplitPoint(total) // largest power of 2 less than total
  207. if index < numLeft {
  208. leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1])
  209. assert(leftHash != nil)
  210. return innerHash(leftHash, innerHashes[len(innerHashes)-1])
  211. }
  212. rightHash := computeHashFromAunts(index-numLeft, total-numLeft, leafHash, innerHashes[:len(innerHashes)-1])
  213. assert(rightHash != nil)
  214. return innerHash(innerHashes[len(innerHashes)-1], rightHash)
  215. }
  216. ```
  217. ### IAVL+ Tree
  218. Because Tendermint only uses a Simple Merkle Tree, application developers are expect to use their own Merkle tree in their applications. For example, the IAVL+ Tree - an immutable self-balancing binary tree for persisting application state is used by the [Cosmos SDK](https://github.com/cosmos/cosmos-sdk/blob/master/docs/clients/lite/specification.md)
  219. ## JSON
  220. ### Amino
  221. Amino also supports JSON encoding - registered types are simply encoded as:
  222. ```
  223. {
  224. "type": "<amino type name>",
  225. "value": <JSON>
  226. }
  227. ```
  228. For instance, an ED25519 PubKey would look like:
  229. ```
  230. {
  231. "type": "tendermint/PubKeyEd25519",
  232. "value": "uZ4h63OFWuQ36ZZ4Bd6NF+/w9fWUwrOncrQsackrsTk="
  233. }
  234. ```
  235. Where the `"value"` is the base64 encoding of the raw pubkey bytes, and the
  236. `"type"` is the amino name for Ed25519 pubkeys.
  237. ### Signed Messages
  238. Signed messages (eg. votes, proposals) in the consensus are encoded using Amino.
  239. When signing, the elements of a message are re-ordered so the fixed-length fields
  240. are first, making it easy to quickly check the type, height, and round.
  241. The `ChainID` is also appended to the end.
  242. We call this encoding the SignBytes. For instance, SignBytes for a vote is the Amino encoding of the following struct:
  243. ```go
  244. type CanonicalVote struct {
  245. Type byte
  246. Height int64 `binary:"fixed64"`
  247. Round int64 `binary:"fixed64"`
  248. BlockID CanonicalBlockID
  249. Timestamp time.Time
  250. ChainID string
  251. }
  252. ```
  253. The field ordering and the fixed sized encoding for the first three fields is optimized to ease parsing of SignBytes
  254. in HSMs. It creates fixed offsets for relevant fields that need to be read in this context.
  255. For more details, see the [signing spec](../consensus/signing.md).
  256. Also, see the motivating discussion in
  257. [#1622](https://github.com/tendermint/tendermint/issues/1622).