You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

320 lines
9.8 KiB

7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
  1. # Tendermint Encoding
  2. ## Amino
  3. Tendermint uses the Protobuf3 derrivative [Amino]() for all data structures.
  4. Think of Amino as an object-oriented Protobuf3 with native JSON support.
  5. The goal of the Amino encoding protocol is to bring parity between application
  6. logic objects and persistence objects.
  7. Please see the [Amino
  8. specification](https://github.com/tendermint/go-amino#amino-encoding-for-go) for
  9. more details.
  10. Notably, every object that satisfies an interface (eg. a particular kind of p2p message,
  11. or a particular kind of pubkey) is registered with a global name, the hash of
  12. which is included in the object's encoding as the so-called "prefix bytes".
  13. We define the `func AminoEncode(obj interface{}) []byte` function to take an
  14. arbitrary object and return the Amino encoded bytes.
  15. ## Byte Arrays
  16. The encoding of a byte array is simply the raw-bytes prefixed with the length of
  17. the array as a `UVarint` (what Protobuf calls a `Varint`).
  18. For details on varints, see the [protobuf
  19. spec](https://developers.google.com/protocol-buffers/docs/encoding#varints).
  20. For example, the byte-array `[0xA, 0xB]` would be encoded as `0x020A0B`,
  21. while a byte-array containing 300 entires beginning with `[0xA, 0xB, ...]` would
  22. be encoded as `0xAC020A0B...` where `0xAC02` is the UVarint encoding of 300.
  23. ## Public Key Cryptography
  24. Tendermint uses Amino to distinguish between different types of private keys,
  25. public keys, and signatures. Additionally, for each public key, Tendermint
  26. defines an Address function that can be used as a more compact identifier in
  27. place of the public key. Here we list the concrete types, their names,
  28. and prefix bytes for public keys and signatures, as well as the address schemes
  29. for each PubKey. Note for brevity we don't
  30. include details of the private keys beyond their type and name, as they can be
  31. derrived the same way as the others using Amino.
  32. All registered objects are encoded by Amino using a 4-byte PrefixBytes that
  33. uniquely identifies the object and includes information about its underlying
  34. type. For details on how PrefixBytes are computed, see the [Amino
  35. spec](https://github.com/tendermint/go-amino#computing-the-prefix-and-disambiguation-bytes).
  36. In what follows, we provide the type names and prefix bytes directly.
  37. Notice that when encoding byte-arrays, the length of the byte-array is appended
  38. to the PrefixBytes. Thus the encoding of a byte array becomes `<PrefixBytes>
  39. <Length> <ByteArray>`
  40. (NOTE: the remainder of this section on Public Key Cryptography can be generated
  41. from [this script](./scripts/crypto.go))
  42. ### PubKeyEd25519
  43. ```
  44. // Name: tendermint/PubKeyEd25519
  45. // PrefixBytes: 0x1624DE62
  46. // Length: 0x20
  47. // Notes: raw 32-byte Ed25519 pubkey
  48. type PubKeyEd25519 [32]byte
  49. func (pubkey PubKeyEd25519) Address() []byte {
  50. // NOTE: hash of the Amino encoded bytes!
  51. return RIPEMD160(AminoEncode(pubkey))
  52. }
  53. ```
  54. For example, the 32-byte Ed25519 pubkey
  55. `CCACD52F9B29D04393F01CD9AF6535455668115641F3D8BAEFD2295F24BAF60E` would be
  56. encoded as
  57. `1624DE6220CCACD52F9B29D04393F01CD9AF6535455668115641F3D8BAEFD2295F24BAF60E`.
  58. The address would then be
  59. `RIPEMD160(0x1624DE6220CCACD52F9B29D04393F01CD9AF6535455668115641F3D8BAEFD2295F24BAF60E)`
  60. or `430FF75BAF1EC4B0D51BB3EEC2955479D0071605`
  61. ### SignatureEd25519
  62. ```
  63. // Name: tendermint/SignatureKeyEd25519
  64. // PrefixBytes: 0x3DA1DB2A
  65. // Length: 0x40
  66. // Notes: raw 64-byte Ed25519 signature
  67. type SignatureEd25519 [64]byte
  68. ```
  69. For example, the 64-byte Ed25519 signature
  70. `1B6034A8ED149D3C94FDA13EC03B26CC0FB264D9B0E47D3FA3DEF9FCDE658E49C80B35F9BE74949356401B15B18FB817D6E54495AD1C4A8401B248466CB0DB0B`
  71. would be encoded as
  72. `3DA1DB2A401B6034A8ED149D3C94FDA13EC03B26CC0FB264D9B0E47D3FA3DEF9FCDE658E49C80B35F9BE74949356401B15B18FB817D6E54495AD1C4A8401B248466CB0DB0B`
  73. ### PrivKeyEd25519
  74. ```
  75. // Name: tendermint/PrivKeyEd25519
  76. // Notes: raw 32-byte priv key concatenated to raw 32-byte pub key
  77. type PrivKeyEd25519 [64]byte
  78. ```
  79. ### PubKeySecp256k1
  80. ```
  81. // Name: tendermint/PubKeySecp256k1
  82. // PrefixBytes: 0xEB5AE982
  83. // Length: 0x21
  84. // Notes: OpenSSL compressed pubkey prefixed with 0x02 or 0x03
  85. type PubKeySecp256k1 [33]byte
  86. func (pubkey PubKeySecp256k1) Address() []byte {
  87. // NOTE: hash of the raw pubkey bytes (not Amino encoded!).
  88. // Compatible with Bitcoin addresses.
  89. return RIPEMD160(SHA256(pubkey[:]))
  90. }
  91. ```
  92. For example, the 33-byte Secp256k1 pubkey
  93. `020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9` would be
  94. encoded as
  95. `EB5AE98221020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9`
  96. The address would then be
  97. `RIPEMD160(SHA256(0x020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9))`
  98. or `0AE5BEE929ABE51BAD345DB925EEA652680783FC`
  99. ### SignatureSecp256k1
  100. ```
  101. // Name: tendermint/SignatureKeySecp256k1
  102. // PrefixBytes: 0x16E1FEEA
  103. // Length: Variable
  104. // Encoding prefix: Variable
  105. // Notes: raw bytes of the Secp256k1 signature
  106. type SignatureSecp256k1 []byte
  107. ```
  108. For example, the Secp256k1 signature
  109. `304402201CD4B8C764D2FD8AF23ECFE6666CA8A53886D47754D951295D2D311E1FEA33BF02201E0F906BB1CF2C30EAACFFB032A7129358AFF96B9F79B06ACFFB18AC90C2ADD7`
  110. would be encoded as
  111. `16E1FEEA46304402201CD4B8C764D2FD8AF23ECFE6666CA8A53886D47754D951295D2D311E1FEA33BF02201E0F906BB1CF2C30EAACFFB032A7129358AFF96B9F79B06ACFFB18AC90C2ADD7`
  112. ### PrivKeySecp256k1
  113. ```
  114. // Name: tendermint/PrivKeySecp256k1
  115. // Notes: raw 32-byte priv key
  116. type PrivKeySecp256k1 [32]byte
  117. ```
  118. ## Other Common Types
  119. ### BitArray
  120. The BitArray is used in block headers and some consensus messages to signal
  121. whether or not something was done by each validator. BitArray is represented
  122. with a struct containing the number of bits (`Bits`) and the bit-array itself
  123. encoded in base64 (`Elems`).
  124. ```go
  125. type BitArray struct {
  126. Bits int
  127. Elems []uint64
  128. }
  129. ```
  130. This type is easily encoded directly by Amino.
  131. Note BitArray receives a special JSON encoding in the form of `x` and `_`
  132. representing `1` and `0`. Ie. the BitArray `10110` would be JSON encoded as
  133. `"x_xx_"`
  134. ### Part
  135. Part is used to break up blocks into pieces that can be gossiped in parallel
  136. and securely verified using a Merkle tree of the parts.
  137. Part contains the index of the part in the larger set (`Index`), the actual
  138. underlying data of the part (`Bytes`), and a simple Merkle proof that the part is contained in
  139. the larger set (`Proof`).
  140. ```go
  141. type Part struct {
  142. Index int
  143. Bytes byte[]
  144. Proof byte[]
  145. }
  146. ```
  147. ### MakeParts
  148. Encode an object using Amino and slice it into parts.
  149. ```go
  150. func MakeParts(obj interface{}, partSize int) []Part
  151. ```
  152. ## Merkle Trees
  153. Simple Merkle trees are used in numerous places in Tendermint to compute a cryptographic digest of a data structure.
  154. RIPEMD160 is always used as the hashing function.
  155. ### Simple Merkle Root
  156. The function `SimpleMerkleRoot` is a simple recursive function defined as follows:
  157. ```go
  158. func SimpleMerkleRoot(hashes [][]byte) []byte{
  159. switch len(hashes) {
  160. case 0:
  161. return nil
  162. case 1:
  163. return hashes[0]
  164. default:
  165. left := SimpleMerkleRoot(hashes[:(len(hashes)+1)/2])
  166. right := SimpleMerkleRoot(hashes[(len(hashes)+1)/2:])
  167. return SimpleConcatHash(left, right)
  168. }
  169. }
  170. func SimpleConcatHash(left, right []byte) []byte{
  171. left = encodeByteSlice(left)
  172. right = encodeByteSlice(right)
  173. return RIPEMD160 (append(left, right))
  174. }
  175. ```
  176. Note that the leaves are Amino encoded as byte-arrays (ie. simple Uvarint length
  177. prefix) before being concatenated together and hashed.
  178. Note: we will abuse notion and invoke `SimpleMerkleRoot` with arguments of type `struct` or type `[]struct`.
  179. For `struct` arguments, we compute a `[][]byte` by sorting elements of the `struct` according to
  180. field name and then hashing them.
  181. For `[]struct` arguments, we compute a `[][]byte` by hashing the individual `struct` elements.
  182. ### Simple Merkle Proof
  183. Proof that a leaf is in a Merkle tree consists of a simple structure:
  184. ```
  185. type SimpleProof struct {
  186. Aunts [][]byte
  187. }
  188. ```
  189. Which is verified using the following:
  190. ```
  191. func (proof SimpleProof) Verify(index, total int, leafHash, rootHash []byte) bool {
  192. computedHash := computeHashFromAunts(index, total, leafHash, proof.Aunts)
  193. return computedHash == rootHash
  194. }
  195. func computeHashFromAunts(index, total int, leafHash []byte, innerHashes [][]byte) []byte{
  196. assert(index < total && index >= 0 && total > 0)
  197. if total == 1{
  198. assert(len(proof.Aunts) == 0)
  199. return leafHash
  200. }
  201. assert(len(innerHashes) > 0)
  202. numLeft := (total + 1) / 2
  203. if index < numLeft {
  204. leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1])
  205. assert(leftHash != nil)
  206. return SimpleHashFromTwoHashes(leftHash, innerHashes[len(innerHashes)-1])
  207. }
  208. rightHash := computeHashFromAunts(index-numLeft, total-numLeft, leafHash, innerHashes[:len(innerHashes)-1])
  209. assert(rightHash != nil)
  210. return SimpleHashFromTwoHashes(innerHashes[len(innerHashes)-1], rightHash)
  211. }
  212. ```
  213. ## JSON
  214. ### Amino
  215. TODO: improve this
  216. Amino also supports JSON encoding - registered types are simply encoded as:
  217. ```
  218. {
  219. "type": "<DisfixBytes>",
  220. "value": <JSON>
  221. }
  222. For instance, an ED25519 PubKey would look like:
  223. ```
  224. {
  225. "type": "AC26791624DE60",
  226. "value": "uZ4h63OFWuQ36ZZ4Bd6NF+/w9fWUwrOncrQsackrsTk="
  227. }
  228. ```
  229. Where the `"value"` is the base64 encoding of the raw pubkey bytes, and the
  230. `"type"` is the full disfix bytes for Ed25519 pubkeys.
  231. ### Signed Messages
  232. Signed messages (eg. votes, proposals) in the consensus are encoded using Amino-JSON, rather than in the standard binary format.
  233. When signing, the elements of a message are sorted by key and the sorted message is embedded in an
  234. outer JSON that includes a `chain_id` field.
  235. We call this encoding the CanonicalSignBytes. For instance, CanonicalSignBytes for a vote would look
  236. like:
  237. ```json
  238. {"chain_id":"my-chain-id","vote":{"block_id":{"hash":DEADBEEF,"parts":{"hash":BEEFDEAD,"total":3}},"height":3,"round":2,"timestamp":1234567890, "type":2}
  239. ```
  240. Note how the fields within each level are sorted.