|
|
@ -1,106 +1,163 @@ |
|
|
|
# Tendermint Encoding |
|
|
|
|
|
|
|
## Binary Serialization (TMBIN) |
|
|
|
## Amino |
|
|
|
|
|
|
|
Tendermint aims to encode data structures in a manner similar to how the corresponding Go structs |
|
|
|
are laid out in memory. |
|
|
|
Variable length items are length-prefixed. |
|
|
|
While the encoding was inspired by Go, it is easily implemented in other languages as well, given its intuitive design. |
|
|
|
Tendermint uses the Protobuf3 derrivative [Amino]() for all data structures. |
|
|
|
Think of Amino as an object-oriented Protobuf3 with native JSON support. |
|
|
|
The goal of the Amino encoding protocol is to bring parity between application |
|
|
|
logic objects and persistence objects. |
|
|
|
|
|
|
|
XXX: This is changing to use real varints and 4-byte-prefixes. |
|
|
|
See https://github.com/tendermint/go-wire/tree/sdk2. |
|
|
|
Please see the [Amino |
|
|
|
specification](https://github.com/tendermint/go-amino#amino-encoding-for-go) for |
|
|
|
more details. |
|
|
|
|
|
|
|
### Fixed Length Integers |
|
|
|
Notably, every object that satisfies an interface (eg. a particular kind of p2p message, |
|
|
|
or a particular kind of pubkey) is registered with a global name, the hash of |
|
|
|
which is included in the object's encoding as the so-called "prefix bytes". |
|
|
|
|
|
|
|
Fixed length integers are encoded in Big-Endian using the specified number of bytes. |
|
|
|
So `uint8` and `int8` use one byte, `uint16` and `int16` use two bytes, |
|
|
|
`uint32` and `int32` use 3 bytes, and `uint64` and `int64` use 4 bytes. |
|
|
|
We define the `func AminoEncode(obj interface{}) []byte` function to take an |
|
|
|
arbitrary object and return the Amino encoded bytes. |
|
|
|
|
|
|
|
Negative integers are encoded via twos-complement. |
|
|
|
## Byte Arrays |
|
|
|
|
|
|
|
Examples: |
|
|
|
The encoding of a byte array is simply the raw-bytes prefixed with the length of |
|
|
|
the array as a `UVarint` (what Protobuf calls a `Varint`). |
|
|
|
|
|
|
|
```go |
|
|
|
encode(uint8(6)) == [0x06] |
|
|
|
encode(uint32(6)) == [0x00, 0x00, 0x00, 0x06] |
|
|
|
|
|
|
|
encode(int8(-6)) == [0xFA] |
|
|
|
encode(int32(-6)) == [0xFF, 0xFF, 0xFF, 0xFA] |
|
|
|
``` |
|
|
|
For details on varints, see the [protobuf |
|
|
|
spec](https://developers.google.com/protocol-buffers/docs/encoding#varints). |
|
|
|
|
|
|
|
### Variable Length Integers |
|
|
|
For example, the byte-array `[0xA, 0xB]` would be encoded as `0x020A0B`, |
|
|
|
while a byte-array containing 300 entires beginning with `[0xA, 0xB, ...]` would |
|
|
|
be encoded as `0xAC020A0B...` where `0xAC02` is the UVarint encoding of 300. |
|
|
|
|
|
|
|
Variable length integers are encoded as length-prefixed Big-Endian integers. |
|
|
|
The length-prefix consists of a single byte and corresponds to the length of the encoded integer. |
|
|
|
## Public Key Cryptography |
|
|
|
|
|
|
|
Negative integers are encoded by flipping the leading bit of the length-prefix to a `1`. |
|
|
|
Tendermint uses Amino to distinguish between different types of private keys, |
|
|
|
public keys, and signatures. Additionally, for each public key, Tendermint |
|
|
|
defines an Address function that can be used as a more compact identifier in |
|
|
|
place of the public key. Here we list the concrete types, their names, |
|
|
|
and prefix bytes for public keys and signatures, as well as the address schemes |
|
|
|
for each PubKey. Note for brevity we don't |
|
|
|
include details of the private keys beyond their type and name, as they can be |
|
|
|
derrived the same way as the others using Amino. |
|
|
|
|
|
|
|
Zero is encoded as `0x00`. It is not length-prefixed. |
|
|
|
All registered objects are encoded by Amino using a 4-byte PrefixBytes that |
|
|
|
uniquely identifies the object and includes information about its underlying |
|
|
|
type. For details on how PrefixBytes are computed, see the [Amino |
|
|
|
spec](https://github.com/tendermint/go-amino#computing-the-prefix-and-disambiguation-bytes). |
|
|
|
|
|
|
|
Examples: |
|
|
|
In what follows, we provide the type names and prefix bytes directly. |
|
|
|
Notice that when encoding byte-arrays, the length of the byte-array is appended |
|
|
|
to the PrefixBytes. Thus the encoding of a byte array becomes `<PrefixBytes> |
|
|
|
<Length> <ByteArray>` |
|
|
|
|
|
|
|
```go |
|
|
|
encode(uint(6)) == [0x01, 0x06] |
|
|
|
encode(uint(70000)) == [0x03, 0x01, 0x11, 0x70] |
|
|
|
(NOTE: the remainder of this section on Public Key Cryptography can be generated |
|
|
|
from [this script](./scripts/crypto.go)) |
|
|
|
|
|
|
|
encode(int(-6)) == [0xF1, 0x06] |
|
|
|
encode(int(-70000)) == [0xF3, 0x01, 0x11, 0x70] |
|
|
|
### PubKeyEd25519 |
|
|
|
|
|
|
|
encode(int(0)) == [0x00] |
|
|
|
``` |
|
|
|
// Name: tendermint/PubKeyEd25519 |
|
|
|
// PrefixBytes: 0x1624DE62 |
|
|
|
// Length: 0x20 |
|
|
|
// Notes: raw 32-byte Ed25519 pubkey |
|
|
|
type PubKeyEd25519 [32]byte |
|
|
|
|
|
|
|
func (pubkey PubKeyEd25519) Address() []byte { |
|
|
|
// NOTE: hash of the Amino encoded bytes! |
|
|
|
return RIPEMD160(AminoEncode(pubkey)) |
|
|
|
} |
|
|
|
``` |
|
|
|
|
|
|
|
### Strings |
|
|
|
|
|
|
|
An encoded string is length-prefixed followed by the underlying bytes of the string. |
|
|
|
The length-prefix is itself encoded as an `int`. |
|
|
|
For example, the 32-byte Ed25519 pubkey |
|
|
|
`CCACD52F9B29D04393F01CD9AF6535455668115641F3D8BAEFD2295F24BAF60E` would be |
|
|
|
encoded as |
|
|
|
`1624DE6220CCACD52F9B29D04393F01CD9AF6535455668115641F3D8BAEFD2295F24BAF60E`. |
|
|
|
|
|
|
|
The empty string is encoded as `0x00`. It is not length-prefixed. |
|
|
|
The address would then be |
|
|
|
`RIPEMD160(0x1624DE6220CCACD52F9B29D04393F01CD9AF6535455668115641F3D8BAEFD2295F24BAF60E)` |
|
|
|
or `430FF75BAF1EC4B0D51BB3EEC2955479D0071605` |
|
|
|
|
|
|
|
Examples: |
|
|
|
### SignatureEd25519 |
|
|
|
|
|
|
|
```go |
|
|
|
encode("") == [0x00] |
|
|
|
encode("a") == [0x01, 0x01, 0x61] |
|
|
|
encode("hello") == [0x01, 0x05, 0x68, 0x65, 0x6C, 0x6C, 0x6F] |
|
|
|
encode("¥") == [0x01, 0x02, 0xC2, 0xA5] |
|
|
|
``` |
|
|
|
// Name: tendermint/SignatureKeyEd25519 |
|
|
|
// PrefixBytes: 0x3DA1DB2A |
|
|
|
// Length: 0x40 |
|
|
|
// Notes: raw 64-byte Ed25519 signature |
|
|
|
type SignatureEd25519 [64]byte |
|
|
|
``` |
|
|
|
|
|
|
|
### Arrays (fixed length) |
|
|
|
For example, the 64-byte Ed25519 signature |
|
|
|
`1B6034A8ED149D3C94FDA13EC03B26CC0FB264D9B0E47D3FA3DEF9FCDE658E49C80B35F9BE74949356401B15B18FB817D6E54495AD1C4A8401B248466CB0DB0B` |
|
|
|
would be encoded as |
|
|
|
`3DA1DB2A401B6034A8ED149D3C94FDA13EC03B26CC0FB264D9B0E47D3FA3DEF9FCDE658E49C80B35F9BE74949356401B15B18FB817D6E54495AD1C4A8401B248466CB0DB0B` |
|
|
|
|
|
|
|
An encoded fix-lengthed array is the concatenation of the encoding of its elements. |
|
|
|
There is no length-prefix. |
|
|
|
### PrivKeyEd25519 |
|
|
|
|
|
|
|
Examples: |
|
|
|
``` |
|
|
|
// Name: tendermint/PrivKeyEd25519 |
|
|
|
// Notes: raw 32-byte priv key concatenated to raw 32-byte pub key |
|
|
|
type PrivKeyEd25519 [64]byte |
|
|
|
``` |
|
|
|
|
|
|
|
```go |
|
|
|
encode([4]int8{1, 2, 3, 4}) == [0x01, 0x02, 0x03, 0x04] |
|
|
|
encode([4]int16{1, 2, 3, 4}) == [0x00, 0x01, 0x00, 0x02, 0x00, 0x03, 0x00, 0x04] |
|
|
|
encode([4]int{1, 2, 3, 4}) == [0x01, 0x01, 0x01, 0x02, 0x01, 0x03, 0x01, 0x04] |
|
|
|
encode([2]string{"abc", "efg"}) == [0x01, 0x03, 0x61, 0x62, 0x63, 0x01, 0x03, 0x65, 0x66, 0x67] |
|
|
|
### PubKeySecp256k1 |
|
|
|
|
|
|
|
``` |
|
|
|
// Name: tendermint/PubKeySecp256k1 |
|
|
|
// PrefixBytes: 0xEB5AE982 |
|
|
|
// Length: 0x21 |
|
|
|
// Notes: OpenSSL compressed pubkey prefixed with 0x02 or 0x03 |
|
|
|
type PubKeySecp256k1 [33]byte |
|
|
|
|
|
|
|
func (pubkey PubKeySecp256k1) Address() []byte { |
|
|
|
// NOTE: hash of the raw pubkey bytes (not Amino encoded!). |
|
|
|
// Compatible with Bitcoin addresses. |
|
|
|
return RIPEMD160(SHA256(pubkey[:])) |
|
|
|
} |
|
|
|
``` |
|
|
|
|
|
|
|
### Slices (variable length) |
|
|
|
For example, the 33-byte Secp256k1 pubkey |
|
|
|
`020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9` would be |
|
|
|
encoded as |
|
|
|
`EB5AE98221020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9` |
|
|
|
|
|
|
|
The address would then be |
|
|
|
`RIPEMD160(SHA256(0x020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9))` |
|
|
|
or `0AE5BEE929ABE51BAD345DB925EEA652680783FC` |
|
|
|
|
|
|
|
An encoded variable-length array is length-prefixed followed by the concatenation of the encoding of |
|
|
|
its elements. |
|
|
|
The length-prefix is itself encoded as an `int`. |
|
|
|
### SignatureSecp256k1 |
|
|
|
|
|
|
|
An empty slice is encoded as `0x00`. It is not length-prefixed. |
|
|
|
``` |
|
|
|
// Name: tendermint/SignatureKeySecp256k1 |
|
|
|
// PrefixBytes: 0x16E1FEEA |
|
|
|
// Length: Variable |
|
|
|
// Encoding prefix: Variable |
|
|
|
// Notes: raw bytes of the Secp256k1 signature |
|
|
|
type SignatureSecp256k1 []byte |
|
|
|
``` |
|
|
|
|
|
|
|
Examples: |
|
|
|
For example, the Secp256k1 signature |
|
|
|
`304402201CD4B8C764D2FD8AF23ECFE6666CA8A53886D47754D951295D2D311E1FEA33BF02201E0F906BB1CF2C30EAACFFB032A7129358AFF96B9F79B06ACFFB18AC90C2ADD7` |
|
|
|
would be encoded as |
|
|
|
`16E1FEEA46304402201CD4B8C764D2FD8AF23ECFE6666CA8A53886D47754D951295D2D311E1FEA33BF02201E0F906BB1CF2C30EAACFFB032A7129358AFF96B9F79B06ACFFB18AC90C2ADD7` |
|
|
|
|
|
|
|
### PrivKeySecp256k1 |
|
|
|
|
|
|
|
```go |
|
|
|
encode([]int8{}) == [0x00] |
|
|
|
encode([]int8{1, 2, 3, 4}) == [0x01, 0x04, 0x01, 0x02, 0x03, 0x04] |
|
|
|
encode([]int16{1, 2, 3, 4}) == [0x01, 0x04, 0x00, 0x01, 0x00, 0x02, 0x00, 0x03, 0x00, 0x04] |
|
|
|
encode([]int{1, 2, 3, 4}) == [0x01, 0x04, 0x01, 0x01, 0x01, 0x02, 0x01, 0x03, 0x01, 0x4] |
|
|
|
encode([]string{"abc", "efg"}) == [0x01, 0x02, 0x01, 0x03, 0x61, 0x62, 0x63, 0x01, 0x03, 0x65, 0x66, 0x67] |
|
|
|
``` |
|
|
|
// Name: tendermint/PrivKeySecp256k1 |
|
|
|
// Notes: raw 32-byte priv key |
|
|
|
type PrivKeySecp256k1 [32]byte |
|
|
|
``` |
|
|
|
|
|
|
|
## Other Common Types |
|
|
|
|
|
|
|
### BitArray |
|
|
|
|
|
|
|
BitArray is encoded as an `int` of the number of bits, and with an array of `uint64` to encode |
|
|
|
value of each array element. |
|
|
|
The BitArray is used in block headers and some consensus messages to signal |
|
|
|
whether or not something was done by each validator. BitArray is represented |
|
|
|
with a struct containing the number of bits (`Bits`) and the bit-array itself |
|
|
|
encoded in base64 (`Elems`). |
|
|
|
|
|
|
|
```go |
|
|
|
type BitArray struct { |
|
|
@ -109,36 +166,35 @@ type BitArray struct { |
|
|
|
} |
|
|
|
``` |
|
|
|
|
|
|
|
### Time |
|
|
|
This type is easily encoded directly by Amino. |
|
|
|
|
|
|
|
Time is encoded as an `int64` of the number of nanoseconds since January 1, 1970, |
|
|
|
rounded to the nearest millisecond. |
|
|
|
Note BitArray receives a special JSON encoding in the form of `x` and `_` |
|
|
|
representing `1` and `0`. Ie. the BitArray `10110` would be JSON encoded as |
|
|
|
`"x_xx_"` |
|
|
|
|
|
|
|
### Part |
|
|
|
|
|
|
|
Times before then are invalid. |
|
|
|
Part is used to break up blocks into pieces that can be gossiped in parallel |
|
|
|
and securely verified using a Merkle tree of the parts. |
|
|
|
|
|
|
|
Examples: |
|
|
|
Part contains the index of the part in the larger set (`Index`), the actual |
|
|
|
underlying data of the part (`Bytes`), and a simple Merkle proof that the part is contained in |
|
|
|
the larger set (`Proof`). |
|
|
|
|
|
|
|
```go |
|
|
|
encode(time.Time("Jan 1 00:00:00 UTC 1970")) == [0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00] |
|
|
|
encode(time.Time("Jan 1 00:00:01 UTC 1970")) == [0x00, 0x00, 0x00, 0x00, 0x3B, 0x9A, 0xCA, 0x00] // 1,000,000,000 ns |
|
|
|
encode(time.Time("Mon Jan 2 15:04:05 -0700 MST 2006")) == [0x0F, 0xC4, 0xBB, 0xC1, 0x53, 0x03, 0x12, 0x00] |
|
|
|
type Part struct { |
|
|
|
Index int |
|
|
|
Bytes byte[] |
|
|
|
Proof byte[] |
|
|
|
} |
|
|
|
``` |
|
|
|
|
|
|
|
### Structs |
|
|
|
|
|
|
|
An encoded struct is the concatenation of the encoding of its elements. |
|
|
|
There is no length-prefix. |
|
|
|
### MakeParts |
|
|
|
|
|
|
|
Examples: |
|
|
|
Encode an object using Amino and slice it into parts. |
|
|
|
|
|
|
|
```go |
|
|
|
type MyStruct struct{ |
|
|
|
A int |
|
|
|
B string |
|
|
|
C time.Time |
|
|
|
} |
|
|
|
encode(MyStruct{4, "hello", time.Time("Mon Jan 2 15:04:05 -0700 MST 2006")}) == |
|
|
|
[0x01, 0x04, 0x01, 0x05, 0x68, 0x65, 0x6C, 0x6C, 0x6F, 0x0F, 0xC4, 0xBB, 0xC1, 0x53, 0x03, 0x12, 0x00] |
|
|
|
func MakeParts(obj interface{}, partSize int) []Part |
|
|
|
``` |
|
|
|
|
|
|
|
## Merkle Trees |
|
|
@ -147,6 +203,8 @@ Simple Merkle trees are used in numerous places in Tendermint to compute a crypt |
|
|
|
|
|
|
|
RIPEMD160 is always used as the hashing function. |
|
|
|
|
|
|
|
### Simple Merkle Root |
|
|
|
|
|
|
|
The function `SimpleMerkleRoot` is a simple recursive function defined as follows: |
|
|
|
|
|
|
|
```go |
|
|
@ -159,20 +217,69 @@ func SimpleMerkleRoot(hashes [][]byte) []byte{ |
|
|
|
default: |
|
|
|
left := SimpleMerkleRoot(hashes[:(len(hashes)+1)/2]) |
|
|
|
right := SimpleMerkleRoot(hashes[(len(hashes)+1)/2:]) |
|
|
|
return RIPEMD160(append(left, right)) |
|
|
|
return SimpleConcatHash(left, right) |
|
|
|
} |
|
|
|
} |
|
|
|
|
|
|
|
func SimpleConcatHash(left, right []byte) []byte{ |
|
|
|
left = encodeByteSlice(left) |
|
|
|
right = encodeByteSlice(right) |
|
|
|
return RIPEMD160 (append(left, right)) |
|
|
|
} |
|
|
|
``` |
|
|
|
|
|
|
|
Note: we abuse notion and call `SimpleMerkleRoot` with arguments of type `struct` or type `[]struct`. |
|
|
|
Note that the leaves are Amino encoded as byte-arrays (ie. simple Uvarint length |
|
|
|
prefix) before being concatenated together and hashed. |
|
|
|
|
|
|
|
Note: we will abuse notion and invoke `SimpleMerkleRoot` with arguments of type `struct` or type `[]struct`. |
|
|
|
For `struct` arguments, we compute a `[][]byte` by sorting elements of the `struct` according to |
|
|
|
field name and then hashing them. |
|
|
|
For `[]struct` arguments, we compute a `[][]byte` by hashing the individual `struct` elements. |
|
|
|
|
|
|
|
## JSON (TMJSON) |
|
|
|
### Simple Merkle Proof |
|
|
|
|
|
|
|
Proof that a leaf is in a Merkle tree consists of a simple structure: |
|
|
|
|
|
|
|
|
|
|
|
``` |
|
|
|
type SimpleProof struct { |
|
|
|
Aunts [][]byte |
|
|
|
} |
|
|
|
``` |
|
|
|
|
|
|
|
Which is verified using the following: |
|
|
|
|
|
|
|
``` |
|
|
|
func (proof SimpleProof) Verify(index, total int, leafHash, rootHash []byte) bool { |
|
|
|
computedHash := computeHashFromAunts(index, total, leafHash, proof.Aunts) |
|
|
|
return computedHash == rootHash |
|
|
|
} |
|
|
|
|
|
|
|
func computeHashFromAunts(index, total int, leafHash []byte, innerHashes [][]byte) []byte{ |
|
|
|
assert(index < total && index >= 0 && total > 0) |
|
|
|
|
|
|
|
if total == 1{ |
|
|
|
assert(len(proof.Aunts) == 0) |
|
|
|
return leafHash |
|
|
|
} |
|
|
|
|
|
|
|
assert(len(innerHashes) > 0) |
|
|
|
|
|
|
|
numLeft := (total + 1) / 2 |
|
|
|
if index < numLeft { |
|
|
|
leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1]) |
|
|
|
assert(leftHash != nil) |
|
|
|
return SimpleHashFromTwoHashes(leftHash, innerHashes[len(innerHashes)-1]) |
|
|
|
} |
|
|
|
rightHash := computeHashFromAunts(index-numLeft, total-numLeft, leafHash, innerHashes[:len(innerHashes)-1]) |
|
|
|
assert(rightHash != nil) |
|
|
|
return SimpleHashFromTwoHashes(innerHashes[len(innerHashes)-1], rightHash) |
|
|
|
} |
|
|
|
``` |
|
|
|
|
|
|
|
## AminoJSON |
|
|
|
|
|
|
|
Signed messages (eg. votes, proposals) in the consensus are encoded in TMJSON, rather than TMBIN. |
|
|
|
TMJSON is JSON where `[]byte` are encoded as uppercase hex, rather than base64. |
|
|
|
Signed messages (eg. votes, proposals) in the consensus are encoded in AminoJSON, rather than binary Amino. |
|
|
|
|
|
|
|
When signing, the elements of a message are sorted by key and the sorted message is embedded in an |
|
|
|
outer JSON that includes a `chain_id` field. |
|
|
@ -185,22 +292,5 @@ like: |
|
|
|
|
|
|
|
Note how the fields within each level are sorted. |
|
|
|
|
|
|
|
## Other |
|
|
|
|
|
|
|
### MakeParts |
|
|
|
|
|
|
|
Encode an object using TMBIN and slice it into parts. |
|
|
|
|
|
|
|
```go |
|
|
|
MakeParts(object, partSize) |
|
|
|
``` |
|
|
|
|
|
|
|
### Part |
|
|
|
|
|
|
|
```go |
|
|
|
type Part struct { |
|
|
|
Index int |
|
|
|
Bytes byte[] |
|
|
|
Proof byte[] |
|
|
|
} |
|
|
|
``` |