From c5576dfa6900152179987cd332f9664c996443a6 Mon Sep 17 00:00:00 2001 From: Marko Date: Tue, 6 Oct 2020 12:40:25 +0200 Subject: [PATCH] spec: protobuf changes (#156) Co-authored-by: Anton Kaliaev --- spec/abci/abci.md | 28 +++++-- spec/abci/apps.md | 30 +++---- spec/abci/client-server.md | 2 +- spec/core/data_structures.md | 31 +++++--- spec/core/encoding.md | 147 +++++++++-------------------------- spec/core/state.md | 72 +++++++++-------- spec/p2p/connection.md | 2 +- spec/p2p/peer.md | 6 +- 8 files changed, 139 insertions(+), 179 deletions(-) diff --git a/spec/abci/abci.md b/spec/abci/abci.md index 21662fc5f..729cada0f 100644 --- a/spec/abci/abci.md +++ b/spec/abci/abci.md @@ -61,6 +61,23 @@ Each event has a `type` which is meant to categorize the event for a particular particular event. Every key and value in an event's attributes must be UTF-8 encoded strings along with the event type itself. +```protobuf +message Event { + string type = 1; + repeated EventAttribute attributes = 2; +} +``` + +The attributes of an `Event` consist of a `key`, `value` and a `index`. The index field notifies the indexer within Tendermint to index the event. This field is non-deterministic and will vary across different nodes in the network. + +```protobuf +message EventAttribute { + bytes key = 1; + bytes value = 2; + bool index = 3; // nondeterministic +} +``` + Example: ```go @@ -563,12 +580,7 @@ via light client. ### PubKey - **Fields**: - - `Type (string)`: Type of the public key. A simple string like `"ed25519"`. - In the future, may indicate a serialization algorithm to parse the `Data`, - for instance `"amino"`. - - `Data ([]byte)`: Public key data. For a simple public key, it's just the - raw bytes. If the `Type` indicates an encoding algorithm, this is the - encoded public key. + - `Sum (oneof PublicKey)`: This field is a Protobuf [`oneof`](https://developers.google.com/protocol-buffers/docs/proto#oneof) - **Usage**: - A generic and extensible typed public key @@ -626,8 +638,8 @@ via light client. ### ValidatorParams - **Fields**: - - `PubKeyTypes ([]string)`: List of accepted pubkey types. Uses same - naming as `PubKey.Type`. + - `PubKeyTypes ([]string)`: List of accepted public key types. + - Uses same naming as `PubKey.Type`. ### VersionParams diff --git a/spec/abci/apps.md b/spec/abci/apps.md index b594de4f9..68aa542a1 100644 --- a/spec/abci/apps.md +++ b/spec/abci/apps.md @@ -250,21 +250,21 @@ blockchain. Updates to the Tendermint validator set can be made by returning `ValidatorUpdate` objects in the `ResponseEndBlock`: -```proto +```protobuf message ValidatorUpdate { - PubKey pub_key + tendermint.crypto.keys.PublicKey pub_key int64 power } -message PubKey { - string type - bytes data -} +message PublicKey { + oneof { + ed25519 bytes = 1; + } ``` The `pub_key` currently supports only one type: -- `type = "ed25519"` and `data = ` +- `type = "ed25519"` The `power` is the new voting power for the validator, with the following rules: @@ -288,7 +288,7 @@ evidence. They can be set in InitChain and updated in EndBlock. ### BlockParams.MaxBytes -The maximum size of a complete Amino encoded block. +The maximum size of a complete Protobuf encoded block. This is enforced by Tendermint consensus. This implies a maximum tx size that is this MaxBytes, less the expected size of @@ -314,6 +314,8 @@ This is enforced by Tendermint consensus. Must have `TimeIotaMs > 0` to ensure time monotonicity. +> *Note: This is not exposed to the application* + ### EvidenceParams.MaxAgeDuration This is the maximum age of evidence in time units. @@ -383,7 +385,7 @@ local node's state - hence they may return stale reads. For reads that require consensus, use a transaction. The most important use of Query is to return Merkle proofs of the application state at some height -that can be used for efficient application-specific lite-clients. +that can be used for efficient application-specific light-clients. Note Tendermint has technically no requirements from the Query message for normal operation - that is, the ABCI app developer need not implement @@ -402,9 +404,9 @@ While some applications keep all relevant state in the transactions themselves (like Bitcoin and its UTXOs), others maintain a separated state that is computed deterministically *from* transactions, but is not contained directly in the transactions themselves (like Ethereum contracts and accounts). -For such applications, the `AppHash` provides a much more efficient way to verify lite-client proofs. +For such applications, the `AppHash` provides a much more efficient way to verify light-client proofs. -ABCI applications can take advantage of more efficient lite-client proofs for +ABCI applications can take advantage of more efficient light-client proofs for their state as follows: - return the Merkle root of the deterministic application state in @@ -413,15 +415,15 @@ their state as follows: - return efficient Merkle proofs about that application state in `ResponseQuery.Proof` that can be verified using the `AppHash` of the corresponding block. -For instance, this allows an application's lite-client to verify proofs of +For instance, this allows an application's light-client to verify proofs of absence in the application state, something which is much less efficient to do using the block hash. Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees, where the leaves of one tree are the root hashes of others. To support this, and the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure: -```proto -message Proof { +```protobuf +message ProofOps { repeated ProofOp ops } diff --git a/spec/abci/client-server.md b/spec/abci/client-server.md index 967199455..865957267 100644 --- a/spec/abci/client-server.md +++ b/spec/abci/client-server.md @@ -41,7 +41,7 @@ See examples, in various stages of maintenance, in ### In Process The simplest implementation uses function calls within Golang. -This means ABCI applications written in Golang can be compiled with TendermintCore and run as a single binary. +This means ABCI applications written in Golang can be compiled with Tendermint Core and run as a single binary. ### GRPC diff --git a/spec/core/data_structures.md b/spec/core/data_structures.md index 51061867e..fc535954c 100644 --- a/spec/core/data_structures.md +++ b/spec/core/data_structures.md @@ -89,11 +89,11 @@ parts (ie. `len(MakeParts(block))`) ```go type BlockID struct { Hash []byte - PartsHeader PartSetHeader + PartSetHeader PartSetHeader } type PartSetHeader struct { - Total int32 + Total uint32 Hash []byte } ``` @@ -125,7 +125,7 @@ validator. It also contains the relevant BlockID, height and round: ```go type Commit struct { Height int64 - Round int + Round int32 BlockID BlockID Signatures []CommitSig } @@ -168,17 +168,28 @@ The vote includes information about the validator signing it. ```go type Vote struct { - Type byte + Type SignedMsgType Height int64 - Round int + Round int32 BlockID BlockID Timestamp Time ValidatorAddress []byte - ValidatorIndex int + ValidatorIndex int32 Signature []byte } ``` +```protobuf +enum SignedMsgType { + SIGNED_MSG_TYPE_UNKNOWN = 0; + // Votes + PREVOTE_TYPE = 1; + PRECOMMIT_TYPE = 2; + // Proposals + PROPOSAL_TYPE = 32; +} +``` + There are two types of votes: a _prevote_ has `vote.Type == 1` and a _precommit_ has `vote.Type == 2`. @@ -412,7 +423,7 @@ first by voting power (descending), then by address (ascending). block.ConsensusHash == state.ConsensusParams.Hash() ``` -Hash of the amino-encoding of a subset of the consensus parameters. +Hash of the protobuf-encoding of a subset of the consensus parameters. ### AppHash @@ -497,9 +508,9 @@ The number of votes in a commit is limited to 10000 (see `types.MaxVotesCount`). ### Vote A vote is a signed message broadcast in the consensus for a particular block at a particular height and round. -When stored in the blockchain or propagated over the network, votes are encoded in Amino. -For signing, votes are represented via `CanonicalVote` and also encoded using amino (protobuf compatible) via -`Vote.SignBytes` which includes the `ChainID`, and uses a different ordering of +When stored in the blockchain or propagated over the network, votes are encoded in Protobuf. +For signing, votes are represented via `CanonicalVote` and also encoded using Protobuf via +`VoteSignBytes` which includes the `ChainID`, and uses a different ordering of the fields. We define a method `Verify` that returns `true` if the signature verifies against the pubkey for the `SignBytes` diff --git a/spec/core/encoding.md b/spec/core/encoding.md index ae6e4e42b..7015f3e34 100644 --- a/spec/core/encoding.md +++ b/spec/core/encoding.md @@ -1,22 +1,10 @@ # Encoding -## Amino +## Protocol Buffers -Tendermint uses the proto3 derivative [Amino](https://github.com/tendermint/go-amino) for all data structures. -Think of Amino as an object-oriented proto3 with native JSON support. -The goal of the Amino encoding protocol is to bring parity between application -logic objects and persistence objects. +Tendermint uses [Protocol Buffers](https://developers.google.com/protocol-buffers), specifically proto3, for all data structures. -Please see the [Amino -specification](https://github.com/tendermint/go-amino#amino-encoding-for-go) for -more details. - -Notably, every object that satisfies an interface (eg. a particular kind of p2p message, -or a particular kind of pubkey) is registered with a global name, the hash of -which is included in the object's encoding as the so-called "prefix bytes". - -We define the `func AminoEncode(obj interface{}) []byte` function to take an -arbitrary object and return the Amino encoded bytes. +Please see the [Proto3 language guide](https://developers.google.com/protocol-buffers/docs/proto3) for more details. ## Byte Arrays @@ -34,47 +22,18 @@ be encoded as `0xAC020A0B...` where `0xAC02` is the UVarint encoding of 300. Tendermint uses `SHA256` as its hash function. Objects are always Amino encoded before being hashed. -So `SHA256(obj)` is short for `SHA256(AminoEncode(obj))`. +So `SHA256(obj)` is short for `SHA256(ProtoEncoding(obj))`. ## Public Key Cryptography -Tendermint uses Amino to distinguish between different types of private keys, -public keys, and signatures. Additionally, for each public key, Tendermint +Tendermint uses Protobuf [Oneof](https://developers.google.com/protocol-buffers/docs/proto3#oneof) +to distinguish between different types public keys, and signatures. +Additionally, for each public key, Tendermint defines an Address function that can be used as a more compact identifier in place of the public key. Here we list the concrete types, their names, and prefix bytes for public keys and signatures, as well as the address schemes for each PubKey. Note for brevity we don't -include details of the private keys beyond their type and name, as they can be -derived the same way as the others using Amino. - -All registered objects are encoded by Amino using a 4-byte PrefixBytes that -uniquely identifies the object and includes information about its underlying -type. For details on how PrefixBytes are computed, see the [Amino -spec](https://github.com/tendermint/go-amino#computing-the-prefix-and-disambiguation-bytes). - -In what follows, we provide the type names and prefix bytes directly. -Notice that when encoding byte-arrays, the length of the byte-array is appended -to the PrefixBytes. Thus the encoding of a byte array becomes ` `. In other words, to encode any type listed below you do not need to be -familiar with amino encoding. -You can simply use below table and concatenate Prefix || Length (of raw bytes) || raw bytes -( while || stands for byte concatenation here). - -| Type | Name | Prefix | Length | Notes | -| ----------------------- | ---------------------------------- | ---------- | -------- | ----- | -| PubKeyEd25519 | tendermint/PubKeyEd25519 | 0x1624DE64 | 0x20 | | -| PubKeySr25519 | tendermint/PubKeySr25519 | 0x0DFB1005 | 0x20 | | -| PubKeySecp256k1 | tendermint/PubKeySecp256k1 | 0xEB5AE987 | 0x21 | | -| PrivKeyEd25519 | tendermint/PrivKeyEd25519 | 0xA3288910 | 0x40 | | -| PrivKeySr25519 | tendermint/PrivKeySr25519 | 0x2F82D78B | 0x20 | | -| PrivKeySecp256k1 | tendermint/PrivKeySecp256k1 | 0xE1B0F79B | 0x20 | | -| PubKeyMultisigThreshold | tendermint/PubKeyMultisigThreshold | 0x22C1F7E2 | variable | | - -### Example - -For example, the 33-byte (or 0x21-byte in hex) Secp256k1 pubkey -`020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9` -would be encoded as -`EB5AE98721020BD40F225A57ED383B440CF073BC5539D0341F5767D2BF2D78406D00475A2EE9` +include details of the private keys beyond their type and name. ### Key Types @@ -92,39 +51,6 @@ address = SHA256(pubkey)[:20] The signature is the raw 64-byte ED25519 signature. -#### Sr25519 - -TODO: pubkey - -The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key: - -```go -address = SHA256(pubkey)[:20] -``` - -The signature is the raw 64-byte ED25519 signature. - -#### Secp256k1 - -TODO: pubkey - -The address is the RIPEMD160 hash of the SHA256 hash of the OpenSSL compressed public key: - -```go -address = RIPEMD160(SHA256(pubkey)) -``` - -This is the same as Bitcoin. - -The signature is the 64-byte concatenation of ECDSA `r` and `s` (ie. `r || s`), -where `s` is lexicographically less than its inverse, to prevent malleability. -This is like Ethereum, but without the extra byte for pubkey recovery, since -Tendermint assumes the pubkey is always provided anyway. - -#### Multisig - -TODO - ## Other Common Types ### BitArray @@ -136,7 +62,7 @@ encoded in base64 (`Elems`). ```go type BitArray struct { - Bits int + Bits int64 Elems []uint64 } ``` @@ -158,7 +84,7 @@ the set (`Proof`). ```go type Part struct { - Index int + Index uint32 Bytes []byte Proof SimpleProof } @@ -168,7 +94,7 @@ See details of SimpleProof, below. ### MakeParts -Encode an object using Amino and slice it into parts. +Encode an object using Protobuf and slice it into parts. Tendermint uses a part size of 65536 bytes, and allows a maximum of 1601 parts (see `types.MaxBlockPartsCount`). This corresponds to the hard-coded block size limit of 100MB. @@ -260,27 +186,27 @@ func Hashes(items [][]byte) [][]byte { ``` Note: we will abuse notion and invoke `MerkleRoot` with arguments of type `struct` or type `[]struct`. -For `struct` arguments, we compute a `[][]byte` containing the amino encoding of each +For `struct` arguments, we compute a `[][]byte` containing the protobuf encoding of each field in the struct, in the same order the fields appear in the struct. -For `[]struct` arguments, we compute a `[][]byte` by amino encoding the individual `struct` elements. +For `[]struct` arguments, we compute a `[][]byte` by protobuf encoding the individual `struct` elements. -### Simple Merkle Proof +### Merkle Proof Proof that a leaf is in a Merkle tree is composed as follows: ```golang -type SimpleProof struct { - Total int - Index int - LeafHash []byte - Aunts [][]byte +type Proof struct { + Total int + Index int + LeafHash []byte + Aunts [][]byte } ``` Which is verified as follows: ```golang -func (proof SimpleProof) Verify(rootHash []byte, leaf []byte) bool { +func (proof Proof) Verify(rootHash []byte, leaf []byte) bool { assert(proof.LeafHash, leafHash(leaf) computedHash := computeHashFromAunts(proof.Index, proof.Total, proof.LeafHash, proof.Aunts) @@ -319,13 +245,13 @@ Because Tendermint only uses a Simple Merkle Tree, application developers are ex ## JSON -### Amino +Tendermint has its own JSON encoding in order to keep backwards compatibility with the prvious RPC layer. -Amino also supports JSON encoding - registered types are simply encoded as: +Registered types are encoded as: ```json { - "type": "", + "type": "", "value": } ``` @@ -340,30 +266,33 @@ For instance, an ED25519 PubKey would look like: ``` Where the `"value"` is the base64 encoding of the raw pubkey bytes, and the -`"type"` is the amino name for Ed25519 pubkeys. +`"type"` is the type name for Ed25519 pubkeys. ### Signed Messages -Signed messages (eg. votes, proposals) in the consensus are encoded using Amino. +Signed messages (eg. votes, proposals) in the consensus are encoded using protobuf. When signing, the elements of a message are re-ordered so the fixed-length fields are first, making it easy to quickly check the type, height, and round. The `ChainID` is also appended to the end. -We call this encoding the SignBytes. For instance, SignBytes for a vote is the Amino encoding of the following struct: - -```go -type CanonicalVote struct { - Type byte - Height int64 `binary:"fixed64"` - Round int64 `binary:"fixed64"` - BlockID CanonicalBlockID - Timestamp time.Time - ChainID string +We call this encoding the SignBytes. For instance, SignBytes for a vote is the protobuf encoding of the following struct: + +```protobuf +message CanonicalVote { + SignedMsgType type = 1; + sfixed64 height = 2; // canonicalization requires fixed size encoding here + sfixed64 round = 3; // canonicalization requires fixed size encoding here + CanonicalBlockID block_id = 4; + google.protobuf.Timestamp timestamp = 5; + string chain_id = 6; } ``` The field ordering and the fixed sized encoding for the first three fields is optimized to ease parsing of SignBytes in HSMs. It creates fixed offsets for relevant fields that need to be read in this context. + +> Note: All canonical messages are length prefixed. + For more details, see the [signing spec](../consensus/signing.md). Also, see the motivating discussion in [#1622](https://github.com/tendermint/tendermint/issues/1622). diff --git a/spec/core/state.md b/spec/core/state.md index 6b12ba4a9..eb020e489 100644 --- a/spec/core/state.md +++ b/spec/core/state.md @@ -7,7 +7,7 @@ necessary for validating new blocks. For instance, the validators set and the re transactions are never included in blocks, but their Merkle roots are - the state keeps track of them. Note that the `State` object itself is an implementation detail, since it is never -included in a block or gossipped over the network, and we never compute +included in a block or gossiped over the network, and we never compute its hash. Thus we do not include here details of how the `State` object is persisted or queried. That said, the types it contains are part of the specification, since their Merkle roots are included in blocks and their values are used in @@ -55,21 +55,24 @@ type Consensus struct { } ``` -### Result - -```go -type Result struct { - Code uint32 - Data []byte +### ResponseDeliverTx + +```protobuf +message ResponseDeliverTx { + uint32 code = 1; + bytes data = 2; + string log = 3; // nondeterministic + string info = 4; // nondeterministic + int64 gas_wanted = 5; + int64 gas_used = 6; + repeated Event events = 7 + [(gogoproto.nullable) = false, (gogoproto.jsontag) = "events,omitempty"]; + string codespace = 8; } ``` -`Result` is the result of executing a transaction against the application. -It returns a result code and an arbitrary byte array (ie. a return value). - -NOTE: the Result needs to be updated to include more fields returned from -processing transactions, like gas variables and events - see -[issue 1007](https://github.com/tendermint/tendermint/issues/1007). +`ResponseDeliverTx` is the result of executing a transaction against the application. +It returns a result code (`uint32`), an arbitrary byte array (`[]byte`) (ie. a return value), Log (`string`), Info (`string`), GasWanted (`int64`), GasUsed (`int64`), Events (`[]Events`) and a Codespace (`string`). ### Validator @@ -110,45 +113,48 @@ Like validator sets, they are set during genesis and can be updated by the appli When hashed, only a subset of the params are included, to allow the params to evolve without breaking the header. -```go -type ConsensusParams struct { - Block - Evidence - Validator - Version +```protobuf +message ConsensusParams { + BlockParams block = 1; + EvidenceParams evidence = 2; + ValidatorParams validator = 3; + VersionParams version = 4; } +``` +```go type hashedParams struct { BlockMaxBytes int64 BlockMaxGas int64 } -func (params ConsensusParams) Hash() []byte { +func HashConsensusParams() []byte { SHA256(hashedParams{ BlockMaxBytes: params.Block.MaxBytes, BlockMaxGas: params.Block.MaxGas, }) } +``` -type BlockParams struct { - MaxBytes int64 - MaxGas int64 - TimeIotaMs int64 +```protobuf +message BlockParams { + int64 max_bytes = 1; + int64 max_gas = 2; + int64 time_iota_ms = 3; // not exposed to the application } -type EvidenceParams struct { - MaxAgeNumBlocks int64 - MaxAgeDuration time.Duration - MaxNum uint32 - ProofTrialPeriod int64 +message EvidenceParams { + int64 max_age_num_blocks = 1; + google.protobuf.Duration max_age_duration = 2; + uint32 max_num = 3; } -type ValidatorParams struct { - PubKeyTypes []string +message ValidatorParams { + repeated string pub_key_types = 1; } -type VersionParams struct { - AppVersion uint64 +message VersionParams { + uint64 AppVersion = 1; } ``` diff --git a/spec/p2p/connection.md b/spec/p2p/connection.md index 6babff589..33178f479 100644 --- a/spec/p2p/connection.md +++ b/spec/p2p/connection.md @@ -38,7 +38,7 @@ type msgPacket struct { } ``` -The `msgPacket` is serialized using [go-amino](https://github.com/tendermint/go-amino) and prefixed with 0x3. +The `msgPacket` is serialized using [Proto3](https://developers.google.com/protocol-buffers/docs/proto3). The received `Bytes` of a sequential set of packets are appended together until a packet with `EOF=1` is received, then the complete serialized message is returned for processing by the `onReceive` function of the corresponding channel. diff --git a/spec/p2p/peer.md b/spec/p2p/peer.md index ca791d799..a81b2a7bd 100644 --- a/spec/p2p/peer.md +++ b/spec/p2p/peer.md @@ -21,7 +21,7 @@ corresponding to ``. This prevents man-in-the-middle attacks on the peer lay All p2p connections use TCP. Upon establishing a successful TCP connection with a peer, -two handhsakes are performed: one for authenticated encryption, and one for Tendermint versioning. +two handshakes are performed: one for authenticated encryption, and one for Tendermint versioning. Both handshakes have configurable timeouts (they should complete quickly). ### Authenticated Encryption Handshake @@ -29,8 +29,8 @@ Both handshakes have configurable timeouts (they should complete quickly). Tendermint implements the Station-to-Station protocol using X25519 keys for Diffie-Helman key-exchange and chacha20poly1305 for encryption. -Previous versions of this protocol suffered from malleability attacks whereas an active man -in the middle attacker could compromise confidentiality as decribed in [Prime, Order Please! +Previous versions of this protocol (0.32 and below) suffered from malleability attacks whereas an active man +in the middle attacker could compromise confidentiality as described in [Prime, Order Please! Revisiting Small Subgroup and Invalid Curve Attacks on Protocols using Diffie-Hellman](https://eprint.iacr.org/2019/526.pdf).