Merge pull request #1056 from tendermint/feature/mempool-spec

WIP: Mempool specification

mempool/docs/README.md

@@ -0,0 +1,11 @@
+# Mempool Specification
+
+This package contains documents specifying the functionality
+of the mempool module.
+
+Components:
+
+* [Config](./config.md) - how to configure it
+* [External Messages](./messages.md) - The messages we accept over p2p and rpc interfaces
+* [Functionality](./functionality.md) - high-level description of the functionality it provides
+* [Concurrency Model](./concurrency.md) - What guarantees we provide, what locks we require.

mempool/docs/concurrency.md

@@ -0,0 +1,8 @@
+# Mempool Concurrency
+
+Look at the concurrency model this uses...
+
+* Receiving CheckTx
+* Broadcasting new tx
+* Interfaces with consensus engine, reap/update while checking
+* Calling the ABCI app (ordering. callbacks. how proxy works alongside the blockchain proxy which actually writes blocks)

mempool/docs/config.md

@@ -0,0 +1,59 @@
+# Mempool Configuration
+
+Here we describe configuration options around mempool.
+For the purposes of this document, they are described
+as command-line flags, but they can also be passed in as
+environmental variables or in the config.toml file. The
+following are all equivalent:
+
+Flag: `--mempool.recheck_empty=false`
+
+Environment: `TM_MEMPOOL_RECHECK_EMPTY=false`
+
+Config:
+```
+[mempool]
+recheck_empty = false
+```
+
+
+## Recheck
+
+`--mempool.recheck=false` (default: true)
+
+`--mempool.recheck_empty=false` (default: true)
+
+Recheck determines if the mempool rechecks all pending
+transactions after a block was committed. Once a block
+is committed, the mempool removes all valid transactions
+that were successfully included in the block.
+
+If `recheck` is true, then it will rerun CheckTx on
+all remaining transactions with the new block state.
+
+If the block contained no transactions, it will skip the
+recheck unless `recheck_empty` is true.
+
+## Broadcast
+
+`--mempool.broadcast=false` (default: true)
+
+Determines whether this node gossips any valid transactions
+that arrive in mempool. Default is to gossip anything that
+passes checktx. If this is disabled, transactions are not
+gossiped, but instead stored locally and added to the next
+block this node is the proposer.
+
+## WalDir
+
+`--mempool.wal_dir=/tmp/gaia/mempool.wal` (default: $TM_HOME/data/mempool.wal)
+
+This defines the directory where mempool writes the write-ahead
+logs. These files can be used to reload unbroadcasted
+transactions if the node crashes.
+
+If the directory passed in is an absolute path, the wal file is
+created there. If the directory is a relative path, the path is
+appended to home directory of the tendermint process to
+generate an absolute path to the wal directory
+(default `$HOME/.tendermint` or set via `TM_HOME` or `--home``)

mempool/docs/functionality.md

@@ -0,0 +1,37 @@
+# Mempool Functionality
+
+The mempool maintains a list of potentially valid transactions,
+both to broadcast to other nodes, as well as to provide to the
+consensus reactor when it is selected as the block proposer.
+
+There are two sides to the mempool state:
+
+* External: get, check, and broadcast new transactions
+* Internal: return valid transaction, update list after block commit
+
+
+## External functionality
+
+External functionality is exposed via network interfaces
+to potentially untrusted actors.
+
+* CheckTx - triggered via RPC or P2P
+* Broadcast - gossip messages after a successful check
+
+## Internal functionality
+
+Internal functionality is exposed via method calls to other
+code compiled into the tendermint binary.
+
+* Reap - get tx to propose in next block
+* Update - remove tx that were included in last block
+* ABCI.CheckTx - call ABCI app to validate the tx
+
+What does it provide the consensus reactor?
+What guarantees does it need from the ABCI app?
+(talk about interleaving processes in concurrency)
+
+## Optimizations
+
+Talk about the LRU cache to make sure we don't process any
+tx that we have seen before

mempool/docs/messages.md

@@ -0,0 +1,60 @@
+# Mempool Messages
+
+## P2P Messages
+
+There is currently only one message that Mempool broadcasts
+and receives over the p2p gossip network (via the reactor):
+`TxMessage`
+
+```go
+// TxMessage is a MempoolMessage containing a transaction.
+type TxMessage struct {
+    Tx types.Tx
+}
+```
+
+TxMessage is go-wire encoded and prepended with `0x1` as a
+"type byte". This is followed by a go-wire encoded byte-slice.
+Prefix of 40=0x28 byte tx is: `0x010128...` followed by
+the actual 40-byte tx. Prefix of 350=0x015e byte tx is:
+`0x0102015e...` followed by the actual 350 byte tx.
+
+(Please see the [go-wire repo](https://github.com/tendermint/go-wire#an-interface-example) for more information)
+
+## RPC Messages
+
+Mempool exposes `CheckTx([]byte)` over the RPC interface.
+
+It can be posted via `broadcast_commit`, `broadcast_sync` or
+`broadcast_async`. They all parse a message with one argument,
+`"tx": "HEX_ENCODED_BINARY"` and differ in only how long they
+wait before returning (sync makes sure CheckTx passes, commit
+makes sure it was included in a signed block).
+
+Request (`POST http://gaia.zone:46657/`):
+```json
+{
+ "id": "",
+ "jsonrpc": "2.0",
+ "method": "broadcast_sync",
+ "params": {
+    "tx": "F012A4BC68..."
+ }
+}
+```
+
+
+Response:
+```json
+{
+ "error": "",
+ "result": {
+     "hash": "E39AAB7A537ABAA237831742DCE1117F187C3C52",
+     "log": "",
+     "data": "",
+     "code": 0
+ },
+ "id": "",
+ "jsonrpc": "2.0"
+}
+```