Browse Source

Merge branch 'develop' into release/v0.25.0

pull/2467/head
Ethan Buchman 6 years ago
parent
commit
b394bd5b5c
10 changed files with 423 additions and 77 deletions
  1. +4
    -1
      Makefile
  2. +7
    -2
      docs/introduction/install.md
  3. +42
    -8
      docs/spec/abci/abci.md
  4. +112
    -14
      docs/spec/abci/apps.md
  5. +59
    -0
      docs/tendermint-core/running-in-production.md
  6. +69
    -14
      mempool/mempool.go
  7. +60
    -3
      mempool/mempool_test.go
  8. +18
    -10
      node/node.go
  9. +31
    -12
      state/execution.go
  10. +21
    -13
      state/services.go

+ 4
- 1
Makefile View File

@ -23,11 +23,14 @@ check: check_tools get_vendor_deps
build: build:
CGO_ENABLED=0 go build $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o build/tendermint ./cmd/tendermint/ CGO_ENABLED=0 go build $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o build/tendermint ./cmd/tendermint/
build_c:
CGO_ENABLED=1 go build $(BUILD_FLAGS) -tags "$(BUILD_TAGS) gcc" -o build/tendermint ./cmd/tendermint/
build_race: build_race:
CGO_ENABLED=0 go build -race $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o build/tendermint ./cmd/tendermint CGO_ENABLED=0 go build -race $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o build/tendermint ./cmd/tendermint
install: install:
CGO_ENABLED=0 go install $(BUILD_FLAGS) -tags $(BUILD_TAGS) ./cmd/tendermint
CGO_ENABLED=0 go install $(BUILD_FLAGS) -tags $(BUILD_TAGS) ./cmd/tendermint
######################################## ########################################
### Protobuf ### Protobuf


+ 7
- 2
docs/introduction/install.md View File

@ -77,8 +77,13 @@ make install
## Compile with CLevelDB support ## Compile with CLevelDB support
Install [LevelDB](https://github.com/google/leveldb) (minimum version is 1.7)
with snappy. Example for Ubuntu:
Install [LevelDB](https://github.com/google/leveldb) (minimum version is 1.7).
Build Tendermint with C libraries: `make build_c`.
### Ubuntu
Install LevelDB with snappy:
``` ```
sudo apt-get update sudo apt-get update


+ 42
- 8
docs/spec/abci/abci.md View File

@ -48,16 +48,50 @@ Keys and values in tags must be UTF-8 encoded strings (e.g.
## Determinism ## Determinism
Some methods (`SetOption, Query, CheckTx, DeliverTx`) return
non-deterministic data in the form of `Info` and `Log`. The `Log` is
intended for the literal output from the application's logger, while the
`Info` is any additional info that should be returned.
All other fields in the `Response*` of all methods must be strictly deterministic.
ABCI applications must implement deterministic finite-state machines to be
securely replicated by the Tendermint consensus. This means block execution
over the Consensus Connection must be strictly deterministic: given the same
ordered set of requests, all nodes will compute identical responses, for all
BeginBlock, DeliverTx, EndBlock, and Commit. This is critical, because the
responses are included in the header of the next block, either via a Merkle root
or directly, so all nodes must agree on exactly what they are.
For this reason, it is recommended that applications not be exposed to any For this reason, it is recommended that applications not be exposed to any
external user or process except via the ABCI connections to a consensus engine external user or process except via the ABCI connections to a consensus engine
like Tendermint Core.
like Tendermint Core. The application must only change its state based on input
from block execution (BeginBlock, DeliverTx, EndBlock, Commit), and not through
any other kind of request. This is the only way to ensure all nodes see the same
transactions and compute the same results.
If there is some non-determinism in the state machine, consensus will eventually
fail as nodes disagree over the correct values for the block header. The
non-determinism must be fixed and the nodes restarted.
Sources of non-determinism in applications may include:
- Hardware failures
- Cosmic rays, overheating, etc.
- Node-dependent state
- Random numbers
- Time
- Underspecification
- Library version changes
- Race conditions
- Floating point numbers
- JSON serialization
- Iterating through hash-tables/maps/dictionaries
- External Sources
- Filesystem
- Network calls (eg. some external REST API service)
See [#56](https://github.com/tendermint/abci/issues/56) for original discussion.
Note that some methods (`SetOption, Query, CheckTx, DeliverTx`) return
explicitly non-deterministic data in the form of `Info` and `Log` fields. The `Log` is
intended for the literal output from the application's logger, while the
`Info` is any additional info that should be returned. These are the only fields
that are not included in block header computations, so we don't need agreement
on them. All other fields in the `Response*` must be strictly deterministic.
## Block Execution ## Block Execution
@ -217,7 +251,7 @@ Commit are included in the header of the next block.
be non-deterministic. be non-deterministic.
- `Info (string)`: Additional information. May - `Info (string)`: Additional information. May
be non-deterministic. be non-deterministic.
- `GasWanted (int64)`: Amount of gas request for transaction.
- `GasWanted (int64)`: Amount of gas requested for transaction.
- `GasUsed (int64)`: Amount of gas consumed by transaction. - `GasUsed (int64)`: Amount of gas consumed by transaction.
- `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing - `Tags ([]cmn.KVPair)`: Key-Value tags for filtering and indexing
transactions (eg. by account). transactions (eg. by account).


+ 112
- 14
docs/spec/abci/apps.md View File

@ -86,18 +86,50 @@ Otherwise it should never be modified.
## Transaction Results ## Transaction Results
`ResponseCheckTx` and `ResponseDeliverTx` contain the same fields, though they
have slightly different effects.
`ResponseCheckTx` and `ResponseDeliverTx` contain the same fields.
In both cases, `Info` and `Log` are non-deterministic values for debugging/convenience purposes
The `Info` and `Log` fields are non-deterministic values for debugging/convenience purposes
that are otherwise ignored. that are otherwise ignored.
In both cases, `GasWanted` and `GasUsed` parameters are currently ignored,
though see issues
[#1861](https://github.com/tendermint/tendermint/issues/1861),
[#2299](https://github.com/tendermint/tendermint/issues/2299) and
[#2310](https://github.com/tendermint/tendermint/issues/2310) for how this may
soon change.
The `Data` field must be strictly deterministic, but can be arbitrary data.
### Gas
Ethereum introduced the notion of `gas` as an absract representation of the
cost of resources used by nodes when processing transactions. Every operation in the
Ethereum Virtual Machine uses some amount of gas, and gas can be accepted at a market-variable price.
Users propose a maximum amount of gas for their transaction; if the tx uses less, they get
the difference credited back. Tendermint adopts a similar abstraction,
though uses it only optionally and weakly, allowing applications to define
their own sense of the cost of execution.
In Tendermint, the `ConsensusParams.BlockSize.MaxGas` limits the amount of `gas` that can be used in a block.
The default value is `-1`, meaning no limit, or that the concept of gas is
meaningless.
Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum
amount of gas the sender of a tx is willing to use, and the later is how much it actually
used. Applications should enforce that `GasUsed <= GasWanted` - ie. tx execution
should halt before it can use more resources than it requested.
When `MaxGas > -1`, Tendermint enforces the following rules:
- `GasWanted <= MaxGas` for all txs in the mempool
- `(sum of GasWanted in a block) <= MaxGas` when proposing a block
If `MaxGas == -1`, no rules about gas are enforced.
Note that Tendermint does not currently enforce anything about Gas in the consensus, only the mempool.
This means it does not guarantee that committed blocks satisfy these rules!
It is the application's responsibility to return non-zero response codes when gas limits are exceeded.
The `GasUsed` field is ignored compltely by Tendermint. That said, applications should enforce:
- `GasUsed <= GasWanted` for any given transaction
- `(sum of GasUsed in a block) <= MaxGas` for every block
In the future, we intend to add a `Priority` field to the responses that can be
used to explicitly prioritize txs in the mempool for inclusion in a block
proposal. See [#1861](https://github.com/tendermint/tendermint/issues/1861).
### CheckTx ### CheckTx
@ -142,9 +174,6 @@ If the list is not empty, Tendermint will use it for the validator set.
This way the application can determine the initial validator set for the This way the application can determine the initial validator set for the
blockchain. blockchain.
ResponseInitChain also includes ConsensusParams, but these are presently
ignored.
### EndBlock ### EndBlock
Updates to the Tendermint validator set can be made by returning Updates to the Tendermint validator set can be made by returning
@ -179,14 +208,74 @@ following rules:
Note the updates returned in block `H` will only take effect at block `H+2`. Note the updates returned in block `H` will only take effect at block `H+2`.
## Consensus Parameters
ConsensusParams enforce certain limits in the blockchain, like the maximum size
of blocks, amount of gas used in a block, and the maximum acceptable age of
evidence. They can be set in InitChain and updated in EndBlock.
### BlockSize.MaxBytes
The maximum size of a complete Amino encoded block.
This is enforced by Tendermint consensus.
This implies a maximum tx size that is this MaxBytes, less the expected size of
the header, the validator set, and any included evidence in the block.
Must have `0 < MaxBytes < 100 MB`.
### BlockSize.MaxGas
The maximum of the sum of `GasWanted` in a proposed block.
This is *not* enforced by Tendermint consensus.
It is left to the app to enforce (ie. if txs are included past the
limit, they should return non-zero codes). It is used by Tendermint to limit the
txs included in a proposed block.
Must have `MaxGas >= -1`.
If `MaxGas == -1`, no limit is enforced.
### EvidenceParams.MaxAge
This is the maximum age of evidence.
This is enforced by Tendermint consensus.
If a block includes evidence older than this, the block will be rejected
(validators won't vote for it).
Must have `0 < MaxAge`.
### Updates
The application may set the consensus params during InitChain, and update them during
EndBlock.
#### InitChain
ResponseInitChain includes a ConsensusParams.
If its nil, Tendermint will use the params loaded in the genesis
file. If it's not nil, Tendermint will use it.
This way the application can determine the initial consensus params for the
blockchain.
#### EndBlock
ResponseEndBlock includes a ConsensusParams.
If its nil, Tendermint will do nothing.
If it's not nil, Tendermint will use it.
This way the application can update the consensus params over time.
Note the updates returned in block `H` will take effect right away for block
`H+1`.
## Query ## Query
Query is a generic method with lots of flexibility to enable diverse sets Query is a generic method with lots of flexibility to enable diverse sets
of queries on application state. Tendermint makes use of Query to filter new peers of queries on application state. Tendermint makes use of Query to filter new peers
based on ID and IP, and exposes Query to the user over RPC. based on ID and IP, and exposes Query to the user over RPC.
Note that calls to Query are not replicated across nodes, but rather query the Note that calls to Query are not replicated across nodes, but rather query the
local node's state - hence they may provide stale reads. For reads that require
consensus, a transaction is required.
local node's state - hence they may return stale reads. For reads that require
consensus, use a transaction.
The most important use of Query is to return Merkle proofs of the application state at some height The most important use of Query is to return Merkle proofs of the application state at some height
that can be used for efficient application-specific lite-clients. that can be used for efficient application-specific lite-clients.
@ -235,6 +324,15 @@ using the following paths, with no additional data:
If either of these queries return a non-zero ABCI code, Tendermint will refuse If either of these queries return a non-zero ABCI code, Tendermint will refuse
to connect to the peer. to connect to the peer.
### Paths
Queries are directed at paths, and may optionally include additional data.
The expectation is for there to be some number of high level paths
differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently,
Tendermint only uses `/p2p`, for filtering peers. For more advanced use, see the
implementation of
[Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/v0.23.1/baseapp/baseapp.go#L333).
## Crash Recovery ## Crash Recovery


+ 59
- 0
docs/tendermint-core/running-in-production.md View File

@ -1,5 +1,41 @@
# Running in production # Running in production
## Database
By default, Tendermint uses the `syndtr/goleveldb` package for it's in-process
key-value database. Unfortunately, this implementation of LevelDB seems to suffer under heavy load (see
[#226](https://github.com/syndtr/goleveldb/issues/226)). It may be best to
install the real C-implementaiton of LevelDB and compile Tendermint to use
that using `make build_c`. See the [install instructions](../introduction/install) for details.
Tendermint keeps multiple distinct LevelDB databases in the `$TMROOT/data`:
- `blockstore.db`: Keeps the entire blockchain - stores blocks,
block commits, and block meta data, each indexed by height. Used to sync new
peers.
- `evidence.db`: Stores all verified evidence of misbehaviour.
- `state.db`: Stores the current blockchain state (ie. height, validators,
consensus params). Only grows if consensus params or validators change. Also
used to temporarily store intermediate results during block processing.
- `tx_index.db`: Indexes txs (and their results) by tx hash and by DeliverTx result tags.
By default, Tendermint will only index txs by their hash, not by their DeliverTx
result tags. See [indexing transactions](../app-dev/indexing-transactions) for
details.
There is no current strategy for pruning the databases. Consider reducing
block production by [controlling empty blocks](../tendermint-core/using-tendermint#No-Empty-Blocks)
or by increasing the `consensus.timeout_commit` param. Note both of these are
local settings and not enforced by the consensus.
We're working on [state
syncing](https://github.com/tendermint/tendermint/issues/828),
which will enable history to be thrown away
and recent application state to be directly synced. We'll need to develop solutions
for archival nodes that allow queries on historical transactions and states.
The Cosmos project has had much success just dumping the latest state of a
blockchain to disk and starting a new chain from that state.
## Logging ## Logging
Default logging level (`main:info,state:info,*:`) should suffice for Default logging level (`main:info,state:info,*:`) should suffice for
@ -11,6 +47,29 @@ you're trying to debug Tendermint or asked to provide logs with debug
logging level, you can do so by running tendermint with logging level, you can do so by running tendermint with
`--log_level="*:debug"`. `--log_level="*:debug"`.
## Write Ahead Logs (WAL)
Tendermint uses write ahead logs for the consensus (`cs.wal`) and the mempool
(`mempool.wal`). Both WALs have a max size of 1GB and are automatically rotated..
The `consensus.wal` is used to ensure we can recover from a crash at any point
in the consensus state machine.
It writes all consensus messages (timeouts, proposals, block part, or vote)
to a single file, flushing to disk before processing messages from its own
validator. Since Tendermint validators are expected to never sign a conflicting vote, the
WAL ensures we can always recover deterministically to the latest state of the consensus without
using the network or re-signing any consensus messages.
If your `consensus.wal` is corrupted, see [below](#WAL-Corruption).
The `mempool.wal` logs all incoming txs before running CheckTx, but is
otherwise not used in any programmatic way. It's just a kind of manual
safe guard. Note the mempool provides no durability guarantees - a tx sent to one or many nodes
may never make it into the blockchain if those nodes crash before being able to
propose it. Clients must monitor their txs by subscribing over websockets,
polling for them, or using `/broadcast_tx_commit`. In the worst case, txs can be
resent from the mempool WAL manually.
## DOS Exposure and Mitigation ## DOS Exposure and Mitigation
Validators are supposed to setup [Sentry Node Validators are supposed to setup [Sentry Node


+ 69
- 14
mempool/mempool.go View File

@ -22,6 +22,16 @@ import (
"github.com/tendermint/tendermint/types" "github.com/tendermint/tendermint/types"
) )
// PreCheckFunc is an optional filter executed before CheckTx and rejects
// transaction if false is returned. An example would be to ensure that a
// transaction doesn't exceeded the block size.
type PreCheckFunc func(types.Tx) bool
// PostCheckFunc is an optional filter executed after CheckTx and rejects
// transaction if false is returned. An example would be to ensure a
// transaction doesn't require more gas than available for the block.
type PostCheckFunc func(types.Tx, *abci.ResponseCheckTx) bool
/* /*
The mempool pushes new txs onto the proxyAppConn. The mempool pushes new txs onto the proxyAppConn.
@ -58,6 +68,27 @@ var (
ErrMempoolIsFull = errors.New("Mempool is full") ErrMempoolIsFull = errors.New("Mempool is full")
) )
// PreCheckAminoMaxBytes checks that the size of the transaction plus the amino
// overhead is smaller or equal to the expected maxBytes.
func PreCheckAminoMaxBytes(maxBytes int64) PreCheckFunc {
return func(tx types.Tx) bool {
// We have to account for the amino overhead in the tx size as well
aminoOverhead := amino.UvarintSize(uint64(len(tx)))
return int64(len(tx)+aminoOverhead) <= maxBytes
}
}
// PostCheckMaxGas checks that the wanted gas is smaller or equal to the passed
// maxGas. Returns true if maxGas is -1.
func PostCheckMaxGas(maxGas int64) PostCheckFunc {
return func(tx types.Tx, res *abci.ResponseCheckTx) bool {
if maxGas == -1 {
return true
}
return res.GasWanted <= maxGas
}
}
// TxID is the hex encoded hash of the bytes as a types.Tx. // TxID is the hex encoded hash of the bytes as a types.Tx.
func TxID(tx []byte) string { func TxID(tx []byte) string {
return fmt.Sprintf("%X", types.Tx(tx).Hash()) return fmt.Sprintf("%X", types.Tx(tx).Hash())
@ -80,8 +111,8 @@ type Mempool struct {
recheckEnd *clist.CElement // re-checking stops here recheckEnd *clist.CElement // re-checking stops here
notifiedTxsAvailable bool notifiedTxsAvailable bool
txsAvailable chan struct{} // fires once for each height, when the mempool is not empty txsAvailable chan struct{} // fires once for each height, when the mempool is not empty
// Filter mempool to only accept txs for which filter(tx) returns true.
filter func(types.Tx) bool
preCheck PreCheckFunc
postCheck PostCheckFunc
// Keep a cache of already-seen txs. // Keep a cache of already-seen txs.
// This reduces the pressure on the proxyApp. // This reduces the pressure on the proxyApp.
@ -141,10 +172,16 @@ func (mem *Mempool) SetLogger(l log.Logger) {
mem.logger = l mem.logger = l
} }
// WithFilter sets a filter for mempool to only accept txs for which f(tx)
// returns true.
func WithFilter(f func(types.Tx) bool) MempoolOption {
return func(mem *Mempool) { mem.filter = f }
// WithPreCheck sets a filter for the mempool to reject a tx if f(tx) returns
// false. This is ran before CheckTx.
func WithPreCheck(f PreCheckFunc) MempoolOption {
return func(mem *Mempool) { mem.preCheck = f }
}
// WithPostCheck sets a filter for the mempool to reject a tx if f(tx) returns
// false. This is ran after CheckTx.
func WithPostCheck(f PostCheckFunc) MempoolOption {
return func(mem *Mempool) { mem.postCheck = f }
} }
// WithMetrics sets the metrics. // WithMetrics sets the metrics.
@ -248,7 +285,7 @@ func (mem *Mempool) CheckTx(tx types.Tx, cb func(*abci.Response)) (err error) {
return ErrMempoolIsFull return ErrMempoolIsFull
} }
if mem.filter != nil && !mem.filter(tx) {
if mem.preCheck != nil && !mem.preCheck(tx) {
return return
} }
@ -298,7 +335,8 @@ func (mem *Mempool) resCbNormal(req *abci.Request, res *abci.Response) {
switch r := res.Value.(type) { switch r := res.Value.(type) {
case *abci.Response_CheckTx: case *abci.Response_CheckTx:
tx := req.GetCheckTx().Tx tx := req.GetCheckTx().Tx
if r.CheckTx.Code == abci.CodeTypeOK {
if (r.CheckTx.Code == abci.CodeTypeOK) &&
mem.isPostCheckPass(tx, r.CheckTx) {
mem.counter++ mem.counter++
memTx := &mempoolTx{ memTx := &mempoolTx{
counter: mem.counter, counter: mem.counter,
@ -326,10 +364,15 @@ func (mem *Mempool) resCbRecheck(req *abci.Request, res *abci.Response) {
case *abci.Response_CheckTx: case *abci.Response_CheckTx:
memTx := mem.recheckCursor.Value.(*mempoolTx) memTx := mem.recheckCursor.Value.(*mempoolTx)
if !bytes.Equal(req.GetCheckTx().Tx, memTx.tx) { if !bytes.Equal(req.GetCheckTx().Tx, memTx.tx) {
cmn.PanicSanity(fmt.Sprintf("Unexpected tx response from proxy during recheck\n"+
"Expected %X, got %X", r.CheckTx.Data, memTx.tx))
cmn.PanicSanity(
fmt.Sprintf(
"Unexpected tx response from proxy during recheck\nExpected %X, got %X",
r.CheckTx.Data,
memTx.tx,
),
)
} }
if r.CheckTx.Code == abci.CodeTypeOK {
if (r.CheckTx.Code == abci.CodeTypeOK) && mem.isPostCheckPass(memTx.tx, r.CheckTx) {
// Good, nothing to do. // Good, nothing to do.
} else { } else {
// Tx became invalidated due to newly committed block. // Tx became invalidated due to newly committed block.
@ -444,7 +487,12 @@ func (mem *Mempool) ReapMaxTxs(max int) types.Txs {
// Update informs the mempool that the given txs were committed and can be discarded. // Update informs the mempool that the given txs were committed and can be discarded.
// NOTE: this should be called *after* block is committed by consensus. // NOTE: this should be called *after* block is committed by consensus.
// NOTE: unsafe; Lock/Unlock must be managed by caller // NOTE: unsafe; Lock/Unlock must be managed by caller
func (mem *Mempool) Update(height int64, txs types.Txs, filter func(types.Tx) bool) error {
func (mem *Mempool) Update(
height int64,
txs types.Txs,
preCheck PreCheckFunc,
postCheck PostCheckFunc,
) error {
// First, create a lookup map of txns in new txs. // First, create a lookup map of txns in new txs.
txsMap := make(map[string]struct{}, len(txs)) txsMap := make(map[string]struct{}, len(txs))
for _, tx := range txs { for _, tx := range txs {
@ -455,8 +503,11 @@ func (mem *Mempool) Update(height int64, txs types.Txs, filter func(types.Tx) bo
mem.height = height mem.height = height
mem.notifiedTxsAvailable = false mem.notifiedTxsAvailable = false
if filter != nil {
mem.filter = filter
if preCheck != nil {
mem.preCheck = preCheck
}
if postCheck != nil {
mem.postCheck = postCheck
} }
// Remove transactions that are already in txs. // Remove transactions that are already in txs.
@ -514,6 +565,10 @@ func (mem *Mempool) recheckTxs(goodTxs []types.Tx) {
mem.proxyAppConn.FlushAsync() mem.proxyAppConn.FlushAsync()
} }
func (mem *Mempool) isPostCheckPass(tx types.Tx, r *abci.ResponseCheckTx) bool {
return mem.postCheck == nil || mem.postCheck(tx, r)
}
//-------------------------------------------------------------------------------- //--------------------------------------------------------------------------------
// mempoolTx is a transaction that successfully ran // mempoolTx is a transaction that successfully ran


+ 60
- 3
mempool/mempool_test.go View File

@ -14,6 +14,7 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
amino "github.com/tendermint/go-amino"
"github.com/tendermint/tendermint/abci/example/counter" "github.com/tendermint/tendermint/abci/example/counter"
"github.com/tendermint/tendermint/abci/example/kvstore" "github.com/tendermint/tendermint/abci/example/kvstore"
abci "github.com/tendermint/tendermint/abci/types" abci "github.com/tendermint/tendermint/abci/types"
@ -119,6 +120,62 @@ func TestReapMaxBytesMaxGas(t *testing.T) {
} }
} }
func TestMempoolFilters(t *testing.T) {
app := kvstore.NewKVStoreApplication()
cc := proxy.NewLocalClientCreator(app)
mempool := newMempoolWithApp(cc)
emptyTxArr := []types.Tx{[]byte{}}
nopPreFilter := func(tx types.Tx) bool { return true }
nopPostFilter := func(tx types.Tx, res *abci.ResponseCheckTx) bool { return true }
// This is the same filter we expect to be used within node/node.go and state/execution.go
nBytePreFilter := func(n int) func(tx types.Tx) bool {
return func(tx types.Tx) bool {
// We have to account for the amino overhead in the tx size as well
aminoOverhead := amino.UvarintSize(uint64(len(tx)))
return (len(tx) + aminoOverhead) <= n
}
}
nGasPostFilter := func(n int64) func(tx types.Tx, res *abci.ResponseCheckTx) bool {
return func(tx types.Tx, res *abci.ResponseCheckTx) bool {
if n == -1 {
return true
}
return res.GasWanted <= n
}
}
// each table driven test creates numTxsToCreate txs with checkTx, and at the end clears all remaining txs.
// each tx has 20 bytes + amino overhead = 21 bytes, 1 gas
tests := []struct {
numTxsToCreate int
preFilter func(tx types.Tx) bool
postFilter func(tx types.Tx, res *abci.ResponseCheckTx) bool
expectedNumTxs int
}{
{10, nopPreFilter, nopPostFilter, 10},
{10, nBytePreFilter(10), nopPostFilter, 0},
{10, nBytePreFilter(20), nopPostFilter, 0},
{10, nBytePreFilter(21), nopPostFilter, 10},
{10, nopPreFilter, nGasPostFilter(-1), 10},
{10, nopPreFilter, nGasPostFilter(0), 0},
{10, nopPreFilter, nGasPostFilter(1), 10},
{10, nopPreFilter, nGasPostFilter(3000), 10},
{10, nBytePreFilter(10), nGasPostFilter(20), 0},
{10, nBytePreFilter(30), nGasPostFilter(20), 10},
{10, nBytePreFilter(21), nGasPostFilter(1), 10},
{10, nBytePreFilter(21), nGasPostFilter(0), 0},
}
for tcIndex, tt := range tests {
mempool.Update(1, emptyTxArr, tt.preFilter, tt.postFilter)
checkTxs(t, mempool, tt.numTxsToCreate)
require.Equal(t, tt.expectedNumTxs, mempool.Size(), "mempool had the incorrect size, on test case %d", tcIndex)
mempool.Flush()
}
}
func TestTxsAvailable(t *testing.T) { func TestTxsAvailable(t *testing.T) {
app := kvstore.NewKVStoreApplication() app := kvstore.NewKVStoreApplication()
cc := proxy.NewLocalClientCreator(app) cc := proxy.NewLocalClientCreator(app)
@ -139,7 +196,7 @@ func TestTxsAvailable(t *testing.T) {
// it should fire once now for the new height // it should fire once now for the new height
// since there are still txs left // since there are still txs left
committedTxs, txs := txs[:50], txs[50:] committedTxs, txs := txs[:50], txs[50:]
if err := mempool.Update(1, committedTxs, nil); err != nil {
if err := mempool.Update(1, committedTxs, nil, nil); err != nil {
t.Error(err) t.Error(err)
} }
ensureFire(t, mempool.TxsAvailable(), timeoutMS) ensureFire(t, mempool.TxsAvailable(), timeoutMS)
@ -151,7 +208,7 @@ func TestTxsAvailable(t *testing.T) {
// now call update with all the txs. it should not fire as there are no txs left // now call update with all the txs. it should not fire as there are no txs left
committedTxs = append(txs, moreTxs...) committedTxs = append(txs, moreTxs...)
if err := mempool.Update(2, committedTxs, nil); err != nil {
if err := mempool.Update(2, committedTxs, nil, nil); err != nil {
t.Error(err) t.Error(err)
} }
ensureNoFire(t, mempool.TxsAvailable(), timeoutMS) ensureNoFire(t, mempool.TxsAvailable(), timeoutMS)
@ -208,7 +265,7 @@ func TestSerialReap(t *testing.T) {
binary.BigEndian.PutUint64(txBytes, uint64(i)) binary.BigEndian.PutUint64(txBytes, uint64(i))
txs = append(txs, txBytes) txs = append(txs, txBytes)
} }
if err := mempool.Update(0, txs, nil); err != nil {
if err := mempool.Update(0, txs, nil, nil); err != nil {
t.Error(err) t.Error(err)
} }
} }


+ 18
- 10
node/node.go View File

@ -7,22 +7,23 @@ import (
"fmt" "fmt"
"net" "net"
"net/http" "net/http"
_ "net/http/pprof"
"strings"
"time" "time"
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp" "github.com/prometheus/client_golang/prometheus/promhttp"
amino "github.com/tendermint/go-amino" amino "github.com/tendermint/go-amino"
abci "github.com/tendermint/tendermint/abci/types"
"github.com/tendermint/tendermint/crypto/ed25519"
cmn "github.com/tendermint/tendermint/libs/common"
dbm "github.com/tendermint/tendermint/libs/db"
"github.com/tendermint/tendermint/libs/log"
abci "github.com/tendermint/tendermint/abci/types"
bc "github.com/tendermint/tendermint/blockchain" bc "github.com/tendermint/tendermint/blockchain"
cfg "github.com/tendermint/tendermint/config" cfg "github.com/tendermint/tendermint/config"
cs "github.com/tendermint/tendermint/consensus" cs "github.com/tendermint/tendermint/consensus"
"github.com/tendermint/tendermint/crypto/ed25519"
"github.com/tendermint/tendermint/evidence" "github.com/tendermint/tendermint/evidence"
cmn "github.com/tendermint/tendermint/libs/common"
dbm "github.com/tendermint/tendermint/libs/db"
"github.com/tendermint/tendermint/libs/log"
mempl "github.com/tendermint/tendermint/mempool" mempl "github.com/tendermint/tendermint/mempool"
"github.com/tendermint/tendermint/p2p" "github.com/tendermint/tendermint/p2p"
"github.com/tendermint/tendermint/p2p/pex" "github.com/tendermint/tendermint/p2p/pex"
@ -40,9 +41,6 @@ import (
"github.com/tendermint/tendermint/types" "github.com/tendermint/tendermint/types"
tmtime "github.com/tendermint/tendermint/types/time" tmtime "github.com/tendermint/tendermint/types/time"
"github.com/tendermint/tendermint/version" "github.com/tendermint/tendermint/version"
_ "net/http/pprof"
"strings"
) )
//------------------------------------------------------------------------------ //------------------------------------------------------------------------------
@ -255,7 +253,17 @@ func NewNode(config *cfg.Config,
proxyApp.Mempool(), proxyApp.Mempool(),
state.LastBlockHeight, state.LastBlockHeight,
mempl.WithMetrics(memplMetrics), mempl.WithMetrics(memplMetrics),
mempl.WithFilter(sm.TxFilter(state)),
mempl.WithPreCheck(
mempl.PreCheckAminoMaxBytes(
types.MaxDataBytesUnknownEvidence(
state.ConsensusParams.BlockSize.MaxBytes,
state.Validators.Size(),
),
),
),
mempl.WithPostCheck(
mempl.PostCheckMaxGas(state.ConsensusParams.BlockSize.MaxGas),
),
) )
mempoolLogger := logger.With("module", "mempool") mempoolLogger := logger.With("module", "mempool")
mempool.SetLogger(mempoolLogger) mempool.SetLogger(mempoolLogger)


+ 31
- 12
state/execution.go View File

@ -7,6 +7,7 @@ import (
abci "github.com/tendermint/tendermint/abci/types" abci "github.com/tendermint/tendermint/abci/types"
dbm "github.com/tendermint/tendermint/libs/db" dbm "github.com/tendermint/tendermint/libs/db"
"github.com/tendermint/tendermint/libs/log" "github.com/tendermint/tendermint/libs/log"
"github.com/tendermint/tendermint/mempool"
"github.com/tendermint/tendermint/proxy" "github.com/tendermint/tendermint/proxy"
"github.com/tendermint/tendermint/types" "github.com/tendermint/tendermint/types"
) )
@ -115,11 +116,16 @@ func (blockExec *BlockExecutor) ApplyBlock(state State, blockID types.BlockID, b
return state, nil return state, nil
} }
// Commit locks the mempool, runs the ABCI Commit message, and updates the mempool.
// Commit locks the mempool, runs the ABCI Commit message, and updates the
// mempool.
// It returns the result of calling abci.Commit (the AppHash), and an error. // It returns the result of calling abci.Commit (the AppHash), and an error.
// The Mempool must be locked during commit and update because state is typically reset on Commit and old txs must be replayed
// against committed state before new txs are run in the mempool, lest they be invalid.
func (blockExec *BlockExecutor) Commit(state State, block *types.Block) ([]byte, error) {
// The Mempool must be locked during commit and update because state is
// typically reset on Commit and old txs must be replayed against committed
// state before new txs are run in the mempool, lest they be invalid.
func (blockExec *BlockExecutor) Commit(
state State,
block *types.Block,
) ([]byte, error) {
blockExec.mempool.Lock() blockExec.mempool.Lock()
defer blockExec.mempool.Unlock() defer blockExec.mempool.Unlock()
@ -134,22 +140,35 @@ func (blockExec *BlockExecutor) Commit(state State, block *types.Block) ([]byte,
// Commit block, get hash back // Commit block, get hash back
res, err := blockExec.proxyApp.CommitSync() res, err := blockExec.proxyApp.CommitSync()
if err != nil { if err != nil {
blockExec.logger.Error("Client error during proxyAppConn.CommitSync", "err", err)
blockExec.logger.Error(
"Client error during proxyAppConn.CommitSync",
"err", err,
)
return nil, err return nil, err
} }
// ResponseCommit has no error code - just data // ResponseCommit has no error code - just data
blockExec.logger.Info("Committed state",
blockExec.logger.Info(
"Committed state",
"height", block.Height, "height", block.Height,
"txs", block.NumTxs, "txs", block.NumTxs,
"appHash", fmt.Sprintf("%X", res.Data))
"appHash", fmt.Sprintf("%X", res.Data),
)
// Update mempool. // Update mempool.
if err := blockExec.mempool.Update(block.Height, block.Txs, TxFilter(state)); err != nil {
return nil, err
}
return res.Data, nil
err = blockExec.mempool.Update(
block.Height,
block.Txs,
mempool.PreCheckAminoMaxBytes(
types.MaxDataBytesUnknownEvidence(
state.ConsensusParams.BlockSize.MaxBytes,
state.Validators.Size(),
),
),
mempool.PostCheckMaxGas(state.ConsensusParams.MaxGas),
)
return res.Data, err
} }
//--------------------------------------------------------- //---------------------------------------------------------


+ 21
- 13
state/services.go View File

@ -2,6 +2,7 @@ package state
import ( import (
abci "github.com/tendermint/tendermint/abci/types" abci "github.com/tendermint/tendermint/abci/types"
"github.com/tendermint/tendermint/mempool"
"github.com/tendermint/tendermint/types" "github.com/tendermint/tendermint/types"
) )
@ -23,7 +24,7 @@ type Mempool interface {
Size() int Size() int
CheckTx(types.Tx, func(*abci.Response)) error CheckTx(types.Tx, func(*abci.Response)) error
ReapMaxBytesMaxGas(maxBytes, maxGas int64) types.Txs ReapMaxBytesMaxGas(maxBytes, maxGas int64) types.Txs
Update(height int64, txs types.Txs, filter func(types.Tx) bool) error
Update(int64, types.Txs, mempool.PreCheckFunc, mempool.PostCheckFunc) error
Flush() Flush()
FlushAppConn() error FlushAppConn() error
@ -36,16 +37,23 @@ type MockMempool struct{}
var _ Mempool = MockMempool{} var _ Mempool = MockMempool{}
func (MockMempool) Lock() {}
func (MockMempool) Unlock() {}
func (MockMempool) Size() int { return 0 }
func (MockMempool) CheckTx(tx types.Tx, cb func(*abci.Response)) error { return nil }
func (MockMempool) ReapMaxBytesMaxGas(maxBytes, maxGas int64) types.Txs { return types.Txs{} }
func (MockMempool) Update(height int64, txs types.Txs, filter func(types.Tx) bool) error { return nil }
func (MockMempool) Flush() {}
func (MockMempool) FlushAppConn() error { return nil }
func (MockMempool) TxsAvailable() <-chan struct{} { return make(chan struct{}) }
func (MockMempool) EnableTxsAvailable() {}
func (MockMempool) Lock() {}
func (MockMempool) Unlock() {}
func (MockMempool) Size() int { return 0 }
func (MockMempool) CheckTx(_ types.Tx, _ func(*abci.Response)) error { return nil }
func (MockMempool) ReapMaxBytesMaxGas(_, _ int64) types.Txs { return types.Txs{} }
func (MockMempool) Update(
_ int64,
_ types.Txs,
_ mempool.PreCheckFunc,
_ mempool.PostCheckFunc,
) error {
return nil
}
func (MockMempool) Flush() {}
func (MockMempool) FlushAppConn() error { return nil }
func (MockMempool) TxsAvailable() <-chan struct{} { return make(chan struct{}) }
func (MockMempool) EnableTxsAvailable() {}
//------------------------------------------------------ //------------------------------------------------------
// blockstore // blockstore
@ -82,5 +90,5 @@ type EvidencePool interface {
type MockEvidencePool struct{} type MockEvidencePool struct{}
func (m MockEvidencePool) PendingEvidence(int64) []types.Evidence { return nil } func (m MockEvidencePool) PendingEvidence(int64) []types.Evidence { return nil }
func (m MockEvidencePool) AddEvidence(types.Evidence) error { return nil }
func (m MockEvidencePool) Update(*types.Block, State) {}
func (m MockEvidencePool) AddEvidence(types.Evidence) error { return nil }
func (m MockEvidencePool) Update(*types.Block, State) {}

Loading…
Cancel
Save