get rid of gox
build target builds inside docker, dev-build - locally
Revert "build target builds inside docker, dev-build - locally"
This reverts commit 8ba89d5e8c.
add build tags to make build/build_race/install
use tendermint's fork of glide instead of tar.gz
remove TMHOME unused var + set length for git hash
get rid of GOTOOLS_CHECK
fixes after review
zip
needed for distribution
Fixes https://github.com/tendermint/tendermint/issues/1189
For every TxEventBuffer.Flush() invoking, we were invoking
a:
b.events = make([]EventDataTx, 0, b.capacity)
whose intention is to innocently clear the events slice but
maintain the underlying capacity.
However, unfortunately this is memory and garbage collection intensive
which is linear in the number of events added. If an attack had access
to our code somehow, invoking .Flush() in tight loops would be a sure
way to cause huge GC pressure, and say if they added about 1e9
events maliciously, every Flush() would take at least 3.2seconds
which is enough to now control our application.
The new using of the capacity preserving slice clearing idiom
takes a constant time regardless of the number of elements with zero
allocations so we are killing many birds with one stone i.e
b.events = b.events[:0]
For benchmarking results, please see
https://gist.github.com/odeke-em/532c14ab67d71c9c0b95518a7a526058
for a reference on how things can get out of hand easily.
https://play.golang.org/p/gN21yO9IRs3
```
func waitWithCancel(f func() *clist.CElement, ctx context.Context) *clist.CElement {
el := make(chan *clist.CElement, 1)
select {
case el <- f():
```
will just run f() blockingly, so this doesn't change much in terms of behavior.
Leaking goroutine:
```
114 @ 0x42f2bc 0x42f3ae 0x440794 0x4403b9 0x468002 0x9fe32d 0x9ff78f 0xa025ed 0x45e571
```
Explanation:
it blocks on an empty clist forever. so unless theres txs coming in,
this go routine will just sit there, holding onto the peer too.
if we're constantly reconnecting to some peer, old instances are not
garbage collected, leading to memory leak.
Fixes https://github.com/cosmos/gaia/issues/108
Previous attempt https://github.com/tendermint/tendermint/pull/1156
* GetByHeight will now switch to using binary search once
we have >=50 items.
* Feedback from @ebuchman to catch a missed spot where
we forgot about lazy sorting that the original code
assumed would always have sorted commits by height.
Added a lazy sorting routine here too.
A test as well to ensure that we always get the properly
sorted and last value.
Addressing PR review requests from @melekes and @ebuchman to
add a test that checks that the heights returned from both are
the same thus providing a perceptible equivalence of the code
linear range search vs binary range search code.
Updates #1021
* Implement a GetHeightBinarySearch method that looks for
the height using the binary search algorithm guaranteeing
worst case iteration time of O(log2(n))
whereas
worst case iteration time of O(n) for the current linear search
So if n we had 500 commits stored by height and sorted, to
trigger the worst case scenario for each, pass in
the most negative height you can find e.g. -1
Linear search: 500 iterations
Binary search: 9 iterations
with n=1000, qHeight = -1
Linear search: 1000 iterations
Binary search: 10 iterations
with n=1e6, qHeight = -1
Linear search: 1e6 iterations
Binary search: 20 iterations
Of course there are realistic expectations e.g. a max of
commits that may be saved so linear search might be useful
for very small size set because it has less preparing overhead
and only ~2 types of comparisons, but nonetheless binary search
shines as soon as we start to hit say 50 commits to search from
as you can see below:
```shell
$ go test -v -run=^$ -bench=MemStore
goos: darwin
goarch: amd64
pkg: github.com/tendermint/tendermint/lite
BenchmarkMemStoreProviderGetByHeightLinearSearch5-4 300000 6491 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightLinearSearch50-4 200000 12064 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightLinearSearch100-4 50000 32987 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightLinearSearch500-4 5000 395521 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightLinearSearch1000-4 500 2940724 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch5-4 300000 6281 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch50-4 200000 10117 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch100-4 100000 18447 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch500-4 20000 89029 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch1000-4 5000 265719 ns/op 1600 B/op 15 allocs/op
PASS
ok github.com/tendermint/tendermint/lite 86.614s
$ go test -v -run=^$ -bench=MemStore
goos: darwin
goarch: amd64
pkg: github.com/tendermint/tendermint/lite
BenchmarkMemStoreProviderGetByHeightLinearSearch5-4 300000 6779 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightLinearSearch50-4 100000 12980 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightLinearSearch100-4 30000 43598 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightLinearSearch500-4 5000 377462 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightLinearSearch1000-4 500 3278122 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch5-4 300000 7084 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch50-4 200000 9852 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch100-4 100000 19020 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch500-4 20000 99463 ns/op 1600 B/op 15 allocs/op
BenchmarkMemStoreProviderGetByHeightBinarySearch1000-4 5000 259293 ns/op 1600 B/op 15 allocs/op
PASS
ok github.com/tendermint/tendermint/lite 86.204s
```
which gives
```shell
$ benchstat old.txt new.txt
name old time/op new time/op delta
MemStoreProviderGetByHeight5-4 6.63µs ± 2% 6.68µs ± 6% ~ (p=1.000 n=2+2)
MemStoreProviderGetByHeight50-4 12.5µs ± 4% 10.0µs ± 1% ~ (p=0.333 n=2+2)
MemStoreProviderGetByHeight100-4 38.3µs ±14% 18.7µs ± 2% ~ (p=0.333 n=2+2)
MemStoreProviderGetByHeight500-4 386µs ± 2% 94µs ± 6% ~ (p=0.333 n=2+2)
MemStoreProviderGetByHeight1000-4 3.11ms ± 5% 0.26ms ± 1% ~ (p=0.333 n=2+2)
```
If need be we can make a hybrid algorithm that switches between the
linear and binary search depending on the number of items.
This is reminiscent of Python's TimSort algorithm.
Ethan Buchman
Anton Kaliaev
Emmanuel Odeke
Luke Schoen
Zarko Milosevic
Zach Ramsay
Adrian Brink
George Ornbo