You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

314 lines
10 KiB

7 years ago
6 years ago
9 years ago
p2p: fix infinite loop in addrbook (#3232) * failing test * fix infinite loop in addrbook There are cases where we only have a small number of addresses marked good ("old"), but the selection mechanism keeps trying to select more of these addresses, and hence ends up in an infinite loop. Here we fix this to only try and select such "old" addresses if we have enough of them. Note this means, if we don't have enough of them, we may return more "new" addresses than otherwise expected by the newSelectionBias. This whole GetSelectionWithBias method probably needs to be rewritten, but this is a quick fix for the issue. * changelog * fix infinite loop if not enough new addrs * fix another potential infinite loop if a.nNew == 0 -> pickFromOldBucket=true, but we don't have enough items (a.nOld > len(oldBucketToAddrsMap) false) * Revert "fix another potential infinite loop" This reverts commit 146540c1125597162bd89820d611f6531f5e5e4b. * check num addresses instead of buckets, new test * fixed the int division * add slack to bias % in test, lint fixes * Added checks for selection content in test * test cleanup * Apply suggestions from code review Co-Authored-By: ebuchman <ethan@coinculture.info> * address review comments * change after Anton's review comments * use the same docker image we use for testing when building a binary for localnet * switch back to circleci classic * more review comments * more review comments * refactor addrbook_test * build linux binary inside docker in attempt to fix ``` --> Running dep + make build-linux GOOS=linux GOARCH=amd64 make build make[1]: Entering directory `/home/circleci/.go_workspace/src/github.com/tendermint/tendermint' CGO_ENABLED=0 go build -ldflags "-X github.com/tendermint/tendermint/version.GitCommit=`git rev-parse --short=8 HEAD`" -tags 'tendermint' -o build/tendermint ./cmd/tendermint/ p2p/pex/addrbook.go:373:13: undefined: math.Round ``` * change dir from /usr to /go * use concrete Go version for localnet binary * check for nil addresses just to be sure
6 years ago
  1. GOTOOLS = \
  2. github.com/mitchellh/gox \
  3. github.com/golang/dep/cmd/dep \
  4. github.com/golangci/golangci-lint/cmd/golangci-lint \
  5. github.com/gogo/protobuf/protoc-gen-gogo \
  6. github.com/square/certstrap
  7. GOBIN?=${GOPATH}/bin
  8. PACKAGES=$(shell go list ./...)
  9. INCLUDE = -I=. -I=${GOPATH}/src -I=${GOPATH}/src/github.com/gogo/protobuf/protobuf
  10. BUILD_TAGS?='tendermint'
  11. BUILD_FLAGS = -ldflags "-X github.com/tendermint/tendermint/version.GitCommit=`git rev-parse --short=8 HEAD`"
  12. all: check build test install
  13. check: check_tools get_vendor_deps
  14. ########################################
  15. ### Build Tendermint
  16. build:
  17. CGO_ENABLED=0 go build $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o build/tendermint ./cmd/tendermint/
  18. build_c:
  19. CGO_ENABLED=1 go build $(BUILD_FLAGS) -tags "$(BUILD_TAGS) gcc" -o build/tendermint ./cmd/tendermint/
  20. build_race:
  21. CGO_ENABLED=0 go build -race $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o build/tendermint ./cmd/tendermint
  22. install:
  23. CGO_ENABLED=0 go install $(BUILD_FLAGS) -tags $(BUILD_TAGS) ./cmd/tendermint
  24. install_c:
  25. CGO_ENABLED=1 go install $(BUILD_FLAGS) -tags "$(BUILD_TAGS) gcc" ./cmd/tendermint
  26. ########################################
  27. ### Protobuf
  28. protoc_all: protoc_libs protoc_merkle protoc_abci protoc_grpc protoc_proto3types
  29. %.pb.go: %.proto
  30. ## If you get the following error,
  31. ## "error while loading shared libraries: libprotobuf.so.14: cannot open shared object file: No such file or directory"
  32. ## See https://stackoverflow.com/a/25518702
  33. ## Note the $< here is substituted for the %.proto
  34. ## Note the $@ here is substituted for the %.pb.go
  35. protoc $(INCLUDE) $< --gogo_out=Mgoogle/protobuf/timestamp.proto=github.com/golang/protobuf/ptypes/timestamp,plugins=grpc:.
  36. ########################################
  37. ### Build ABCI
  38. # see protobuf section above
  39. protoc_abci: abci/types/types.pb.go
  40. protoc_proto3types: types/proto3/block.pb.go
  41. build_abci:
  42. @go build -i ./abci/cmd/...
  43. install_abci:
  44. @go install ./abci/cmd/...
  45. ########################################
  46. ### Distribution
  47. # dist builds binaries for all platforms and packages them for distribution
  48. # TODO add abci to these scripts
  49. dist:
  50. @BUILD_TAGS=$(BUILD_TAGS) sh -c "'$(CURDIR)/scripts/dist.sh'"
  51. ########################################
  52. ### Tools & dependencies
  53. check_tools:
  54. @# https://stackoverflow.com/a/25668869
  55. @echo "Found tools: $(foreach tool,$(notdir $(GOTOOLS)),\
  56. $(if $(shell which $(tool)),$(tool),$(error "No $(tool) in PATH")))"
  57. get_tools:
  58. @echo "--> Installing tools"
  59. ./scripts/get_tools.sh
  60. update_tools:
  61. @echo "--> Updating tools"
  62. ./scripts/get_tools.sh
  63. #Update dependencies
  64. get_vendor_deps:
  65. @echo "--> Running dep"
  66. @dep ensure
  67. #For ABCI and libs
  68. get_protoc:
  69. @# https://github.com/google/protobuf/releases
  70. curl -L https://github.com/google/protobuf/releases/download/v3.6.1/protobuf-cpp-3.6.1.tar.gz | tar xvz && \
  71. cd protobuf-3.6.1 && \
  72. DIST_LANG=cpp ./configure && \
  73. make && \
  74. make check && \
  75. sudo make install && \
  76. sudo ldconfig && \
  77. cd .. && \
  78. rm -rf protobuf-3.6.1
  79. draw_deps:
  80. @# requires brew install graphviz or apt-get install graphviz
  81. go get github.com/RobotsAndPencils/goviz
  82. @goviz -i github.com/tendermint/tendermint/cmd/tendermint -d 3 | dot -Tpng -o dependency-graph.png
  83. get_deps_bin_size:
  84. @# Copy of build recipe with additional flags to perform binary size analysis
  85. $(eval $(shell go build -work -a $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o build/tendermint ./cmd/tendermint/ 2>&1))
  86. @find $(WORK) -type f -name "*.a" | xargs -I{} du -hxs "{}" | sort -rh | sed -e s:${WORK}/::g > deps_bin_size.log
  87. @echo "Results can be found here: $(CURDIR)/deps_bin_size.log"
  88. ########################################
  89. ### Libs
  90. protoc_libs: libs/common/types.pb.go
  91. gen_certs: clean_certs
  92. ## Generating certificates for TLS testing...
  93. certstrap init --common-name "tendermint.com" --passphrase ""
  94. certstrap request-cert -ip "::" --passphrase ""
  95. certstrap sign "::" --CA "tendermint.com" --passphrase ""
  96. mv out/::.crt out/::.key db/remotedb
  97. clean_certs:
  98. ## Cleaning TLS testing certificates...
  99. rm -rf out
  100. rm -f db/remotedb/::.crt db/remotedb/::.key
  101. test_libs: gen_certs
  102. GOCACHE=off go test -tags gcc $(PACKAGES)
  103. make clean_certs
  104. grpc_dbserver:
  105. protoc -I db/remotedb/proto/ db/remotedb/proto/defs.proto --go_out=plugins=grpc:db/remotedb/proto
  106. protoc_grpc: rpc/grpc/types.pb.go
  107. protoc_merkle: crypto/merkle/merkle.pb.go
  108. ########################################
  109. ### Testing
  110. ## required to be run first by most tests
  111. build_docker_test_image:
  112. docker build -t tester -f ./test/docker/Dockerfile .
  113. ### coverage, app, persistence, and libs tests
  114. test_cover:
  115. # run the go unit tests with coverage
  116. bash test/test_cover.sh
  117. test_apps:
  118. # run the app tests using bash
  119. # requires `abci-cli` and `tendermint` binaries installed
  120. bash test/app/test.sh
  121. test_abci_apps:
  122. bash abci/tests/test_app/test.sh
  123. test_abci_cli:
  124. # test the cli against the examples in the tutorial at:
  125. # ./docs/abci-cli.md
  126. # if test fails, update the docs ^
  127. @ bash abci/tests/test_cli/test.sh
  128. test_persistence:
  129. # run the persistence tests using bash
  130. # requires `abci-cli` installed
  131. docker run --name run_persistence -t tester bash test/persist/test_failure_indices.sh
  132. # TODO undockerize
  133. # bash test/persist/test_failure_indices.sh
  134. test_p2p:
  135. docker rm -f rsyslog || true
  136. rm -rf test/logs || true
  137. mkdir test/logs
  138. cd test/
  139. docker run -d -v "logs:/var/log/" -p 127.0.0.1:5514:514/udp --name rsyslog voxxit/rsyslog
  140. cd ..
  141. # requires 'tester' the image from above
  142. bash test/p2p/test.sh tester
  143. # the `docker cp` takes a really long time; uncomment for debugging
  144. #
  145. # mkdir -p test/p2p/logs && docker cp rsyslog:/var/log test/p2p/logs
  146. test_integrations:
  147. make build_docker_test_image
  148. make get_tools
  149. make get_vendor_deps
  150. make install
  151. make test_cover
  152. make test_apps
  153. make test_abci_apps
  154. make test_abci_cli
  155. make test_libs
  156. make test_persistence
  157. make test_p2p
  158. test_release:
  159. @go test -tags release $(PACKAGES)
  160. test100:
  161. @for i in {1..100}; do make test; done
  162. vagrant_test:
  163. vagrant up
  164. vagrant ssh -c 'make test_integrations'
  165. ### go tests
  166. test:
  167. @echo "--> Running go test"
  168. @GOCACHE=off go test -p 1 $(PACKAGES)
  169. test_race:
  170. @echo "--> Running go test --race"
  171. @GOCACHE=off go test -p 1 -v -race $(PACKAGES)
  172. # uses https://github.com/sasha-s/go-deadlock/ to detect potential deadlocks
  173. test_with_deadlock:
  174. make set_with_deadlock
  175. make test
  176. make cleanup_after_test_with_deadlock
  177. set_with_deadlock:
  178. find . -name "*.go" | grep -v "vendor/" | xargs -n 1 sed -i.bak 's/sync.RWMutex/deadlock.RWMutex/'
  179. find . -name "*.go" | grep -v "vendor/" | xargs -n 1 sed -i.bak 's/sync.Mutex/deadlock.Mutex/'
  180. find . -name "*.go" | grep -v "vendor/" | xargs -n 1 goimports -w
  181. # cleanes up after you ran test_with_deadlock
  182. cleanup_after_test_with_deadlock:
  183. find . -name "*.go" | grep -v "vendor/" | xargs -n 1 sed -i.bak 's/deadlock.RWMutex/sync.RWMutex/'
  184. find . -name "*.go" | grep -v "vendor/" | xargs -n 1 sed -i.bak 's/deadlock.Mutex/sync.Mutex/'
  185. find . -name "*.go" | grep -v "vendor/" | xargs -n 1 goimports -w
  186. ########################################
  187. ### Formatting, linting, and vetting
  188. fmt:
  189. @go fmt ./...
  190. lint:
  191. @echo "--> Running linter"
  192. @golangci-lint run
  193. DESTINATION = ./index.html.md
  194. rpc-docs:
  195. cat rpc/core/slate_header.txt > $(DESTINATION)
  196. godoc2md -template rpc/core/doc_template.txt github.com/tendermint/tendermint/rpc/core | grep -v -e "pipe.go" -e "routes.go" -e "dev.go" | sed 's,/src/target,https://github.com/tendermint/tendermint/tree/master/rpc/core,' >> $(DESTINATION)
  197. check_dep:
  198. dep status >> /dev/null
  199. !(grep -n branch Gopkg.toml)
  200. ###########################################################
  201. ### Docker image
  202. build-docker:
  203. cp build/tendermint DOCKER/tendermint
  204. docker build --label=tendermint --tag="tendermint/tendermint" DOCKER
  205. rm -rf DOCKER/tendermint
  206. ###########################################################
  207. ### Local testnet using docker
  208. # Build linux binary on other platforms
  209. build-linux: get_tools get_vendor_deps
  210. GOOS=linux GOARCH=amd64 $(MAKE) build
  211. build-docker-localnode:
  212. @cd networks/local && make
  213. # Run a 4-node testnet locally
  214. localnet-start: localnet-stop
  215. @if ! [ -f build/node0/config/genesis.json ]; then docker run --rm -v $(CURDIR)/build:/tendermint:Z tendermint/localnode testnet --v 4 --o . --populate-persistent-peers --starting-ip-address 192.167.10.2 ; fi
  216. docker-compose up
  217. # Stop testnet
  218. localnet-stop:
  219. docker-compose down
  220. ###########################################################
  221. ### Remote full-nodes (sentry) using terraform and ansible
  222. # Server management
  223. sentry-start:
  224. @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi
  225. @if ! [ -f $(HOME)/.ssh/id_rsa.pub ]; then ssh-keygen ; fi
  226. cd networks/remote/terraform && terraform init && terraform apply -var DO_API_TOKEN="$(DO_API_TOKEN)" -var SSH_KEY_FILE="$(HOME)/.ssh/id_rsa.pub"
  227. @if ! [ -f $(CURDIR)/build/node0/config/genesis.json ]; then docker run --rm -v $(CURDIR)/build:/tendermint:Z tendermint/localnode testnet --v 0 --n 4 --o . ; fi
  228. cd networks/remote/ansible && ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml
  229. @echo "Next step: Add your validator setup in the genesis.json and config.tml files and run \"make sentry-config\". (Public key of validator, chain ID, peer IP and node ID.)"
  230. # Configuration management
  231. sentry-config:
  232. cd networks/remote/ansible && ansible-playbook -i inventory/digital_ocean.py -l sentrynet config.yml -e BINARY=$(CURDIR)/build/tendermint -e CONFIGDIR=$(CURDIR)/build
  233. sentry-stop:
  234. @if [ -z "$(DO_API_TOKEN)" ]; then echo "DO_API_TOKEN environment variable not set." ; false ; fi
  235. cd networks/remote/terraform && terraform destroy -var DO_API_TOKEN="$(DO_API_TOKEN)" -var SSH_KEY_FILE="$(HOME)/.ssh/id_rsa.pub"
  236. # meant for the CI, inspect script & adapt accordingly
  237. build-slate:
  238. bash scripts/slate.sh
  239. # To avoid unintended conflicts with file names, always add to .PHONY
  240. # unless there is a reason not to.
  241. # https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html
  242. .PHONY: check build build_race build_abci dist install install_abci check_dep check_tools get_tools update_tools get_vendor_deps draw_deps get_protoc protoc_abci protoc_libs gen_certs clean_certs grpc_dbserver test_cover test_apps test_persistence test_p2p test test_race test_integrations test_release test100 vagrant_test fmt rpc-docs build-linux localnet-start localnet-stop build-docker build-docker-localnode sentry-start sentry-config sentry-stop build-slate protoc_grpc protoc_all build_c install_c test_with_deadlock cleanup_after_test_with_deadlock lint