* libs/common: Refactor libs/common 5
- move mathematical functions and types out of `libs/common` to math pkg
- move net functions out of `libs/common` to net pkg
- move string functions out of `libs/common` to strings pkg
- move async functions out of `libs/common` to async pkg
- move bit functions out of `libs/common` to bits pkg
- move cmap functions out of `libs/common` to cmap pkg
- move os functions out of `libs/common` to os pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix testing issues
* fix tests
closes#41417
woooooooooooooooooo kill the cmn pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* fix goimport issues
* run gofmt
- tm-bench has a deprecation warning for 5 releases now, with the major release coming I removed the file and updated the docs to point to `tm-load-test` located in the interchainio repo
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
https://www.jsonrpc.org/specification
What is done in this PR:
JSONRPCClient: validate that Response.ID matches Request.ID I wanted
to do the same for the WSClient, but since we're sending events as
responses, not notifications, checking IDs would require storing
them in memory indefinitely (and we won't be able to remove them
upon client unsubscribing because ID is different then).
Request.ID is now optional. Notification is a Request without an ID.
Previously "" or 0 were considered as notifications
Remove #event suffix from ID from an event response (partially fixes
#2949) ID must be either string, int or null AND must be equal to
request's ID. Now, because we've implemented events as responses, WS
clients are tripping when they see Response.ID("0#event") !=
Request.ID("0"). Implementing events as requests would require a lot
of time (~ 2 days to completely rewrite WS client and server)
generate unique ID for each request
switch to integer IDs instead of "json-client-XYZ"
id=0 method=/subscribe
id=0 result=...
id=1 method=/abci_query
id=1 result=...
> send events (resulting from /subscribe) as requests+notifications (not
responses)
this will require a lot of work. probably not worth it
* rpc: generate an unique ID for each request
in conformance with JSON-RPC spec
* WSClient: check for unsolicited responses
* fix golangci warnings
* save commit
* fix errors
* remove ID from responses from subscribe
Refs #2949
* clients are safe for concurrent access
* tm-bench: switch to int ID
* fixes after my own review
* comment out sentIDs in WSClient
see commit body for the reason
* remove body.Close
it will be closed automatically
* stop ws connection outside of write/read routines
also, use t.Rate in tm-bench indexer when calculating ID
fix gocritic issues
* update swagger.yaml
* Apply suggestions from code review
* fix stylecheck and golint linter warnings
* update changelog
* update changelog2
## Issue:
Removed BlockMeta in ResultBlock in favor of BlockId for /block
Added block_size to BlockMeta this is reflected in /blockchain
fixes#3188
added breaking as some clients may be using header from blockmeta instead of block in /block
## Commits:
* cleanup block path
Remove duplication of data in `/block`
fixes#3188
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* Remove duplication of data in `/block`
- Created a secondary type to be used for /block
fixes#3188
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* remove commented out code
* fix ci
* add changelog_pending entry
* remove extra variable
* update swagger
* change int to uint for blocksize
* fix swagger
* remove extensive comments
* update changelog
* fix conflicts after merge
* use int for BlockSize and NumTxs in BlockMeta
- with 99.9% guarantee, the size of either will never reach int32
- most of the Go "Size" stdlib functions return int
* Removal of TotalTx & NumTx
- Removed totalTx and numTx
closes#2521
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* abci proto changes
* proto number fix
* txfilter_test fix
* comments on PR
* further changes
* bring back metrics
* fix indexer
* fix TestBlockMaxDataBytes and TestBlockMaxDataBytesUnknownEvidence
* indexer service back to header
* statistics.go fix
* fix ci
* listen for blocks, not headers
to be able to record txs throughput
* fix TestNetworkNewBlock
* fix tests
* fix tests in types package
* fixes after Anton's review
* fix tests
* bring back `consensus_total_txs` metric
I mistakenly thought it was removed.
* improve changelog
* remove LastBlockTotalTx from state
* docs: remove getNumTxs from BeginBlock Java example
* Fix long line errors in abci, crypto, and libs packages
* Fix long lines in p2p and rpc packages
* Fix long lines in abci, state, and tools packages
* Fix long lines in behaviour and blockchain packages
* Fix long lines in cmd and config packages
* Begin fixing long lines in consensus package
* Finish fixing long lines in consensus package
* Add lll exclusion for lines containing URLs
* Fix long lines in crypto package
* Fix long lines in evidence package
* Fix long lines in mempool and node packages
* Fix long lines in libs package
* Fix long lines in lite package
* Fix new long line in node package
* Fix long lines in p2p package
* Ignore gocritic warning
* Fix long lines in privval package
* Fix long lines in rpc package
* Fix long lines in scripts package
* Fix long lines in state package
* Fix long lines in tools package
* Fix long lines in types package
* Enable lll linter
* New lint version upgrade
- linter was upgraded
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* enable-a;; is deprecated
* minor change
* another try
* some more changes
* some more changes
* reenable prealloc
* add version till bot is fixed
* Remove traces oaf `github.com/tendermint/abci`
- removed abci dockerfile as it was still referencing `github.com/tendermint/abci`
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* nor change to install of abci
* use abci-cli instead of tendermint node
* remove traces of Dockerfile.develop
also use latest Go in Dockerfile.testing
* update docker readme
* remove wrapping because it will look awful on docker hub
* tm-monitor: tweaked formatting of start time and avg tx throughput.
* tm-monitor: update health when validator number is updated.
* Updated CHANGELOG_PENDING
* Added PR number to CHANGELOG_PENDING.
Improves `tm-monitor` formatting of start time (RFC1123 without unnecessary precision) and avg tx throughput (three decimal places). The old tx throughput display was confusing during local testing where the tx rate is low and displayed as 0.
Also updates the monitor health whenever the validator number changes. It otherwise starts with moderate health and fails to update this once it discovers the validators, leading to incorrect health reporting and invalid uptime statistics. Let me know if you would like me to submit this as a separate PR.
### Before:
```
2019-09-29 20:40:00.992834 +0200 CEST m=+0.024057059 up -92030989600.42%
Height: 2518
Avg block time: 1275.496 ms
Avg tx throughput: 0 per sec
Avg block latency: 2.464 ms
Active nodes: 4/4 (health: moderate) Validators: 4
NAME HEIGHT BLOCK LATENCY ONLINE VALIDATOR
localhost:26657 2518 0.935 ms true true
localhost:26660 2518 0.710 ms true true
localhost:26662 2518 0.708 ms true true
localhost:26664 2518 0.717 ms true true
```
### After:
```
Sun, 29 Sep 2019 20:21:59 +0200 up 100.00%
Height: 2480
Avg block time: 1361.445 ms
Avg tx throughput: 0.735 per sec
Avg block latency: 4.232 ms
Active nodes: 4/4 (health: full) Validators: 4
NAME HEIGHT BLOCK LATENCY ONLINE VALIDATOR
localhost:26657 2480 1.174 ms true true
localhost:26660 2480 1.037 ms true true
localhost:26662 2480 0.981 ms true true
localhost:26664 2480 0.995 ms true true
```
- Added a deprecation warining in for deprecation of tm-bench in favor of tm-load-test
- With the merging of this pr we can close tm-bench related issues.
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* Remove unnecessary type conversions
* Consolidate repeated strings into consts
* Clothe return statements
* Update blockchain/v1/reactor_fsm_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
This PR repairs linter errors seen when running the following commands:
golangci-lint run --no-config --disable-all=true --enable=unconvert
golangci-lint run --no-config --disable-all=true --enable=goconst
golangci-lint run --no-config --disable-all=true --enable=nakedret
Contributes to #3262
This PR is related to #3107 and a continuation of #3351
It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction.
Given two hosts A and B:
Host A is listener/client
Host B is dialer/server (contains the secret key)
When A requires a signature, it needs to wait for B to dial in before it can issue a request.
A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect.
The original rationale behind this design was based on security.
Host B only allows outbound connections to a list of whitelisted hosts.
It is not possible to reach B unless B dials in. There are no listening/open ports in B.
This PR results in the following changes:
Refactors ping/heartbeat to avoid previously existing race conditions.
Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow.
Unifies and abstracts away the differences between unix and tcp sockets.
A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj)
The signer request handler (server side) is customizable to increase testability.
Updates and extends unit tests
A high level overview of the classes is as follows:
Transport (endpoints): The following classes take care of establishing a connection
SignerDialerEndpoint
SignerListeningEndpoint
SignerEndpoint groups common functionality (read/write/timeouts/etc.)
Signing (client/server): The following classes take care of exchanging request/responses
SignerClient
SignerServer
This PR also closes#3601
Commits:
* refactoring - work in progress
* reworking unit tests
* Encapsulating and fixing unit tests
* Improve tests
* Clean up
* Fix/improve unit tests
* clean up tests
* Improving service endpoint
* fixing unit test
* fix linter issues
* avoid invalid cache values (improve later?)
* complete implementation
* wip
* improved connection loop
* Improve reconnections + fixing unit tests
* addressing comments
* small formatting changes
* clean up
* Update node/node.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client_test.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* check during initialization
* dropping connecting when writing fails
* removing break
* use t.log instead
* unifying and using cmn.GetFreePort()
* review fixes
* reordering and unifying drop connection
* closing instead of signalling
* refactored service loop
* removed superfluous brackets
* GetPubKey can return errors
* Revert "GetPubKey can return errors"
This reverts commit 68c06f19b4.
* adding entry to changelog
* Update CHANGELOG_PENDING.md
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_listener_endpoint_test.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* updating node.go
* review fixes
* fixes linter
* fixing unit test
* small fixes in comments
* addressing review comments
* addressing review comments 2
* reverting suggestion
* Update privval/signer_client_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update privval/signer_client_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update privval/signer_listener_endpoint_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* do not expose brokenSignerDialerEndpoint
* clean up logging
* unifying methods
shorten test time
signer also drops
* reenabling pings
* improving testability + unit test
* fixing go fmt + unit test
* remove unused code
* Addressing review comments
* simplifying connection workflow
* fix linter/go import issue
* using base service quit
* updating comment
* Simplifying design + adjusting names
* fixing linter issues
* refactoring test harness + fixes
* Addressing review comments
* cleaning up
* adding additional error check
cleanup to add linter
grpc change:
https://godoc.org/google.golang.org/grpc#WithContextDialerhttps://godoc.org/google.golang.org/grpc#WithDialer
grpc/grpc-go#2627
prometheous change:
due to UninstrumentedHandler, being deprecated in the future
empty branch = empty if or else statement
didn't delete them entirely but commented
couldn't find a reason to have them
could not replicate the issue #3406
but if want to keep it commented then we should comment out the if statement as well
* Renamed wire.go to codec.go
- Wire was the previous name of amino
- Codec describes the file better than `wire` & `amino`
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* ide error
* rename amino.go to codec.go
* Fix rpc handle for tm-monitor
Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>
* go imports file
Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>
* go imports file
Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>
* fix RPCUnmonitor too
Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>
* Update to using go mod from dep
* Remove references to make get_vendor_deps
* Specify go version
* Set GO111MODULE=on and add -mod=readonly
* Fix exported env
* switch to using go1.12 everywhere
* Fix test scripts
* Typo:
* Prepend GO111MODULE=on
* remove dep cache
* Revert "remove dep cache"
This reverts commit 45117bda
Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* bring back the dependency cache and change it to cache modules instead
of vendored deps; also:
- bump version for dependency cache
- bump version on pkg-cache (includes modules directory)
Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* remove some more traces of dep:
- remove Gopkg.(toml | lock)
- update contributing guidlines
- set global default in circleci (GO111MODULE=on)
Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* global var failed for `test_cover` with
`go: unknown environment setting GO111MODULE=true`
although the var was `GO111MODULE: on`
Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* Changelog pending entry
Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* Add bbolt dependency to go.mod
Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* move -mod=readonly to build flags
In order to re-enable the test harness for the KMS (see
tendermint/kms#227), we need some marginally more realistic proposals
and votes. This is because the KMS does some additional sanity checks
now to ensure the height and round are increasing over time.
* Make sure config.TimeoutBroadcastTxCommit < rpcserver.WriteTimeout()
* remove redundant comment
* libs/rpc/http_server: move Read/WriteTimeout into Config
* increase defaults for read/write timeouts
Based on this article
https://www.digitalocean.com/community/tutorials/how-to-optimize-nginx-configuration
* WriteTimeout should be larger than TimeoutBroadcastTxCommit
* set a deadline for subscribing to txs
* extract duration into const
* add two changelog entries
* Update CHANGELOG_PENDING.md
Co-Authored-By: melekes <anton.kalyaev@gmail.com>
* Update CHANGELOG_PENDING.md
Co-Authored-By: melekes <anton.kalyaev@gmail.com>
* 12 -> 10
* changelog
* changelog
* make BlockTimeIota a consensus parameter, not a locally configurable option
Refs #2920
* make TimeIota int64 ms
Refs #2920
* update Gopkg.toml
* fixes after Ethan's review
* fix TestRemoteSignerProposalSigningFailed
* update changelog
This issue is related to #3107
This is a first renaming/refactoring step before reworking and removing heartbeats.
As discussed with @Liamsi , we preferred to go for a couple of independent and separate PRs to simplify review work.
The changes:
Help to clarify the relation between the validator and remote signer endpoints
Differentiate between timeouts and deadlines
Prepare to encapsulate networking related code behind RemoteSigner in the next PR
My intention is to separate and encapsulate the "network related" code from the actual signer.
SignerRemote ---(uses/contains)--> SignerValidatorEndpoint <--(connects to)--> SignerServiceEndpoint ---> SignerService (future.. not here yet but would like to decouple too)
All reconnection/heartbeat/whatever code goes in the endpoints. Signer[Remote/Service] do not need to know about that.
I agree Endpoint may not be the perfect name. I tried to find something "Go-ish" enough. It is a common name in go-kit, kubernetes, etc.
Right now:
SignerValidatorEndpoint:
handles the listener
contains SignerRemote
Implements the PrivValidator interface
connects and sets a connection object in a contained SignerRemote
delegates PrivValidator some calls to SignerRemote which in turn uses the conn object that was set externally
SignerRemote:
Implements the PrivValidator interface
read/writes from a connection object directly
handles heartbeats
SignerServiceEndpoint:
Does most things in a single place
delegates to a PrivValidator IIRC.
* cleanup
* Refactoring step 1
* Refactoring step 2
* move messages to another file
* mark for future work / next steps
* mark deprecated classes in docs
* Fix linter problems
* additional linter fixes
* libs/common: TrapSignal accepts logger as a first parameter
and does not block anymore
* previously it was dumping "captured ..." msg to os.Stdout
* TrapSignal should not be responsible for blocking thread of execution
Refs #3238
* exit with zero (0) code upon receiving SIGTERM/SIGINT
Refs #3238
* fix formatting in docs/app-dev/abci-cli.md
Co-Authored-By: melekes <anton.kalyaev@gmail.com>
* fix formatting in docs/app-dev/abci-cli.md
Co-Authored-By: melekes <anton.kalyaev@gmail.com>
* not related to linter: remove obsolete constants:
- `Insecure` and `Secure` and type `Security` are not used anywhere
* not related to linter: update example
- NewInsecure was deleted; change example to NewRemoteDB
* address: Binds to all network interfaces (gosec):
- bind to localhost instead of 0.0.0.0
- regenerate test key and cert for this purpose (was valid for ::) and
otherwise we would see:
transport: authentication handshake failed: x509: certificate is
valid for ::, not 127.0.0.1\"
(used https://github.com/google/keytransparency/blob/master/scripts/gen_server_keys.sh
to regenerate certs)
* use sha256 in tests instead of md5; time difference is negligible
* nolint usage of math/rand in test and add comment on its import
- crypto/rand is slower and we do not need sth more secure in tests
* enable linter in circle-ci
* another nolint math/rand in test
* replace another occurrence of md5
* consistent comment about importing math/rand
* WIP: Starts adding remote signer test harness
This commit adds a new command to Tendermint to allow for us to build a
standalone binary to test remote signers such as KMS
(https://github.com/tendermint/kms).
Right now, all it does is test that the local public key matches the
public key reported by the client, and fails at the point where it
attempts to get the client to sign a proposal.
* Fixes typo
* Fixes proposal validation test
This commit fixes the proposal validation test as per #3149. It also
moves the test harness into its own internal package to isolate its
exports from the `privval` package.
* Adds vote signing validation
* Applying recommendations from #3149
* Adds function descriptions for test harness
* Adds ability to ask remote signer to shut down
Prior to this commit, the remote signer needs to manually be shut down,
which is not ideal for automated testing. This commit allows us to send
a poison pill message to the KMS to let it shut down gracefully once
testing is done (whether the tests pass or fail).
* Adds tests for remote signer test harness
This commit makes some minor modifications to a few files to allow for
testing of the remote signer test harness. Two tests are added here:
checking for a fully successful (the ideal) case, and for the case where
the maximum number of retries has been reached when attempting to accept
incoming connections from the remote signer.
* Condenses serialization of proposals and votes using existing Tendermint functions
* Removes now-unnecessary amino import and codec
* Adds error message for vote signing failure
* Adds key extraction command for integration test
Took the code from here:
https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to
create a simple utility command to extract a key from a local Tendermint
validator for use in KMS integration testing.
* Makes path expansion success non-compulsory
* Fixes segfault on SIGTERM
We need an additional variable to keep track of whether we're
successfully connected, otherwise hitting Ctrl+Break during execution
causes a segmentation fault. This now allows for a clean shutdown.
* Consolidates shutdown checks
* Adds comments indicating codes for easy lookup
* Adds Docker build for remote signer harness
Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to
override the image name and Dockerfile using environment variables.
Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow
for building the `remote_val_harness` Docker image.
* Adds build_remote_val_harness_docker_image to .PHONY
* Removes remote signer poison pill messaging functionality
* Reduces fluff code in command line parsing
As per
https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788,
this reduces the amount of fluff code in the PR down to the bare
minimum.
* Fixes ordering of error check and info log
* Moves remove_val_harness cmd into tools folder
It seems to make sense to rather keep the remote signer test harness in
its own tool folder (now rather named `tm-signer-harness` to keep with
the tool naming convention). It is actually a separate tool, not meant
to be one of the core binaries, but supplementary and supportive.
* Updates documentation for tm-signer-harness
* Refactors flag parsing to be more compact and less redundant
* Adds version sub-command help
* Removes extraneous flags parsing
* Adds CHANGELOG_PENDING entry for tm-signer-harness
* Improves test coverage
Adds a few extra parameters to the `MockPV` type to fake broken vote and
proposal signing. Also adds some more tests for the test harness so as
to increase coverage for failed cases.
* Fixes formatting for CHANGELOG_PENDING.md
* Fix formatting for documentation config
* Point users towards official Tendermint docs for tools documentation
* Point users towards official Tendermint docs for tm-signer-harness
* Remove extraneous constant
* Rename TestHarness.sc to TestHarness.spv for naming consistency
* Refactor to remove redundant goroutine
* Refactor conditional to cleaner switch statement and better error handling for listener protocol
* Remove extraneous goroutine
* Add note about installing tmkms via Cargo
* Fix typo in naming of output signing key
* Add note about where to find chain ID
* Replace /home/user with ~/ for brevity
* Fixes "signer.key" typo
* Minor edits for clarification for tm-signer-harness bulid/setup process
* Fixed accepting integer IDs in requests for Tendermint RPC server (#2366)
* added a wrapper interface `jsonrpcid` that represents both string and int IDs in JSON-RPC requests/responses + custom JSON unmarshallers
* changed client-side code in RPC that uses it
* added extra tests for integer IDs
* updated CHANGELOG_PENDING, as suggested by PR instructions
* addressed PR comments
* added table driven tests for request type marshalling/unmarshalling
* expanded handler test to check IDs
* changed pending changelog note
* changed json rpc request/response unmarshalling to use empty interfaces and type switches on ID
* some cleanup
* Decouple StartHTTP{,AndTLS}Server from Listen()
This should help solve cosmos/cosmos-sdk#2715
* Fix small mistake
* Update StartGRPCServer
* s/rpc/rpcserver/
* Start grpccore.StartGRPCServer in a goroutine
* Reinstate l.Close()
* Fix rpc/lib/test/main.go
* Update code comment
* update changelog and comments
* fix tm-monitor. more comments
Currently the top level directory contains basically all of the code
for the crypto package. This PR moves the crypto code into submodules
in a similar manner to what `golang/x/crypto` does. This improves code
organization.
Ref discussion: https://github.com/tendermint/tendermint/pull/1966Closes#1956
This PR moves statistics to its own file, seperates getBlockMetas
into its own function, and removes the timeEnd parameter from
calculate statistics. The ending time is now computed directly from
the start time and the duration, to enforce that we only collect
data for the provided duration.
Marko
dongsamb
Leslie Cheung
Anton Kaliaev
Phil Salant
Erik Grinaker
Juan Leni
Karoly Albert Szabo
Jack Zampolin
Thane Thomson
Ismail Khoffi
Juan Leni
Dev Ojha
Jernej Kos
Tomas Tauber
Alessio Treglia
zhangzheng
bradyjoestar
ValarDragon