You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

427 lines
13 KiB

Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: improve Remote Signer implementation (#3351) This issue is related to #3107 This is a first renaming/refactoring step before reworking and removing heartbeats. As discussed with @Liamsi , we preferred to go for a couple of independent and separate PRs to simplify review work. The changes: Help to clarify the relation between the validator and remote signer endpoints Differentiate between timeouts and deadlines Prepare to encapsulate networking related code behind RemoteSigner in the next PR My intention is to separate and encapsulate the "network related" code from the actual signer. SignerRemote ---(uses/contains)--> SignerValidatorEndpoint <--(connects to)--> SignerServiceEndpoint ---> SignerService (future.. not here yet but would like to decouple too) All reconnection/heartbeat/whatever code goes in the endpoints. Signer[Remote/Service] do not need to know about that. I agree Endpoint may not be the perfect name. I tried to find something "Go-ish" enough. It is a common name in go-kit, kubernetes, etc. Right now: SignerValidatorEndpoint: handles the listener contains SignerRemote Implements the PrivValidator interface connects and sets a connection object in a contained SignerRemote delegates PrivValidator some calls to SignerRemote which in turn uses the conn object that was set externally SignerRemote: Implements the PrivValidator interface read/writes from a connection object directly handles heartbeats SignerServiceEndpoint: Does most things in a single place delegates to a PrivValidator IIRC. * cleanup * Refactoring step 1 * Refactoring step 2 * move messages to another file * mark for future work / next steps * mark deprecated classes in docs * Fix linter problems * additional linter fixes
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: improve Remote Signer implementation (#3351) This issue is related to #3107 This is a first renaming/refactoring step before reworking and removing heartbeats. As discussed with @Liamsi , we preferred to go for a couple of independent and separate PRs to simplify review work. The changes: Help to clarify the relation between the validator and remote signer endpoints Differentiate between timeouts and deadlines Prepare to encapsulate networking related code behind RemoteSigner in the next PR My intention is to separate and encapsulate the "network related" code from the actual signer. SignerRemote ---(uses/contains)--> SignerValidatorEndpoint <--(connects to)--> SignerServiceEndpoint ---> SignerService (future.. not here yet but would like to decouple too) All reconnection/heartbeat/whatever code goes in the endpoints. Signer[Remote/Service] do not need to know about that. I agree Endpoint may not be the perfect name. I tried to find something "Go-ish" enough. It is a common name in go-kit, kubernetes, etc. Right now: SignerValidatorEndpoint: handles the listener contains SignerRemote Implements the PrivValidator interface connects and sets a connection object in a contained SignerRemote delegates PrivValidator some calls to SignerRemote which in turn uses the conn object that was set externally SignerRemote: Implements the PrivValidator interface read/writes from a connection object directly handles heartbeats SignerServiceEndpoint: Does most things in a single place delegates to a PrivValidator IIRC. * cleanup * Refactoring step 1 * Refactoring step 2 * move messages to another file * mark for future work / next steps * mark deprecated classes in docs * Fix linter problems * additional linter fixes
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
Add remote signer test harness (KMS) (#3149) * WIP: Starts adding remote signer test harness This commit adds a new command to Tendermint to allow for us to build a standalone binary to test remote signers such as KMS (https://github.com/tendermint/kms). Right now, all it does is test that the local public key matches the public key reported by the client, and fails at the point where it attempts to get the client to sign a proposal. * Fixes typo * Fixes proposal validation test This commit fixes the proposal validation test as per #3149. It also moves the test harness into its own internal package to isolate its exports from the `privval` package. * Adds vote signing validation * Applying recommendations from #3149 * Adds function descriptions for test harness * Adds ability to ask remote signer to shut down Prior to this commit, the remote signer needs to manually be shut down, which is not ideal for automated testing. This commit allows us to send a poison pill message to the KMS to let it shut down gracefully once testing is done (whether the tests pass or fail). * Adds tests for remote signer test harness This commit makes some minor modifications to a few files to allow for testing of the remote signer test harness. Two tests are added here: checking for a fully successful (the ideal) case, and for the case where the maximum number of retries has been reached when attempting to accept incoming connections from the remote signer. * Condenses serialization of proposals and votes using existing Tendermint functions * Removes now-unnecessary amino import and codec * Adds error message for vote signing failure * Adds key extraction command for integration test Took the code from here: https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to create a simple utility command to extract a key from a local Tendermint validator for use in KMS integration testing. * Makes path expansion success non-compulsory * Fixes segfault on SIGTERM We need an additional variable to keep track of whether we're successfully connected, otherwise hitting Ctrl+Break during execution causes a segmentation fault. This now allows for a clean shutdown. * Consolidates shutdown checks * Adds comments indicating codes for easy lookup * Adds Docker build for remote signer harness Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to override the image name and Dockerfile using environment variables. Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow for building the `remote_val_harness` Docker image. * Adds build_remote_val_harness_docker_image to .PHONY * Removes remote signer poison pill messaging functionality * Reduces fluff code in command line parsing As per https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788, this reduces the amount of fluff code in the PR down to the bare minimum. * Fixes ordering of error check and info log * Moves remove_val_harness cmd into tools folder It seems to make sense to rather keep the remote signer test harness in its own tool folder (now rather named `tm-signer-harness` to keep with the tool naming convention). It is actually a separate tool, not meant to be one of the core binaries, but supplementary and supportive. * Updates documentation for tm-signer-harness * Refactors flag parsing to be more compact and less redundant * Adds version sub-command help * Removes extraneous flags parsing * Adds CHANGELOG_PENDING entry for tm-signer-harness * Improves test coverage Adds a few extra parameters to the `MockPV` type to fake broken vote and proposal signing. Also adds some more tests for the test harness so as to increase coverage for failed cases. * Fixes formatting for CHANGELOG_PENDING.md * Fix formatting for documentation config * Point users towards official Tendermint docs for tools documentation * Point users towards official Tendermint docs for tm-signer-harness * Remove extraneous constant * Rename TestHarness.sc to TestHarness.spv for naming consistency * Refactor to remove redundant goroutine * Refactor conditional to cleaner switch statement and better error handling for listener protocol * Remove extraneous goroutine * Add note about installing tmkms via Cargo * Fix typo in naming of output signing key * Add note about where to find chain ID * Replace /home/user with ~/ for brevity * Fixes "signer.key" typo * Minor edits for clarification for tm-signer-harness bulid/setup process
6 years ago
  1. package internal
  2. import (
  3. "bytes"
  4. "context"
  5. "fmt"
  6. "net"
  7. "os"
  8. "os/signal"
  9. "time"
  10. "github.com/tendermint/tendermint/crypto/tmhash"
  11. "github.com/tendermint/tendermint/crypto/ed25519"
  12. "github.com/tendermint/tendermint/privval"
  13. "github.com/tendermint/tendermint/state"
  14. "github.com/tendermint/tendermint/libs/log"
  15. tmnet "github.com/tendermint/tendermint/libs/net"
  16. tmos "github.com/tendermint/tendermint/libs/os"
  17. tmproto "github.com/tendermint/tendermint/proto/tendermint/types"
  18. "github.com/tendermint/tendermint/types"
  19. )
  20. // Test harness error codes (which act as exit codes when the test harness fails).
  21. const (
  22. NoError int = iota // 0
  23. ErrInvalidParameters // 1
  24. ErrMaxAcceptRetriesReached // 2
  25. ErrFailedToLoadGenesisFile // 3
  26. ErrFailedToCreateListener // 4
  27. ErrFailedToStartListener // 5
  28. ErrInterrupted // 6
  29. ErrOther // 7
  30. ErrTestPublicKeyFailed // 8
  31. ErrTestSignProposalFailed // 9
  32. ErrTestSignVoteFailed // 10
  33. )
  34. var voteTypes = []tmproto.SignedMsgType{tmproto.PrevoteType, tmproto.PrecommitType}
  35. // TestHarnessError allows us to keep track of which exit code should be used
  36. // when exiting the main program.
  37. type TestHarnessError struct {
  38. Code int // The exit code to return
  39. Err error // The original error
  40. Info string // Any additional information
  41. }
  42. var _ error = (*TestHarnessError)(nil)
  43. // TestHarness allows for testing of a remote signer to ensure compatibility
  44. // with this version of Tendermint.
  45. type TestHarness struct {
  46. addr string
  47. signerClient *privval.SignerClient
  48. fpv *privval.FilePV
  49. chainID string
  50. acceptRetries int
  51. logger log.Logger
  52. exitWhenComplete bool
  53. exitCode int
  54. }
  55. // TestHarnessConfig provides configuration to set up a remote signer test
  56. // harness.
  57. type TestHarnessConfig struct {
  58. BindAddr string
  59. KeyFile string
  60. StateFile string
  61. GenesisFile string
  62. AcceptDeadline time.Duration
  63. ConnDeadline time.Duration
  64. AcceptRetries int
  65. SecretConnKey ed25519.PrivKey
  66. ExitWhenComplete bool // Whether or not to call os.Exit when the harness has completed.
  67. }
  68. // timeoutError can be used to check if an error returned from the netp package
  69. // was due to a timeout.
  70. type timeoutError interface {
  71. Timeout() bool
  72. }
  73. // NewTestHarness will load Tendermint data from the given files (including
  74. // validator public/private keypairs and chain details) and create a new
  75. // harness.
  76. func NewTestHarness(logger log.Logger, cfg TestHarnessConfig) (*TestHarness, error) {
  77. keyFile := ExpandPath(cfg.KeyFile)
  78. stateFile := ExpandPath(cfg.StateFile)
  79. logger.Info("Loading private validator configuration", "keyFile", keyFile, "stateFile", stateFile)
  80. // NOTE: LoadFilePV ultimately calls os.Exit on failure. No error will be
  81. // returned if this call fails.
  82. fpv, err := privval.LoadFilePV(keyFile, stateFile)
  83. if err != nil {
  84. return nil, err
  85. }
  86. genesisFile := ExpandPath(cfg.GenesisFile)
  87. logger.Info("Loading chain ID from genesis file", "genesisFile", genesisFile)
  88. st, err := state.MakeGenesisDocFromFile(genesisFile)
  89. if err != nil {
  90. return nil, newTestHarnessError(ErrFailedToLoadGenesisFile, err, genesisFile)
  91. }
  92. logger.Info("Loaded genesis file", "chainID", st.ChainID)
  93. spv, err := newTestHarnessListener(logger, cfg)
  94. if err != nil {
  95. return nil, newTestHarnessError(ErrFailedToCreateListener, err, "")
  96. }
  97. signerClient, err := privval.NewSignerClient(spv, st.ChainID)
  98. if err != nil {
  99. return nil, newTestHarnessError(ErrFailedToCreateListener, err, "")
  100. }
  101. return &TestHarness{
  102. addr: cfg.BindAddr,
  103. signerClient: signerClient,
  104. fpv: fpv,
  105. chainID: st.ChainID,
  106. acceptRetries: cfg.AcceptRetries,
  107. logger: logger,
  108. exitWhenComplete: cfg.ExitWhenComplete,
  109. exitCode: 0,
  110. }, nil
  111. }
  112. // Run will execute the tests associated with this test harness. The intention
  113. // here is to call this from one's `main` function, as the way it succeeds or
  114. // fails at present is to call os.Exit() with an exit code related to the error
  115. // that caused the tests to fail, or exit code 0 on success.
  116. func (th *TestHarness) Run() {
  117. c := make(chan os.Signal, 1)
  118. signal.Notify(c, os.Interrupt)
  119. go func() {
  120. for sig := range c {
  121. th.logger.Info("Caught interrupt, terminating...", "sig", sig)
  122. th.Shutdown(newTestHarnessError(ErrInterrupted, nil, ""))
  123. }
  124. }()
  125. th.logger.Info("Starting test harness")
  126. accepted := false
  127. var startErr error
  128. for acceptRetries := th.acceptRetries; acceptRetries > 0; acceptRetries-- {
  129. th.logger.Info("Attempting to accept incoming connection", "acceptRetries", acceptRetries)
  130. if err := th.signerClient.WaitForConnection(10 * time.Millisecond); err != nil {
  131. // if it wasn't a timeout error
  132. if _, ok := err.(timeoutError); !ok {
  133. th.logger.Error("Failed to start listener", "err", err)
  134. th.Shutdown(newTestHarnessError(ErrFailedToStartListener, err, ""))
  135. // we need the return statements in case this is being run
  136. // from a unit test - otherwise this function will just die
  137. // when os.Exit is called
  138. return
  139. }
  140. startErr = err
  141. } else {
  142. th.logger.Info("Accepted external connection")
  143. accepted = true
  144. break
  145. }
  146. }
  147. if !accepted {
  148. th.logger.Error("Maximum accept retries reached", "acceptRetries", th.acceptRetries)
  149. th.Shutdown(newTestHarnessError(ErrMaxAcceptRetriesReached, startErr, ""))
  150. return
  151. }
  152. // Run the tests
  153. if err := th.TestPublicKey(); err != nil {
  154. th.Shutdown(err)
  155. return
  156. }
  157. if err := th.TestSignProposal(); err != nil {
  158. th.Shutdown(err)
  159. return
  160. }
  161. if err := th.TestSignVote(); err != nil {
  162. th.Shutdown(err)
  163. return
  164. }
  165. th.logger.Info("SUCCESS! All tests passed.")
  166. th.Shutdown(nil)
  167. }
  168. // TestPublicKey just validates that we can (1) fetch the public key from the
  169. // remote signer, and (2) it matches the public key we've configured for our
  170. // local Tendermint version.
  171. func (th *TestHarness) TestPublicKey() error {
  172. th.logger.Info("TEST: Public key of remote signer")
  173. fpvk, err := th.fpv.GetPubKey(context.Background())
  174. if err != nil {
  175. return err
  176. }
  177. th.logger.Info("Local", "pubKey", fpvk)
  178. sck, err := th.signerClient.GetPubKey(context.Background())
  179. if err != nil {
  180. return err
  181. }
  182. th.logger.Info("Remote", "pubKey", sck)
  183. if !bytes.Equal(fpvk.Bytes(), sck.Bytes()) {
  184. th.logger.Error("FAILED: Local and remote public keys do not match")
  185. return newTestHarnessError(ErrTestPublicKeyFailed, nil, "")
  186. }
  187. return nil
  188. }
  189. // TestSignProposal makes sure the remote signer can successfully sign
  190. // proposals.
  191. func (th *TestHarness) TestSignProposal() error {
  192. th.logger.Info("TEST: Signing of proposals")
  193. // sha256 hash of "hash"
  194. hash := tmhash.Sum([]byte("hash"))
  195. prop := &types.Proposal{
  196. Type: tmproto.ProposalType,
  197. Height: 100,
  198. Round: 0,
  199. POLRound: -1,
  200. BlockID: types.BlockID{
  201. Hash: hash,
  202. PartSetHeader: types.PartSetHeader{
  203. Hash: hash,
  204. Total: 1000000,
  205. },
  206. },
  207. Timestamp: time.Now(),
  208. }
  209. p := prop.ToProto()
  210. propBytes := types.ProposalSignBytes(th.chainID, p)
  211. if err := th.signerClient.SignProposal(context.Background(), th.chainID, p); err != nil {
  212. th.logger.Error("FAILED: Signing of proposal", "err", err)
  213. return newTestHarnessError(ErrTestSignProposalFailed, err, "")
  214. }
  215. prop.Signature = p.Signature
  216. th.logger.Debug("Signed proposal", "prop", prop)
  217. // first check that it's a basically valid proposal
  218. if err := prop.ValidateBasic(); err != nil {
  219. th.logger.Error("FAILED: Signed proposal is invalid", "err", err)
  220. return newTestHarnessError(ErrTestSignProposalFailed, err, "")
  221. }
  222. sck, err := th.signerClient.GetPubKey(context.Background())
  223. if err != nil {
  224. return err
  225. }
  226. // now validate the signature on the proposal
  227. if sck.VerifySignature(propBytes, prop.Signature) {
  228. th.logger.Info("Successfully validated proposal signature")
  229. } else {
  230. th.logger.Error("FAILED: Proposal signature validation failed")
  231. return newTestHarnessError(ErrTestSignProposalFailed, nil, "signature validation failed")
  232. }
  233. return nil
  234. }
  235. // TestSignVote makes sure the remote signer can successfully sign all kinds of
  236. // votes.
  237. func (th *TestHarness) TestSignVote() error {
  238. th.logger.Info("TEST: Signing of votes")
  239. for _, voteType := range voteTypes {
  240. th.logger.Info("Testing vote type", "type", voteType)
  241. hash := tmhash.Sum([]byte("hash"))
  242. vote := &types.Vote{
  243. Type: voteType,
  244. Height: 101,
  245. Round: 0,
  246. BlockID: types.BlockID{
  247. Hash: hash,
  248. PartSetHeader: types.PartSetHeader{
  249. Hash: hash,
  250. Total: 1000000,
  251. },
  252. },
  253. ValidatorIndex: 0,
  254. ValidatorAddress: tmhash.SumTruncated([]byte("addr")),
  255. Timestamp: time.Now(),
  256. }
  257. v := vote.ToProto()
  258. voteBytes := types.VoteSignBytes(th.chainID, v)
  259. // sign the vote
  260. if err := th.signerClient.SignVote(context.Background(), th.chainID, v); err != nil {
  261. th.logger.Error("FAILED: Signing of vote", "err", err)
  262. return newTestHarnessError(ErrTestSignVoteFailed, err, fmt.Sprintf("voteType=%d", voteType))
  263. }
  264. vote.Signature = v.Signature
  265. th.logger.Debug("Signed vote", "vote", vote)
  266. // validate the contents of the vote
  267. if err := vote.ValidateBasic(); err != nil {
  268. th.logger.Error("FAILED: Signed vote is invalid", "err", err)
  269. return newTestHarnessError(ErrTestSignVoteFailed, err, fmt.Sprintf("voteType=%d", voteType))
  270. }
  271. sck, err := th.signerClient.GetPubKey(context.Background())
  272. if err != nil {
  273. return err
  274. }
  275. // now validate the signature on the proposal
  276. if sck.VerifySignature(voteBytes, vote.Signature) {
  277. th.logger.Info("Successfully validated vote signature", "type", voteType)
  278. } else {
  279. th.logger.Error("FAILED: Vote signature validation failed", "type", voteType)
  280. return newTestHarnessError(ErrTestSignVoteFailed, nil, "signature validation failed")
  281. }
  282. }
  283. return nil
  284. }
  285. // Shutdown will kill the test harness and attempt to close all open sockets
  286. // gracefully. If the supplied error is nil, it is assumed that the exit code
  287. // should be 0. If err is not nil, it will exit with an exit code related to the
  288. // error.
  289. func (th *TestHarness) Shutdown(err error) {
  290. var exitCode int
  291. if err == nil {
  292. exitCode = NoError
  293. } else if therr, ok := err.(*TestHarnessError); ok {
  294. exitCode = therr.Code
  295. } else {
  296. exitCode = ErrOther
  297. }
  298. th.exitCode = exitCode
  299. // in case sc.Stop() takes too long
  300. if th.exitWhenComplete {
  301. go func() {
  302. time.Sleep(time.Duration(5) * time.Second)
  303. th.logger.Error("Forcibly exiting program after timeout")
  304. os.Exit(exitCode)
  305. }()
  306. }
  307. err = th.signerClient.Close()
  308. if err != nil {
  309. th.logger.Error("Failed to cleanly stop listener: %s", err.Error())
  310. }
  311. if th.exitWhenComplete {
  312. os.Exit(exitCode)
  313. }
  314. }
  315. // newTestHarnessListener creates our client instance which we will use for testing.
  316. func newTestHarnessListener(logger log.Logger, cfg TestHarnessConfig) (*privval.SignerListenerEndpoint, error) {
  317. proto, addr := tmnet.ProtocolAndAddress(cfg.BindAddr)
  318. if proto == "unix" {
  319. // make sure the socket doesn't exist - if so, try to delete it
  320. if tmos.FileExists(addr) {
  321. if err := os.Remove(addr); err != nil {
  322. logger.Error("Failed to remove existing Unix domain socket", "addr", addr)
  323. return nil, err
  324. }
  325. }
  326. }
  327. ln, err := net.Listen(proto, addr)
  328. if err != nil {
  329. return nil, err
  330. }
  331. logger.Info("Listening", "proto", proto, "addr", addr)
  332. var svln net.Listener
  333. switch proto {
  334. case "unix":
  335. unixLn := privval.NewUnixListener(ln)
  336. privval.UnixListenerTimeoutAccept(cfg.AcceptDeadline)(unixLn)
  337. privval.UnixListenerTimeoutReadWrite(cfg.ConnDeadline)(unixLn)
  338. svln = unixLn
  339. case "tcp":
  340. tcpLn := privval.NewTCPListener(ln, cfg.SecretConnKey)
  341. privval.TCPListenerTimeoutAccept(cfg.AcceptDeadline)(tcpLn)
  342. privval.TCPListenerTimeoutReadWrite(cfg.ConnDeadline)(tcpLn)
  343. logger.Info("Resolved TCP address for listener", "addr", tcpLn.Addr())
  344. svln = tcpLn
  345. default:
  346. _ = ln.Close()
  347. logger.Error("Unsupported protocol (must be unix:// or tcp://)", "proto", proto)
  348. return nil, newTestHarnessError(ErrInvalidParameters, nil, fmt.Sprintf("Unsupported protocol: %s", proto))
  349. }
  350. return privval.NewSignerListenerEndpoint(logger, svln), nil
  351. }
  352. func newTestHarnessError(code int, err error, info string) *TestHarnessError {
  353. return &TestHarnessError{
  354. Code: code,
  355. Err: err,
  356. Info: info,
  357. }
  358. }
  359. func (e *TestHarnessError) Error() string {
  360. var msg string
  361. switch e.Code {
  362. case ErrInvalidParameters:
  363. msg = "Invalid parameters supplied to application"
  364. case ErrMaxAcceptRetriesReached:
  365. msg = "Maximum accept retries reached"
  366. case ErrFailedToLoadGenesisFile:
  367. msg = "Failed to load genesis file"
  368. case ErrFailedToCreateListener:
  369. msg = "Failed to create listener"
  370. case ErrFailedToStartListener:
  371. msg = "Failed to start listener"
  372. case ErrInterrupted:
  373. msg = "Interrupted"
  374. case ErrTestPublicKeyFailed:
  375. msg = "Public key validation test failed"
  376. case ErrTestSignProposalFailed:
  377. msg = "Proposal signing validation test failed"
  378. case ErrTestSignVoteFailed:
  379. msg = "Vote signing validation test failed"
  380. default:
  381. msg = "Unknown error"
  382. }
  383. if len(e.Info) > 0 {
  384. msg = fmt.Sprintf("%s: %s", msg, e.Info)
  385. }
  386. if e.Err != nil {
  387. msg = fmt.Sprintf("%s (original error: %s)", msg, e.Err.Error())
  388. }
  389. return msg
  390. }