Add remote signer test harness (KMS) (#3149)

* WIP: Starts adding remote signer test harness

This commit adds a new command to Tendermint to allow for us to build a
standalone binary to test remote signers such as KMS
(https://github.com/tendermint/kms).

Right now, all it does is test that the local public key matches the
public key reported by the client, and fails at the point where it
attempts to get the client to sign a proposal.

* Fixes typo

* Fixes proposal validation test

This commit fixes the proposal validation test as per #3149. It also
moves the test harness into its own internal package to isolate its
exports from the `privval` package.

* Adds vote signing validation

* Applying recommendations from #3149

* Adds function descriptions for test harness

* Adds ability to ask remote signer to shut down

Prior to this commit, the remote signer needs to manually be shut down,
which is not ideal for automated testing. This commit allows us to send
a poison pill message to the KMS to let it shut down gracefully once
testing is done (whether the tests pass or fail).

* Adds tests for remote signer test harness

This commit makes some minor modifications to a few files to allow for
testing of the remote signer test harness. Two tests are added here:
checking for a fully successful (the ideal) case, and for the case where
the maximum number of retries has been reached when attempting to accept
incoming connections from the remote signer.

* Condenses serialization of proposals and votes using existing Tendermint functions

* Removes now-unnecessary amino import and codec

* Adds error message for vote signing failure

* Adds key extraction command for integration test

Took the code from here:
https://gist.github.com/Liamsi/a80993f24bff574bbfdbbfa9efa84bc7 to
create a simple utility command to extract a key from a local Tendermint
validator for use in KMS integration testing.

* Makes path expansion success non-compulsory

* Fixes segfault on SIGTERM

We need an additional variable to keep track of whether we're
successfully connected, otherwise hitting Ctrl+Break during execution
causes a segmentation fault. This now allows for a clean shutdown.

* Consolidates shutdown checks

* Adds comments indicating codes for easy lookup

* Adds Docker build for remote signer harness

Updates the `DOCKER/build.sh` and `DOCKER/push.sh` files to allow one to
override the image name and Dockerfile using environment variables.
Updates the primary `Makefile` as well as the `DOCKER/Makefile` to allow
for building the `remote_val_harness` Docker image.

* Adds build_remote_val_harness_docker_image to .PHONY

* Removes remote signer poison pill messaging functionality

* Reduces fluff code in command line parsing

As per
https://github.com/tendermint/tendermint/pull/3149#pullrequestreview-196171788,
this reduces the amount of fluff code in the PR down to the bare
minimum.

* Fixes ordering of error check and info log

* Moves remove_val_harness cmd into tools folder

It seems to make sense to rather keep the remote signer test harness in
its own tool folder (now rather named `tm-signer-harness` to keep with
the tool naming convention). It is actually a separate tool, not meant
to be one of the core binaries, but supplementary and supportive.

* Updates documentation for tm-signer-harness

* Refactors flag parsing to be more compact and less redundant

* Adds version sub-command help

* Removes extraneous flags parsing

* Adds CHANGELOG_PENDING entry for tm-signer-harness

* Improves test coverage

Adds a few extra parameters to the `MockPV` type to fake broken vote and
proposal signing. Also adds some more tests for the test harness so as
to increase coverage for failed cases.

* Fixes formatting for CHANGELOG_PENDING.md

* Fix formatting for documentation config

* Point users towards official Tendermint docs for tools documentation

* Point users towards official Tendermint docs for tm-signer-harness

* Remove extraneous constant

* Rename TestHarness.sc to TestHarness.spv for naming consistency

* Refactor to remove redundant goroutine

* Refactor conditional to cleaner switch statement and better error handling for listener protocol

* Remove extraneous goroutine

* Add note about installing tmkms via Cargo

* Fix typo in naming of output signing key

* Add note about where to find chain ID

* Replace /home/user with ~/ for brevity

* Fixes "signer.key" typo

* Minor edits for clarification for tm-signer-harness bulid/setup process

CHANGELOG_PENDING.md

@@ -20,9 +20,10 @@ Special thanks to external contributors on this release:
 ### FEATURES:
 
 ### IMPROVEMENTS:
-- [tools] add go-deadlock tool to help detect deadlocks
-- [crypto] \#3163 use ethereum's libsecp256k1 go-wrapper for signatures when cgo is available
-- [crypto] \#3162 wrap btcd instead of forking it to keep up with fixes (used if cgo is not available)
+- [tools] Add go-deadlock tool to help detect deadlocks
+- [tools] \#3106 Add tm-signer-harness test harness for remote signers
+- [crypto] \#3163 Use ethereum's libsecp256k1 go-wrapper for signatures when cgo is available
+- [crypto] \#3162 Wrap btcd instead of forking it to keep up with fixes (used if cgo is not available)
 
 ### BUG FIXES:
 - [node] \#3186 EventBus and indexerService should be started before first block (for replay last block on handshake) execution

docs/.vuepress/config.js

@@ -79,10 +79,11 @@ module.exports = {
         title: "Tools",
         collapsable: false,
         children:  [
-	  "/tools/",
-	  "/tools/benchmarking",
-	  "/tools/monitoring"
-	]
+          "/tools/",
+          "/tools/benchmarking",
+          "/tools/monitoring",
+          "/tools/remote-signer-validation"
+        ]
       },
       {
         title: "Tendermint Spec",

docs/tools/README.md

@@ -1,4 +1,7 @@
 # Overview
 
-Tendermint comes with some tools for [benchmarking](./benchmarking.md)
-and [monitoring](./monitoring.md).
+Tendermint comes with some tools for:
+
+* [Benchmarking](./benchmarking.md)
+* [Monitoring](./monitoring.md)
+* [Validation of remote signers](./remote-signer-validation.md)

docs/tools/remote-signer-validation.md

@@ -0,0 +1,146 @@
+# tm-signer-harness
+
+Located under the `tools/tm-signer-harness` folder in the [Tendermint
+repository](https://github.com/tendermint/tendermint).
+
+The Tendermint remote signer test harness facilitates integration testing
+between Tendermint and remote signers such as
+[KMS](https://github.com/tendermint/kms). Such remote signers allow for signing
+of important Tendermint messages using
+[HSMs](https://en.wikipedia.org/wiki/Hardware_security_module), providing
+additional security.
+
+When executed, `tm-signer-harness`:
+
+1. Runs a listener (either TCP or Unix sockets).
+2. Waits for a connection from the remote signer.
+3. Upon connection from the remote signer, executes a number of automated tests
+   to ensure compatibility.
+4. Upon successful validation, the harness process exits with a 0 exit code.
+   Upon validation failure, it exits with a particular exit code related to the
+   error.
+
+## Prerequisites
+Requires the same prerequisites as for building
+[Tendermint](https://github.com/tendermint/tendermint).
+
+## Building
+From the `tools/tm-signer-harness` directory in your Tendermint source
+repository, simply run:
+
+```bash
+make
+
+# To have global access to this executable
+make install
+```
+
+## Docker Image
+To build a Docker image containing the `tm-signer-harness`, also from the
+`tools/tm-signer-harness` directory of your Tendermint source repo, simply run:
+
+```bash
+make docker-image
+```
+
+## Running against KMS
+As an example of how to use `tm-signer-harness`, the following instructions show
+you how to execute its tests against [KMS](https://github.com/tendermint/kms).
+For this example, we will make use of the **software signing module in KMS**, as
+the hardware signing module requires a physical
+[YubiHSM](https://www.yubico.com/products/yubihsm/) device.
+
+### Step 1: Install KMS on your local machine
+See the [KMS repo](https://github.com/tendermint/kms) for details on how to set
+KMS up on your local machine.
+
+If you have [Rust](https://www.rust-lang.org/) installed on your local machine,
+you can simply install KMS by:
+
+```bash
+cargo install tmkms
+```
+
+### Step 2: Make keys for KMS
+The KMS software signing module needs a key with which to sign messages. In our
+example, we will simply export a signing key from our local Tendermint instance.
+
+```bash
+# Will generate all necessary Tendermint configuration files, including:
+# - ~/.tendermint/config/priv_validator_key.json
+# - ~/.tendermint/data/priv_validator_state.json
+tendermint init
+
+# Extract the signing key from our local Tendermint instance
+tm-signer-harness extract_key \      # Use the "extract_key" command
+    -tmhome ~/.tendermint \          # Where to find the Tendermint home directory
+    -output ./signing.key            # Where to write the key
+```
+
+Also, because we want KMS to connect to `tm-signer-harness`, we will need to
+provide a secret connection key from KMS' side:
+
+```bash
+tmkms keygen secret_connection.key
+```
+
+### Step 3: Configure and run KMS
+KMS needs some configuration to tell it to use the softer signing module as well
+as the `signing.key` file we just generated. Save the following to a file called
+`tmkms.toml`:
+
+```toml
+[[validator]]
+addr = "tcp://127.0.0.1:61219"         # This is where we will find tm-signer-harness.
+chain_id = "test-chain-0XwP5E"         # The Tendermint chain ID for which KMS will be signing (found in ~/.tendermint/config/genesis.json).
+reconnect = true                       # true is the default
+secret_key = "./secret_connection.key" # Where to find our secret connection key.
+
+[[providers.softsign]]
+id = "test-chain-0XwP5E"               # The Tendermint chain ID for which KMS will be signing (same as validator.chain_id above).
+path = "./signing.key"                 # The signing key we extracted earlier.
+```
+
+Then run KMS with this configuration:
+
+```bash
+tmkms start -c tmkms.toml
+```
+
+This will start KMS, which will repeatedly try to connect to
+`tcp://127.0.0.1:61219` until it is successful.
+
+### Step 4: Run tm-signer-harness
+Now we get to run the signer test harness:
+
+```bash
+tm-signer-harness run \             # The "run" command executes the tests
+    -addr tcp://127.0.0.1:61219 \   # The address we promised KMS earlier
+    -tmhome ~/.tendermint           # Where to find our Tendermint configuration/data files.
+```
+
+If the current version of Tendermint and KMS are compatible, `tm-signer-harness`
+should now exit with a 0 exit code. If they are somehow not compatible, it
+should exit with a meaningful non-zero exit code (see the exit codes below).
+
+### Step 5: Shut down KMS
+Simply hit Ctrl+Break on your KMS instance (or use the `kill` command in Linux)
+to terminate it gracefully.
+
+## Exit Code Meanings
+The following list shows the various exit codes from `tm-signer-harness` and
+their meanings:
+
+| Exit Code | Description |
+| --- | --- |
+| 0 | Success! |
+| 1 | Invalid command line parameters supplied to `tm-signer-harness` |
+| 2 | Maximum number of accept retries reached (the `-accept-retries` parameter) |
+| 3 | Failed to load `${TMHOME}/config/genesis.json` |
+| 4 | Failed to create listener specified by `-addr` parameter |
+| 5 | Failed to start listener |
+| 6 | Interrupted by `SIGINT` (e.g. when hitting Ctrl+Break or Ctrl+C) |
+| 7 | Other unknown error |
+| 8 | Test 1 failed: public key mismatch |
+| 9 | Test 2 failed: signing of proposals failed |
+| 10 | Test 3 failed: signing of votes failed |

tools/README.md

@@ -1,3 +1,5 @@
 # tools
 
-Tools for working with tendermint and associated technologies. Documentation can be found in the `README.md` of each the `tm-bench/` and `tm-monitor/` directories.
+Tools for working with Tendermint and associated technologies. Documentation for
+these tools can be found online in the [Tendermint tools
+documentation](https://tendermint.com/docs/tools/).

tools/tm-signer-harness/Dockerfile

@@ -0,0 +1,4 @@
+ARG TENDERMINT_VERSION=latest
+FROM tendermint/tendermint:${TENDERMINT_VERSION}
+
+COPY tm-signer-harness /usr/bin/tm-signer-harness

tools/tm-signer-harness/Makefile

@@ -0,0 +1,20 @@
+.PHONY: build install docker-image
+
+TENDERMINT_VERSION?=latest
+BUILD_TAGS?='tendermint'
+BUILD_FLAGS = -ldflags "-X github.com/tendermint/tendermint/version.GitCommit=`git rev-parse --short=8 HEAD`"
+
+.DEFAULT_GOAL := build
+
+build:
+	CGO_ENABLED=0 go build $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o ../../build/tm-signer-harness main.go
+
+install:
+	CGO_ENABLED=0 go install $(BUILD_FLAGS) -tags $(BUILD_TAGS) .
+
+docker-image:
+	GOOS=linux GOARCH=amd64 go build $(BUILD_FLAGS) -tags $(BUILD_TAGS) -o tm-signer-harness main.go
+	docker build \
+		--build-arg TENDERMINT_VERSION=$(TENDERMINT_VERSION) \
+		-t tendermint/tm-signer-harness:$(TENDERMINT_VERSION) .
+	rm -rf tm-signer-harness

tools/tm-signer-harness/README.md

@@ -0,0 +1,5 @@
+# tm-signer-harness
+
+See the [`tm-signer-harness`
+documentation](https://tendermint.com/docs/tools/remote-signer-validation.html)
+for more details.

tools/tm-signer-harness/internal/test_harness.go

@@ -0,0 +1,392 @@
+package internal
+
+import (
+	"fmt"
+	"net"
+	"os"
+	"os/signal"
+	"time"
+
+	"github.com/tendermint/tendermint/crypto/tmhash"
+
+	"github.com/tendermint/tendermint/crypto/ed25519"
+	"github.com/tendermint/tendermint/privval"
+	"github.com/tendermint/tendermint/state"
+
+	cmn "github.com/tendermint/tendermint/libs/common"
+	"github.com/tendermint/tendermint/libs/log"
+	"github.com/tendermint/tendermint/types"
+)
+
+// Test harness error codes (which act as exit codes when the test harness fails).
+const (
+	NoError                    int = iota // 0
+	ErrInvalidParameters                  // 1
+	ErrMaxAcceptRetriesReached            // 2
+	ErrFailedToLoadGenesisFile            // 3
+	ErrFailedToCreateListener             // 4
+	ErrFailedToStartListener              // 5
+	ErrInterrupted                        // 6
+	ErrOther                              // 7
+	ErrTestPublicKeyFailed                // 8
+	ErrTestSignProposalFailed             // 9
+	ErrTestSignVoteFailed                 // 10
+)
+
+var voteTypes = []types.SignedMsgType{types.PrevoteType, types.PrecommitType}
+
+// TestHarnessError allows us to keep track of which exit code should be used
+// when exiting the main program.
+type TestHarnessError struct {
+	Code int    // The exit code to return
+	Err  error  // The original error
+	Info string // Any additional information
+}
+
+var _ error = (*TestHarnessError)(nil)
+
+// TestHarness allows for testing of a remote signer to ensure compatibility
+// with this version of Tendermint.
+type TestHarness struct {
+	addr             string
+	spv              *privval.SocketVal
+	fpv              *privval.FilePV
+	chainID          string
+	acceptRetries    int
+	logger           log.Logger
+	exitWhenComplete bool
+	exitCode         int
+}
+
+// TestHarnessConfig provides configuration to set up a remote signer test
+// harness.
+type TestHarnessConfig struct {
+	BindAddr string
+
+	KeyFile     string
+	StateFile   string
+	GenesisFile string
+
+	AcceptDeadline time.Duration
+	ConnDeadline   time.Duration
+	AcceptRetries  int
+
+	SecretConnKey ed25519.PrivKeyEd25519
+
+	ExitWhenComplete bool // Whether or not to call os.Exit when the harness has completed.
+}
+
+// timeoutError can be used to check if an error returned from the netp package
+// was due to a timeout.
+type timeoutError interface {
+	Timeout() bool
+}
+
+// NewTestHarness will load Tendermint data from the given files (including
+// validator public/private keypairs and chain details) and create a new
+// harness.
+func NewTestHarness(logger log.Logger, cfg TestHarnessConfig) (*TestHarness, error) {
+	keyFile := ExpandPath(cfg.KeyFile)
+	stateFile := ExpandPath(cfg.StateFile)
+	logger.Info("Loading private validator configuration", "keyFile", keyFile, "stateFile", stateFile)
+	// NOTE: LoadFilePV ultimately calls os.Exit on failure. No error will be
+	// returned if this call fails.
+	fpv := privval.LoadFilePV(keyFile, stateFile)
+
+	genesisFile := ExpandPath(cfg.GenesisFile)
+	logger.Info("Loading chain ID from genesis file", "genesisFile", genesisFile)
+	st, err := state.MakeGenesisDocFromFile(genesisFile)
+	if err != nil {
+		return nil, newTestHarnessError(ErrFailedToLoadGenesisFile, err, genesisFile)
+	}
+	logger.Info("Loaded genesis file", "chainID", st.ChainID)
+
+	spv, err := newTestHarnessSocketVal(logger, cfg)
+	if err != nil {
+		return nil, newTestHarnessError(ErrFailedToCreateListener, err, "")
+	}
+
+	return &TestHarness{
+		addr:             cfg.BindAddr,
+		spv:              spv,
+		fpv:              fpv,
+		chainID:          st.ChainID,
+		acceptRetries:    cfg.AcceptRetries,
+		logger:           logger,
+		exitWhenComplete: cfg.ExitWhenComplete,
+		exitCode:         0,
+	}, nil
+}
+
+// Run will execute the tests associated with this test harness. The intention
+// here is to call this from one's `main` function, as the way it succeeds or
+// fails at present is to call os.Exit() with an exit code related to the error
+// that caused the tests to fail, or exit code 0 on success.
+func (th *TestHarness) Run() {
+	c := make(chan os.Signal, 1)
+	signal.Notify(c, os.Interrupt)
+	go func() {
+		for sig := range c {
+			th.logger.Info("Caught interrupt, terminating...", "sig", sig)
+			th.Shutdown(newTestHarnessError(ErrInterrupted, nil, ""))
+		}
+	}()
+
+	th.logger.Info("Starting test harness")
+	accepted := false
+	var startErr error
+	for acceptRetries := th.acceptRetries; acceptRetries > 0; acceptRetries-- {
+		th.logger.Info("Attempting to accept incoming connection", "acceptRetries", acceptRetries)
+		if err := th.spv.Start(); err != nil {
+			// if it wasn't a timeout error
+			if _, ok := err.(timeoutError); !ok {
+				th.logger.Error("Failed to start listener", "err", err)
+				th.Shutdown(newTestHarnessError(ErrFailedToStartListener, err, ""))
+				// we need the return statements in case this is being run
+				// from a unit test - otherwise this function will just die
+				// when os.Exit is called
+				return
+			}
+			startErr = err
+		} else {
+			accepted = true
+			break
+		}
+	}
+	if !accepted {
+		th.logger.Error("Maximum accept retries reached", "acceptRetries", th.acceptRetries)
+		th.Shutdown(newTestHarnessError(ErrMaxAcceptRetriesReached, startErr, ""))
+		return
+	}
+
+	// Run the tests
+	if err := th.TestPublicKey(); err != nil {
+		th.Shutdown(err)
+		return
+	}
+	if err := th.TestSignProposal(); err != nil {
+		th.Shutdown(err)
+		return
+	}
+	if err := th.TestSignVote(); err != nil {
+		th.Shutdown(err)
+		return
+	}
+	th.logger.Info("SUCCESS! All tests passed.")
+	th.Shutdown(nil)
+}
+
+// TestPublicKey just validates that we can (1) fetch the public key from the
+// remote signer, and (2) it matches the public key we've configured for our
+// local Tendermint version.
+func (th *TestHarness) TestPublicKey() error {
+	th.logger.Info("TEST: Public key of remote signer")
+	th.logger.Info("Local", "pubKey", th.fpv.GetPubKey())
+	th.logger.Info("Remote", "pubKey", th.spv.GetPubKey())
+	if th.fpv.GetPubKey() != th.spv.GetPubKey() {
+		th.logger.Error("FAILED: Local and remote public keys do not match")
+		return newTestHarnessError(ErrTestPublicKeyFailed, nil, "")
+	}
+	return nil
+}
+
+// TestSignProposal makes sure the remote signer can successfully sign
+// proposals.
+func (th *TestHarness) TestSignProposal() error {
+	th.logger.Info("TEST: Signing of proposals")
+	// sha256 hash of "hash"
+	hash := tmhash.Sum([]byte("hash"))
+	prop := &types.Proposal{
+		Type:     types.ProposalType,
+		Height:   12345,
+		Round:    23456,
+		POLRound: -1,
+		BlockID: types.BlockID{
+			Hash: hash,
+			PartsHeader: types.PartSetHeader{
+				Hash:  hash,
+				Total: 1000000,
+			},
+		},
+		Timestamp: time.Now(),
+	}
+	propBytes := prop.SignBytes(th.chainID)
+	if err := th.spv.SignProposal(th.chainID, prop); err != nil {
+		th.logger.Error("FAILED: Signing of proposal", "err", err)
+		return newTestHarnessError(ErrTestSignProposalFailed, err, "")
+	}
+	th.logger.Debug("Signed proposal", "prop", prop)
+	// first check that it's a basically valid proposal
+	if err := prop.ValidateBasic(); err != nil {
+		th.logger.Error("FAILED: Signed proposal is invalid", "err", err)
+		return newTestHarnessError(ErrTestSignProposalFailed, err, "")
+	}
+	// now validate the signature on the proposal
+	if th.spv.GetPubKey().VerifyBytes(propBytes, prop.Signature) {
+		th.logger.Info("Successfully validated proposal signature")
+	} else {
+		th.logger.Error("FAILED: Proposal signature validation failed")
+		return newTestHarnessError(ErrTestSignProposalFailed, nil, "signature validation failed")
+	}
+	return nil
+}
+
+// TestSignVote makes sure the remote signer can successfully sign all kinds of
+// votes.
+func (th *TestHarness) TestSignVote() error {
+	th.logger.Info("TEST: Signing of votes")
+	for _, voteType := range voteTypes {
+		th.logger.Info("Testing vote type", "type", voteType)
+		hash := tmhash.Sum([]byte("hash"))
+		vote := &types.Vote{
+			Type:   voteType,
+			Height: 12345,
+			Round:  23456,
+			BlockID: types.BlockID{
+				Hash: hash,
+				PartsHeader: types.PartSetHeader{
+					Hash:  hash,
+					Total: 1000000,
+				},
+			},
+			ValidatorIndex:   0,
+			ValidatorAddress: tmhash.SumTruncated([]byte("addr")),
+			Timestamp:        time.Now(),
+		}
+		voteBytes := vote.SignBytes(th.chainID)
+		// sign the vote
+		if err := th.spv.SignVote(th.chainID, vote); err != nil {
+			th.logger.Error("FAILED: Signing of vote", "err", err)
+			return newTestHarnessError(ErrTestSignVoteFailed, err, fmt.Sprintf("voteType=%d", voteType))
+		}
+		th.logger.Debug("Signed vote", "vote", vote)
+		// validate the contents of the vote
+		if err := vote.ValidateBasic(); err != nil {
+			th.logger.Error("FAILED: Signed vote is invalid", "err", err)
+			return newTestHarnessError(ErrTestSignVoteFailed, err, fmt.Sprintf("voteType=%d", voteType))
+		}
+		// now validate the signature on the proposal
+		if th.spv.GetPubKey().VerifyBytes(voteBytes, vote.Signature) {
+			th.logger.Info("Successfully validated vote signature", "type", voteType)
+		} else {
+			th.logger.Error("FAILED: Vote signature validation failed", "type", voteType)
+			return newTestHarnessError(ErrTestSignVoteFailed, nil, "signature validation failed")
+		}
+	}
+	return nil
+}
+
+// Shutdown will kill the test harness and attempt to close all open sockets
+// gracefully. If the supplied error is nil, it is assumed that the exit code
+// should be 0. If err is not nil, it will exit with an exit code related to the
+// error.
+func (th *TestHarness) Shutdown(err error) {
+	var exitCode int
+
+	if err == nil {
+		exitCode = NoError
+	} else if therr, ok := err.(*TestHarnessError); ok {
+		exitCode = therr.Code
+	} else {
+		exitCode = ErrOther
+	}
+	th.exitCode = exitCode
+
+	// in case sc.Stop() takes too long
+	if th.exitWhenComplete {
+		go func() {
+			time.Sleep(time.Duration(5) * time.Second)
+			th.logger.Error("Forcibly exiting program after timeout")
+			os.Exit(exitCode)
+		}()
+	}
+
+	if th.spv.IsRunning() {
+		if err := th.spv.Stop(); err != nil {
+			th.logger.Error("Failed to cleanly stop listener: %s", err.Error())
+		}
+	}
+
+	if th.exitWhenComplete {
+		os.Exit(exitCode)
+	}
+}
+
+// newTestHarnessSocketVal creates our client instance which we will use for
+// testing.
+func newTestHarnessSocketVal(logger log.Logger, cfg TestHarnessConfig) (*privval.SocketVal, error) {
+	proto, addr := cmn.ProtocolAndAddress(cfg.BindAddr)
+	if proto == "unix" {
+		// make sure the socket doesn't exist - if so, try to delete it
+		if cmn.FileExists(addr) {
+			if err := os.Remove(addr); err != nil {
+				logger.Error("Failed to remove existing Unix domain socket", "addr", addr)
+				return nil, err
+			}
+		}
+	}
+	ln, err := net.Listen(proto, addr)
+	if err != nil {
+		return nil, err
+	}
+	logger.Info("Listening at", "proto", proto, "addr", addr)
+	var svln net.Listener
+	switch proto {
+	case "unix":
+		unixLn := privval.NewUnixListener(ln)
+		privval.UnixListenerAcceptDeadline(cfg.AcceptDeadline)(unixLn)
+		privval.UnixListenerConnDeadline(cfg.ConnDeadline)(unixLn)
+		svln = unixLn
+	case "tcp":
+		tcpLn := privval.NewTCPListener(ln, cfg.SecretConnKey)
+		privval.TCPListenerAcceptDeadline(cfg.AcceptDeadline)(tcpLn)
+		privval.TCPListenerConnDeadline(cfg.ConnDeadline)(tcpLn)
+		logger.Info("Resolved TCP address for listener", "addr", tcpLn.Addr())
+		svln = tcpLn
+	default:
+		logger.Error("Unsupported protocol (must be unix:// or tcp://)", "proto", proto)
+		return nil, newTestHarnessError(ErrInvalidParameters, nil, fmt.Sprintf("Unsupported protocol: %s", proto))
+	}
+	return privval.NewSocketVal(logger, svln), nil
+}
+
+func newTestHarnessError(code int, err error, info string) *TestHarnessError {
+	return &TestHarnessError{
+		Code: code,
+		Err:  err,
+		Info: info,
+	}
+}
+
+func (e *TestHarnessError) Error() string {
+	var msg string
+	switch e.Code {
+	case ErrInvalidParameters:
+		msg = "Invalid parameters supplied to application"
+	case ErrMaxAcceptRetriesReached:
+		msg = "Maximum accept retries reached"
+	case ErrFailedToLoadGenesisFile:
+		msg = "Failed to load genesis file"
+	case ErrFailedToCreateListener:
+		msg = "Failed to create listener"
+	case ErrFailedToStartListener:
+		msg = "Failed to start listener"
+	case ErrInterrupted:
+		msg = "Interrupted"
+	case ErrTestPublicKeyFailed:
+		msg = "Public key validation test failed"
+	case ErrTestSignProposalFailed:
+		msg = "Proposal signing validation test failed"
+	case ErrTestSignVoteFailed:
+		msg = "Vote signing validation test failed"
+	default:
+		msg = "Unknown error"
+	}
+	if len(e.Info) > 0 {
+		msg = fmt.Sprintf("%s: %s", msg, e.Info)
+	}
+	if e.Err != nil {
+		msg = fmt.Sprintf("%s (original error: %s)", msg, e.Err.Error())
+	}
+	return msg
+}

tools/tm-signer-harness/internal/test_harness_test.go

@@ -0,0 +1,201 @@
+package internal
+
+import (
+	"fmt"
+	"io/ioutil"
+	"net"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/tendermint/tendermint/crypto"
+	"github.com/tendermint/tendermint/privval"
+	"github.com/tendermint/tendermint/types"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"github.com/tendermint/tendermint/crypto/ed25519"
+	"github.com/tendermint/tendermint/libs/log"
+)
+
+const (
+	keyFileContents = `{
+	"address": "D08FCA3BA74CF17CBFC15E64F9505302BB0E2748",
+	"pub_key": {
+		"type": "tendermint/PubKeyEd25519",
+		"value": "ZCsuTjaczEyon70nmKxwvwu+jqrbq5OH3yQjcK0SFxc="
+	},
+	"priv_key": {
+		"type": "tendermint/PrivKeyEd25519",
+		"value": "8O39AkQsoe1sBQwud/Kdul8lg8K9SFsql9aZvwXQSt1kKy5ONpzMTKifvSeYrHC/C76Oqturk4ffJCNwrRIXFw=="
+	}
+}`
+
+	stateFileContents = `{
+	"height": "0",
+	"round": "0",
+	"step": 0
+}`
+
+	genesisFileContents = `{
+	"genesis_time": "2019-01-15T11:56:34.8963Z",
+	"chain_id": "test-chain-0XwP5E",
+	"consensus_params": {
+		"block_size": {
+		"max_bytes": "22020096",
+		"max_gas": "-1"
+		},
+		"evidence": {
+		"max_age": "100000"
+		},
+		"validator": {
+		"pub_key_types": [
+			"ed25519"
+		]
+		}
+	},
+	"validators": [
+		{
+		"address": "D08FCA3BA74CF17CBFC15E64F9505302BB0E2748",
+		"pub_key": {
+			"type": "tendermint/PubKeyEd25519",
+			"value": "ZCsuTjaczEyon70nmKxwvwu+jqrbq5OH3yQjcK0SFxc="
+		},
+		"power": "10",
+		"name": ""
+		}
+	],
+	"app_hash": ""
+}`
+
+	defaultConnDeadline = 100
+)
+
+func TestRemoteSignerTestHarnessMaxAcceptRetriesReached(t *testing.T) {
+	cfg := makeConfig(t, 1, 2)
+	defer cleanup(cfg)
+
+	th, err := NewTestHarness(log.TestingLogger(), cfg)
+	require.NoError(t, err)
+	th.Run()
+	assert.Equal(t, ErrMaxAcceptRetriesReached, th.exitCode)
+}
+
+func TestRemoteSignerTestHarnessSuccessfulRun(t *testing.T) {
+	harnessTest(
+		t,
+		func(th *TestHarness) *privval.RemoteSigner {
+			return newMockRemoteSigner(t, th, th.fpv.Key.PrivKey, false, false)
+		},
+		NoError,
+	)
+}
+
+func TestRemoteSignerPublicKeyCheckFailed(t *testing.T) {
+	harnessTest(
+		t,
+		func(th *TestHarness) *privval.RemoteSigner {
+			return newMockRemoteSigner(t, th, ed25519.GenPrivKey(), false, false)
+		},
+		ErrTestPublicKeyFailed,
+	)
+}
+
+func TestRemoteSignerProposalSigningFailed(t *testing.T) {
+	harnessTest(
+		t,
+		func(th *TestHarness) *privval.RemoteSigner {
+			return newMockRemoteSigner(t, th, th.fpv.Key.PrivKey, true, false)
+		},
+		ErrTestSignProposalFailed,
+	)
+}
+
+func TestRemoteSignerVoteSigningFailed(t *testing.T) {
+	harnessTest(
+		t,
+		func(th *TestHarness) *privval.RemoteSigner {
+			return newMockRemoteSigner(t, th, th.fpv.Key.PrivKey, false, true)
+		},
+		ErrTestSignVoteFailed,
+	)
+}
+
+func newMockRemoteSigner(t *testing.T, th *TestHarness, privKey crypto.PrivKey, breakProposalSigning bool, breakVoteSigning bool) *privval.RemoteSigner {
+	return privval.NewRemoteSigner(
+		th.logger,
+		th.chainID,
+		types.NewMockPVWithParams(privKey, breakProposalSigning, breakVoteSigning),
+		privval.DialTCPFn(
+			th.addr,
+			time.Duration(defaultConnDeadline)*time.Millisecond,
+			ed25519.GenPrivKey(),
+		),
+	)
+}
+
+// For running relatively standard tests.
+func harnessTest(t *testing.T, rsMaker func(th *TestHarness) *privval.RemoteSigner, expectedExitCode int) {
+	cfg := makeConfig(t, 100, 3)
+	defer cleanup(cfg)
+
+	th, err := NewTestHarness(log.TestingLogger(), cfg)
+	require.NoError(t, err)
+	donec := make(chan struct{})
+	go func() {
+		defer close(donec)
+		th.Run()
+	}()
+
+	rs := rsMaker(th)
+	require.NoError(t, rs.Start())
+	assert.True(t, rs.IsRunning())
+	defer rs.Stop()
+
+	<-donec
+	assert.Equal(t, expectedExitCode, th.exitCode)
+}
+
+func makeConfig(t *testing.T, acceptDeadline, acceptRetries int) TestHarnessConfig {
+	return TestHarnessConfig{
+		BindAddr:         testFreeTCPAddr(t),
+		KeyFile:          makeTempFile("tm-testharness-keyfile", keyFileContents),
+		StateFile:        makeTempFile("tm-testharness-statefile", stateFileContents),
+		GenesisFile:      makeTempFile("tm-testharness-genesisfile", genesisFileContents),
+		AcceptDeadline:   time.Duration(acceptDeadline) * time.Millisecond,
+		ConnDeadline:     time.Duration(defaultConnDeadline) * time.Millisecond,
+		AcceptRetries:    acceptRetries,
+		SecretConnKey:    ed25519.GenPrivKey(),
+		ExitWhenComplete: false,
+	}
+}
+
+func cleanup(cfg TestHarnessConfig) {
+	os.Remove(cfg.KeyFile)
+	os.Remove(cfg.StateFile)
+	os.Remove(cfg.GenesisFile)
+}
+
+func makeTempFile(name, content string) string {
+	tempFile, err := ioutil.TempFile("", fmt.Sprintf("%s-*", name))
+	if err != nil {
+		panic(err)
+	}
+	if _, err := tempFile.Write([]byte(content)); err != nil {
+		tempFile.Close()
+		panic(err)
+	}
+	if err := tempFile.Close(); err != nil {
+		panic(err)
+	}
+	return tempFile.Name()
+}
+
+// testFreeTCPAddr claims a free port so we don't block on listener being ready.
+func testFreeTCPAddr(t *testing.T) string {
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	require.NoError(t, err)
+	defer ln.Close()
+
+	return fmt.Sprintf("127.0.0.1:%d", ln.Addr().(*net.TCPAddr).Port)
+}

tools/tm-signer-harness/internal/utils.go

@@ -0,0 +1,25 @@
+package internal
+
+import (
+	"os/user"
+	"path/filepath"
+	"strings"
+)
+
+// ExpandPath will check if the given path begins with a "~" symbol, and if so,
+// will expand it to become the user's home directory. If it fails to expand the
+// path it will automatically return the original path itself.
+func ExpandPath(path string) string {
+	usr, err := user.Current()
+	if err != nil {
+		return path
+	}
+
+	if path == "~" {
+		return usr.HomeDir
+	} else if strings.HasPrefix(path, "~/") {
+		return filepath.Join(usr.HomeDir, path[2:])
+	}
+
+	return path
+}

tools/tm-signer-harness/main.go

@@ -0,0 +1,174 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"time"
+
+	"github.com/tendermint/tendermint/crypto/ed25519"
+	"github.com/tendermint/tendermint/libs/log"
+	"github.com/tendermint/tendermint/privval"
+	"github.com/tendermint/tendermint/tools/tm-signer-harness/internal"
+	"github.com/tendermint/tendermint/version"
+)
+
+const (
+	defaultAcceptRetries    = 100
+	defaultBindAddr         = "tcp://127.0.0.1:0"
+	defaultTMHome           = "~/.tendermint"
+	defaultAcceptDeadline   = 1
+	defaultConnDeadline     = 3
+	defaultExtractKeyOutput = "./signing.key"
+)
+
+var logger = log.NewTMLogger(log.NewSyncWriter(os.Stdout))
+
+// Command line flags
+var (
+	flagAcceptRetries int
+	flagBindAddr      string
+	flagTMHome        string
+	flagKeyOutputPath string
+)
+
+// Command line commands
+var (
+	rootCmd       *flag.FlagSet
+	runCmd        *flag.FlagSet
+	extractKeyCmd *flag.FlagSet
+	versionCmd    *flag.FlagSet
+)
+
+func init() {
+	rootCmd = flag.NewFlagSet("root", flag.ExitOnError)
+	rootCmd.Usage = func() {
+		fmt.Println(`Remote signer test harness for Tendermint.
+
+Usage:
+  tm-signer-harness <command> [flags]
+
+Available Commands:
+  extract_key        Extracts a signing key from a local Tendermint instance
+  help               Help on the available commands
+  run                Runs the test harness
+  version            Display version information and exit
+
+Use "tm-signer-harness help <command>" for more information about that command.`)
+		fmt.Println("")
+	}
+
+	runCmd = flag.NewFlagSet("run", flag.ExitOnError)
+	runCmd.IntVar(&flagAcceptRetries, "accept-retries", defaultAcceptRetries, "The number of attempts to listen for incoming connections")
+	runCmd.StringVar(&flagBindAddr, "addr", defaultBindAddr, "Bind to this address for the testing")
+	runCmd.StringVar(&flagTMHome, "tmhome", defaultTMHome, "Path to the Tendermint home directory")
+	runCmd.Usage = func() {
+		fmt.Println(`Runs the remote signer test harness for Tendermint.
+
+Usage:
+  tm-signer-harness run [flags]
+
+Flags:`)
+		runCmd.PrintDefaults()
+		fmt.Println("")
+	}
+
+	extractKeyCmd = flag.NewFlagSet("extract_key", flag.ExitOnError)
+	extractKeyCmd.StringVar(&flagKeyOutputPath, "output", defaultExtractKeyOutput, "Path to which signing key should be written")
+	extractKeyCmd.StringVar(&flagTMHome, "tmhome", defaultTMHome, "Path to the Tendermint home directory")
+	extractKeyCmd.Usage = func() {
+		fmt.Println(`Extracts a signing key from a local Tendermint instance for use in the remote
+signer under test.
+
+Usage:
+  tm-signer-harness extract_key [flags]
+
+Flags:`)
+		extractKeyCmd.PrintDefaults()
+		fmt.Println("")
+	}
+
+	versionCmd = flag.NewFlagSet("version", flag.ExitOnError)
+	versionCmd.Usage = func() {
+		fmt.Println(`
+Prints the Tendermint version for which this remote signer harness was built.
+
+Usage:
+  tm-signer-harness version`)
+		fmt.Println("")
+	}
+}
+
+func runTestHarness(acceptRetries int, bindAddr, tmhome string) {
+	tmhome = internal.ExpandPath(tmhome)
+	cfg := internal.TestHarnessConfig{
+		BindAddr:         bindAddr,
+		KeyFile:          filepath.Join(tmhome, "config", "priv_validator_key.json"),
+		StateFile:        filepath.Join(tmhome, "data", "priv_validator_state.json"),
+		GenesisFile:      filepath.Join(tmhome, "config", "genesis.json"),
+		AcceptDeadline:   time.Duration(defaultAcceptDeadline) * time.Second,
+		AcceptRetries:    acceptRetries,
+		ConnDeadline:     time.Duration(defaultConnDeadline) * time.Second,
+		SecretConnKey:    ed25519.GenPrivKey(),
+		ExitWhenComplete: true,
+	}
+	harness, err := internal.NewTestHarness(logger, cfg)
+	if err != nil {
+		logger.Error(err.Error())
+		if therr, ok := err.(*internal.TestHarnessError); ok {
+			os.Exit(therr.Code)
+		}
+		os.Exit(internal.ErrOther)
+	}
+	harness.Run()
+}
+
+func extractKey(tmhome, outputPath string) {
+	keyFile := filepath.Join(internal.ExpandPath(tmhome), "config", "priv_validator_key.json")
+	stateFile := filepath.Join(internal.ExpandPath(tmhome), "data", "priv_validator_state.json")
+	fpv := privval.LoadFilePV(keyFile, stateFile)
+	pkb := [64]byte(fpv.Key.PrivKey.(ed25519.PrivKeyEd25519))
+	if err := ioutil.WriteFile(internal.ExpandPath(outputPath), pkb[:32], 0644); err != nil {
+		logger.Info("Failed to write private key", "output", outputPath, "err", err)
+		os.Exit(1)
+	}
+	logger.Info("Successfully wrote private key", "output", outputPath)
+}
+
+func main() {
+	rootCmd.Parse(os.Args[1:])
+	if rootCmd.NArg() == 0 || (rootCmd.NArg() == 1 && rootCmd.Arg(0) == "help") {
+		rootCmd.Usage()
+		os.Exit(0)
+	}
+
+	logger = log.NewFilter(logger, log.AllowInfo())
+
+	switch rootCmd.Arg(0) {
+	case "help":
+		switch rootCmd.Arg(1) {
+		case "run":
+			runCmd.Usage()
+		case "extract_key":
+			extractKeyCmd.Usage()
+		case "version":
+			versionCmd.Usage()
+		default:
+			fmt.Printf("Unrecognized command: %s\n", rootCmd.Arg(1))
+			os.Exit(1)
+		}
+	case "run":
+		runCmd.Parse(os.Args[2:])
+		runTestHarness(flagAcceptRetries, flagBindAddr, flagTMHome)
+	case "extract_key":
+		extractKeyCmd.Parse(os.Args[2:])
+		extractKey(flagTMHome, flagKeyOutputPath)
+	case "version":
+		fmt.Println(version.Version)
+	default:
+		fmt.Printf("Unrecognized command: %s\n", flag.Arg(0))
+		os.Exit(1)
+	}
+}

types/priv_validator.go

@@ -43,11 +43,20 @@ func (pvs PrivValidatorsByAddress) Swap(i, j int) {
 // MockPV implements PrivValidator without any safety or persistence.
 // Only use it for testing.
 type MockPV struct {
-	privKey crypto.PrivKey
+	privKey              crypto.PrivKey
+	breakProposalSigning bool
+	breakVoteSigning     bool
 }
 
 func NewMockPV() *MockPV {
-	return &MockPV{ed25519.GenPrivKey()}
+	return &MockPV{ed25519.GenPrivKey(), false, false}
+}
+
+// NewMockPVWithParams allows one to create a MockPV instance, but with finer
+// grained control over the operation of the mock validator. This is useful for
+// mocking test failures.
+func NewMockPVWithParams(privKey crypto.PrivKey, breakProposalSigning, breakVoteSigning bool) *MockPV {
+	return &MockPV{privKey, breakProposalSigning, breakVoteSigning}
 }
 
 // Implements PrivValidator.
@@ -57,7 +66,11 @@ func (pv *MockPV) GetPubKey() crypto.PubKey {
 
 // Implements PrivValidator.
 func (pv *MockPV) SignVote(chainID string, vote *Vote) error {
-	signBytes := vote.SignBytes(chainID)
+	useChainID := chainID
+	if pv.breakVoteSigning {
+		useChainID = "incorrect-chain-id"
+	}
+	signBytes := vote.SignBytes(useChainID)
 	sig, err := pv.privKey.Sign(signBytes)
 	if err != nil {
 		return err
@@ -68,7 +81,11 @@ func (pv *MockPV) SignVote(chainID string, vote *Vote) error {
 
 // Implements PrivValidator.
 func (pv *MockPV) SignProposal(chainID string, proposal *Proposal) error {
-	signBytes := proposal.SignBytes(chainID)
+	useChainID := chainID
+	if pv.breakProposalSigning {
+		useChainID = "incorrect-chain-id"
+	}
+	signBytes := proposal.SignBytes(useChainID)
 	sig, err := pv.privKey.Sign(signBytes)
 	if err != nil {
 		return err
@@ -107,5 +124,5 @@ func (pv *erroringMockPV) SignProposal(chainID string, proposal *Proposal) error
 
 // NewErroringMockPV returns a MockPV that fails on each signing request. Again, for testing only.
 func NewErroringMockPV() *erroringMockPV {
-	return &erroringMockPV{&MockPV{ed25519.GenPrivKey()}}
+	return &erroringMockPV{&MockPV{ed25519.GenPrivKey(), false, false}}
 }