You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

166 lines
4.6 KiB

Close and retry a RemoteSigner on err (#2923) * Close and recreate a RemoteSigner on err * Update changelog * Address Anton's comments / suggestions: - update changelog - restart TCPVal - shut down on `ErrUnexpectedResponse` * re-init remote signer client with fresh connection if Ping fails - add/update TODOs in secret connection - rename tcp.go -> tcp_client.go, same with ipc to clarify their purpose * account for `conn returned by waitConnection can be `nil` - also add TODO about RemoteSigner conn field * Tests for retrying: IPC / TCP - shorter info log on success - set conn and use it in tests to close conn * Tests for retrying: IPC / TCP - shorter info log on success - set conn and use it in tests to close conn - add rwmutex for conn field in IPC * comments and doc.go * fix ipc tests. fixes #2677 * use constants for tests * cleanup some error statements * fixes #2784, race in tests * remove print statement * minor fixes from review * update comment on sts spec * cosmetics * p2p/conn: add failing tests * p2p/conn: make SecretConnection thread safe * changelog * IPCVal signer refactor - use a .reset() method - don't use embedded RemoteSignerClient - guard RemoteSignerClient with mutex - drop the .conn - expose Close() on RemoteSignerClient * apply IPCVal refactor to TCPVal * remove mtx from RemoteSignerClient * consolidate IPCVal and TCPVal, fixes #3104 - done in tcp_client.go - now called SocketVal - takes a listener in the constructor - make tcpListener and unixListener contain all the differences * delete ipc files * introduce unix and tcp dialer for RemoteSigner * rename files - drop tcp_ prefix - rename priv_validator.go to file.go * bring back listener options * fix node * fix priv_val_server * fix node test * minor cleanup and comments
6 years ago
Close and retry a RemoteSigner on err (#2923) * Close and recreate a RemoteSigner on err * Update changelog * Address Anton's comments / suggestions: - update changelog - restart TCPVal - shut down on `ErrUnexpectedResponse` * re-init remote signer client with fresh connection if Ping fails - add/update TODOs in secret connection - rename tcp.go -> tcp_client.go, same with ipc to clarify their purpose * account for `conn returned by waitConnection can be `nil` - also add TODO about RemoteSigner conn field * Tests for retrying: IPC / TCP - shorter info log on success - set conn and use it in tests to close conn * Tests for retrying: IPC / TCP - shorter info log on success - set conn and use it in tests to close conn - add rwmutex for conn field in IPC * comments and doc.go * fix ipc tests. fixes #2677 * use constants for tests * cleanup some error statements * fixes #2784, race in tests * remove print statement * minor fixes from review * update comment on sts spec * cosmetics * p2p/conn: add failing tests * p2p/conn: make SecretConnection thread safe * changelog * IPCVal signer refactor - use a .reset() method - don't use embedded RemoteSignerClient - guard RemoteSignerClient with mutex - drop the .conn - expose Close() on RemoteSignerClient * apply IPCVal refactor to TCPVal * remove mtx from RemoteSignerClient * consolidate IPCVal and TCPVal, fixes #3104 - done in tcp_client.go - now called SocketVal - takes a listener in the constructor - make tcpListener and unixListener contain all the differences * delete ipc files * introduce unix and tcp dialer for RemoteSigner * rename files - drop tcp_ prefix - rename priv_validator.go to file.go * bring back listener options * fix node * fix priv_val_server * fix node test * minor cleanup and comments
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
  1. package main
  2. import (
  3. "context"
  4. "crypto/tls"
  5. "crypto/x509"
  6. "flag"
  7. "fmt"
  8. "io/ioutil"
  9. "net"
  10. "net/http"
  11. "os"
  12. "time"
  13. grpc_prometheus "github.com/grpc-ecosystem/go-grpc-prometheus"
  14. "github.com/prometheus/client_golang/prometheus"
  15. "github.com/prometheus/client_golang/prometheus/promhttp"
  16. "google.golang.org/grpc"
  17. "google.golang.org/grpc/credentials"
  18. "github.com/tendermint/tendermint/libs/log"
  19. tmnet "github.com/tendermint/tendermint/libs/net"
  20. tmos "github.com/tendermint/tendermint/libs/os"
  21. "github.com/tendermint/tendermint/privval"
  22. grpcprivval "github.com/tendermint/tendermint/privval/grpc"
  23. privvalproto "github.com/tendermint/tendermint/proto/tendermint/privval"
  24. )
  25. var (
  26. // Create a metrics registry.
  27. reg = prometheus.NewRegistry()
  28. // Create some standard server metrics.
  29. grpcMetrics = grpc_prometheus.NewServerMetrics()
  30. )
  31. func main() {
  32. var (
  33. addr = flag.String("addr", "127.0.0.1:26659", "Address to listen on (host:port)")
  34. chainID = flag.String("chain-id", "mychain", "chain id")
  35. privValKeyPath = flag.String("priv-key", "", "priv val key file path")
  36. privValStatePath = flag.String("priv-state", "", "priv val state file path")
  37. insecure = flag.Bool("insecure", false, "allow server to run insecurely (no TLS)")
  38. certFile = flag.String("certfile", "", "absolute path to server certificate")
  39. keyFile = flag.String("keyfile", "", "absolute path to server key")
  40. rootCA = flag.String("rootcafile", "", "absolute path to root CA")
  41. prometheusAddr = flag.String("prometheus-addr", "", "address for prometheus endpoint (host:port)")
  42. logger = log.MustNewDefaultLogger(log.LogFormatPlain, log.LogLevelInfo, false).
  43. With("module", "priv_val")
  44. )
  45. flag.Parse()
  46. logger.Info(
  47. "Starting private validator",
  48. "addr", *addr,
  49. "chainID", *chainID,
  50. "privKeyPath", *privValKeyPath,
  51. "privStatePath", *privValStatePath,
  52. "insecure", *insecure,
  53. "certFile", *certFile,
  54. "keyFile", *keyFile,
  55. "rootCA", *rootCA,
  56. )
  57. pv, err := privval.LoadFilePV(*privValKeyPath, *privValStatePath)
  58. if err != nil {
  59. fmt.Fprint(os.Stderr, err)
  60. os.Exit(1)
  61. }
  62. opts := []grpc.ServerOption{}
  63. if !*insecure {
  64. certificate, err := tls.LoadX509KeyPair(*certFile, *keyFile)
  65. if err != nil {
  66. fmt.Fprintf(os.Stderr, "failed to load X509 key pair: %v", err)
  67. os.Exit(1)
  68. }
  69. certPool := x509.NewCertPool()
  70. bs, err := ioutil.ReadFile(*rootCA)
  71. if err != nil {
  72. fmt.Fprintf(os.Stderr, "failed to read client ca cert: %s", err)
  73. os.Exit(1)
  74. }
  75. if ok := certPool.AppendCertsFromPEM(bs); !ok {
  76. fmt.Fprintf(os.Stderr, "failed to append client certs")
  77. os.Exit(1)
  78. }
  79. tlsConfig := &tls.Config{
  80. ClientAuth: tls.RequireAndVerifyClientCert,
  81. Certificates: []tls.Certificate{certificate},
  82. ClientCAs: certPool,
  83. MinVersion: tls.VersionTLS13,
  84. }
  85. creds := grpc.Creds(credentials.NewTLS(tlsConfig))
  86. opts = append(opts, creds)
  87. logger.Info("SignerServer: Creating security credentials")
  88. } else {
  89. logger.Info("SignerServer: You are using an insecure gRPC connection!")
  90. }
  91. // add prometheus metrics for unary RPC calls
  92. opts = append(opts, grpc.UnaryInterceptor(grpc_prometheus.UnaryServerInterceptor))
  93. ss := grpcprivval.NewSignerServer(*chainID, pv, logger)
  94. protocol, address := tmnet.ProtocolAndAddress(*addr)
  95. lis, err := net.Listen(protocol, address)
  96. if err != nil {
  97. fmt.Fprintf(os.Stderr, "SignerServer: Failed to listen %v", err)
  98. os.Exit(1)
  99. }
  100. s := grpc.NewServer(opts...)
  101. privvalproto.RegisterPrivValidatorAPIServer(s, ss)
  102. var httpSrv *http.Server
  103. if *prometheusAddr != "" {
  104. httpSrv = registerPrometheus(*prometheusAddr, s)
  105. }
  106. logger.Info("SignerServer: Starting grpc server")
  107. if err := s.Serve(lis); err != nil {
  108. fmt.Fprintf(os.Stderr, "Unable to listen on port %s: %v", *addr, err)
  109. os.Exit(1)
  110. }
  111. // Stop upon receiving SIGTERM or CTRL-C.
  112. tmos.TrapSignal(logger, func() {
  113. logger.Debug("SignerServer: calling Close")
  114. if *prometheusAddr != "" {
  115. ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
  116. defer cancel()
  117. if err := httpSrv.Shutdown(ctx); err != nil {
  118. fmt.Fprintf(os.Stderr, "Unable to stop http server: %v", err)
  119. os.Exit(1)
  120. }
  121. }
  122. s.GracefulStop()
  123. })
  124. // Run forever.
  125. select {}
  126. }
  127. func registerPrometheus(addr string, s *grpc.Server) *http.Server {
  128. // Initialize all metrics.
  129. grpcMetrics.InitializeMetrics(s)
  130. // create http server to serve prometheus
  131. httpServer := &http.Server{Handler: promhttp.HandlerFor(reg, promhttp.HandlerOpts{}), Addr: addr}
  132. go func() {
  133. if err := httpServer.ListenAndServe(); err != nil {
  134. fmt.Fprintf(os.Stderr, "Unable to start a http server: %v", err)
  135. os.Exit(1)
  136. }
  137. }()
  138. return httpServer
  139. }