You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

168 lines
4.7 KiB

Close and retry a RemoteSigner on err (#2923) * Close and recreate a RemoteSigner on err * Update changelog * Address Anton's comments / suggestions: - update changelog - restart TCPVal - shut down on `ErrUnexpectedResponse` * re-init remote signer client with fresh connection if Ping fails - add/update TODOs in secret connection - rename tcp.go -> tcp_client.go, same with ipc to clarify their purpose * account for `conn returned by waitConnection can be `nil` - also add TODO about RemoteSigner conn field * Tests for retrying: IPC / TCP - shorter info log on success - set conn and use it in tests to close conn * Tests for retrying: IPC / TCP - shorter info log on success - set conn and use it in tests to close conn - add rwmutex for conn field in IPC * comments and doc.go * fix ipc tests. fixes #2677 * use constants for tests * cleanup some error statements * fixes #2784, race in tests * remove print statement * minor fixes from review * update comment on sts spec * cosmetics * p2p/conn: add failing tests * p2p/conn: make SecretConnection thread safe * changelog * IPCVal signer refactor - use a .reset() method - don't use embedded RemoteSignerClient - guard RemoteSignerClient with mutex - drop the .conn - expose Close() on RemoteSignerClient * apply IPCVal refactor to TCPVal * remove mtx from RemoteSignerClient * consolidate IPCVal and TCPVal, fixes #3104 - done in tcp_client.go - now called SocketVal - takes a listener in the constructor - make tcpListener and unixListener contain all the differences * delete ipc files * introduce unix and tcp dialer for RemoteSigner * rename files - drop tcp_ prefix - rename priv_validator.go to file.go * bring back listener options * fix node * fix priv_val_server * fix node test * minor cleanup and comments
6 years ago
Close and retry a RemoteSigner on err (#2923) * Close and recreate a RemoteSigner on err * Update changelog * Address Anton's comments / suggestions: - update changelog - restart TCPVal - shut down on `ErrUnexpectedResponse` * re-init remote signer client with fresh connection if Ping fails - add/update TODOs in secret connection - rename tcp.go -> tcp_client.go, same with ipc to clarify their purpose * account for `conn returned by waitConnection can be `nil` - also add TODO about RemoteSigner conn field * Tests for retrying: IPC / TCP - shorter info log on success - set conn and use it in tests to close conn * Tests for retrying: IPC / TCP - shorter info log on success - set conn and use it in tests to close conn - add rwmutex for conn field in IPC * comments and doc.go * fix ipc tests. fixes #2677 * use constants for tests * cleanup some error statements * fixes #2784, race in tests * remove print statement * minor fixes from review * update comment on sts spec * cosmetics * p2p/conn: add failing tests * p2p/conn: make SecretConnection thread safe * changelog * IPCVal signer refactor - use a .reset() method - don't use embedded RemoteSignerClient - guard RemoteSignerClient with mutex - drop the .conn - expose Close() on RemoteSignerClient * apply IPCVal refactor to TCPVal * remove mtx from RemoteSignerClient * consolidate IPCVal and TCPVal, fixes #3104 - done in tcp_client.go - now called SocketVal - takes a listener in the constructor - make tcpListener and unixListener contain all the differences * delete ipc files * introduce unix and tcp dialer for RemoteSigner * rename files - drop tcp_ prefix - rename priv_validator.go to file.go * bring back listener options * fix node * fix priv_val_server * fix node test * minor cleanup and comments
6 years ago
privval: refactor Remote signers (#3370) This PR is related to #3107 and a continuation of #3351 It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction. Given two hosts A and B: Host A is listener/client Host B is dialer/server (contains the secret key) When A requires a signature, it needs to wait for B to dial in before it can issue a request. A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect. The original rationale behind this design was based on security. Host B only allows outbound connections to a list of whitelisted hosts. It is not possible to reach B unless B dials in. There are no listening/open ports in B. This PR results in the following changes: Refactors ping/heartbeat to avoid previously existing race conditions. Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow. Unifies and abstracts away the differences between unix and tcp sockets. A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj) The signer request handler (server side) is customizable to increase testability. Updates and extends unit tests A high level overview of the classes is as follows: Transport (endpoints): The following classes take care of establishing a connection SignerDialerEndpoint SignerListeningEndpoint SignerEndpoint groups common functionality (read/write/timeouts/etc.) Signing (client/server): The following classes take care of exchanging request/responses SignerClient SignerServer This PR also closes #3601 Commits: * refactoring - work in progress * reworking unit tests * Encapsulating and fixing unit tests * Improve tests * Clean up * Fix/improve unit tests * clean up tests * Improving service endpoint * fixing unit test * fix linter issues * avoid invalid cache values (improve later?) * complete implementation * wip * improved connection loop * Improve reconnections + fixing unit tests * addressing comments * small formatting changes * clean up * Update node/node.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * check during initialization * dropping connecting when writing fails * removing break * use t.log instead * unifying and using cmn.GetFreePort() * review fixes * reordering and unifying drop connection * closing instead of signalling * refactored service loop * removed superfluous brackets * GetPubKey can return errors * Revert "GetPubKey can return errors" This reverts commit 68c06f19b4650389d7e5ab1659b318889028202c. * adding entry to changelog * Update CHANGELOG_PENDING.md Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_client.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_dialer_endpoint.go Co-Authored-By: jleni <juan.leni@zondax.ch> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: jleni <juan.leni@zondax.ch> * updating node.go * review fixes * fixes linter * fixing unit test * small fixes in comments * addressing review comments * addressing review comments 2 * reverting suggestion * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_client_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * Update privval/signer_listener_endpoint_test.go Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com> * do not expose brokenSignerDialerEndpoint * clean up logging * unifying methods shorten test time signer also drops * reenabling pings * improving testability + unit test * fixing go fmt + unit test * remove unused code * Addressing review comments * simplifying connection workflow * fix linter/go import issue * using base service quit * updating comment * Simplifying design + adjusting names * fixing linter issues * refactoring test harness + fixes * Addressing review comments * cleaning up * adding additional error check
5 years ago
  1. package main
  2. import (
  3. "context"
  4. "crypto/tls"
  5. "crypto/x509"
  6. "flag"
  7. "fmt"
  8. "net"
  9. "net/http"
  10. "os"
  11. "time"
  12. grpc_prometheus "github.com/grpc-ecosystem/go-grpc-prometheus"
  13. "github.com/prometheus/client_golang/prometheus"
  14. "github.com/prometheus/client_golang/prometheus/promhttp"
  15. "google.golang.org/grpc"
  16. "google.golang.org/grpc/credentials"
  17. "github.com/tendermint/tendermint/libs/log"
  18. tmnet "github.com/tendermint/tendermint/libs/net"
  19. tmos "github.com/tendermint/tendermint/libs/os"
  20. "github.com/tendermint/tendermint/privval"
  21. grpcprivval "github.com/tendermint/tendermint/privval/grpc"
  22. privvalproto "github.com/tendermint/tendermint/proto/tendermint/privval"
  23. )
  24. var (
  25. // Create a metrics registry.
  26. reg = prometheus.NewRegistry()
  27. // Create some standard server metrics.
  28. grpcMetrics = grpc_prometheus.NewServerMetrics()
  29. )
  30. func main() {
  31. var (
  32. addr = flag.String("addr", "127.0.0.1:26659", "Address to listen on (host:port)")
  33. chainID = flag.String("chain-id", "mychain", "chain id")
  34. privValKeyPath = flag.String("priv-key", "", "priv val key file path")
  35. privValStatePath = flag.String("priv-state", "", "priv val state file path")
  36. insecure = flag.Bool("insecure", false, "allow server to run insecurely (no TLS)")
  37. certFile = flag.String("certfile", "", "absolute path to server certificate")
  38. keyFile = flag.String("keyfile", "", "absolute path to server key")
  39. rootCA = flag.String("rootcafile", "", "absolute path to root CA")
  40. prometheusAddr = flag.String("prometheus-addr", "", "address for prometheus endpoint (host:port)")
  41. logger = log.MustNewDefaultLogger(log.LogFormatPlain, log.LogLevelInfo, false).
  42. With("module", "priv_val")
  43. )
  44. flag.Parse()
  45. ctx, cancel := context.WithCancel(context.Background())
  46. defer cancel()
  47. logger.Info(
  48. "Starting private validator",
  49. "addr", *addr,
  50. "chainID", *chainID,
  51. "privKeyPath", *privValKeyPath,
  52. "privStatePath", *privValStatePath,
  53. "insecure", *insecure,
  54. "certFile", *certFile,
  55. "keyFile", *keyFile,
  56. "rootCA", *rootCA,
  57. )
  58. pv, err := privval.LoadFilePV(*privValKeyPath, *privValStatePath)
  59. if err != nil {
  60. fmt.Fprint(os.Stderr, err)
  61. os.Exit(1)
  62. }
  63. opts := []grpc.ServerOption{}
  64. if !*insecure {
  65. certificate, err := tls.LoadX509KeyPair(*certFile, *keyFile)
  66. if err != nil {
  67. fmt.Fprintf(os.Stderr, "failed to load X509 key pair: %v", err)
  68. os.Exit(1)
  69. }
  70. certPool := x509.NewCertPool()
  71. bs, err := os.ReadFile(*rootCA)
  72. if err != nil {
  73. fmt.Fprintf(os.Stderr, "failed to read client ca cert: %s", err)
  74. os.Exit(1)
  75. }
  76. if ok := certPool.AppendCertsFromPEM(bs); !ok {
  77. fmt.Fprintf(os.Stderr, "failed to append client certs")
  78. os.Exit(1)
  79. }
  80. tlsConfig := &tls.Config{
  81. ClientAuth: tls.RequireAndVerifyClientCert,
  82. Certificates: []tls.Certificate{certificate},
  83. ClientCAs: certPool,
  84. MinVersion: tls.VersionTLS13,
  85. }
  86. creds := grpc.Creds(credentials.NewTLS(tlsConfig))
  87. opts = append(opts, creds)
  88. logger.Info("SignerServer: Creating security credentials")
  89. } else {
  90. logger.Info("SignerServer: You are using an insecure gRPC connection!")
  91. }
  92. // add prometheus metrics for unary RPC calls
  93. opts = append(opts, grpc.UnaryInterceptor(grpc_prometheus.UnaryServerInterceptor))
  94. ss := grpcprivval.NewSignerServer(*chainID, pv, logger)
  95. protocol, address := tmnet.ProtocolAndAddress(*addr)
  96. lis, err := net.Listen(protocol, address)
  97. if err != nil {
  98. fmt.Fprintf(os.Stderr, "SignerServer: Failed to listen %v", err)
  99. os.Exit(1)
  100. }
  101. s := grpc.NewServer(opts...)
  102. privvalproto.RegisterPrivValidatorAPIServer(s, ss)
  103. var httpSrv *http.Server
  104. if *prometheusAddr != "" {
  105. httpSrv = registerPrometheus(*prometheusAddr, s)
  106. }
  107. logger.Info("SignerServer: Starting grpc server")
  108. if err := s.Serve(lis); err != nil {
  109. fmt.Fprintf(os.Stderr, "Unable to listen on port %s: %v", *addr, err)
  110. os.Exit(1)
  111. }
  112. // Stop upon receiving SIGTERM or CTRL-C.
  113. tmos.TrapSignal(ctx, logger, func() {
  114. logger.Debug("SignerServer: calling Close")
  115. if *prometheusAddr != "" {
  116. ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
  117. defer cancel()
  118. if err := httpSrv.Shutdown(ctx); err != nil {
  119. fmt.Fprintf(os.Stderr, "Unable to stop http server: %v", err)
  120. os.Exit(1)
  121. }
  122. }
  123. s.GracefulStop()
  124. })
  125. // Run forever.
  126. select {}
  127. }
  128. func registerPrometheus(addr string, s *grpc.Server) *http.Server {
  129. // Initialize all metrics.
  130. grpcMetrics.InitializeMetrics(s)
  131. // create http server to serve prometheus
  132. httpServer := &http.Server{Handler: promhttp.HandlerFor(reg, promhttp.HandlerOpts{}), Addr: addr}
  133. go func() {
  134. if err := httpServer.ListenAndServe(); err != nil {
  135. fmt.Fprintf(os.Stderr, "Unable to start a http server: %v", err)
  136. os.Exit(1)
  137. }
  138. }()
  139. return httpServer
  140. }