You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

1110 lines
31 KiB

p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
p2p: make PeerManager.DialNext() and EvictNext() block (#5947) See #5936 and #5938 for background. The plan was initially to have `DialNext()` and `EvictNext()` return a channel. However, implementing this became unnecessarily complicated and error-prone. As an example, the channel would be both consumed and populated (via method calls) by the same driving method (e.g. `Router.dialPeers()`) which could easily cause deadlocks where a method call blocked while sending on the channel that the caller itself was responsible for consuming (but couldn't since it was busy making the method call). It would also require a set of goroutines in the peer manager that would interact with the goroutines in the router in non-obvious ways, and fully populating the channel on startup could cause deadlocks with other startup tasks. Several issues like these made the solution hard to reason about. I therefore simply made `DialNext()` and `EvictNext()` block until the next peer was available, using internal triggers to wake these methods up in a non-blocking fashion when any relevant state changes occurred. This proved much simpler to reason about, since there are no goroutines in the peer manager (except for trivial retry timers), nor any blocking channel sends, and it instead relies entirely on the existing goroutine structure of the router for concurrency. This also happens to be the same pattern used by the `Transport.Accept()` API, following Go stdlib conventions, so all router goroutines end up using a consistent pattern as well.
4 years ago
  1. package p2p
  2. import (
  3. "context"
  4. "errors"
  5. "fmt"
  6. "io"
  7. "math/rand"
  8. "net"
  9. "runtime"
  10. "sync"
  11. "time"
  12. "github.com/gogo/protobuf/proto"
  13. "github.com/tendermint/tendermint/crypto"
  14. "github.com/tendermint/tendermint/libs/log"
  15. "github.com/tendermint/tendermint/libs/service"
  16. "github.com/tendermint/tendermint/types"
  17. )
  18. const queueBufferDefault = 32
  19. // Envelope contains a message with sender/receiver routing info.
  20. type Envelope struct {
  21. From types.NodeID // sender (empty if outbound)
  22. To types.NodeID // receiver (empty if inbound)
  23. Broadcast bool // send to all connected peers (ignores To)
  24. Message proto.Message // message payload
  25. // channelID is for internal Router use, set on outbound messages to inform
  26. // the sendPeer() goroutine which transport channel to use.
  27. //
  28. // FIXME: If we migrate the Transport API to a byte-oriented multi-stream
  29. // API, this will no longer be necessary since each channel will be mapped
  30. // onto a stream during channel/peer setup. See:
  31. // https://github.com/tendermint/spec/pull/227
  32. channelID ChannelID
  33. }
  34. // PeerError is a peer error reported via Channel.Error.
  35. //
  36. // FIXME: This currently just disconnects the peer, which is too simplistic.
  37. // For example, some errors should be logged, some should cause disconnects,
  38. // and some should ban the peer.
  39. //
  40. // FIXME: This should probably be replaced by a more general PeerBehavior
  41. // concept that can mark good and bad behavior and contributes to peer scoring.
  42. // It should possibly also allow reactors to request explicit actions, e.g.
  43. // disconnection or banning, in addition to doing this based on aggregates.
  44. type PeerError struct {
  45. NodeID types.NodeID
  46. Err error
  47. }
  48. // Channel is a bidirectional channel to exchange Protobuf messages with peers,
  49. // wrapped in Envelope to specify routing info (i.e. sender/receiver).
  50. type Channel struct {
  51. ID ChannelID
  52. In <-chan Envelope // inbound messages (peers to reactors)
  53. Out chan<- Envelope // outbound messages (reactors to peers)
  54. Error chan<- PeerError // peer error reporting
  55. messageType proto.Message // the channel's message type, used for unmarshaling
  56. closeCh chan struct{}
  57. closeOnce sync.Once
  58. }
  59. // NewChannel creates a new channel. It is primarily for internal and test
  60. // use, reactors should use Router.OpenChannel().
  61. func NewChannel(
  62. id ChannelID,
  63. messageType proto.Message,
  64. inCh <-chan Envelope,
  65. outCh chan<- Envelope,
  66. errCh chan<- PeerError,
  67. ) *Channel {
  68. return &Channel{
  69. ID: id,
  70. messageType: messageType,
  71. In: inCh,
  72. Out: outCh,
  73. Error: errCh,
  74. closeCh: make(chan struct{}),
  75. }
  76. }
  77. // Close closes the channel. Future sends on Out and Error will panic. The In
  78. // channel remains open to avoid having to synchronize Router senders, which
  79. // should use Done() to detect channel closure instead.
  80. func (c *Channel) Close() {
  81. c.closeOnce.Do(func() {
  82. close(c.closeCh)
  83. close(c.Out)
  84. close(c.Error)
  85. })
  86. }
  87. // Done returns a channel that's closed when Channel.Close() is called.
  88. func (c *Channel) Done() <-chan struct{} {
  89. return c.closeCh
  90. }
  91. // Wrapper is a Protobuf message that can contain a variety of inner messages
  92. // (e.g. via oneof fields). If a Channel's message type implements Wrapper, the
  93. // Router will automatically wrap outbound messages and unwrap inbound messages,
  94. // such that reactors do not have to do this themselves.
  95. type Wrapper interface {
  96. proto.Message
  97. // Wrap will take a message and wrap it in this one if possible.
  98. Wrap(proto.Message) error
  99. // Unwrap will unwrap the inner message contained in this message.
  100. Unwrap() (proto.Message, error)
  101. }
  102. // RouterOptions specifies options for a Router.
  103. type RouterOptions struct {
  104. // ResolveTimeout is the timeout for resolving NodeAddress URLs.
  105. // 0 means no timeout.
  106. ResolveTimeout time.Duration
  107. // DialTimeout is the timeout for dialing a peer. 0 means no timeout.
  108. DialTimeout time.Duration
  109. // HandshakeTimeout is the timeout for handshaking with a peer. 0 means
  110. // no timeout.
  111. HandshakeTimeout time.Duration
  112. // QueueType must be, "priority", or "fifo". Defaults to
  113. // "fifo".
  114. QueueType string
  115. // MaxIncomingConnectionAttempts rate limits the number of incoming connection
  116. // attempts per IP address. Defaults to 100.
  117. MaxIncomingConnectionAttempts uint
  118. // IncomingConnectionWindow describes how often an IP address
  119. // can attempt to create a new connection. Defaults to 10
  120. // milliseconds, and cannot be less than 1 millisecond.
  121. IncomingConnectionWindow time.Duration
  122. // FilterPeerByIP is used by the router to inject filtering
  123. // behavior for new incoming connections. The router passes
  124. // the remote IP of the incoming connection the port number as
  125. // arguments. Functions should return an error to reject the
  126. // peer.
  127. FilterPeerByIP func(context.Context, net.IP, uint16) error
  128. // FilterPeerByID is used by the router to inject filtering
  129. // behavior for new incoming connections. The router passes
  130. // the NodeID of the node before completing the connection,
  131. // but this occurs after the handshake is complete. Filter by
  132. // IP address to filter before the handshake. Functions should
  133. // return an error to reject the peer.
  134. FilterPeerByID func(context.Context, types.NodeID) error
  135. // DialSleep controls the amount of time that the router
  136. // sleeps between dialing peers. If not set, a default value
  137. // is used that sleeps for a (random) amount of time up to 3
  138. // seconds between submitting each peer to be dialed.
  139. DialSleep func(context.Context)
  140. // NumConcrruentDials controls how many parallel go routines
  141. // are used to dial peers. This defaults to the value of
  142. // runtime.NumCPU.
  143. NumConcurrentDials func() int
  144. }
  145. const (
  146. queueTypeFifo = "fifo"
  147. queueTypePriority = "priority"
  148. )
  149. // Validate validates router options.
  150. func (o *RouterOptions) Validate() error {
  151. switch o.QueueType {
  152. case "":
  153. o.QueueType = queueTypeFifo
  154. case queueTypeFifo, queueTypePriority:
  155. // pass
  156. default:
  157. return fmt.Errorf("queue type %q is not supported", o.QueueType)
  158. }
  159. switch {
  160. case o.IncomingConnectionWindow == 0:
  161. o.IncomingConnectionWindow = 100 * time.Millisecond
  162. case o.IncomingConnectionWindow < time.Millisecond:
  163. return fmt.Errorf("incomming connection window must be grater than 1m [%s]",
  164. o.IncomingConnectionWindow)
  165. }
  166. if o.MaxIncomingConnectionAttempts == 0 {
  167. o.MaxIncomingConnectionAttempts = 100
  168. }
  169. return nil
  170. }
  171. // Router manages peer connections and routes messages between peers and reactor
  172. // channels. It takes a PeerManager for peer lifecycle management (e.g. which
  173. // peers to dial and when) and a set of Transports for connecting and
  174. // communicating with peers.
  175. //
  176. // On startup, three main goroutines are spawned to maintain peer connections:
  177. //
  178. // dialPeers(): in a loop, calls PeerManager.DialNext() to get the next peer
  179. // address to dial and spawns a goroutine that dials the peer, handshakes
  180. // with it, and begins to route messages if successful.
  181. //
  182. // acceptPeers(): in a loop, waits for an inbound connection via
  183. // Transport.Accept() and spawns a goroutine that handshakes with it and
  184. // begins to route messages if successful.
  185. //
  186. // evictPeers(): in a loop, calls PeerManager.EvictNext() to get the next
  187. // peer to evict, and disconnects it by closing its message queue.
  188. //
  189. // When a peer is connected, an outbound peer message queue is registered in
  190. // peerQueues, and routePeer() is called to spawn off two additional goroutines:
  191. //
  192. // sendPeer(): waits for an outbound message from the peerQueues queue,
  193. // marshals it, and passes it to the peer transport which delivers it.
  194. //
  195. // receivePeer(): waits for an inbound message from the peer transport,
  196. // unmarshals it, and passes it to the appropriate inbound channel queue
  197. // in channelQueues.
  198. //
  199. // When a reactor opens a channel via OpenChannel, an inbound channel message
  200. // queue is registered in channelQueues, and a channel goroutine is spawned:
  201. //
  202. // routeChannel(): waits for an outbound message from the channel, looks
  203. // up the recipient peer's outbound message queue in peerQueues, and submits
  204. // the message to it.
  205. //
  206. // All channel sends in the router are blocking. It is the responsibility of the
  207. // queue interface in peerQueues and channelQueues to prioritize and drop
  208. // messages as appropriate during contention to prevent stalls and ensure good
  209. // quality of service.
  210. type Router struct {
  211. *service.BaseService
  212. logger log.Logger
  213. metrics *Metrics
  214. options RouterOptions
  215. nodeInfo types.NodeInfo
  216. privKey crypto.PrivKey
  217. peerManager *PeerManager
  218. chDescs []*ChannelDescriptor
  219. transports []Transport
  220. endpoints []Endpoint
  221. connTracker connectionTracker
  222. protocolTransports map[Protocol]Transport
  223. stopCh chan struct{} // signals Router shutdown
  224. peerMtx sync.RWMutex
  225. peerQueues map[types.NodeID]queue // outbound messages per peer for all channels
  226. // the channels that the peer queue has open
  227. peerChannels map[types.NodeID]channelIDs
  228. queueFactory func(int) queue
  229. // FIXME: We don't strictly need to use a mutex for this if we seal the
  230. // channels on router start. This depends on whether we want to allow
  231. // dynamic channels in the future.
  232. channelMtx sync.RWMutex
  233. channelQueues map[ChannelID]queue // inbound messages from all peers to a single channel
  234. channelMessages map[ChannelID]proto.Message
  235. }
  236. // NewRouter creates a new Router. The given Transports must already be
  237. // listening on appropriate interfaces, and will be closed by the Router when it
  238. // stops.
  239. func NewRouter(
  240. logger log.Logger,
  241. metrics *Metrics,
  242. nodeInfo types.NodeInfo,
  243. privKey crypto.PrivKey,
  244. peerManager *PeerManager,
  245. transports []Transport,
  246. endpoints []Endpoint,
  247. options RouterOptions,
  248. ) (*Router, error) {
  249. if err := options.Validate(); err != nil {
  250. return nil, err
  251. }
  252. router := &Router{
  253. logger: logger,
  254. metrics: metrics,
  255. nodeInfo: nodeInfo,
  256. privKey: privKey,
  257. connTracker: newConnTracker(
  258. options.MaxIncomingConnectionAttempts,
  259. options.IncomingConnectionWindow,
  260. ),
  261. chDescs: make([]*ChannelDescriptor, 0),
  262. transports: transports,
  263. endpoints: endpoints,
  264. protocolTransports: map[Protocol]Transport{},
  265. peerManager: peerManager,
  266. options: options,
  267. stopCh: make(chan struct{}),
  268. channelQueues: map[ChannelID]queue{},
  269. channelMessages: map[ChannelID]proto.Message{},
  270. peerQueues: map[types.NodeID]queue{},
  271. peerChannels: make(map[types.NodeID]channelIDs),
  272. }
  273. router.BaseService = service.NewBaseService(logger, "router", router)
  274. qf, err := router.createQueueFactory()
  275. if err != nil {
  276. return nil, err
  277. }
  278. router.queueFactory = qf
  279. for _, transport := range transports {
  280. for _, protocol := range transport.Protocols() {
  281. if _, ok := router.protocolTransports[protocol]; !ok {
  282. router.protocolTransports[protocol] = transport
  283. }
  284. }
  285. }
  286. return router, nil
  287. }
  288. func (r *Router) createQueueFactory() (func(int) queue, error) {
  289. switch r.options.QueueType {
  290. case queueTypeFifo:
  291. return newFIFOQueue, nil
  292. case queueTypePriority:
  293. return func(size int) queue {
  294. if size%2 != 0 {
  295. size++
  296. }
  297. q := newPQScheduler(r.logger, r.metrics, r.chDescs, uint(size)/2, uint(size)/2, defaultCapacity)
  298. q.start()
  299. return q
  300. }, nil
  301. default:
  302. return nil, fmt.Errorf("cannot construct queue of type %q", r.options.QueueType)
  303. }
  304. }
  305. // OpenChannel opens a new channel for the given message type. The caller must
  306. // close the channel when done, before stopping the Router. messageType is the
  307. // type of message passed through the channel (used for unmarshaling), which can
  308. // implement Wrapper to automatically (un)wrap multiple message types in a
  309. // wrapper message. The caller may provide a size to make the channel buffered,
  310. // which internally makes the inbound, outbound, and error channel buffered.
  311. func (r *Router) OpenChannel(chDesc *ChannelDescriptor) (*Channel, error) {
  312. r.channelMtx.Lock()
  313. defer r.channelMtx.Unlock()
  314. id := chDesc.ID
  315. if _, ok := r.channelQueues[id]; ok {
  316. return nil, fmt.Errorf("channel %v already exists", id)
  317. }
  318. r.chDescs = append(r.chDescs, chDesc)
  319. messageType := chDesc.MessageType
  320. queue := r.queueFactory(chDesc.RecvBufferCapacity)
  321. outCh := make(chan Envelope, chDesc.RecvBufferCapacity)
  322. errCh := make(chan PeerError, chDesc.RecvBufferCapacity)
  323. channel := NewChannel(id, messageType, queue.dequeue(), outCh, errCh)
  324. var wrapper Wrapper
  325. if w, ok := messageType.(Wrapper); ok {
  326. wrapper = w
  327. }
  328. r.channelQueues[id] = queue
  329. r.channelMessages[id] = messageType
  330. // add the channel to the nodeInfo if it's not already there.
  331. r.nodeInfo.AddChannel(uint16(chDesc.ID))
  332. for _, t := range r.transports {
  333. t.AddChannelDescriptors([]*ChannelDescriptor{chDesc})
  334. }
  335. go func() {
  336. defer func() {
  337. r.channelMtx.Lock()
  338. delete(r.channelQueues, id)
  339. delete(r.channelMessages, id)
  340. r.channelMtx.Unlock()
  341. queue.close()
  342. }()
  343. r.routeChannel(id, outCh, errCh, wrapper)
  344. }()
  345. return channel, nil
  346. }
  347. // routeChannel receives outbound channel messages and routes them to the
  348. // appropriate peer. It also receives peer errors and reports them to the peer
  349. // manager. It returns when either the outbound channel or error channel is
  350. // closed, or the Router is stopped. wrapper is an optional message wrapper
  351. // for messages, see Wrapper for details.
  352. func (r *Router) routeChannel(
  353. chID ChannelID,
  354. outCh <-chan Envelope,
  355. errCh <-chan PeerError,
  356. wrapper Wrapper,
  357. ) {
  358. for {
  359. select {
  360. case envelope, ok := <-outCh:
  361. if !ok {
  362. return
  363. }
  364. // Mark the envelope with the channel ID to allow sendPeer() to pass
  365. // it on to Transport.SendMessage().
  366. envelope.channelID = chID
  367. // wrap the message in a wrapper message, if requested
  368. if wrapper != nil {
  369. msg := proto.Clone(wrapper)
  370. if err := msg.(Wrapper).Wrap(envelope.Message); err != nil {
  371. r.Logger.Error("failed to wrap message", "channel", chID, "err", err)
  372. continue
  373. }
  374. envelope.Message = msg
  375. }
  376. // collect peer queues to pass the message via
  377. var queues []queue
  378. if envelope.Broadcast {
  379. r.peerMtx.RLock()
  380. queues = make([]queue, 0, len(r.peerQueues))
  381. for nodeID, q := range r.peerQueues {
  382. peerChs := r.peerChannels[nodeID]
  383. // check whether the peer is receiving on that channel
  384. if _, ok := peerChs[chID]; ok {
  385. queues = append(queues, q)
  386. }
  387. }
  388. r.peerMtx.RUnlock()
  389. } else {
  390. r.peerMtx.RLock()
  391. q, ok := r.peerQueues[envelope.To]
  392. contains := false
  393. if ok {
  394. peerChs := r.peerChannels[envelope.To]
  395. // check whether the peer is receiving on that channel
  396. _, contains = peerChs[chID]
  397. }
  398. r.peerMtx.RUnlock()
  399. if !ok {
  400. r.logger.Debug("dropping message for unconnected peer", "peer", envelope.To, "channel", chID)
  401. continue
  402. }
  403. if !contains {
  404. // reactor tried to send a message across a channel that the
  405. // peer doesn't have available. This is a known issue due to
  406. // how peer subscriptions work:
  407. // https://github.com/tendermint/tendermint/issues/6598
  408. continue
  409. }
  410. queues = []queue{q}
  411. }
  412. // send message to peers
  413. for _, q := range queues {
  414. start := time.Now().UTC()
  415. select {
  416. case q.enqueue() <- envelope:
  417. r.metrics.RouterPeerQueueSend.Observe(time.Since(start).Seconds())
  418. case <-q.closed():
  419. r.logger.Debug("dropping message for unconnected peer", "peer", envelope.To, "channel", chID)
  420. case <-r.stopCh:
  421. return
  422. }
  423. }
  424. case peerError, ok := <-errCh:
  425. if !ok {
  426. return
  427. }
  428. r.logger.Error("peer error, evicting", "peer", peerError.NodeID, "err", peerError.Err)
  429. r.peerManager.Errored(peerError.NodeID, peerError.Err)
  430. case <-r.stopCh:
  431. return
  432. }
  433. }
  434. }
  435. func (r *Router) numConccurentDials() int {
  436. if r.options.NumConcurrentDials == nil {
  437. return runtime.NumCPU()
  438. }
  439. return r.options.NumConcurrentDials()
  440. }
  441. func (r *Router) filterPeersIP(ctx context.Context, ip net.IP, port uint16) error {
  442. if r.options.FilterPeerByIP == nil {
  443. return nil
  444. }
  445. return r.options.FilterPeerByIP(ctx, ip, port)
  446. }
  447. func (r *Router) filterPeersID(ctx context.Context, id types.NodeID) error {
  448. if r.options.FilterPeerByID == nil {
  449. return nil
  450. }
  451. return r.options.FilterPeerByID(ctx, id)
  452. }
  453. func (r *Router) dialSleep(ctx context.Context) {
  454. if r.options.DialSleep == nil {
  455. const (
  456. maxDialerInterval = 3000
  457. minDialerInterval = 250
  458. )
  459. // nolint:gosec // G404: Use of weak random number generator
  460. dur := time.Duration(rand.Int63n(maxDialerInterval-minDialerInterval+1) + minDialerInterval)
  461. timer := time.NewTimer(dur * time.Millisecond)
  462. defer timer.Stop()
  463. select {
  464. case <-ctx.Done():
  465. case <-timer.C:
  466. }
  467. return
  468. }
  469. r.options.DialSleep(ctx)
  470. }
  471. // acceptPeers accepts inbound connections from peers on the given transport,
  472. // and spawns goroutines that route messages to/from them.
  473. func (r *Router) acceptPeers(transport Transport) {
  474. r.logger.Debug("starting accept routine", "transport", transport)
  475. ctx := r.stopCtx()
  476. for {
  477. conn, err := transport.Accept()
  478. switch err {
  479. case nil:
  480. case io.EOF:
  481. r.logger.Debug("stopping accept routine", "transport", transport)
  482. return
  483. default:
  484. r.logger.Error("failed to accept connection", "transport", transport, "err", err)
  485. return
  486. }
  487. incomingIP := conn.RemoteEndpoint().IP
  488. if err := r.connTracker.AddConn(incomingIP); err != nil {
  489. closeErr := conn.Close()
  490. r.logger.Debug("rate limiting incoming peer",
  491. "err", err,
  492. "ip", incomingIP.String(),
  493. "close_err", closeErr,
  494. )
  495. return
  496. }
  497. // Spawn a goroutine for the handshake, to avoid head-of-line blocking.
  498. go r.openConnection(ctx, conn)
  499. }
  500. }
  501. func (r *Router) openConnection(ctx context.Context, conn Connection) {
  502. defer conn.Close()
  503. defer r.connTracker.RemoveConn(conn.RemoteEndpoint().IP)
  504. re := conn.RemoteEndpoint()
  505. incomingIP := re.IP
  506. if err := r.filterPeersIP(ctx, incomingIP, re.Port); err != nil {
  507. r.logger.Debug("peer filtered by IP", "ip", incomingIP.String(), "err", err)
  508. return
  509. }
  510. // FIXME: The peer manager may reject the peer during Accepted()
  511. // after we've handshaked with the peer (to find out which peer it
  512. // is). However, because the handshake has no ack, the remote peer
  513. // will think the handshake was successful and start sending us
  514. // messages.
  515. //
  516. // This can cause problems in tests, where a disconnection can cause
  517. // the local node to immediately redial, while the remote node may
  518. // not have completed the disconnection yet and therefore reject the
  519. // reconnection attempt (since it thinks we're still connected from
  520. // before).
  521. //
  522. // The Router should do the handshake and have a final ack/fail
  523. // message to make sure both ends have accepted the connection, such
  524. // that it can be coordinated with the peer manager.
  525. peerInfo, err := r.handshakePeer(ctx, conn, "")
  526. switch {
  527. case errors.Is(err, context.Canceled):
  528. return
  529. case err != nil:
  530. r.logger.Error("peer handshake failed", "endpoint", conn, "err", err)
  531. return
  532. }
  533. if err := r.filterPeersID(ctx, peerInfo.NodeID); err != nil {
  534. r.logger.Debug("peer filtered by node ID", "node", peerInfo.NodeID, "err", err)
  535. return
  536. }
  537. if err := r.runWithPeerMutex(func() error { return r.peerManager.Accepted(peerInfo.NodeID) }); err != nil {
  538. r.logger.Error("failed to accept connection",
  539. "op", "incoming/accepted", "peer", peerInfo.NodeID, "err", err)
  540. return
  541. }
  542. r.routePeer(peerInfo.NodeID, conn, toChannelIDs(peerInfo.Channels))
  543. }
  544. // dialPeers maintains outbound connections to peers by dialing them.
  545. func (r *Router) dialPeers() {
  546. r.logger.Debug("starting dial routine")
  547. ctx := r.stopCtx()
  548. addresses := make(chan NodeAddress)
  549. wg := &sync.WaitGroup{}
  550. // Start a limited number of goroutines to dial peers in
  551. // parallel. the goal is to avoid starting an unbounded number
  552. // of goroutines thereby spamming the network, but also being
  553. // able to add peers at a reasonable pace, though the number
  554. // is somewhat arbitrary. The action is further throttled by a
  555. // sleep after sending to the addresses channel.
  556. for i := 0; i < r.numConccurentDials(); i++ {
  557. wg.Add(1)
  558. go func() {
  559. defer wg.Done()
  560. for {
  561. select {
  562. case <-ctx.Done():
  563. return
  564. case address := <-addresses:
  565. r.connectPeer(ctx, address)
  566. }
  567. }
  568. }()
  569. }
  570. LOOP:
  571. for {
  572. address, err := r.peerManager.DialNext(ctx)
  573. switch {
  574. case errors.Is(err, context.Canceled):
  575. r.logger.Debug("stopping dial routine")
  576. break LOOP
  577. case err != nil:
  578. r.logger.Error("failed to find next peer to dial", "err", err)
  579. break LOOP
  580. }
  581. select {
  582. case addresses <- address:
  583. // this jitters the frequency that we call
  584. // DialNext and prevents us from attempting to
  585. // create connections too quickly.
  586. r.dialSleep(ctx)
  587. continue
  588. case <-ctx.Done():
  589. close(addresses)
  590. break LOOP
  591. }
  592. }
  593. wg.Wait()
  594. }
  595. func (r *Router) connectPeer(ctx context.Context, address NodeAddress) {
  596. conn, err := r.dialPeer(ctx, address)
  597. switch {
  598. case errors.Is(err, context.Canceled):
  599. return
  600. case err != nil:
  601. r.logger.Error("failed to dial peer", "peer", address, "err", err)
  602. if err = r.peerManager.DialFailed(address); err != nil {
  603. r.logger.Error("failed to report dial failure", "peer", address, "err", err)
  604. }
  605. return
  606. }
  607. peerInfo, err := r.handshakePeer(ctx, conn, address.NodeID)
  608. switch {
  609. case errors.Is(err, context.Canceled):
  610. conn.Close()
  611. return
  612. case err != nil:
  613. r.logger.Error("failed to handshake with peer", "peer", address, "err", err)
  614. if err = r.peerManager.DialFailed(address); err != nil {
  615. r.logger.Error("failed to report dial failure", "peer", address, "err", err)
  616. }
  617. conn.Close()
  618. return
  619. }
  620. if err := r.runWithPeerMutex(func() error { return r.peerManager.Dialed(address) }); err != nil {
  621. r.logger.Error("failed to dial peer",
  622. "op", "outgoing/dialing", "peer", address.NodeID, "err", err)
  623. conn.Close()
  624. return
  625. }
  626. // routePeer (also) calls connection close
  627. go r.routePeer(address.NodeID, conn, toChannelIDs(peerInfo.Channels))
  628. }
  629. func (r *Router) getOrMakeQueue(peerID types.NodeID, channels channelIDs) queue {
  630. r.peerMtx.Lock()
  631. defer r.peerMtx.Unlock()
  632. if peerQueue, ok := r.peerQueues[peerID]; ok {
  633. return peerQueue
  634. }
  635. peerQueue := r.queueFactory(queueBufferDefault)
  636. r.peerQueues[peerID] = peerQueue
  637. r.peerChannels[peerID] = channels
  638. return peerQueue
  639. }
  640. // dialPeer connects to a peer by dialing it.
  641. func (r *Router) dialPeer(ctx context.Context, address NodeAddress) (Connection, error) {
  642. resolveCtx := ctx
  643. if r.options.ResolveTimeout > 0 {
  644. var cancel context.CancelFunc
  645. resolveCtx, cancel = context.WithTimeout(resolveCtx, r.options.ResolveTimeout)
  646. defer cancel()
  647. }
  648. r.logger.Debug("resolving peer address", "peer", address)
  649. endpoints, err := address.Resolve(resolveCtx)
  650. switch {
  651. case err != nil:
  652. return nil, fmt.Errorf("failed to resolve address %q: %w", address, err)
  653. case len(endpoints) == 0:
  654. return nil, fmt.Errorf("address %q did not resolve to any endpoints", address)
  655. }
  656. for _, endpoint := range endpoints {
  657. transport, ok := r.protocolTransports[endpoint.Protocol]
  658. if !ok {
  659. r.logger.Error("no transport found for protocol", "endpoint", endpoint)
  660. continue
  661. }
  662. dialCtx := ctx
  663. if r.options.DialTimeout > 0 {
  664. var cancel context.CancelFunc
  665. dialCtx, cancel = context.WithTimeout(dialCtx, r.options.DialTimeout)
  666. defer cancel()
  667. }
  668. // FIXME: When we dial and handshake the peer, we should pass it
  669. // appropriate address(es) it can use to dial us back. It can't use our
  670. // remote endpoint, since TCP uses different port numbers for outbound
  671. // connections than it does for inbound. Also, we may need to vary this
  672. // by the peer's endpoint, since e.g. a peer on 192.168.0.0 can reach us
  673. // on a private address on this endpoint, but a peer on the public
  674. // Internet can't and needs a different public address.
  675. conn, err := transport.Dial(dialCtx, endpoint)
  676. if err != nil {
  677. r.logger.Error("failed to dial endpoint", "peer", address.NodeID, "endpoint", endpoint, "err", err)
  678. } else {
  679. r.logger.Debug("dialed peer", "peer", address.NodeID, "endpoint", endpoint)
  680. return conn, nil
  681. }
  682. }
  683. return nil, errors.New("all endpoints failed")
  684. }
  685. // handshakePeer handshakes with a peer, validating the peer's information. If
  686. // expectID is given, we check that the peer's info matches it.
  687. func (r *Router) handshakePeer(
  688. ctx context.Context,
  689. conn Connection,
  690. expectID types.NodeID,
  691. ) (types.NodeInfo, error) {
  692. if r.options.HandshakeTimeout > 0 {
  693. var cancel context.CancelFunc
  694. ctx, cancel = context.WithTimeout(ctx, r.options.HandshakeTimeout)
  695. defer cancel()
  696. }
  697. peerInfo, peerKey, err := conn.Handshake(ctx, r.nodeInfo, r.privKey)
  698. if err != nil {
  699. return peerInfo, err
  700. }
  701. if err = peerInfo.Validate(); err != nil {
  702. return peerInfo, fmt.Errorf("invalid handshake NodeInfo: %w", err)
  703. }
  704. if types.NodeIDFromPubKey(peerKey) != peerInfo.NodeID {
  705. return peerInfo, fmt.Errorf("peer's public key did not match its node ID %q (expected %q)",
  706. peerInfo.NodeID, types.NodeIDFromPubKey(peerKey))
  707. }
  708. if expectID != "" && expectID != peerInfo.NodeID {
  709. return peerInfo, fmt.Errorf("expected to connect with peer %q, got %q",
  710. expectID, peerInfo.NodeID)
  711. }
  712. if err := r.nodeInfo.CompatibleWith(peerInfo); err != nil {
  713. return peerInfo, ErrRejected{
  714. err: err,
  715. id: peerInfo.ID(),
  716. isIncompatible: true,
  717. }
  718. }
  719. return peerInfo, nil
  720. }
  721. func (r *Router) runWithPeerMutex(fn func() error) error {
  722. r.peerMtx.Lock()
  723. defer r.peerMtx.Unlock()
  724. return fn()
  725. }
  726. // routePeer routes inbound and outbound messages between a peer and the reactor
  727. // channels. It will close the given connection and send queue when done, or if
  728. // they are closed elsewhere it will cause this method to shut down and return.
  729. func (r *Router) routePeer(peerID types.NodeID, conn Connection, channels channelIDs) {
  730. r.metrics.Peers.Add(1)
  731. r.peerManager.Ready(peerID)
  732. sendQueue := r.getOrMakeQueue(peerID, channels)
  733. defer func() {
  734. r.peerMtx.Lock()
  735. delete(r.peerQueues, peerID)
  736. delete(r.peerChannels, peerID)
  737. r.peerMtx.Unlock()
  738. sendQueue.close()
  739. r.peerManager.Disconnected(peerID)
  740. r.metrics.Peers.Add(-1)
  741. }()
  742. r.logger.Info("peer connected", "peer", peerID, "endpoint", conn)
  743. errCh := make(chan error, 2)
  744. go func() {
  745. errCh <- r.receivePeer(peerID, conn)
  746. }()
  747. go func() {
  748. errCh <- r.sendPeer(peerID, conn, sendQueue)
  749. }()
  750. err := <-errCh
  751. _ = conn.Close()
  752. sendQueue.close()
  753. if e := <-errCh; err == nil {
  754. // The first err was nil, so we update it with the second err, which may
  755. // or may not be nil.
  756. err = e
  757. }
  758. switch err {
  759. case nil, io.EOF:
  760. r.logger.Info("peer disconnected", "peer", peerID, "endpoint", conn)
  761. default:
  762. r.logger.Error("peer failure", "peer", peerID, "endpoint", conn, "err", err)
  763. }
  764. }
  765. // receivePeer receives inbound messages from a peer, deserializes them and
  766. // passes them on to the appropriate channel.
  767. func (r *Router) receivePeer(peerID types.NodeID, conn Connection) error {
  768. for {
  769. chID, bz, err := conn.ReceiveMessage()
  770. if err != nil {
  771. return err
  772. }
  773. r.channelMtx.RLock()
  774. queue, ok := r.channelQueues[chID]
  775. messageType := r.channelMessages[chID]
  776. r.channelMtx.RUnlock()
  777. if !ok {
  778. r.logger.Debug("dropping message for unknown channel", "peer", peerID, "channel", chID)
  779. continue
  780. }
  781. msg := proto.Clone(messageType)
  782. if err := proto.Unmarshal(bz, msg); err != nil {
  783. r.logger.Error("message decoding failed, dropping message", "peer", peerID, "err", err)
  784. continue
  785. }
  786. if wrapper, ok := msg.(Wrapper); ok {
  787. msg, err = wrapper.Unwrap()
  788. if err != nil {
  789. r.logger.Error("failed to unwrap message", "err", err)
  790. continue
  791. }
  792. }
  793. start := time.Now().UTC()
  794. select {
  795. case queue.enqueue() <- Envelope{From: peerID, Message: msg}:
  796. r.metrics.PeerReceiveBytesTotal.With(
  797. "chID", fmt.Sprint(chID),
  798. "peer_id", string(peerID),
  799. "message_type", r.metrics.ValueToMetricLabel(msg)).Add(float64(proto.Size(msg)))
  800. r.metrics.RouterChannelQueueSend.Observe(time.Since(start).Seconds())
  801. r.logger.Debug("received message", "peer", peerID, "message", msg)
  802. case <-queue.closed():
  803. r.logger.Debug("channel closed, dropping message", "peer", peerID, "channel", chID)
  804. case <-r.stopCh:
  805. return nil
  806. }
  807. }
  808. }
  809. // sendPeer sends queued messages to a peer.
  810. func (r *Router) sendPeer(peerID types.NodeID, conn Connection, peerQueue queue) error {
  811. for {
  812. start := time.Now().UTC()
  813. select {
  814. case envelope := <-peerQueue.dequeue():
  815. r.metrics.RouterPeerQueueRecv.Observe(time.Since(start).Seconds())
  816. if envelope.Message == nil {
  817. r.logger.Error("dropping nil message", "peer", peerID)
  818. continue
  819. }
  820. bz, err := proto.Marshal(envelope.Message)
  821. if err != nil {
  822. r.logger.Error("failed to marshal message", "peer", peerID, "err", err)
  823. continue
  824. }
  825. if err = conn.SendMessage(envelope.channelID, bz); err != nil {
  826. return err
  827. }
  828. r.logger.Debug("sent message", "peer", envelope.To, "message", envelope.Message)
  829. case <-peerQueue.closed():
  830. return nil
  831. case <-r.stopCh:
  832. return nil
  833. }
  834. }
  835. }
  836. // evictPeers evicts connected peers as requested by the peer manager.
  837. func (r *Router) evictPeers() {
  838. r.logger.Debug("starting evict routine")
  839. ctx := r.stopCtx()
  840. for {
  841. peerID, err := r.peerManager.EvictNext(ctx)
  842. switch {
  843. case errors.Is(err, context.Canceled):
  844. r.logger.Debug("stopping evict routine")
  845. return
  846. case err != nil:
  847. r.logger.Error("failed to find next peer to evict", "err", err)
  848. return
  849. }
  850. r.logger.Info("evicting peer", "peer", peerID)
  851. r.peerMtx.RLock()
  852. queue, ok := r.peerQueues[peerID]
  853. r.peerMtx.RUnlock()
  854. if ok {
  855. queue.close()
  856. }
  857. }
  858. }
  859. // NodeInfo returns a copy of the current NodeInfo. Used for testing.
  860. func (r *Router) NodeInfo() types.NodeInfo {
  861. return r.nodeInfo.Copy()
  862. }
  863. // OnStart implements service.Service.
  864. func (r *Router) OnStart() error {
  865. for _, transport := range r.transports {
  866. for _, endpoint := range r.endpoints {
  867. if err := transport.Listen(endpoint); err != nil {
  868. return err
  869. }
  870. }
  871. }
  872. r.Logger.Info(
  873. "starting router",
  874. "node_id", r.nodeInfo.NodeID,
  875. "channels", r.nodeInfo.Channels,
  876. "listen_addr", r.nodeInfo.ListenAddr,
  877. "transports", len(r.transports),
  878. )
  879. go r.dialPeers()
  880. go r.evictPeers()
  881. for _, transport := range r.transports {
  882. go r.acceptPeers(transport)
  883. }
  884. return nil
  885. }
  886. // OnStop implements service.Service.
  887. //
  888. // All channels must be closed by OpenChannel() callers before stopping the
  889. // router, to prevent blocked channel sends in reactors. Channels are not closed
  890. // here, since that would cause any reactor senders to panic, so it is the
  891. // sender's responsibility.
  892. func (r *Router) OnStop() {
  893. // Signal router shutdown.
  894. close(r.stopCh)
  895. // Close transport listeners (unblocks Accept calls).
  896. for _, transport := range r.transports {
  897. if err := transport.Close(); err != nil {
  898. r.logger.Error("failed to close transport", "transport", transport, "err", err)
  899. }
  900. }
  901. // Collect all remaining queues, and wait for them to close.
  902. queues := []queue{}
  903. r.channelMtx.RLock()
  904. for _, q := range r.channelQueues {
  905. queues = append(queues, q)
  906. }
  907. r.channelMtx.RUnlock()
  908. r.peerMtx.RLock()
  909. for _, q := range r.peerQueues {
  910. queues = append(queues, q)
  911. }
  912. r.peerMtx.RUnlock()
  913. for _, q := range queues {
  914. <-q.closed()
  915. }
  916. }
  917. // stopCtx returns a new context that is canceled when the router stops.
  918. func (r *Router) stopCtx() context.Context {
  919. ctx, cancel := context.WithCancel(context.Background())
  920. go func() {
  921. <-r.stopCh
  922. cancel()
  923. }()
  924. return ctx
  925. }
  926. type channelIDs map[ChannelID]struct{}
  927. func toChannelIDs(bytes []byte) channelIDs {
  928. c := make(map[ChannelID]struct{}, len(bytes))
  929. for _, b := range bytes {
  930. c[ChannelID(b)] = struct{}{}
  931. }
  932. return c
  933. }