You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

774 lines
23 KiB

8 years ago
8 years ago
7 years ago
8 years ago
8 years ago
8 years ago
7 years ago
7 years ago
7 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
new pubsub package comment out failing consensus tests for now rewrite rpc httpclient to use new pubsub package import pubsub as tmpubsub, query as tmquery make event IDs constants EventKey -> EventTypeKey rename EventsPubsub to PubSub mempool does not use pubsub rename eventsSub to pubsub new subscribe API fix channel size issues and consensus tests bugs refactor rpc client add missing discardFromChan method add mutex rename pubsub to eventBus remove IsRunning from WSRPCConnection interface (not needed) add a comment in broadcastNewRoundStepsAndVotes rename registerEventCallbacks to broadcastNewRoundStepsAndVotes See https://dave.cheney.net/2014/03/19/channel-axioms stop eventBuses after reactor tests remove unnecessary Unsubscribe return subscribe helper function move discardFromChan to where it is used subscribe now returns an err this gives us ability to refuse to subscribe if pubsub is at its max capacity. use context for control overflow cache queries handle err when subscribing in replay_test rename testClientID to testSubscriber extract var set channel buffer capacity to 1 in replay_file fix byzantine_test unsubscribe from single event, not all events refactor httpclient to return events to appropriate channels return failing testReplayCrashBeforeWriteVote test fix TestValidatorSetChanges refactor code a bit fix testReplayCrashBeforeWriteVote add comment fix TestValidatorSetChanges fixes from Bucky's review update comment [ci skip] test TxEventBuffer update changelog fix TestValidatorSetChanges (2nd attempt) only do wg.Done when no errors benchmark event bus create pubsub server inside NewEventBus only expose config params (later if needed) set buffer capacity to 0 so we are not testing cache new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ} This should allow to subscribe to all transactions! or a specific one using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'" use TimeoutCommit instead of afterPublishEventNewBlockTimeout TimeoutCommit is the time a node waits after committing a block, before it goes into the next height. So it will finish everything from the last block, but then wait a bit. The idea is this gives it time to hear more votes from other validators, to strengthen the commit it includes in the next block. But it also gives it time to hear about new transactions. waitForBlockWithUpdatedVals rewrite WAL crash tests Task: test that we can recover from any WAL crash. Solution: the old tests were relying on event hub being run in the same thread (we were injecting the private validator's last signature). when considering a rewrite, we considered two possible solutions: write a "fuzzy" testing system where WAL is crashing upon receiving a new message, or inject failures and trigger them in tests using something like https://github.com/coreos/gofail. remove sleep no cs.Lock around wal.Save test different cases (empty block, non-empty block, ...) comments add comments test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks fixes as per Bucky's last review reset subscriptions on UnsubscribeAll use a simple counter to track message for which we panicked also, set a smaller part size for all test cases
8 years ago
new pubsub package comment out failing consensus tests for now rewrite rpc httpclient to use new pubsub package import pubsub as tmpubsub, query as tmquery make event IDs constants EventKey -> EventTypeKey rename EventsPubsub to PubSub mempool does not use pubsub rename eventsSub to pubsub new subscribe API fix channel size issues and consensus tests bugs refactor rpc client add missing discardFromChan method add mutex rename pubsub to eventBus remove IsRunning from WSRPCConnection interface (not needed) add a comment in broadcastNewRoundStepsAndVotes rename registerEventCallbacks to broadcastNewRoundStepsAndVotes See https://dave.cheney.net/2014/03/19/channel-axioms stop eventBuses after reactor tests remove unnecessary Unsubscribe return subscribe helper function move discardFromChan to where it is used subscribe now returns an err this gives us ability to refuse to subscribe if pubsub is at its max capacity. use context for control overflow cache queries handle err when subscribing in replay_test rename testClientID to testSubscriber extract var set channel buffer capacity to 1 in replay_file fix byzantine_test unsubscribe from single event, not all events refactor httpclient to return events to appropriate channels return failing testReplayCrashBeforeWriteVote test fix TestValidatorSetChanges refactor code a bit fix testReplayCrashBeforeWriteVote add comment fix TestValidatorSetChanges fixes from Bucky's review update comment [ci skip] test TxEventBuffer update changelog fix TestValidatorSetChanges (2nd attempt) only do wg.Done when no errors benchmark event bus create pubsub server inside NewEventBus only expose config params (later if needed) set buffer capacity to 0 so we are not testing cache new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ} This should allow to subscribe to all transactions! or a specific one using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'" use TimeoutCommit instead of afterPublishEventNewBlockTimeout TimeoutCommit is the time a node waits after committing a block, before it goes into the next height. So it will finish everything from the last block, but then wait a bit. The idea is this gives it time to hear more votes from other validators, to strengthen the commit it includes in the next block. But it also gives it time to hear about new transactions. waitForBlockWithUpdatedVals rewrite WAL crash tests Task: test that we can recover from any WAL crash. Solution: the old tests were relying on event hub being run in the same thread (we were injecting the private validator's last signature). when considering a rewrite, we considered two possible solutions: write a "fuzzy" testing system where WAL is crashing upon receiving a new message, or inject failures and trigger them in tests using something like https://github.com/coreos/gofail. remove sleep no cs.Lock around wal.Save test different cases (empty block, non-empty block, ...) comments add comments test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks fixes as per Bucky's last review reset subscriptions on UnsubscribeAll use a simple counter to track message for which we panicked also, set a smaller part size for all test cases
8 years ago
new pubsub package comment out failing consensus tests for now rewrite rpc httpclient to use new pubsub package import pubsub as tmpubsub, query as tmquery make event IDs constants EventKey -> EventTypeKey rename EventsPubsub to PubSub mempool does not use pubsub rename eventsSub to pubsub new subscribe API fix channel size issues and consensus tests bugs refactor rpc client add missing discardFromChan method add mutex rename pubsub to eventBus remove IsRunning from WSRPCConnection interface (not needed) add a comment in broadcastNewRoundStepsAndVotes rename registerEventCallbacks to broadcastNewRoundStepsAndVotes See https://dave.cheney.net/2014/03/19/channel-axioms stop eventBuses after reactor tests remove unnecessary Unsubscribe return subscribe helper function move discardFromChan to where it is used subscribe now returns an err this gives us ability to refuse to subscribe if pubsub is at its max capacity. use context for control overflow cache queries handle err when subscribing in replay_test rename testClientID to testSubscriber extract var set channel buffer capacity to 1 in replay_file fix byzantine_test unsubscribe from single event, not all events refactor httpclient to return events to appropriate channels return failing testReplayCrashBeforeWriteVote test fix TestValidatorSetChanges refactor code a bit fix testReplayCrashBeforeWriteVote add comment fix TestValidatorSetChanges fixes from Bucky's review update comment [ci skip] test TxEventBuffer update changelog fix TestValidatorSetChanges (2nd attempt) only do wg.Done when no errors benchmark event bus create pubsub server inside NewEventBus only expose config params (later if needed) set buffer capacity to 0 so we are not testing cache new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ} This should allow to subscribe to all transactions! or a specific one using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'" use TimeoutCommit instead of afterPublishEventNewBlockTimeout TimeoutCommit is the time a node waits after committing a block, before it goes into the next height. So it will finish everything from the last block, but then wait a bit. The idea is this gives it time to hear more votes from other validators, to strengthen the commit it includes in the next block. But it also gives it time to hear about new transactions. waitForBlockWithUpdatedVals rewrite WAL crash tests Task: test that we can recover from any WAL crash. Solution: the old tests were relying on event hub being run in the same thread (we were injecting the private validator's last signature). when considering a rewrite, we considered two possible solutions: write a "fuzzy" testing system where WAL is crashing upon receiving a new message, or inject failures and trigger them in tests using something like https://github.com/coreos/gofail. remove sleep no cs.Lock around wal.Save test different cases (empty block, non-empty block, ...) comments add comments test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks fixes as per Bucky's last review reset subscriptions on UnsubscribeAll use a simple counter to track message for which we panicked also, set a smaller part size for all test cases
8 years ago
8 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
new pubsub package comment out failing consensus tests for now rewrite rpc httpclient to use new pubsub package import pubsub as tmpubsub, query as tmquery make event IDs constants EventKey -> EventTypeKey rename EventsPubsub to PubSub mempool does not use pubsub rename eventsSub to pubsub new subscribe API fix channel size issues and consensus tests bugs refactor rpc client add missing discardFromChan method add mutex rename pubsub to eventBus remove IsRunning from WSRPCConnection interface (not needed) add a comment in broadcastNewRoundStepsAndVotes rename registerEventCallbacks to broadcastNewRoundStepsAndVotes See https://dave.cheney.net/2014/03/19/channel-axioms stop eventBuses after reactor tests remove unnecessary Unsubscribe return subscribe helper function move discardFromChan to where it is used subscribe now returns an err this gives us ability to refuse to subscribe if pubsub is at its max capacity. use context for control overflow cache queries handle err when subscribing in replay_test rename testClientID to testSubscriber extract var set channel buffer capacity to 1 in replay_file fix byzantine_test unsubscribe from single event, not all events refactor httpclient to return events to appropriate channels return failing testReplayCrashBeforeWriteVote test fix TestValidatorSetChanges refactor code a bit fix testReplayCrashBeforeWriteVote add comment fix TestValidatorSetChanges fixes from Bucky's review update comment [ci skip] test TxEventBuffer update changelog fix TestValidatorSetChanges (2nd attempt) only do wg.Done when no errors benchmark event bus create pubsub server inside NewEventBus only expose config params (later if needed) set buffer capacity to 0 so we are not testing cache new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ} This should allow to subscribe to all transactions! or a specific one using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'" use TimeoutCommit instead of afterPublishEventNewBlockTimeout TimeoutCommit is the time a node waits after committing a block, before it goes into the next height. So it will finish everything from the last block, but then wait a bit. The idea is this gives it time to hear more votes from other validators, to strengthen the commit it includes in the next block. But it also gives it time to hear about new transactions. waitForBlockWithUpdatedVals rewrite WAL crash tests Task: test that we can recover from any WAL crash. Solution: the old tests were relying on event hub being run in the same thread (we were injecting the private validator's last signature). when considering a rewrite, we considered two possible solutions: write a "fuzzy" testing system where WAL is crashing upon receiving a new message, or inject failures and trigger them in tests using something like https://github.com/coreos/gofail. remove sleep no cs.Lock around wal.Save test different cases (empty block, non-empty block, ...) comments add comments test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks fixes as per Bucky's last review reset subscriptions on UnsubscribeAll use a simple counter to track message for which we panicked also, set a smaller part size for all test cases
8 years ago
new pubsub package comment out failing consensus tests for now rewrite rpc httpclient to use new pubsub package import pubsub as tmpubsub, query as tmquery make event IDs constants EventKey -> EventTypeKey rename EventsPubsub to PubSub mempool does not use pubsub rename eventsSub to pubsub new subscribe API fix channel size issues and consensus tests bugs refactor rpc client add missing discardFromChan method add mutex rename pubsub to eventBus remove IsRunning from WSRPCConnection interface (not needed) add a comment in broadcastNewRoundStepsAndVotes rename registerEventCallbacks to broadcastNewRoundStepsAndVotes See https://dave.cheney.net/2014/03/19/channel-axioms stop eventBuses after reactor tests remove unnecessary Unsubscribe return subscribe helper function move discardFromChan to where it is used subscribe now returns an err this gives us ability to refuse to subscribe if pubsub is at its max capacity. use context for control overflow cache queries handle err when subscribing in replay_test rename testClientID to testSubscriber extract var set channel buffer capacity to 1 in replay_file fix byzantine_test unsubscribe from single event, not all events refactor httpclient to return events to appropriate channels return failing testReplayCrashBeforeWriteVote test fix TestValidatorSetChanges refactor code a bit fix testReplayCrashBeforeWriteVote add comment fix TestValidatorSetChanges fixes from Bucky's review update comment [ci skip] test TxEventBuffer update changelog fix TestValidatorSetChanges (2nd attempt) only do wg.Done when no errors benchmark event bus create pubsub server inside NewEventBus only expose config params (later if needed) set buffer capacity to 0 so we are not testing cache new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ} This should allow to subscribe to all transactions! or a specific one using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'" use TimeoutCommit instead of afterPublishEventNewBlockTimeout TimeoutCommit is the time a node waits after committing a block, before it goes into the next height. So it will finish everything from the last block, but then wait a bit. The idea is this gives it time to hear more votes from other validators, to strengthen the commit it includes in the next block. But it also gives it time to hear about new transactions. waitForBlockWithUpdatedVals rewrite WAL crash tests Task: test that we can recover from any WAL crash. Solution: the old tests were relying on event hub being run in the same thread (we were injecting the private validator's last signature). when considering a rewrite, we considered two possible solutions: write a "fuzzy" testing system where WAL is crashing upon receiving a new message, or inject failures and trigger them in tests using something like https://github.com/coreos/gofail. remove sleep no cs.Lock around wal.Save test different cases (empty block, non-empty block, ...) comments add comments test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks fixes as per Bucky's last review reset subscriptions on UnsubscribeAll use a simple counter to track message for which we panicked also, set a smaller part size for all test cases
8 years ago
7 years ago
new pubsub package comment out failing consensus tests for now rewrite rpc httpclient to use new pubsub package import pubsub as tmpubsub, query as tmquery make event IDs constants EventKey -> EventTypeKey rename EventsPubsub to PubSub mempool does not use pubsub rename eventsSub to pubsub new subscribe API fix channel size issues and consensus tests bugs refactor rpc client add missing discardFromChan method add mutex rename pubsub to eventBus remove IsRunning from WSRPCConnection interface (not needed) add a comment in broadcastNewRoundStepsAndVotes rename registerEventCallbacks to broadcastNewRoundStepsAndVotes See https://dave.cheney.net/2014/03/19/channel-axioms stop eventBuses after reactor tests remove unnecessary Unsubscribe return subscribe helper function move discardFromChan to where it is used subscribe now returns an err this gives us ability to refuse to subscribe if pubsub is at its max capacity. use context for control overflow cache queries handle err when subscribing in replay_test rename testClientID to testSubscriber extract var set channel buffer capacity to 1 in replay_file fix byzantine_test unsubscribe from single event, not all events refactor httpclient to return events to appropriate channels return failing testReplayCrashBeforeWriteVote test fix TestValidatorSetChanges refactor code a bit fix testReplayCrashBeforeWriteVote add comment fix TestValidatorSetChanges fixes from Bucky's review update comment [ci skip] test TxEventBuffer update changelog fix TestValidatorSetChanges (2nd attempt) only do wg.Done when no errors benchmark event bus create pubsub server inside NewEventBus only expose config params (later if needed) set buffer capacity to 0 so we are not testing cache new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ} This should allow to subscribe to all transactions! or a specific one using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'" use TimeoutCommit instead of afterPublishEventNewBlockTimeout TimeoutCommit is the time a node waits after committing a block, before it goes into the next height. So it will finish everything from the last block, but then wait a bit. The idea is this gives it time to hear more votes from other validators, to strengthen the commit it includes in the next block. But it also gives it time to hear about new transactions. waitForBlockWithUpdatedVals rewrite WAL crash tests Task: test that we can recover from any WAL crash. Solution: the old tests were relying on event hub being run in the same thread (we were injecting the private validator's last signature). when considering a rewrite, we considered two possible solutions: write a "fuzzy" testing system where WAL is crashing upon receiving a new message, or inject failures and trigger them in tests using something like https://github.com/coreos/gofail. remove sleep no cs.Lock around wal.Save test different cases (empty block, non-empty block, ...) comments add comments test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks fixes as per Bucky's last review reset subscriptions on UnsubscribeAll use a simple counter to track message for which we panicked also, set a smaller part size for all test cases
8 years ago
7 years ago
8 years ago
7 years ago
new pubsub package comment out failing consensus tests for now rewrite rpc httpclient to use new pubsub package import pubsub as tmpubsub, query as tmquery make event IDs constants EventKey -> EventTypeKey rename EventsPubsub to PubSub mempool does not use pubsub rename eventsSub to pubsub new subscribe API fix channel size issues and consensus tests bugs refactor rpc client add missing discardFromChan method add mutex rename pubsub to eventBus remove IsRunning from WSRPCConnection interface (not needed) add a comment in broadcastNewRoundStepsAndVotes rename registerEventCallbacks to broadcastNewRoundStepsAndVotes See https://dave.cheney.net/2014/03/19/channel-axioms stop eventBuses after reactor tests remove unnecessary Unsubscribe return subscribe helper function move discardFromChan to where it is used subscribe now returns an err this gives us ability to refuse to subscribe if pubsub is at its max capacity. use context for control overflow cache queries handle err when subscribing in replay_test rename testClientID to testSubscriber extract var set channel buffer capacity to 1 in replay_file fix byzantine_test unsubscribe from single event, not all events refactor httpclient to return events to appropriate channels return failing testReplayCrashBeforeWriteVote test fix TestValidatorSetChanges refactor code a bit fix testReplayCrashBeforeWriteVote add comment fix TestValidatorSetChanges fixes from Bucky's review update comment [ci skip] test TxEventBuffer update changelog fix TestValidatorSetChanges (2nd attempt) only do wg.Done when no errors benchmark event bus create pubsub server inside NewEventBus only expose config params (later if needed) set buffer capacity to 0 so we are not testing cache new tx event format: key = "Tx" plus a tag {"tx.hash": XYZ} This should allow to subscribe to all transactions! or a specific one using a query: "tm.events.type = Tx and tx.hash = '013ABF99434...'" use TimeoutCommit instead of afterPublishEventNewBlockTimeout TimeoutCommit is the time a node waits after committing a block, before it goes into the next height. So it will finish everything from the last block, but then wait a bit. The idea is this gives it time to hear more votes from other validators, to strengthen the commit it includes in the next block. But it also gives it time to hear about new transactions. waitForBlockWithUpdatedVals rewrite WAL crash tests Task: test that we can recover from any WAL crash. Solution: the old tests were relying on event hub being run in the same thread (we were injecting the private validator's last signature). when considering a rewrite, we considered two possible solutions: write a "fuzzy" testing system where WAL is crashing upon receiving a new message, or inject failures and trigger them in tests using something like https://github.com/coreos/gofail. remove sleep no cs.Lock around wal.Save test different cases (empty block, non-empty block, ...) comments add comments test 4 cases: empty block, non-empty block, non-empty block with smaller part size, many blocks fixes as per Bucky's last review reset subscriptions on UnsubscribeAll use a simple counter to track message for which we panicked also, set a smaller part size for all test cases
8 years ago
8 years ago
  1. package rpcserver
  2. import (
  3. "bytes"
  4. "encoding/hex"
  5. "encoding/json"
  6. "fmt"
  7. "io/ioutil"
  8. "net/http"
  9. "reflect"
  10. "runtime/debug"
  11. "sort"
  12. "strings"
  13. "time"
  14. "github.com/gorilla/websocket"
  15. "github.com/pkg/errors"
  16. types "github.com/tendermint/tendermint/rpc/lib/types"
  17. cmn "github.com/tendermint/tmlibs/common"
  18. "github.com/tendermint/tmlibs/log"
  19. )
  20. // RegisterRPCFuncs adds a route for each function in the funcMap, as well as general jsonrpc and websocket handlers for all functions.
  21. // "result" is the interface on which the result objects are registered, and is popualted with every RPCResponse
  22. func RegisterRPCFuncs(mux *http.ServeMux, funcMap map[string]*RPCFunc, logger log.Logger) {
  23. // HTTP endpoints
  24. for funcName, rpcFunc := range funcMap {
  25. mux.HandleFunc("/"+funcName, makeHTTPHandler(rpcFunc, logger))
  26. }
  27. // JSONRPC endpoints
  28. mux.HandleFunc("/", makeJSONRPCHandler(funcMap, logger))
  29. }
  30. //-------------------------------------
  31. // function introspection
  32. // RPCFunc contains the introspected type information for a function
  33. type RPCFunc struct {
  34. f reflect.Value // underlying rpc function
  35. args []reflect.Type // type of each function arg
  36. returns []reflect.Type // type of each return arg
  37. argNames []string // name of each argument
  38. ws bool // websocket only
  39. }
  40. // NewRPCFunc wraps a function for introspection.
  41. // f is the function, args are comma separated argument names
  42. func NewRPCFunc(f interface{}, args string) *RPCFunc {
  43. return newRPCFunc(f, args, false)
  44. }
  45. // NewWSRPCFunc wraps a function for introspection and use in the websockets.
  46. func NewWSRPCFunc(f interface{}, args string) *RPCFunc {
  47. return newRPCFunc(f, args, true)
  48. }
  49. func newRPCFunc(f interface{}, args string, ws bool) *RPCFunc {
  50. var argNames []string
  51. if args != "" {
  52. argNames = strings.Split(args, ",")
  53. }
  54. return &RPCFunc{
  55. f: reflect.ValueOf(f),
  56. args: funcArgTypes(f),
  57. returns: funcReturnTypes(f),
  58. argNames: argNames,
  59. ws: ws,
  60. }
  61. }
  62. // return a function's argument types
  63. func funcArgTypes(f interface{}) []reflect.Type {
  64. t := reflect.TypeOf(f)
  65. n := t.NumIn()
  66. typez := make([]reflect.Type, n)
  67. for i := 0; i < n; i++ {
  68. typez[i] = t.In(i)
  69. }
  70. return typez
  71. }
  72. // return a function's return types
  73. func funcReturnTypes(f interface{}) []reflect.Type {
  74. t := reflect.TypeOf(f)
  75. n := t.NumOut()
  76. typez := make([]reflect.Type, n)
  77. for i := 0; i < n; i++ {
  78. typez[i] = t.Out(i)
  79. }
  80. return typez
  81. }
  82. // function introspection
  83. //-----------------------------------------------------------------------------
  84. // rpc.json
  85. // jsonrpc calls grab the given method's function info and runs reflect.Call
  86. func makeJSONRPCHandler(funcMap map[string]*RPCFunc, logger log.Logger) http.HandlerFunc {
  87. return func(w http.ResponseWriter, r *http.Request) {
  88. b, _ := ioutil.ReadAll(r.Body)
  89. // if its an empty request (like from a browser),
  90. // just display a list of functions
  91. if len(b) == 0 {
  92. writeListOfEndpoints(w, r, funcMap)
  93. return
  94. }
  95. var request types.RPCRequest
  96. err := json.Unmarshal(b, &request)
  97. if err != nil {
  98. WriteRPCResponseHTTP(w, types.RPCParseError("", errors.Wrap(err, "Error unmarshalling request")))
  99. return
  100. }
  101. // A Notification is a Request object without an "id" member.
  102. // The Server MUST NOT reply to a Notification, including those that are within a batch request.
  103. if request.ID == "" {
  104. logger.Debug("HTTPJSONRPC received a notification, skipping... (please send a non-empty ID if you want to call a method)")
  105. return
  106. }
  107. if len(r.URL.Path) > 1 {
  108. WriteRPCResponseHTTP(w, types.RPCInvalidRequestError(request.ID, errors.Errorf("Path %s is invalid", r.URL.Path)))
  109. return
  110. }
  111. rpcFunc := funcMap[request.Method]
  112. if rpcFunc == nil || rpcFunc.ws {
  113. WriteRPCResponseHTTP(w, types.RPCMethodNotFoundError(request.ID))
  114. return
  115. }
  116. var args []reflect.Value
  117. if len(request.Params) > 0 {
  118. args, err = jsonParamsToArgsRPC(rpcFunc, request.Params)
  119. if err != nil {
  120. WriteRPCResponseHTTP(w, types.RPCInvalidParamsError(request.ID, errors.Wrap(err, "Error converting json params to arguments")))
  121. return
  122. }
  123. }
  124. returns := rpcFunc.f.Call(args)
  125. logger.Info("HTTPJSONRPC", "method", request.Method, "args", args, "returns", returns)
  126. result, err := unreflectResult(returns)
  127. if err != nil {
  128. WriteRPCResponseHTTP(w, types.RPCInternalError(request.ID, err))
  129. return
  130. }
  131. WriteRPCResponseHTTP(w, types.NewRPCSuccessResponse(request.ID, result))
  132. }
  133. }
  134. func mapParamsToArgs(rpcFunc *RPCFunc, params map[string]*json.RawMessage, argsOffset int) ([]reflect.Value, error) {
  135. values := make([]reflect.Value, len(rpcFunc.argNames))
  136. for i, argName := range rpcFunc.argNames {
  137. argType := rpcFunc.args[i+argsOffset]
  138. if p, ok := params[argName]; ok && p != nil && len(*p) > 0 {
  139. val := reflect.New(argType)
  140. err := json.Unmarshal(*p, val.Interface())
  141. if err != nil {
  142. return nil, err
  143. }
  144. values[i] = val.Elem()
  145. } else { // use default for that type
  146. values[i] = reflect.Zero(argType)
  147. }
  148. }
  149. return values, nil
  150. }
  151. func arrayParamsToArgs(rpcFunc *RPCFunc, params []*json.RawMessage, argsOffset int) ([]reflect.Value, error) {
  152. if len(rpcFunc.argNames) != len(params) {
  153. return nil, errors.Errorf("Expected %v parameters (%v), got %v (%v)",
  154. len(rpcFunc.argNames), rpcFunc.argNames, len(params), params)
  155. }
  156. values := make([]reflect.Value, len(params))
  157. for i, p := range params {
  158. argType := rpcFunc.args[i+argsOffset]
  159. val := reflect.New(argType)
  160. err := json.Unmarshal(*p, val.Interface())
  161. if err != nil {
  162. return nil, err
  163. }
  164. values[i] = val.Elem()
  165. }
  166. return values, nil
  167. }
  168. // raw is unparsed json (from json.RawMessage) encoding either a map or an array.
  169. //
  170. // argsOffset should be 0 for RPC calls, and 1 for WS requests, where len(rpcFunc.args) != len(rpcFunc.argNames).
  171. // Example:
  172. // rpcFunc.args = [rpctypes.WSRPCContext string]
  173. // rpcFunc.argNames = ["arg"]
  174. func jsonParamsToArgs(rpcFunc *RPCFunc, raw []byte, argsOffset int) ([]reflect.Value, error) {
  175. // first, try to get the map..
  176. var m map[string]*json.RawMessage
  177. err := json.Unmarshal(raw, &m)
  178. if err == nil {
  179. return mapParamsToArgs(rpcFunc, m, argsOffset)
  180. }
  181. // otherwise, try an array
  182. var a []*json.RawMessage
  183. err = json.Unmarshal(raw, &a)
  184. if err == nil {
  185. return arrayParamsToArgs(rpcFunc, a, argsOffset)
  186. }
  187. // otherwise, bad format, we cannot parse
  188. return nil, errors.Errorf("Unknown type for JSON params: %v. Expected map or array", err)
  189. }
  190. // Convert a []interface{} OR a map[string]interface{} to properly typed values
  191. func jsonParamsToArgsRPC(rpcFunc *RPCFunc, params json.RawMessage) ([]reflect.Value, error) {
  192. return jsonParamsToArgs(rpcFunc, params, 0)
  193. }
  194. // Same as above, but with the first param the websocket connection
  195. func jsonParamsToArgsWS(rpcFunc *RPCFunc, params json.RawMessage, wsCtx types.WSRPCContext) ([]reflect.Value, error) {
  196. values, err := jsonParamsToArgs(rpcFunc, params, 1)
  197. if err != nil {
  198. return nil, err
  199. }
  200. return append([]reflect.Value{reflect.ValueOf(wsCtx)}, values...), nil
  201. }
  202. // rpc.json
  203. //-----------------------------------------------------------------------------
  204. // rpc.http
  205. // convert from a function name to the http handler
  206. func makeHTTPHandler(rpcFunc *RPCFunc, logger log.Logger) func(http.ResponseWriter, *http.Request) {
  207. // Exception for websocket endpoints
  208. if rpcFunc.ws {
  209. return func(w http.ResponseWriter, r *http.Request) {
  210. WriteRPCResponseHTTP(w, types.RPCMethodNotFoundError(""))
  211. }
  212. }
  213. // All other endpoints
  214. return func(w http.ResponseWriter, r *http.Request) {
  215. logger.Debug("HTTP HANDLER", "req", r)
  216. args, err := httpParamsToArgs(rpcFunc, r)
  217. if err != nil {
  218. WriteRPCResponseHTTP(w, types.RPCInvalidParamsError("", errors.Wrap(err, "Error converting http params to arguments")))
  219. return
  220. }
  221. returns := rpcFunc.f.Call(args)
  222. logger.Info("HTTPRestRPC", "method", r.URL.Path, "args", args, "returns", returns)
  223. result, err := unreflectResult(returns)
  224. if err != nil {
  225. WriteRPCResponseHTTP(w, types.RPCInternalError("", err))
  226. return
  227. }
  228. WriteRPCResponseHTTP(w, types.NewRPCSuccessResponse("", result))
  229. }
  230. }
  231. // Covert an http query to a list of properly typed values.
  232. // To be properly decoded the arg must be a concrete type from tendermint (if its an interface).
  233. func httpParamsToArgs(rpcFunc *RPCFunc, r *http.Request) ([]reflect.Value, error) {
  234. values := make([]reflect.Value, len(rpcFunc.args))
  235. for i, name := range rpcFunc.argNames {
  236. argType := rpcFunc.args[i]
  237. values[i] = reflect.Zero(argType) // set default for that type
  238. arg := GetParam(r, name)
  239. // log.Notice("param to arg", "argType", argType, "name", name, "arg", arg)
  240. if "" == arg {
  241. continue
  242. }
  243. v, err, ok := nonJsonToArg(argType, arg)
  244. if err != nil {
  245. return nil, err
  246. }
  247. if ok {
  248. values[i] = v
  249. continue
  250. }
  251. values[i], err = _jsonStringToArg(argType, arg)
  252. if err != nil {
  253. return nil, err
  254. }
  255. }
  256. return values, nil
  257. }
  258. func _jsonStringToArg(ty reflect.Type, arg string) (reflect.Value, error) {
  259. v := reflect.New(ty)
  260. err := json.Unmarshal([]byte(arg), v.Interface())
  261. if err != nil {
  262. return v, err
  263. }
  264. v = v.Elem()
  265. return v, nil
  266. }
  267. func nonJsonToArg(ty reflect.Type, arg string) (reflect.Value, error, bool) {
  268. isQuotedString := strings.HasPrefix(arg, `"`) && strings.HasSuffix(arg, `"`)
  269. isHexString := strings.HasPrefix(strings.ToLower(arg), "0x")
  270. expectingString := ty.Kind() == reflect.String
  271. expectingByteSlice := ty.Kind() == reflect.Slice && ty.Elem().Kind() == reflect.Uint8
  272. if isHexString {
  273. if !expectingString && !expectingByteSlice {
  274. err := errors.Errorf("Got a hex string arg, but expected '%s'",
  275. ty.Kind().String())
  276. return reflect.ValueOf(nil), err, false
  277. }
  278. var value []byte
  279. value, err := hex.DecodeString(arg[2:])
  280. if err != nil {
  281. return reflect.ValueOf(nil), err, false
  282. }
  283. if ty.Kind() == reflect.String {
  284. return reflect.ValueOf(string(value)), nil, true
  285. }
  286. return reflect.ValueOf([]byte(value)), nil, true
  287. }
  288. if isQuotedString && expectingByteSlice {
  289. v := reflect.New(reflect.TypeOf(""))
  290. err := json.Unmarshal([]byte(arg), v.Interface())
  291. if err != nil {
  292. return reflect.ValueOf(nil), err, false
  293. }
  294. v = v.Elem()
  295. return reflect.ValueOf([]byte(v.String())), nil, true
  296. }
  297. return reflect.ValueOf(nil), nil, false
  298. }
  299. // rpc.http
  300. //-----------------------------------------------------------------------------
  301. // rpc.websocket
  302. const (
  303. defaultWSWriteChanCapacity = 1000
  304. defaultWSWriteWait = 10 * time.Second
  305. defaultWSReadWait = 30 * time.Second
  306. defaultWSPingPeriod = (defaultWSReadWait * 9) / 10
  307. )
  308. // a single websocket connection contains listener id, underlying ws
  309. // connection, and the event switch for subscribing to events.
  310. //
  311. // In case of an error, the connection is stopped.
  312. type wsConnection struct {
  313. cmn.BaseService
  314. remoteAddr string
  315. baseConn *websocket.Conn
  316. writeChan chan types.RPCResponse
  317. funcMap map[string]*RPCFunc
  318. subscriptions map[string]interface{}
  319. // write channel capacity
  320. writeChanCapacity int
  321. // each write times out after this.
  322. writeWait time.Duration
  323. // Connection times out if we haven't received *anything* in this long, not even pings.
  324. readWait time.Duration
  325. // Send pings to server with this period. Must be less than readWait, but greater than zero.
  326. pingPeriod time.Duration
  327. // called before stopping the connection.
  328. onDisconnect func(remoteAddr string)
  329. }
  330. // NewWSConnection wraps websocket.Conn.
  331. //
  332. // See the commentary on the func(*wsConnection) functions for a detailed
  333. // description of how to configure ping period and pong wait time. NOTE: if the
  334. // write buffer is full, pongs may be dropped, which may cause clients to
  335. // disconnect. see https://github.com/gorilla/websocket/issues/97
  336. func NewWSConnection(baseConn *websocket.Conn, funcMap map[string]*RPCFunc, options ...func(*wsConnection)) *wsConnection {
  337. wsc := &wsConnection{
  338. remoteAddr: baseConn.RemoteAddr().String(),
  339. baseConn: baseConn,
  340. funcMap: funcMap,
  341. subscriptions: make(map[string]interface{}),
  342. writeWait: defaultWSWriteWait,
  343. writeChanCapacity: defaultWSWriteChanCapacity,
  344. readWait: defaultWSReadWait,
  345. pingPeriod: defaultWSPingPeriod,
  346. }
  347. for _, option := range options {
  348. option(wsc)
  349. }
  350. wsc.BaseService = *cmn.NewBaseService(nil, "wsConnection", wsc)
  351. return wsc
  352. }
  353. // WriteWait sets the amount of time to wait before a websocket write times out.
  354. // It should only be used in the constructor - not Goroutine-safe.
  355. func WriteWait(writeWait time.Duration) func(*wsConnection) {
  356. return func(wsc *wsConnection) {
  357. wsc.writeWait = writeWait
  358. }
  359. }
  360. // WriteChanCapacity sets the capacity of the websocket write channel.
  361. // It should only be used in the constructor - not Goroutine-safe.
  362. func WriteChanCapacity(cap int) func(*wsConnection) {
  363. return func(wsc *wsConnection) {
  364. wsc.writeChanCapacity = cap
  365. }
  366. }
  367. // ReadWait sets the amount of time to wait before a websocket read times out.
  368. // It should only be used in the constructor - not Goroutine-safe.
  369. func ReadWait(readWait time.Duration) func(*wsConnection) {
  370. return func(wsc *wsConnection) {
  371. wsc.readWait = readWait
  372. }
  373. }
  374. // PingPeriod sets the duration for sending websocket pings.
  375. // It should only be used in the constructor - not Goroutine-safe.
  376. func PingPeriod(pingPeriod time.Duration) func(*wsConnection) {
  377. return func(wsc *wsConnection) {
  378. wsc.pingPeriod = pingPeriod
  379. }
  380. }
  381. // OnDisconnect called before stopping the connection.
  382. // It should only be used in the constructor - not Goroutine-safe.
  383. func OnDisconnect(cb func(remoteAddr string)) func(*wsConnection) {
  384. return func(wsc *wsConnection) {
  385. wsc.onDisconnect = cb
  386. }
  387. }
  388. // OnStart implements cmn.Service by starting the read and write routines. It
  389. // blocks until the connection closes.
  390. func (wsc *wsConnection) OnStart() error {
  391. wsc.writeChan = make(chan types.RPCResponse, wsc.writeChanCapacity)
  392. // Read subscriptions/unsubscriptions to events
  393. go wsc.readRoutine()
  394. // Write responses, BLOCKING.
  395. wsc.writeRoutine()
  396. return nil
  397. }
  398. // OnStop implements cmn.Service by calling OnDisconnect callback.
  399. func (wsc *wsConnection) OnStop() {
  400. // Both read and write loops close the websocket connection when they exit their loops.
  401. // The writeChan is never closed, to allow WriteRPCResponse() to fail.
  402. if wsc.onDisconnect != nil {
  403. wsc.onDisconnect(wsc.remoteAddr)
  404. }
  405. }
  406. // GetRemoteAddr returns the remote address of the underlying connection.
  407. // It implements WSRPCConnection
  408. func (wsc *wsConnection) GetRemoteAddr() string {
  409. return wsc.remoteAddr
  410. }
  411. // WriteRPCResponse pushes a response to the writeChan, and blocks until it is accepted.
  412. // It implements WSRPCConnection. It is Goroutine-safe.
  413. func (wsc *wsConnection) WriteRPCResponse(resp types.RPCResponse) {
  414. select {
  415. case <-wsc.Quit:
  416. return
  417. case wsc.writeChan <- resp:
  418. }
  419. }
  420. // TryWriteRPCResponse attempts to push a response to the writeChan, but does not block.
  421. // It implements WSRPCConnection. It is Goroutine-safe
  422. func (wsc *wsConnection) TryWriteRPCResponse(resp types.RPCResponse) bool {
  423. select {
  424. case <-wsc.Quit:
  425. return false
  426. case wsc.writeChan <- resp:
  427. return true
  428. default:
  429. return false
  430. }
  431. }
  432. func (wsc *wsConnection) AddSubscription(query string, data interface{}) error {
  433. if _, ok := wsc.subscriptions[query]; ok {
  434. return errors.New("Already subscribed")
  435. }
  436. wsc.subscriptions[query] = data
  437. return nil
  438. }
  439. func (wsc *wsConnection) DeleteSubscription(query string) (interface{}, bool) {
  440. data, ok := wsc.subscriptions[query]
  441. if ok {
  442. delete(wsc.subscriptions, query)
  443. return data, true
  444. }
  445. return nil, false
  446. }
  447. func (wsc *wsConnection) DeleteAllSubscriptions() {
  448. wsc.subscriptions = make(map[string]interface{})
  449. }
  450. // Read from the socket and subscribe to or unsubscribe from events
  451. func (wsc *wsConnection) readRoutine() {
  452. defer func() {
  453. if r := recover(); r != nil {
  454. err, ok := r.(error)
  455. if !ok {
  456. err = fmt.Errorf("WSJSONRPC: %v", r)
  457. }
  458. wsc.Logger.Error("Panic in WSJSONRPC handler", "err", err, "stack", string(debug.Stack()))
  459. wsc.WriteRPCResponse(types.RPCInternalError("unknown", err))
  460. go wsc.readRoutine()
  461. } else {
  462. wsc.baseConn.Close()
  463. }
  464. }()
  465. wsc.baseConn.SetPongHandler(func(m string) error {
  466. return wsc.baseConn.SetReadDeadline(time.Now().Add(wsc.readWait))
  467. })
  468. for {
  469. select {
  470. case <-wsc.Quit:
  471. return
  472. default:
  473. // reset deadline for every type of message (control or data)
  474. wsc.baseConn.SetReadDeadline(time.Now().Add(wsc.readWait))
  475. var in []byte
  476. _, in, err := wsc.baseConn.ReadMessage()
  477. if err != nil {
  478. if websocket.IsCloseError(err, websocket.CloseNormalClosure) {
  479. wsc.Logger.Info("Client closed the connection")
  480. } else {
  481. wsc.Logger.Error("Failed to read request", "err", err)
  482. }
  483. wsc.Stop()
  484. return
  485. }
  486. var request types.RPCRequest
  487. err = json.Unmarshal(in, &request)
  488. if err != nil {
  489. wsc.WriteRPCResponse(types.RPCParseError("", errors.Wrap(err, "Error unmarshaling request")))
  490. continue
  491. }
  492. // A Notification is a Request object without an "id" member.
  493. // The Server MUST NOT reply to a Notification, including those that are within a batch request.
  494. if request.ID == "" {
  495. wsc.Logger.Debug("WSJSONRPC received a notification, skipping... (please send a non-empty ID if you want to call a method)")
  496. continue
  497. }
  498. // Now, fetch the RPCFunc and execute it.
  499. rpcFunc := wsc.funcMap[request.Method]
  500. if rpcFunc == nil {
  501. wsc.WriteRPCResponse(types.RPCMethodNotFoundError(request.ID))
  502. continue
  503. }
  504. var args []reflect.Value
  505. if rpcFunc.ws {
  506. wsCtx := types.WSRPCContext{Request: request, WSRPCConnection: wsc}
  507. if len(request.Params) > 0 {
  508. args, err = jsonParamsToArgsWS(rpcFunc, request.Params, wsCtx)
  509. }
  510. } else {
  511. if len(request.Params) > 0 {
  512. args, err = jsonParamsToArgsRPC(rpcFunc, request.Params)
  513. }
  514. }
  515. if err != nil {
  516. wsc.WriteRPCResponse(types.RPCInternalError(request.ID, errors.Wrap(err, "Error converting json params to arguments")))
  517. continue
  518. }
  519. returns := rpcFunc.f.Call(args)
  520. // TODO: Need to encode args/returns to string if we want to log them
  521. wsc.Logger.Info("WSJSONRPC", "method", request.Method)
  522. result, err := unreflectResult(returns)
  523. if err != nil {
  524. wsc.WriteRPCResponse(types.RPCInternalError(request.ID, err))
  525. continue
  526. } else {
  527. wsc.WriteRPCResponse(types.NewRPCSuccessResponse(request.ID, result))
  528. continue
  529. }
  530. }
  531. }
  532. }
  533. // receives on a write channel and writes out on the socket
  534. func (wsc *wsConnection) writeRoutine() {
  535. pingTicker := time.NewTicker(wsc.pingPeriod)
  536. defer func() {
  537. pingTicker.Stop()
  538. wsc.baseConn.Close()
  539. }()
  540. // https://github.com/gorilla/websocket/issues/97
  541. pongs := make(chan string, 1)
  542. wsc.baseConn.SetPingHandler(func(m string) error {
  543. select {
  544. case pongs <- m:
  545. default:
  546. }
  547. return nil
  548. })
  549. for {
  550. select {
  551. case m := <-pongs:
  552. err := wsc.writeMessageWithDeadline(websocket.PongMessage, []byte(m))
  553. if err != nil {
  554. wsc.Logger.Info("Failed to write pong (client may disconnect)", "err", err)
  555. }
  556. case <-pingTicker.C:
  557. err := wsc.writeMessageWithDeadline(websocket.PingMessage, []byte{})
  558. if err != nil {
  559. wsc.Logger.Error("Failed to write ping", "err", err)
  560. wsc.Stop()
  561. return
  562. }
  563. case msg := <-wsc.writeChan:
  564. jsonBytes, err := json.MarshalIndent(msg, "", " ")
  565. if err != nil {
  566. wsc.Logger.Error("Failed to marshal RPCResponse to JSON", "err", err)
  567. } else {
  568. if err = wsc.writeMessageWithDeadline(websocket.TextMessage, jsonBytes); err != nil {
  569. wsc.Logger.Error("Failed to write response", "err", err)
  570. wsc.Stop()
  571. return
  572. }
  573. }
  574. case <-wsc.Quit:
  575. return
  576. }
  577. }
  578. }
  579. // All writes to the websocket must (re)set the write deadline.
  580. // If some writes don't set it while others do, they may timeout incorrectly (https://github.com/tendermint/tendermint/issues/553)
  581. func (wsc *wsConnection) writeMessageWithDeadline(msgType int, msg []byte) error {
  582. wsc.baseConn.SetWriteDeadline(time.Now().Add(wsc.writeWait))
  583. return wsc.baseConn.WriteMessage(msgType, msg)
  584. }
  585. //----------------------------------------
  586. // WebsocketManager is the main manager for all websocket connections.
  587. // It holds the event switch and a map of functions for routing.
  588. // NOTE: The websocket path is defined externally, e.g. in node/node.go
  589. type WebsocketManager struct {
  590. websocket.Upgrader
  591. funcMap map[string]*RPCFunc
  592. logger log.Logger
  593. wsConnOptions []func(*wsConnection)
  594. }
  595. // NewWebsocketManager returns a new WebsocketManager that routes according to
  596. // the given funcMap and connects to the server with the given connection
  597. // options.
  598. func NewWebsocketManager(funcMap map[string]*RPCFunc, wsConnOptions ...func(*wsConnection)) *WebsocketManager {
  599. return &WebsocketManager{
  600. funcMap: funcMap,
  601. Upgrader: websocket.Upgrader{
  602. CheckOrigin: func(r *http.Request) bool {
  603. // TODO ???
  604. return true
  605. },
  606. },
  607. logger: log.NewNopLogger(),
  608. wsConnOptions: wsConnOptions,
  609. }
  610. }
  611. // SetLogger sets the logger.
  612. func (wm *WebsocketManager) SetLogger(l log.Logger) {
  613. wm.logger = l
  614. }
  615. // WebsocketHandler upgrades the request/response (via http.Hijack) and starts the wsConnection.
  616. func (wm *WebsocketManager) WebsocketHandler(w http.ResponseWriter, r *http.Request) {
  617. wsConn, err := wm.Upgrade(w, r, nil)
  618. if err != nil {
  619. // TODO - return http error
  620. wm.logger.Error("Failed to upgrade to websocket connection", "err", err)
  621. return
  622. }
  623. // register connection
  624. con := NewWSConnection(wsConn, wm.funcMap, wm.wsConnOptions...)
  625. con.SetLogger(wm.logger.With("remote", wsConn.RemoteAddr()))
  626. wm.logger.Info("New websocket connection", "remote", con.remoteAddr)
  627. con.Start() // Blocking
  628. }
  629. // rpc.websocket
  630. //-----------------------------------------------------------------------------
  631. // NOTE: assume returns is result struct and error. If error is not nil, return it
  632. func unreflectResult(returns []reflect.Value) (interface{}, error) {
  633. errV := returns[1]
  634. if errV.Interface() != nil {
  635. return nil, errors.Errorf("%v", errV.Interface())
  636. }
  637. rv := returns[0]
  638. // the result is a registered interface,
  639. // we need a pointer to it so we can marshal with type byte
  640. rvp := reflect.New(rv.Type())
  641. rvp.Elem().Set(rv)
  642. return rvp.Interface(), nil
  643. }
  644. // writes a list of available rpc endpoints as an html page
  645. func writeListOfEndpoints(w http.ResponseWriter, r *http.Request, funcMap map[string]*RPCFunc) {
  646. noArgNames := []string{}
  647. argNames := []string{}
  648. for name, funcData := range funcMap {
  649. if len(funcData.args) == 0 {
  650. noArgNames = append(noArgNames, name)
  651. } else {
  652. argNames = append(argNames, name)
  653. }
  654. }
  655. sort.Strings(noArgNames)
  656. sort.Strings(argNames)
  657. buf := new(bytes.Buffer)
  658. buf.WriteString("<html><body>")
  659. buf.WriteString("<br>Available endpoints:<br>")
  660. for _, name := range noArgNames {
  661. link := fmt.Sprintf("http://%s/%s", r.Host, name)
  662. buf.WriteString(fmt.Sprintf("<a href=\"%s\">%s</a></br>", link, link))
  663. }
  664. buf.WriteString("<br>Endpoints that require arguments:<br>")
  665. for _, name := range argNames {
  666. link := fmt.Sprintf("http://%s/%s?", r.Host, name)
  667. funcData := funcMap[name]
  668. for i, argName := range funcData.argNames {
  669. link += argName + "=_"
  670. if i < len(funcData.argNames)-1 {
  671. link += "&"
  672. }
  673. }
  674. buf.WriteString(fmt.Sprintf("<a href=\"%s\">%s</a></br>", link, link))
  675. }
  676. buf.WriteString("</body></html>")
  677. w.Header().Set("Content-Type", "text/html")
  678. w.WriteHeader(200)
  679. w.Write(buf.Bytes())
  680. }