You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

588 lines
15 KiB

abci: Refactor tagging events using list of lists (#3643) ## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test
5 years ago
abci: Refactor tagging events using list of lists (#3643) ## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test
5 years ago
7 years ago
abci: Refactor tagging events using list of lists (#3643) ## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test
5 years ago
abci: Refactor tagging events using list of lists (#3643) ## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test
5 years ago
abci: Refactor tagging events using list of lists (#3643) ## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test
5 years ago
abci: Refactor tagging events using list of lists (#3643) ## PR This PR introduces a fundamental breaking change to the structure of ABCI response and tx tags and the way they're processed. Namely, the SDK can support more complex and aggregated events for distribution and slashing. In addition, block responses can include duplicate keys in events. Implement new Event type. An event has a type and a list of KV pairs (ie. list-of-lists). Typical events may look like: "rewards": [{"amount": "5000uatom", "validator": "...", "recipient": "..."}] "sender": [{"address": "...", "balance": "100uatom"}] The events are indexed by {even.type}.{even.attribute[i].key}/.... In this case a client would subscribe or query for rewards.recipient='...' ABCI response types and related types now include Events []Event instead of Tags []cmn.KVPair. PubSub logic now publishes/matches against map[string][]string instead of map[string]string to support duplicate keys in response events (from #1385). A match is successful if the value is found in the slice of strings. closes: #1859 closes: #2905 ## Commits: * Implement Event ABCI type and updates responses to use events * Update messages_test.go * Update kvstore.go * Update event_bus.go * Update subscription.go * Update pubsub.go * Update kvstore.go * Update query logic to handle slice of strings in events * Update Empty#Matches and unit tests * Update pubsub logic * Update EventBus#Publish * Update kv tx indexer * Update godocs * Update ResultEvent to use slice of strings; update RPC * Update more tests * Update abci.md * Check for key in validateAndStringifyEvents * Fix KV indexer to skip empty keys * Fix linting errors * Update CHANGELOG_PENDING.md * Update docs/spec/abci/abci.md Co-Authored-By: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * Update abci/types/types.proto Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update docs/spec/abci/abci.md Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update libs/pubsub/query/query.go Co-Authored-By: Ethan Buchman <ethan@coinculture.info> * Update match function to match if ANY value matches * Implement TestSubscribeDuplicateKeys * Update TestMatches to include multi-key test cases * Update events.go * Update Query interface godoc * Update match godoc * Add godoc for matchValue * DRY-up tx indexing * Return error from PublishWithEvents in EventBus#Publish * Update PublishEventNewBlockHeader to return an error * Fix build * Update events doc in ABCI * Update ABCI events godoc * Implement TestEventBusPublishEventTxDuplicateKeys * Update TestSubscribeDuplicateKeys to be table-driven * Remove mod file * Remove markdown from events godoc * Implement TestTxSearchDeprecatedIndexing test
5 years ago
  1. package kv
  2. import (
  3. "context"
  4. "encoding/hex"
  5. "fmt"
  6. "strconv"
  7. "strings"
  8. "github.com/gogo/protobuf/proto"
  9. "github.com/google/orderedcode"
  10. dbm "github.com/tendermint/tm-db"
  11. abci "github.com/tendermint/tendermint/abci/types"
  12. "github.com/tendermint/tendermint/internal/pubsub/query"
  13. "github.com/tendermint/tendermint/internal/pubsub/query/syntax"
  14. indexer "github.com/tendermint/tendermint/internal/state/indexer"
  15. "github.com/tendermint/tendermint/types"
  16. )
  17. var _ indexer.TxIndexer = (*TxIndex)(nil)
  18. // TxIndex is the simplest possible indexer
  19. // It is backed by two kv stores:
  20. // 1. txhash - result (primary key)
  21. // 2. event - txhash (secondary key)
  22. type TxIndex struct {
  23. store dbm.DB
  24. }
  25. // NewTxIndex creates new KV indexer.
  26. func NewTxIndex(store dbm.DB) *TxIndex {
  27. return &TxIndex{
  28. store: store,
  29. }
  30. }
  31. // Get gets transaction from the TxIndex storage and returns it or nil if the
  32. // transaction is not found.
  33. func (txi *TxIndex) Get(hash []byte) (*abci.TxResult, error) {
  34. if len(hash) == 0 {
  35. return nil, indexer.ErrorEmptyHash
  36. }
  37. rawBytes, err := txi.store.Get(primaryKey(hash))
  38. if err != nil {
  39. panic(err)
  40. }
  41. if rawBytes == nil {
  42. return nil, nil
  43. }
  44. txResult := new(abci.TxResult)
  45. err = proto.Unmarshal(rawBytes, txResult)
  46. if err != nil {
  47. return nil, fmt.Errorf("error reading TxResult: %w", err)
  48. }
  49. return txResult, nil
  50. }
  51. // Index indexes transactions using the given list of events. Each key
  52. // that indexed from the tx's events is a composite of the event type and the
  53. // respective attribute's key delimited by a "." (eg. "account.number").
  54. // Any event with an empty type is not indexed.
  55. func (txi *TxIndex) Index(results []*abci.TxResult) error {
  56. b := txi.store.NewBatch()
  57. defer b.Close()
  58. for _, result := range results {
  59. hash := types.Tx(result.Tx).Hash()
  60. // index tx by events
  61. err := txi.indexEvents(result, hash, b)
  62. if err != nil {
  63. return err
  64. }
  65. // index by height (always)
  66. err = b.Set(KeyFromHeight(result), hash)
  67. if err != nil {
  68. return err
  69. }
  70. rawBytes, err := proto.Marshal(result)
  71. if err != nil {
  72. return err
  73. }
  74. // index by hash (always)
  75. err = b.Set(primaryKey(hash), rawBytes)
  76. if err != nil {
  77. return err
  78. }
  79. }
  80. return b.WriteSync()
  81. }
  82. func (txi *TxIndex) indexEvents(result *abci.TxResult, hash []byte, store dbm.Batch) error {
  83. for _, event := range result.Result.Events {
  84. // only index events with a non-empty type
  85. if len(event.Type) == 0 {
  86. continue
  87. }
  88. for _, attr := range event.Attributes {
  89. if len(attr.Key) == 0 {
  90. continue
  91. }
  92. // index if `index: true` is set
  93. compositeTag := fmt.Sprintf("%s.%s", event.Type, attr.Key)
  94. // ensure event does not conflict with a reserved prefix key
  95. if compositeTag == types.TxHashKey || compositeTag == types.TxHeightKey {
  96. return fmt.Errorf("event type and attribute key \"%s\" is reserved; please use a different key", compositeTag)
  97. }
  98. if attr.GetIndex() {
  99. err := store.Set(keyFromEvent(compositeTag, attr.Value, result), hash)
  100. if err != nil {
  101. return err
  102. }
  103. }
  104. }
  105. }
  106. return nil
  107. }
  108. // Search performs a search using the given query.
  109. //
  110. // It breaks the query into conditions (like "tx.height > 5"). For each
  111. // condition, it queries the DB index. One special use cases here: (1) if
  112. // "tx.hash" is found, it returns tx result for it (2) for range queries it is
  113. // better for the client to provide both lower and upper bounds, so we are not
  114. // performing a full scan. Results from querying indexes are then intersected
  115. // and returned to the caller, in no particular order.
  116. //
  117. // Search will exit early and return any result fetched so far,
  118. // when a message is received on the context chan.
  119. func (txi *TxIndex) Search(ctx context.Context, q *query.Query) ([]*abci.TxResult, error) {
  120. select {
  121. case <-ctx.Done():
  122. return make([]*abci.TxResult, 0), nil
  123. default:
  124. }
  125. var hashesInitialized bool
  126. filteredHashes := make(map[string][]byte)
  127. // get a list of conditions (like "tx.height > 5")
  128. conditions := q.Syntax()
  129. // if there is a hash condition, return the result immediately
  130. hash, ok, err := lookForHash(conditions)
  131. if err != nil {
  132. return nil, fmt.Errorf("error during searching for a hash in the query: %w", err)
  133. } else if ok {
  134. res, err := txi.Get(hash)
  135. switch {
  136. case err != nil:
  137. return []*abci.TxResult{}, fmt.Errorf("error while retrieving the result: %w", err)
  138. case res == nil:
  139. return []*abci.TxResult{}, nil
  140. default:
  141. return []*abci.TxResult{res}, nil
  142. }
  143. }
  144. // conditions to skip because they're handled before "everything else"
  145. skipIndexes := make([]int, 0)
  146. // extract ranges
  147. // if both upper and lower bounds exist, it's better to get them in order not
  148. // no iterate over kvs that are not within range.
  149. ranges, rangeIndexes := indexer.LookForRanges(conditions)
  150. if len(ranges) > 0 {
  151. skipIndexes = append(skipIndexes, rangeIndexes...)
  152. for _, qr := range ranges {
  153. if !hashesInitialized {
  154. filteredHashes = txi.matchRange(ctx, qr, prefixFromCompositeKey(qr.Key), filteredHashes, true)
  155. hashesInitialized = true
  156. // Ignore any remaining conditions if the first condition resulted
  157. // in no matches (assuming implicit AND operand).
  158. if len(filteredHashes) == 0 {
  159. break
  160. }
  161. } else {
  162. filteredHashes = txi.matchRange(ctx, qr, prefixFromCompositeKey(qr.Key), filteredHashes, false)
  163. }
  164. }
  165. }
  166. // if there is a height condition ("tx.height=3"), extract it
  167. height := lookForHeight(conditions)
  168. // for all other conditions
  169. for i, c := range conditions {
  170. if intInSlice(i, skipIndexes) {
  171. continue
  172. }
  173. if !hashesInitialized {
  174. filteredHashes = txi.match(ctx, c, prefixForCondition(c, height), filteredHashes, true)
  175. hashesInitialized = true
  176. // Ignore any remaining conditions if the first condition resulted
  177. // in no matches (assuming implicit AND operand).
  178. if len(filteredHashes) == 0 {
  179. break
  180. }
  181. } else {
  182. filteredHashes = txi.match(ctx, c, prefixForCondition(c, height), filteredHashes, false)
  183. }
  184. }
  185. results := make([]*abci.TxResult, 0, len(filteredHashes))
  186. hashes:
  187. for _, h := range filteredHashes {
  188. res, err := txi.Get(h)
  189. if err != nil {
  190. return nil, fmt.Errorf("failed to get Tx{%X}: %w", h, err)
  191. }
  192. results = append(results, res)
  193. // Potentially exit early.
  194. select {
  195. case <-ctx.Done():
  196. break hashes
  197. default:
  198. }
  199. }
  200. return results, nil
  201. }
  202. func lookForHash(conditions []syntax.Condition) (hash []byte, ok bool, err error) {
  203. for _, c := range conditions {
  204. if c.Tag == types.TxHashKey {
  205. decoded, err := hex.DecodeString(c.Arg.Value())
  206. return decoded, true, err
  207. }
  208. }
  209. return
  210. }
  211. // lookForHeight returns a height if there is an "height=X" condition.
  212. func lookForHeight(conditions []syntax.Condition) (height int64) {
  213. for _, c := range conditions {
  214. if c.Tag == types.TxHeightKey && c.Op == syntax.TEq {
  215. return int64(c.Arg.Number())
  216. }
  217. }
  218. return 0
  219. }
  220. // match returns all matching txs by hash that meet a given condition and start
  221. // key. An already filtered result (filteredHashes) is provided such that any
  222. // non-intersecting matches are removed.
  223. //
  224. // NOTE: filteredHashes may be empty if no previous condition has matched.
  225. func (txi *TxIndex) match(
  226. ctx context.Context,
  227. c syntax.Condition,
  228. startKeyBz []byte,
  229. filteredHashes map[string][]byte,
  230. firstRun bool,
  231. ) map[string][]byte {
  232. // A previous match was attempted but resulted in no matches, so we return
  233. // no matches (assuming AND operand).
  234. if !firstRun && len(filteredHashes) == 0 {
  235. return filteredHashes
  236. }
  237. tmpHashes := make(map[string][]byte)
  238. switch {
  239. case c.Op == syntax.TEq:
  240. it, err := dbm.IteratePrefix(txi.store, startKeyBz)
  241. if err != nil {
  242. panic(err)
  243. }
  244. defer it.Close()
  245. iterEqual:
  246. for ; it.Valid(); it.Next() {
  247. tmpHashes[string(it.Value())] = it.Value()
  248. // Potentially exit early.
  249. select {
  250. case <-ctx.Done():
  251. break iterEqual
  252. default:
  253. }
  254. }
  255. if err := it.Error(); err != nil {
  256. panic(err)
  257. }
  258. case c.Op == syntax.TExists:
  259. // XXX: can't use startKeyBz here because c.Operand is nil
  260. // (e.g. "account.owner/<nil>/" won't match w/ a single row)
  261. it, err := dbm.IteratePrefix(txi.store, prefixFromCompositeKey(c.Tag))
  262. if err != nil {
  263. panic(err)
  264. }
  265. defer it.Close()
  266. iterExists:
  267. for ; it.Valid(); it.Next() {
  268. tmpHashes[string(it.Value())] = it.Value()
  269. // Potentially exit early.
  270. select {
  271. case <-ctx.Done():
  272. break iterExists
  273. default:
  274. }
  275. }
  276. if err := it.Error(); err != nil {
  277. panic(err)
  278. }
  279. case c.Op == syntax.TContains:
  280. // XXX: startKey does not apply here.
  281. // For example, if startKey = "account.owner/an/" and search query = "account.owner CONTAINS an"
  282. // we can't iterate with prefix "account.owner/an/" because we might miss keys like "account.owner/Ulan/"
  283. it, err := dbm.IteratePrefix(txi.store, prefixFromCompositeKey(c.Tag))
  284. if err != nil {
  285. panic(err)
  286. }
  287. defer it.Close()
  288. iterContains:
  289. for ; it.Valid(); it.Next() {
  290. value, err := parseValueFromKey(it.Key())
  291. if err != nil {
  292. continue
  293. }
  294. if strings.Contains(value, c.Arg.Value()) {
  295. tmpHashes[string(it.Value())] = it.Value()
  296. }
  297. // Potentially exit early.
  298. select {
  299. case <-ctx.Done():
  300. break iterContains
  301. default:
  302. }
  303. }
  304. if err := it.Error(); err != nil {
  305. panic(err)
  306. }
  307. default:
  308. panic("other operators should be handled already")
  309. }
  310. if len(tmpHashes) == 0 || firstRun {
  311. // Either:
  312. //
  313. // 1. Regardless if a previous match was attempted, which may have had
  314. // results, but no match was found for the current condition, then we
  315. // return no matches (assuming AND operand).
  316. //
  317. // 2. A previous match was not attempted, so we return all results.
  318. return tmpHashes
  319. }
  320. // Remove/reduce matches in filteredHashes that were not found in this
  321. // match (tmpHashes).
  322. for k := range filteredHashes {
  323. if tmpHashes[k] == nil {
  324. delete(filteredHashes, k)
  325. // Potentially exit early.
  326. select {
  327. case <-ctx.Done():
  328. break
  329. default:
  330. }
  331. }
  332. }
  333. return filteredHashes
  334. }
  335. // matchRange returns all matching txs by hash that meet a given queryRange and
  336. // start key. An already filtered result (filteredHashes) is provided such that
  337. // any non-intersecting matches are removed.
  338. //
  339. // NOTE: filteredHashes may be empty if no previous condition has matched.
  340. func (txi *TxIndex) matchRange(
  341. ctx context.Context,
  342. qr indexer.QueryRange,
  343. startKey []byte,
  344. filteredHashes map[string][]byte,
  345. firstRun bool,
  346. ) map[string][]byte {
  347. // A previous match was attempted but resulted in no matches, so we return
  348. // no matches (assuming AND operand).
  349. if !firstRun && len(filteredHashes) == 0 {
  350. return filteredHashes
  351. }
  352. tmpHashes := make(map[string][]byte)
  353. lowerBound := qr.LowerBoundValue()
  354. upperBound := qr.UpperBoundValue()
  355. it, err := dbm.IteratePrefix(txi.store, startKey)
  356. if err != nil {
  357. panic(err)
  358. }
  359. defer it.Close()
  360. iter:
  361. for ; it.Valid(); it.Next() {
  362. value, err := parseValueFromKey(it.Key())
  363. if err != nil {
  364. continue
  365. }
  366. if _, ok := qr.AnyBound().(int64); ok {
  367. v, err := strconv.ParseInt(value, 10, 64)
  368. if err != nil {
  369. continue iter
  370. }
  371. include := true
  372. if lowerBound != nil && v < lowerBound.(int64) {
  373. include = false
  374. }
  375. if upperBound != nil && v > upperBound.(int64) {
  376. include = false
  377. }
  378. if include {
  379. tmpHashes[string(it.Value())] = it.Value()
  380. }
  381. // XXX: passing time in a ABCI Events is not yet implemented
  382. // case time.Time:
  383. // v := strconv.ParseInt(extractValueFromKey(it.Key()), 10, 64)
  384. // if v == r.upperBound {
  385. // break
  386. // }
  387. }
  388. // Potentially exit early.
  389. select {
  390. case <-ctx.Done():
  391. break iter
  392. default:
  393. }
  394. }
  395. if err := it.Error(); err != nil {
  396. panic(err)
  397. }
  398. if len(tmpHashes) == 0 || firstRun {
  399. // Either:
  400. //
  401. // 1. Regardless if a previous match was attempted, which may have had
  402. // results, but no match was found for the current condition, then we
  403. // return no matches (assuming AND operand).
  404. //
  405. // 2. A previous match was not attempted, so we return all results.
  406. return tmpHashes
  407. }
  408. // Remove/reduce matches in filteredHashes that were not found in this
  409. // match (tmpHashes).
  410. for k := range filteredHashes {
  411. if tmpHashes[k] == nil {
  412. delete(filteredHashes, k)
  413. // Potentially exit early.
  414. select {
  415. case <-ctx.Done():
  416. break
  417. default:
  418. }
  419. }
  420. }
  421. return filteredHashes
  422. }
  423. // ########################## Keys #############################
  424. //
  425. // The indexer has two types of kv stores:
  426. // 1. txhash - result (primary key)
  427. // 2. event - txhash (secondary key)
  428. //
  429. // The event key can be decomposed into 4 parts.
  430. // 1. A composite key which can be any string.
  431. // Usually something like "tx.height" or "account.owner"
  432. // 2. A value. That corresponds to the key. In the above
  433. // example the value could be "5" or "Ivan"
  434. // 3. The height of the Tx that aligns with the key and value.
  435. // 4. The index of the Tx that aligns with the key and value
  436. // the hash/primary key
  437. func primaryKey(hash []byte) []byte {
  438. key, err := orderedcode.Append(
  439. nil,
  440. types.TxHashKey,
  441. string(hash),
  442. )
  443. if err != nil {
  444. panic(err)
  445. }
  446. return key
  447. }
  448. // The event/secondary key
  449. func secondaryKey(compositeKey, value string, height int64, index uint32) []byte {
  450. key, err := orderedcode.Append(
  451. nil,
  452. compositeKey,
  453. value,
  454. height,
  455. int64(index),
  456. )
  457. if err != nil {
  458. panic(err)
  459. }
  460. return key
  461. }
  462. // parseValueFromKey parses an event key and extracts out the value, returning an error if one arises.
  463. // This will also involve ensuring that the key has the correct format.
  464. // CONTRACT: function doesn't check that the prefix is correct. This should have already been done by the iterator
  465. func parseValueFromKey(key []byte) (string, error) {
  466. var (
  467. compositeKey, value string
  468. height, index int64
  469. )
  470. remaining, err := orderedcode.Parse(string(key), &compositeKey, &value, &height, &index)
  471. if err != nil {
  472. return "", err
  473. }
  474. if len(remaining) != 0 {
  475. return "", fmt.Errorf("unexpected remainder in key: %s", remaining)
  476. }
  477. return value, nil
  478. }
  479. func keyFromEvent(compositeKey string, value string, result *abci.TxResult) []byte {
  480. return secondaryKey(compositeKey, value, result.Height, result.Index)
  481. }
  482. func KeyFromHeight(result *abci.TxResult) []byte {
  483. return secondaryKey(types.TxHeightKey, fmt.Sprintf("%d", result.Height), result.Height, result.Index)
  484. }
  485. // Prefixes: these represent an initial part of the key and are used by iterators to iterate over a small
  486. // section of the kv store during searches.
  487. func prefixFromCompositeKey(compositeKey string) []byte {
  488. key, err := orderedcode.Append(nil, compositeKey)
  489. if err != nil {
  490. panic(err)
  491. }
  492. return key
  493. }
  494. func prefixFromCompositeKeyAndValue(compositeKey, value string) []byte {
  495. key, err := orderedcode.Append(nil, compositeKey, value)
  496. if err != nil {
  497. panic(err)
  498. }
  499. return key
  500. }
  501. // a small utility function for getting a keys prefix based on a condition and a height
  502. func prefixForCondition(c syntax.Condition, height int64) []byte {
  503. key := prefixFromCompositeKeyAndValue(c.Tag, c.Arg.Value())
  504. if height > 0 {
  505. var err error
  506. key, err = orderedcode.Append(key, height)
  507. if err != nil {
  508. panic(err)
  509. }
  510. }
  511. return key
  512. }