You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

118 lines
3.5 KiB

crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
6 years ago
lint: Enable Golint (#4212) * Fix many golint errors * Fix golint errors in the 'lite' package * Don't export Pool.store * Fix typo * Revert unwanted changes * Fix errors in counter package * Fix linter errors in kvstore package * Fix linter error in example package * Fix error in tests package * Fix linter errors in v2 package * Fix linter errors in consensus package * Fix linter errors in evidence package * Fix linter error in fail package * Fix linter errors in query package * Fix linter errors in core package * Fix linter errors in node package * Fix linter errors in mempool package * Fix linter error in conn package * Fix linter errors in pex package * Rename PEXReactor export to Reactor * Fix linter errors in trust package * Fix linter errors in upnp package * Fix linter errors in p2p package * Fix linter errors in proxy package * Fix linter errors in mock_test package * Fix linter error in client_test package * Fix linter errors in coretypes package * Fix linter errors in coregrpc package * Fix linter errors in rpcserver package * Fix linter errors in rpctypes package * Fix linter errors in rpctest package * Fix linter error in json2wal script * Fix linter error in wal2json script * Fix linter errors in kv package * Fix linter error in state package * Fix linter error in grpc_client * Fix linter errors in types package * Fix linter error in version package * Fix remaining errors * Address review comments * Fix broken tests * Reconcile package coregrpc * Fix golangci bot error * Fix new golint errors * Fix broken reference * Enable golint linter * minor changes to bring golint into line * fix failing test * fix pex reactor naming * address PR comments
5 years ago
crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
6 years ago
lint: Enable Golint (#4212) * Fix many golint errors * Fix golint errors in the 'lite' package * Don't export Pool.store * Fix typo * Revert unwanted changes * Fix errors in counter package * Fix linter errors in kvstore package * Fix linter error in example package * Fix error in tests package * Fix linter errors in v2 package * Fix linter errors in consensus package * Fix linter errors in evidence package * Fix linter error in fail package * Fix linter errors in query package * Fix linter errors in core package * Fix linter errors in node package * Fix linter errors in mempool package * Fix linter error in conn package * Fix linter errors in pex package * Rename PEXReactor export to Reactor * Fix linter errors in trust package * Fix linter errors in upnp package * Fix linter errors in p2p package * Fix linter errors in proxy package * Fix linter errors in mock_test package * Fix linter error in client_test package * Fix linter errors in coretypes package * Fix linter errors in coregrpc package * Fix linter errors in rpcserver package * Fix linter errors in rpctypes package * Fix linter errors in rpctest package * Fix linter error in json2wal script * Fix linter error in wal2json script * Fix linter errors in kv package * Fix linter error in state package * Fix linter error in grpc_client * Fix linter errors in types package * Fix linter error in version package * Fix remaining errors * Address review comments * Fix broken tests * Reconcile package coregrpc * Fix golangci bot error * Fix new golint errors * Fix broken reference * Enable golint linter * minor changes to bring golint into line * fix failing test * fix pex reactor naming * address PR comments
5 years ago
crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
6 years ago
  1. package merkle
  2. import (
  3. "math/bits"
  4. )
  5. // SimpleHashFromByteSlices computes a Merkle tree where the leaves are the byte slice,
  6. // in the provided order.
  7. func SimpleHashFromByteSlices(items [][]byte) []byte {
  8. switch len(items) {
  9. case 0:
  10. return nil
  11. case 1:
  12. return leafHash(items[0])
  13. default:
  14. k := getSplitPoint(len(items))
  15. left := SimpleHashFromByteSlices(items[:k])
  16. right := SimpleHashFromByteSlices(items[k:])
  17. return innerHash(left, right)
  18. }
  19. }
  20. // SimpleHashFromByteSliceIterative is an iterative alternative to
  21. // SimpleHashFromByteSlice motivated by potential performance improvements.
  22. // (#2611) had suggested that an iterative version of
  23. // SimpleHashFromByteSlice would be faster, presumably because
  24. // we can envision some overhead accumulating from stack
  25. // frames and function calls. Additionally, a recursive algorithm risks
  26. // hitting the stack limit and causing a stack overflow should the tree
  27. // be too large.
  28. //
  29. // Provided here is an iterative alternative, a simple test to assert
  30. // correctness and a benchmark. On the performance side, there appears to
  31. // be no overall difference:
  32. //
  33. // BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op
  34. // BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op
  35. //
  36. // On the surface it might seem that the additional overhead is due to
  37. // the different allocation patterns of the implementations. The recursive
  38. // version uses a single [][]byte slices which it then re-slices at each level of the tree.
  39. // The iterative version reproduces [][]byte once within the function and
  40. // then rewrites sub-slices of that array at each level of the tree.
  41. //
  42. // Experimenting by modifying the code to simply calculate the
  43. // hash and not store the result show little to no difference in performance.
  44. //
  45. // These preliminary results suggest:
  46. //
  47. // 1. The performance of the SimpleHashFromByteSlice is pretty good
  48. // 2. Go has low overhead for recursive functions
  49. // 3. The performance of the SimpleHashFromByteSlice routine is dominated
  50. // by the actual hashing of data
  51. //
  52. // Although this work is in no way exhaustive, point #3 suggests that
  53. // optimization of this routine would need to take an alternative
  54. // approach to make significant improvements on the current performance.
  55. //
  56. // Finally, considering that the recursive implementation is easier to
  57. // read, it might not be worthwhile to switch to a less intuitive
  58. // implementation for so little benefit.
  59. func SimpleHashFromByteSlicesIterative(input [][]byte) []byte {
  60. items := make([][]byte, len(input))
  61. for i, leaf := range input {
  62. items[i] = leafHash(leaf)
  63. }
  64. size := len(items)
  65. for {
  66. switch size {
  67. case 0:
  68. return nil
  69. case 1:
  70. return items[0]
  71. default:
  72. rp := 0 // read position
  73. wp := 0 // write position
  74. for rp < size {
  75. if rp+1 < size {
  76. items[wp] = innerHash(items[rp], items[rp+1])
  77. rp += 2
  78. } else {
  79. items[wp] = items[rp]
  80. rp++
  81. }
  82. wp++
  83. }
  84. size = wp
  85. }
  86. }
  87. }
  88. // SimpleHashFromMap computes a Merkle tree from sorted map.
  89. // Like calling SimpleHashFromHashers with
  90. // `item = []byte(Hash(key) | Hash(value))`,
  91. // sorted by `item`.
  92. func SimpleHashFromMap(m map[string][]byte) []byte {
  93. sm := newSimpleMap()
  94. for k, v := range m {
  95. sm.Set(k, v)
  96. }
  97. return sm.Hash()
  98. }
  99. // getSplitPoint returns the largest power of 2 less than length
  100. func getSplitPoint(length int) int {
  101. if length < 1 {
  102. panic("Trying to split a tree with size < 1")
  103. }
  104. uLength := uint(length)
  105. bitlen := bits.Len(uLength)
  106. k := 1 << uint(bitlen-1)
  107. if k == length {
  108. k >>= 1
  109. }
  110. return k
  111. }