You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

160 lines
4.3 KiB

crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
5 years ago
crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
5 years ago
crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
5 years ago
crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
5 years ago
crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
5 years ago
crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
5 years ago
crypto: Proof of Concept for iterative version of SimpleHashFromByteSlices (#2611) (#3530) (#2611) had suggested that an iterative version of SimpleHashFromByteSlice would be faster, presumably because we can envision some overhead accumulating from stack frames and function calls. Additionally, a recursive algorithm risks hitting the stack limit and causing a stack overflow should the tree be too large. Provided here is an iterative alternative, a simple test to assert correctness and a benchmark. On the performance side, there appears to be no overall difference: ``` BenchmarkSimpleHashAlternatives/recursive-4 20000 77677 ns/op BenchmarkSimpleHashAlternatives/iterative-4 20000 76802 ns/op ``` On the surface it might seem that the additional overhead is due to the different allocation patterns of the implementations. The recursive version uses a single `[][]byte` slices which it then re-slices at each level of the tree. The iterative version reproduces `[][]byte` once within the function and then rewrites sub-slices of that array at each level of the tree. Eexperimenting by modifying the code to simply calculate the hash and not store the result show little to no difference in performance. These preliminary results suggest: 1. The performance of the current implementation is pretty good 2. Go has low overhead for recursive functions 3. The performance of the SimpleHashFromByteSlice routine is dominated by the actual hashing of data Although this work is in no way exhaustive, point #3 suggests that optimizations of this routine would need to take an alternative approach to make significant improvements on the current performance. Finally, considering that the recursive implementation is easier to read, it might not be worthwhile to switch to a less intuitive implementation for so little benefit. * re-add slice re-writing * [crypto] Document SimpleHashFromByteSlicesIterative
5 years ago
  1. package merkle
  2. import (
  3. "encoding/hex"
  4. "testing"
  5. "github.com/stretchr/testify/assert"
  6. "github.com/stretchr/testify/require"
  7. "github.com/tendermint/tendermint/crypto/tmhash"
  8. ctest "github.com/tendermint/tendermint/internal/libs/test"
  9. tmrand "github.com/tendermint/tendermint/libs/rand"
  10. )
  11. type testItem []byte
  12. func (tI testItem) Hash() []byte {
  13. return []byte(tI)
  14. }
  15. func TestHashFromByteSlices(t *testing.T) {
  16. testcases := map[string]struct {
  17. slices [][]byte
  18. expectHash string // in hex format
  19. }{
  20. "nil": {nil, "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"},
  21. "empty": {[][]byte{}, "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"},
  22. "single": {[][]byte{{1, 2, 3}}, "054edec1d0211f624fed0cbca9d4f9400b0e491c43742af2c5b0abebf0c990d8"},
  23. "single blank": {[][]byte{{}}, "6e340b9cffb37a989ca544e6bb780a2c78901d3fb33738768511a30617afa01d"},
  24. "two": {[][]byte{{1, 2, 3}, {4, 5, 6}}, "82e6cfce00453804379b53962939eaa7906b39904be0813fcadd31b100773c4b"},
  25. "many": {
  26. [][]byte{{1, 2}, {3, 4}, {5, 6}, {7, 8}, {9, 10}},
  27. "f326493eceab4f2d9ffbc78c59432a0a005d6ea98392045c74df5d14a113be18",
  28. },
  29. }
  30. for name, tc := range testcases {
  31. tc := tc
  32. t.Run(name, func(t *testing.T) {
  33. hash := HashFromByteSlices(tc.slices)
  34. assert.Equal(t, tc.expectHash, hex.EncodeToString(hash))
  35. })
  36. }
  37. }
  38. func TestProof(t *testing.T) {
  39. // Try an empty proof first
  40. rootHash, proofs := ProofsFromByteSlices([][]byte{})
  41. require.Equal(t, "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", hex.EncodeToString(rootHash))
  42. require.Empty(t, proofs)
  43. total := 100
  44. items := make([][]byte, total)
  45. for i := 0; i < total; i++ {
  46. items[i] = testItem(tmrand.Bytes(tmhash.Size))
  47. }
  48. rootHash = HashFromByteSlices(items)
  49. rootHash2, proofs := ProofsFromByteSlices(items)
  50. require.Equal(t, rootHash, rootHash2, "Unmatched root hashes: %X vs %X", rootHash, rootHash2)
  51. // For each item, check the trail.
  52. for i, item := range items {
  53. proof := proofs[i]
  54. // Check total/index
  55. require.EqualValues(t, proof.Index, i, "Unmatched indicies: %d vs %d", proof.Index, i)
  56. require.EqualValues(t, proof.Total, total, "Unmatched totals: %d vs %d", proof.Total, total)
  57. // Verify success
  58. err := proof.Verify(rootHash, item)
  59. require.NoError(t, err, "Verification failed: %v.", err)
  60. // Trail too long should make it fail
  61. origAunts := proof.Aunts
  62. proof.Aunts = append(proof.Aunts, tmrand.Bytes(32))
  63. err = proof.Verify(rootHash, item)
  64. require.Error(t, err, "Expected verification to fail for wrong trail length")
  65. proof.Aunts = origAunts
  66. // Trail too short should make it fail
  67. proof.Aunts = proof.Aunts[0 : len(proof.Aunts)-1]
  68. err = proof.Verify(rootHash, item)
  69. require.Error(t, err, "Expected verification to fail for wrong trail length")
  70. proof.Aunts = origAunts
  71. // Mutating the itemHash should make it fail.
  72. err = proof.Verify(rootHash, ctest.MutateByteSlice(item))
  73. require.Error(t, err, "Expected verification to fail for mutated leaf hash")
  74. // Mutating the rootHash should make it fail.
  75. err = proof.Verify(ctest.MutateByteSlice(rootHash), item)
  76. require.Error(t, err, "Expected verification to fail for mutated root hash")
  77. }
  78. }
  79. func TestHashAlternatives(t *testing.T) {
  80. total := 100
  81. items := make([][]byte, total)
  82. for i := 0; i < total; i++ {
  83. items[i] = testItem(tmrand.Bytes(tmhash.Size))
  84. }
  85. rootHash1 := HashFromByteSlicesIterative(items)
  86. rootHash2 := HashFromByteSlices(items)
  87. require.Equal(t, rootHash1, rootHash2, "Unmatched root hashes: %X vs %X", rootHash1, rootHash2)
  88. }
  89. func BenchmarkHashAlternatives(b *testing.B) {
  90. total := 100
  91. items := make([][]byte, total)
  92. for i := 0; i < total; i++ {
  93. items[i] = testItem(tmrand.Bytes(tmhash.Size))
  94. }
  95. b.ResetTimer()
  96. b.Run("recursive", func(b *testing.B) {
  97. for i := 0; i < b.N; i++ {
  98. _ = HashFromByteSlices(items)
  99. }
  100. })
  101. b.Run("iterative", func(b *testing.B) {
  102. for i := 0; i < b.N; i++ {
  103. _ = HashFromByteSlicesIterative(items)
  104. }
  105. })
  106. }
  107. func Test_getSplitPoint(t *testing.T) {
  108. tests := []struct {
  109. length int64
  110. want int64
  111. }{
  112. {1, 0},
  113. {2, 1},
  114. {3, 2},
  115. {4, 2},
  116. {5, 4},
  117. {10, 8},
  118. {20, 16},
  119. {100, 64},
  120. {255, 128},
  121. {256, 128},
  122. {257, 256},
  123. }
  124. for _, tt := range tests {
  125. got := getSplitPoint(tt.length)
  126. require.EqualValues(t, tt.want, got, "getSplitPoint(%d) = %v, want %v", tt.length, got, tt.want)
  127. }
  128. }