You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

933 lines
28 KiB

10 years ago
10 years ago
10 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
Normalize priorities to not exceed total voting power (#3049) * more proposer priority tests - test that we don't reset to zero when updating / adding - test that same power validators alternate * add another test to track / simulate similar behaviour as in #2960 * address some of Chris' review comments * address some more of Chris' review comments * temporarily pushing branch with the following changes: The total power might change if: - a validator is added - a validator is removed - a validator is updated Decrement the accums (of all validators) directly after any of these events (by the inverse of the change) * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * Fix 2960 by re-normalizing / scaling priorities to be in bounds of total power, additionally: - remove heap where it doesn't make sense - avg. only at the end of IncrementProposerPriority instead of each iteration - update (and slightly improve) TestAveragingInIncrementProposerPriorityWithVotingPower to reflect above changes * fix tests * add comment * update changelog pending & some minor changes * comment about division will floor the result & fix typo * Update TestLargeGenesisValidator: - remove TODO and increase large genesis validator's voting power accordingly * move changelog entry to P2P Protocol * Ceil instead of flooring when dividing & update test * quickly fix failing TestProposerPriorityDoesNotGetResetToZero: - divide by Ceil((maxPriority - minPriority) / 2*totalVotingPower) * fix typo: rename getValWitMostPriority -> getValWithMostPriority * test proposer frequencies * return absolute value for diff. keep testing * use for loop for div * cleanup, more tests * spellcheck * get rid of using floats: manually ceil where necessary * Remove float, simplify, fix tests to match chris's proof (#3157)
6 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
10 years ago
10 years ago
10 years ago
10 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: prevent spurious validator power overflow warnings when changing the validator set (#4183) Fix for #4164 The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed. Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower. Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied. Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update. ## Commits: * tests for overflow detection and prevention * test fix * more tests * fix the false overflow warnings and golint * scopelint warning fix * review comments * variant with using sort by amount of change in power * compute separately number new validators in update * types: use a switch in processChanges * more review comments * types: use HasAddress in numNewValidators * types: refactor verifyUpdates copy updates, sort them by delta and use resulting slice to calculate tvpAfterUpdatesBeforeRemovals. * remove unused structs * review comments * update changelog
5 years ago
types: verify commit fully Since the light client work introduced in v0.33 it appears full nodes are no longer fully verifying commit signatures during block execution - they stop after +2/3. See in VerifyCommit: https://github.com/tendermint/tendermint/blob/0c7fd316eb006c0afc13996c00ac8bde1078b32c/types/validator_set.go#L700-L703 This means proposers can propose blocks that contain valid +2/3 signatures and then the rest of the signatures can be whatever they want. They can claim that all the other validators signed just by including a CommitSig with arbitrary signature data. While this doesn't seem to impact safety of Tendermint per se, it means that Commits may contain a lot of invalid data. This is already true of blocks, since they can include invalid txs filled with garbage, but in that case the application knows they they are invalid and can punish the proposer. But since applications dont verify commit signatures directly (they trust tendermint to do that), they won't be able to detect it. This can impact incentivization logic in the application that depends on the LastCommitInfo sent in BeginBlock, which includes which validators signed. For instance, Gaia incentivizes proposers with a bonus for including more than +2/3 of the signatures. But a proposer can now claim that bonus just by including arbitrary data for the final -1/3 of validators without actually waiting for their signatures. There may be other tricks that can be played because of this. In general, the full node should be a fully verifying machine. While it's true that the light client can avoid verifying all signatures by stopping after +2/3, the full node can not. Thus the light client and full node should use distinct VerifyCommit functions if one is going to stop after +2/3 or otherwise perform less validation (for instance light clients can also skip verifying votes for nil while full nodes can not). See a commit with a bad signature that verifies here: 56367fd. From what I can tell, Tendermint will go on to think this commit is valid and forward this data to the app, so the app will think the second validator actually signed when it clearly did not.
4 years ago
types: verify commit fully Since the light client work introduced in v0.33 it appears full nodes are no longer fully verifying commit signatures during block execution - they stop after +2/3. See in VerifyCommit: https://github.com/tendermint/tendermint/blob/0c7fd316eb006c0afc13996c00ac8bde1078b32c/types/validator_set.go#L700-L703 This means proposers can propose blocks that contain valid +2/3 signatures and then the rest of the signatures can be whatever they want. They can claim that all the other validators signed just by including a CommitSig with arbitrary signature data. While this doesn't seem to impact safety of Tendermint per se, it means that Commits may contain a lot of invalid data. This is already true of blocks, since they can include invalid txs filled with garbage, but in that case the application knows they they are invalid and can punish the proposer. But since applications dont verify commit signatures directly (they trust tendermint to do that), they won't be able to detect it. This can impact incentivization logic in the application that depends on the LastCommitInfo sent in BeginBlock, which includes which validators signed. For instance, Gaia incentivizes proposers with a bonus for including more than +2/3 of the signatures. But a proposer can now claim that bonus just by including arbitrary data for the final -1/3 of validators without actually waiting for their signatures. There may be other tricks that can be played because of this. In general, the full node should be a fully verifying machine. While it's true that the light client can avoid verifying all signatures by stopping after +2/3, the full node can not. Thus the light client and full node should use distinct VerifyCommit functions if one is going to stop after +2/3 or otherwise perform less validation (for instance light clients can also skip verifying votes for nil while full nodes can not). See a commit with a bad signature that verifies here: 56367fd. From what I can tell, Tendermint will go on to think this commit is valid and forward this data to the app, so the app will think the second validator actually signed when it clearly did not.
4 years ago
types: verify commit fully Since the light client work introduced in v0.33 it appears full nodes are no longer fully verifying commit signatures during block execution - they stop after +2/3. See in VerifyCommit: https://github.com/tendermint/tendermint/blob/0c7fd316eb006c0afc13996c00ac8bde1078b32c/types/validator_set.go#L700-L703 This means proposers can propose blocks that contain valid +2/3 signatures and then the rest of the signatures can be whatever they want. They can claim that all the other validators signed just by including a CommitSig with arbitrary signature data. While this doesn't seem to impact safety of Tendermint per se, it means that Commits may contain a lot of invalid data. This is already true of blocks, since they can include invalid txs filled with garbage, but in that case the application knows they they are invalid and can punish the proposer. But since applications dont verify commit signatures directly (they trust tendermint to do that), they won't be able to detect it. This can impact incentivization logic in the application that depends on the LastCommitInfo sent in BeginBlock, which includes which validators signed. For instance, Gaia incentivizes proposers with a bonus for including more than +2/3 of the signatures. But a proposer can now claim that bonus just by including arbitrary data for the final -1/3 of validators without actually waiting for their signatures. There may be other tricks that can be played because of this. In general, the full node should be a fully verifying machine. While it's true that the light client can avoid verifying all signatures by stopping after +2/3, the full node can not. Thus the light client and full node should use distinct VerifyCommit functions if one is going to stop after +2/3 or otherwise perform less validation (for instance light clients can also skip verifying votes for nil while full nodes can not). See a commit with a bad signature that verifies here: 56367fd. From what I can tell, Tendermint will go on to think this commit is valid and forward this data to the app, so the app will think the second validator actually signed when it clearly did not.
4 years ago
lite2: light client with weak subjectivity (#3989) Refs #1771 ADR: https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-044-lite-client-with-weak-subjectivity.md ## Commits: * add Verifier and VerifyCommitTrusting * add two more checks make trustLevel an option * float32 for trustLevel * check newHeader time * started writing lite Client * unify Verify methods * ensure h2.Header.bfttime < h1.Header.bfttime + tp * move trust checks into Verify function * add more comments * more docs * started writing tests * unbonding period failures * tests are green * export ErrNewHeaderTooFarIntoFuture * make golangci happy * test for non-adjusted headers * more precision * providers and stores * VerifyHeader and VerifyHeaderAtHeight funcs * fix compile errors * remove lastVerifiedHeight, persist new trusted header * sequential verification * remove TrustedStore option * started writing tests for light client * cover basic cases for linear verification * bisection tests PASS * rename BisectingVerification to SkippingVerification * refactor the code * add TrustedHeader method * consolidate sequential verification tests * consolidate skipping verification tests * rename trustedVals to trustedNextVals * start writing docs * ValidateTrustLevel func and ErrOldHeaderExpired error * AutoClient and example tests * fix errors * update doc * remove ErrNewHeaderTooFarIntoFuture This check is unnecessary given existing a) ErrOldHeaderExpired b) h2.Time > now checks. * return an error if we're at more recent height * add comments * add LastSignedHeaderHeight method to Store I think it's fine if Store tracks last height * copy over proxy from old lite package * make TrustedHeader return latest if height=0 * modify LastSignedHeaderHeight to return an error if no headers exist * copy over proxy impl * refactor proxy and start http lite client * Tx and BlockchainInfo methods * Block method * commit method * code compiles again * lite client compiles * extract updateLiteClientIfNeededTo func * move final parts * add placeholder for tests * force usage of lite http client in proxy * comment out query tests for now * explicitly mention tp: trusting period * verify nextVals in VerifyHeader * refactor bisection * move the NextValidatorsHash check into updateTrustedHeaderAndVals + update the comment * add ConsensusParams method to RPC client * add ConsensusParams to rpc/mock/client * change trustLevel type to a new cmn.Fraction type + update SkippingVerification comment * stress out trustLevel is only used for non-adjusted headers * fixes after Fede's review Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * compare newHeader with a header from an alternative provider * save pivot header Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349122824 * check header can still be trusted in TrustedHeader Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349101424 * lite: update Validators and Block endpoints - Block no longer contains BlockMeta - Validators now accept two additional params: page and perPage * make linter happy
5 years ago
lite: follow up from #3989 (#4209) * rename adjusted to adjacent Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r352140829 * rename ErrTooMuchChange to ErrNotEnoughVotingPowerSigned Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r352142785 * verify commit is properly signed * remove no longer trusted headers * restore trustedHeader and trustedNextVals * check trustedHeader using options Refs https://github.com/tendermint/tendermint/pull/4209#issuecomment-562462165 * use correct var when checking if headers are adjacent in bisection func + replace TODO with a comment https://github.com/tendermint/tendermint/pull/3989#discussion_r352125455 * return header in VerifyHeaderAtHeight because that way we avoid DB call + add godoc comments + check if there are no headers yet in AutoClient https://github.com/tendermint/tendermint/pull/3989#pullrequestreview-315454506 * TestVerifyAdjacentHeaders: add 2 more test-cases + add TestVerifyReturnsErrorIfTrustLevelIsInvalid * lite: avoid overflow when parsing key in db store! * lite: rename AutoClient#Err to Errs * lite: add a test for AutoClient * lite: fix keyPattern and call itr.Next in db store * lite: add two tests for db store * lite: add TestClientRemovesNoLongerTrustedHeaders * lite: test Client#Cleanup * lite: test restoring trustedHeader https://github.com/tendermint/tendermint/pull/4209#issuecomment-562462165 * lite: comment out unused code in test_helpers * fix TestVerifyReturnsErrorIfTrustLevelIsInvalid after merge * change defaultRemoveNoLongerTrustedHeadersPeriod and add docs * write more doc * lite: uncomment testable examples * use stdlog.Fatal to stop AutoClient tests * make lll linter happy * separate errors for 2 cases - the validator set of a skipped header cannot be trusted, i.e. <1/3rd of h1 validator set has signed (new error, something like ErrNewValSetCantBeTrusted) - the validator set is trusted but < 2/3rds has signed (ErrNewHeaderCantBeTrusted) https://github.com/tendermint/tendermint/pull/4209#discussion_r360331253 * remove all headers (even the last one) that are outside of the trusting period. By doing this, we avoid checking the trustedHeader's hash in checkTrustedHeaderUsingOptions (case #1). https://github.com/tendermint/tendermint/pull/4209#discussion_r360332460 * explain restoreTrustedHeaderAndNextVals better https://github.com/tendermint/tendermint/pull/4209#discussion_r360602328 * add ConfirmationFunction option for optionally prompting for user input Y/n before removing headers Refs https://github.com/tendermint/tendermint/pull/4209#discussion_r360602945 * make cleaning optional https://github.com/tendermint/tendermint/pull/4209#discussion_r364838189 * return error when user refused to remove headers * check for double votes in VerifyCommitTrusting * leave only ErrNewValSetCantBeTrusted error to differenciate between h2Vals.VerifyCommit and h1NextVals.VerifyCommitTrusting * fix example tests * remove unnecessary if condition https://github.com/tendermint/tendermint/pull/4209#discussion_r365171847 It will be handled by the above switch. * verifyCommitBasic does not depend on vals Co-authored-by: Marko <marbar3778@yahoo.com>
5 years ago
lite: follow up from #3989 (#4209) * rename adjusted to adjacent Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r352140829 * rename ErrTooMuchChange to ErrNotEnoughVotingPowerSigned Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r352142785 * verify commit is properly signed * remove no longer trusted headers * restore trustedHeader and trustedNextVals * check trustedHeader using options Refs https://github.com/tendermint/tendermint/pull/4209#issuecomment-562462165 * use correct var when checking if headers are adjacent in bisection func + replace TODO with a comment https://github.com/tendermint/tendermint/pull/3989#discussion_r352125455 * return header in VerifyHeaderAtHeight because that way we avoid DB call + add godoc comments + check if there are no headers yet in AutoClient https://github.com/tendermint/tendermint/pull/3989#pullrequestreview-315454506 * TestVerifyAdjacentHeaders: add 2 more test-cases + add TestVerifyReturnsErrorIfTrustLevelIsInvalid * lite: avoid overflow when parsing key in db store! * lite: rename AutoClient#Err to Errs * lite: add a test for AutoClient * lite: fix keyPattern and call itr.Next in db store * lite: add two tests for db store * lite: add TestClientRemovesNoLongerTrustedHeaders * lite: test Client#Cleanup * lite: test restoring trustedHeader https://github.com/tendermint/tendermint/pull/4209#issuecomment-562462165 * lite: comment out unused code in test_helpers * fix TestVerifyReturnsErrorIfTrustLevelIsInvalid after merge * change defaultRemoveNoLongerTrustedHeadersPeriod and add docs * write more doc * lite: uncomment testable examples * use stdlog.Fatal to stop AutoClient tests * make lll linter happy * separate errors for 2 cases - the validator set of a skipped header cannot be trusted, i.e. <1/3rd of h1 validator set has signed (new error, something like ErrNewValSetCantBeTrusted) - the validator set is trusted but < 2/3rds has signed (ErrNewHeaderCantBeTrusted) https://github.com/tendermint/tendermint/pull/4209#discussion_r360331253 * remove all headers (even the last one) that are outside of the trusting period. By doing this, we avoid checking the trustedHeader's hash in checkTrustedHeaderUsingOptions (case #1). https://github.com/tendermint/tendermint/pull/4209#discussion_r360332460 * explain restoreTrustedHeaderAndNextVals better https://github.com/tendermint/tendermint/pull/4209#discussion_r360602328 * add ConfirmationFunction option for optionally prompting for user input Y/n before removing headers Refs https://github.com/tendermint/tendermint/pull/4209#discussion_r360602945 * make cleaning optional https://github.com/tendermint/tendermint/pull/4209#discussion_r364838189 * return error when user refused to remove headers * check for double votes in VerifyCommitTrusting * leave only ErrNewValSetCantBeTrusted error to differenciate between h2Vals.VerifyCommit and h1NextVals.VerifyCommitTrusting * fix example tests * remove unnecessary if condition https://github.com/tendermint/tendermint/pull/4209#discussion_r365171847 It will be handled by the above switch. * verifyCommitBasic does not depend on vals Co-authored-by: Marko <marbar3778@yahoo.com>
5 years ago
lite2: light client with weak subjectivity (#3989) Refs #1771 ADR: https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-044-lite-client-with-weak-subjectivity.md ## Commits: * add Verifier and VerifyCommitTrusting * add two more checks make trustLevel an option * float32 for trustLevel * check newHeader time * started writing lite Client * unify Verify methods * ensure h2.Header.bfttime < h1.Header.bfttime + tp * move trust checks into Verify function * add more comments * more docs * started writing tests * unbonding period failures * tests are green * export ErrNewHeaderTooFarIntoFuture * make golangci happy * test for non-adjusted headers * more precision * providers and stores * VerifyHeader and VerifyHeaderAtHeight funcs * fix compile errors * remove lastVerifiedHeight, persist new trusted header * sequential verification * remove TrustedStore option * started writing tests for light client * cover basic cases for linear verification * bisection tests PASS * rename BisectingVerification to SkippingVerification * refactor the code * add TrustedHeader method * consolidate sequential verification tests * consolidate skipping verification tests * rename trustedVals to trustedNextVals * start writing docs * ValidateTrustLevel func and ErrOldHeaderExpired error * AutoClient and example tests * fix errors * update doc * remove ErrNewHeaderTooFarIntoFuture This check is unnecessary given existing a) ErrOldHeaderExpired b) h2.Time > now checks. * return an error if we're at more recent height * add comments * add LastSignedHeaderHeight method to Store I think it's fine if Store tracks last height * copy over proxy from old lite package * make TrustedHeader return latest if height=0 * modify LastSignedHeaderHeight to return an error if no headers exist * copy over proxy impl * refactor proxy and start http lite client * Tx and BlockchainInfo methods * Block method * commit method * code compiles again * lite client compiles * extract updateLiteClientIfNeededTo func * move final parts * add placeholder for tests * force usage of lite http client in proxy * comment out query tests for now * explicitly mention tp: trusting period * verify nextVals in VerifyHeader * refactor bisection * move the NextValidatorsHash check into updateTrustedHeaderAndVals + update the comment * add ConsensusParams method to RPC client * add ConsensusParams to rpc/mock/client * change trustLevel type to a new cmn.Fraction type + update SkippingVerification comment * stress out trustLevel is only used for non-adjusted headers * fixes after Fede's review Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com> * compare newHeader with a header from an alternative provider * save pivot header Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349122824 * check header can still be trusted in TrustedHeader Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349101424 * lite: update Validators and Block endpoints - Block no longer contains BlockMeta - Validators now accept two additional params: page and perPage * make linter happy
5 years ago
lite: follow up from #3989 (#4209) * rename adjusted to adjacent Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r352140829 * rename ErrTooMuchChange to ErrNotEnoughVotingPowerSigned Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r352142785 * verify commit is properly signed * remove no longer trusted headers * restore trustedHeader and trustedNextVals * check trustedHeader using options Refs https://github.com/tendermint/tendermint/pull/4209#issuecomment-562462165 * use correct var when checking if headers are adjacent in bisection func + replace TODO with a comment https://github.com/tendermint/tendermint/pull/3989#discussion_r352125455 * return header in VerifyHeaderAtHeight because that way we avoid DB call + add godoc comments + check if there are no headers yet in AutoClient https://github.com/tendermint/tendermint/pull/3989#pullrequestreview-315454506 * TestVerifyAdjacentHeaders: add 2 more test-cases + add TestVerifyReturnsErrorIfTrustLevelIsInvalid * lite: avoid overflow when parsing key in db store! * lite: rename AutoClient#Err to Errs * lite: add a test for AutoClient * lite: fix keyPattern and call itr.Next in db store * lite: add two tests for db store * lite: add TestClientRemovesNoLongerTrustedHeaders * lite: test Client#Cleanup * lite: test restoring trustedHeader https://github.com/tendermint/tendermint/pull/4209#issuecomment-562462165 * lite: comment out unused code in test_helpers * fix TestVerifyReturnsErrorIfTrustLevelIsInvalid after merge * change defaultRemoveNoLongerTrustedHeadersPeriod and add docs * write more doc * lite: uncomment testable examples * use stdlog.Fatal to stop AutoClient tests * make lll linter happy * separate errors for 2 cases - the validator set of a skipped header cannot be trusted, i.e. <1/3rd of h1 validator set has signed (new error, something like ErrNewValSetCantBeTrusted) - the validator set is trusted but < 2/3rds has signed (ErrNewHeaderCantBeTrusted) https://github.com/tendermint/tendermint/pull/4209#discussion_r360331253 * remove all headers (even the last one) that are outside of the trusting period. By doing this, we avoid checking the trustedHeader's hash in checkTrustedHeaderUsingOptions (case #1). https://github.com/tendermint/tendermint/pull/4209#discussion_r360332460 * explain restoreTrustedHeaderAndNextVals better https://github.com/tendermint/tendermint/pull/4209#discussion_r360602328 * add ConfirmationFunction option for optionally prompting for user input Y/n before removing headers Refs https://github.com/tendermint/tendermint/pull/4209#discussion_r360602945 * make cleaning optional https://github.com/tendermint/tendermint/pull/4209#discussion_r364838189 * return error when user refused to remove headers * check for double votes in VerifyCommitTrusting * leave only ErrNewValSetCantBeTrusted error to differenciate between h2Vals.VerifyCommit and h1NextVals.VerifyCommitTrusting * fix example tests * remove unnecessary if condition https://github.com/tendermint/tendermint/pull/4209#discussion_r365171847 It will be handled by the above switch. * verifyCommitBasic does not depend on vals Co-authored-by: Marko <marbar3778@yahoo.com>
5 years ago
  1. package types
  2. import (
  3. "bytes"
  4. "errors"
  5. "fmt"
  6. "math"
  7. "math/big"
  8. "sort"
  9. "strings"
  10. "github.com/tendermint/tendermint/crypto/merkle"
  11. tmmath "github.com/tendermint/tendermint/libs/math"
  12. tmproto "github.com/tendermint/tendermint/proto/tendermint/types"
  13. )
  14. const (
  15. // MaxTotalVotingPower - the maximum allowed total voting power.
  16. // It needs to be sufficiently small to, in all cases:
  17. // 1. prevent clipping in incrementProposerPriority()
  18. // 2. let (diff+diffMax-1) not overflow in IncrementProposerPriority()
  19. // (Proof of 1 is tricky, left to the reader).
  20. // It could be higher, but this is sufficiently large for our purposes,
  21. // and leaves room for defensive purposes.
  22. MaxTotalVotingPower = int64(math.MaxInt64) / 8
  23. // PriorityWindowSizeFactor - is a constant that when multiplied with the
  24. // total voting power gives the maximum allowed distance between validator
  25. // priorities.
  26. PriorityWindowSizeFactor = 2
  27. )
  28. // ErrTotalVotingPowerOverflow is returned if the total voting power of the
  29. // resulting validator set exceeds MaxTotalVotingPower.
  30. var ErrTotalVotingPowerOverflow = fmt.Errorf("total voting power of resulting valset exceeds max %d",
  31. MaxTotalVotingPower)
  32. // ValidatorSet represent a set of *Validator at a given height.
  33. //
  34. // The validators can be fetched by address or index.
  35. // The index is in order of .VotingPower, so the indices are fixed for all
  36. // rounds of a given blockchain height - ie. the validators are sorted by their
  37. // voting power (descending). Secondary index - .Address (ascending).
  38. //
  39. // On the other hand, the .ProposerPriority of each validator and the
  40. // designated .GetProposer() of a set changes every round, upon calling
  41. // .IncrementProposerPriority().
  42. //
  43. // NOTE: Not goroutine-safe.
  44. // NOTE: All get/set to validators should copy the value for safety.
  45. type ValidatorSet struct {
  46. // NOTE: persisted via reflect, must be exported.
  47. Validators []*Validator `json:"validators"`
  48. Proposer *Validator `json:"proposer"`
  49. // cached (unexported)
  50. totalVotingPower int64
  51. }
  52. // NewValidatorSet initializes a ValidatorSet by copying over the values from
  53. // `valz`, a list of Validators. If valz is nil or empty, the new ValidatorSet
  54. // will have an empty list of Validators.
  55. //
  56. // The addresses of validators in `valz` must be unique otherwise the function
  57. // panics.
  58. //
  59. // Note the validator set size has an implied limit equal to that of the
  60. // MaxVotesCount - commits by a validator set larger than this will fail
  61. // validation.
  62. func NewValidatorSet(valz []*Validator) *ValidatorSet {
  63. vals := &ValidatorSet{}
  64. err := vals.updateWithChangeSet(valz, false)
  65. if err != nil {
  66. panic(fmt.Errorf("cannot create validator set: %w", err))
  67. }
  68. if len(valz) > 0 {
  69. vals.IncrementProposerPriority(1)
  70. }
  71. return vals
  72. }
  73. func (vals *ValidatorSet) ValidateBasic() error {
  74. if vals.IsNilOrEmpty() {
  75. return errors.New("validator set is nil or empty")
  76. }
  77. for idx, val := range vals.Validators {
  78. if err := val.ValidateBasic(); err != nil {
  79. return fmt.Errorf("invalid validator #%d: %w", idx, err)
  80. }
  81. }
  82. if err := vals.Proposer.ValidateBasic(); err != nil {
  83. return fmt.Errorf("proposer failed validate basic, error: %w", err)
  84. }
  85. return nil
  86. }
  87. // IsNilOrEmpty returns true if validator set is nil or empty.
  88. func (vals *ValidatorSet) IsNilOrEmpty() bool {
  89. return vals == nil || len(vals.Validators) == 0
  90. }
  91. // CopyIncrementProposerPriority increments ProposerPriority and updates the
  92. // proposer on a copy, and returns it.
  93. func (vals *ValidatorSet) CopyIncrementProposerPriority(times int32) *ValidatorSet {
  94. copy := vals.Copy()
  95. copy.IncrementProposerPriority(times)
  96. return copy
  97. }
  98. // IncrementProposerPriority increments ProposerPriority of each validator and
  99. // updates the proposer. Panics if validator set is empty.
  100. // `times` must be positive.
  101. func (vals *ValidatorSet) IncrementProposerPriority(times int32) {
  102. if vals.IsNilOrEmpty() {
  103. panic("empty validator set")
  104. }
  105. if times <= 0 {
  106. panic("Cannot call IncrementProposerPriority with non-positive times")
  107. }
  108. // Cap the difference between priorities to be proportional to 2*totalPower by
  109. // re-normalizing priorities, i.e., rescale all priorities by multiplying with:
  110. // 2*totalVotingPower/(maxPriority - minPriority)
  111. diffMax := PriorityWindowSizeFactor * vals.TotalVotingPower()
  112. vals.RescalePriorities(diffMax)
  113. vals.shiftByAvgProposerPriority()
  114. var proposer *Validator
  115. // Call IncrementProposerPriority(1) times times.
  116. for i := int32(0); i < times; i++ {
  117. proposer = vals.incrementProposerPriority()
  118. }
  119. vals.Proposer = proposer
  120. }
  121. // RescalePriorities rescales the priorities such that the distance between the
  122. // maximum and minimum is smaller than `diffMax`. Panics if validator set is
  123. // empty.
  124. func (vals *ValidatorSet) RescalePriorities(diffMax int64) {
  125. if vals.IsNilOrEmpty() {
  126. panic("empty validator set")
  127. }
  128. // NOTE: This check is merely a sanity check which could be
  129. // removed if all tests would init. voting power appropriately;
  130. // i.e. diffMax should always be > 0
  131. if diffMax <= 0 {
  132. return
  133. }
  134. // Calculating ceil(diff/diffMax):
  135. // Re-normalization is performed by dividing by an integer for simplicity.
  136. // NOTE: This may make debugging priority issues easier as well.
  137. diff := computeMaxMinPriorityDiff(vals)
  138. ratio := (diff + diffMax - 1) / diffMax
  139. if diff > diffMax {
  140. for _, val := range vals.Validators {
  141. val.ProposerPriority /= ratio
  142. }
  143. }
  144. }
  145. func (vals *ValidatorSet) incrementProposerPriority() *Validator {
  146. for _, val := range vals.Validators {
  147. // Check for overflow for sum.
  148. newPrio := safeAddClip(val.ProposerPriority, val.VotingPower)
  149. val.ProposerPriority = newPrio
  150. }
  151. // Decrement the validator with most ProposerPriority.
  152. mostest := vals.getValWithMostPriority()
  153. // Mind the underflow.
  154. mostest.ProposerPriority = safeSubClip(mostest.ProposerPriority, vals.TotalVotingPower())
  155. return mostest
  156. }
  157. // Should not be called on an empty validator set.
  158. func (vals *ValidatorSet) computeAvgProposerPriority() int64 {
  159. n := int64(len(vals.Validators))
  160. sum := big.NewInt(0)
  161. for _, val := range vals.Validators {
  162. sum.Add(sum, big.NewInt(val.ProposerPriority))
  163. }
  164. avg := sum.Div(sum, big.NewInt(n))
  165. if avg.IsInt64() {
  166. return avg.Int64()
  167. }
  168. // This should never happen: each val.ProposerPriority is in bounds of int64.
  169. panic(fmt.Sprintf("Cannot represent avg ProposerPriority as an int64 %v", avg))
  170. }
  171. // Compute the difference between the max and min ProposerPriority of that set.
  172. func computeMaxMinPriorityDiff(vals *ValidatorSet) int64 {
  173. if vals.IsNilOrEmpty() {
  174. panic("empty validator set")
  175. }
  176. max := int64(math.MinInt64)
  177. min := int64(math.MaxInt64)
  178. for _, v := range vals.Validators {
  179. if v.ProposerPriority < min {
  180. min = v.ProposerPriority
  181. }
  182. if v.ProposerPriority > max {
  183. max = v.ProposerPriority
  184. }
  185. }
  186. diff := max - min
  187. if diff < 0 {
  188. return -1 * diff
  189. }
  190. return diff
  191. }
  192. func (vals *ValidatorSet) getValWithMostPriority() *Validator {
  193. var res *Validator
  194. for _, val := range vals.Validators {
  195. res = res.CompareProposerPriority(val)
  196. }
  197. return res
  198. }
  199. func (vals *ValidatorSet) shiftByAvgProposerPriority() {
  200. if vals.IsNilOrEmpty() {
  201. panic("empty validator set")
  202. }
  203. avgProposerPriority := vals.computeAvgProposerPriority()
  204. for _, val := range vals.Validators {
  205. val.ProposerPriority = safeSubClip(val.ProposerPriority, avgProposerPriority)
  206. }
  207. }
  208. // Makes a copy of the validator list.
  209. func validatorListCopy(valsList []*Validator) []*Validator {
  210. if valsList == nil {
  211. return nil
  212. }
  213. valsCopy := make([]*Validator, len(valsList))
  214. for i, val := range valsList {
  215. valsCopy[i] = val.Copy()
  216. }
  217. return valsCopy
  218. }
  219. // Copy each validator into a new ValidatorSet.
  220. func (vals *ValidatorSet) Copy() *ValidatorSet {
  221. return &ValidatorSet{
  222. Validators: validatorListCopy(vals.Validators),
  223. Proposer: vals.Proposer,
  224. totalVotingPower: vals.totalVotingPower,
  225. }
  226. }
  227. // HasAddress returns true if address given is in the validator set, false -
  228. // otherwise.
  229. func (vals *ValidatorSet) HasAddress(address []byte) bool {
  230. for _, val := range vals.Validators {
  231. if bytes.Equal(val.Address, address) {
  232. return true
  233. }
  234. }
  235. return false
  236. }
  237. // GetByAddress returns an index of the validator with address and validator
  238. // itself (copy) if found. Otherwise, -1 and nil are returned.
  239. func (vals *ValidatorSet) GetByAddress(address []byte) (index int32, val *Validator) {
  240. for idx, val := range vals.Validators {
  241. if bytes.Equal(val.Address, address) {
  242. return int32(idx), val.Copy()
  243. }
  244. }
  245. return -1, nil
  246. }
  247. // GetByIndex returns the validator's address and validator itself (copy) by
  248. // index.
  249. // It returns nil values if index is less than 0 or greater or equal to
  250. // len(ValidatorSet.Validators).
  251. func (vals *ValidatorSet) GetByIndex(index int32) (address []byte, val *Validator) {
  252. if index < 0 || int(index) >= len(vals.Validators) {
  253. return nil, nil
  254. }
  255. val = vals.Validators[index]
  256. return val.Address, val.Copy()
  257. }
  258. // Size returns the length of the validator set.
  259. func (vals *ValidatorSet) Size() int {
  260. return len(vals.Validators)
  261. }
  262. // Forces recalculation of the set's total voting power.
  263. // Panics if total voting power is bigger than MaxTotalVotingPower.
  264. func (vals *ValidatorSet) updateTotalVotingPower() {
  265. sum := int64(0)
  266. for _, val := range vals.Validators {
  267. // mind overflow
  268. sum = safeAddClip(sum, val.VotingPower)
  269. if sum > MaxTotalVotingPower {
  270. panic(fmt.Sprintf(
  271. "Total voting power should be guarded to not exceed %v; got: %v",
  272. MaxTotalVotingPower,
  273. sum))
  274. }
  275. }
  276. vals.totalVotingPower = sum
  277. }
  278. // TotalVotingPower returns the sum of the voting powers of all validators.
  279. // It recomputes the total voting power if required.
  280. func (vals *ValidatorSet) TotalVotingPower() int64 {
  281. if vals.totalVotingPower == 0 {
  282. vals.updateTotalVotingPower()
  283. }
  284. return vals.totalVotingPower
  285. }
  286. // GetProposer returns the current proposer. If the validator set is empty, nil
  287. // is returned.
  288. func (vals *ValidatorSet) GetProposer() (proposer *Validator) {
  289. if len(vals.Validators) == 0 {
  290. return nil
  291. }
  292. if vals.Proposer == nil {
  293. vals.Proposer = vals.findProposer()
  294. }
  295. return vals.Proposer.Copy()
  296. }
  297. func (vals *ValidatorSet) findProposer() *Validator {
  298. var proposer *Validator
  299. for _, val := range vals.Validators {
  300. if proposer == nil || !bytes.Equal(val.Address, proposer.Address) {
  301. proposer = proposer.CompareProposerPriority(val)
  302. }
  303. }
  304. return proposer
  305. }
  306. // Hash returns the Merkle root hash build using validators (as leaves) in the
  307. // set.
  308. func (vals *ValidatorSet) Hash() []byte {
  309. bzs := make([][]byte, len(vals.Validators))
  310. for i, val := range vals.Validators {
  311. bzs[i] = val.Bytes()
  312. }
  313. return merkle.HashFromByteSlices(bzs)
  314. }
  315. // Iterate will run the given function over the set.
  316. func (vals *ValidatorSet) Iterate(fn func(index int, val *Validator) bool) {
  317. for i, val := range vals.Validators {
  318. stop := fn(i, val.Copy())
  319. if stop {
  320. break
  321. }
  322. }
  323. }
  324. // Checks changes against duplicates, splits the changes in updates and
  325. // removals, sorts them by address.
  326. //
  327. // Returns:
  328. // updates, removals - the sorted lists of updates and removals
  329. // err - non-nil if duplicate entries or entries with negative voting power are seen
  330. //
  331. // No changes are made to 'origChanges'.
  332. func processChanges(origChanges []*Validator) (updates, removals []*Validator, err error) {
  333. // Make a deep copy of the changes and sort by address.
  334. changes := validatorListCopy(origChanges)
  335. sort.Sort(ValidatorsByAddress(changes))
  336. removals = make([]*Validator, 0, len(changes))
  337. updates = make([]*Validator, 0, len(changes))
  338. var prevAddr Address
  339. // Scan changes by address and append valid validators to updates or removals lists.
  340. for _, valUpdate := range changes {
  341. if bytes.Equal(valUpdate.Address, prevAddr) {
  342. err = fmt.Errorf("duplicate entry %v in %v", valUpdate, changes)
  343. return nil, nil, err
  344. }
  345. switch {
  346. case valUpdate.VotingPower < 0:
  347. err = fmt.Errorf("voting power can't be negative: %d", valUpdate.VotingPower)
  348. return nil, nil, err
  349. case valUpdate.VotingPower > MaxTotalVotingPower:
  350. err = fmt.Errorf("to prevent clipping/overflow, voting power can't be higher than %d, got %d",
  351. MaxTotalVotingPower, valUpdate.VotingPower)
  352. return nil, nil, err
  353. case valUpdate.VotingPower == 0:
  354. removals = append(removals, valUpdate)
  355. default:
  356. updates = append(updates, valUpdate)
  357. }
  358. prevAddr = valUpdate.Address
  359. }
  360. return updates, removals, err
  361. }
  362. // verifyUpdates verifies a list of updates against a validator set, making sure the allowed
  363. // total voting power would not be exceeded if these updates would be applied to the set.
  364. //
  365. // Inputs:
  366. // updates - a list of proper validator changes, i.e. they have been verified by processChanges for duplicates
  367. // and invalid values.
  368. // vals - the original validator set. Note that vals is NOT modified by this function.
  369. // removedPower - the total voting power that will be removed after the updates are verified and applied.
  370. //
  371. // Returns:
  372. // tvpAfterUpdatesBeforeRemovals - the new total voting power if these updates would be applied without the removals.
  373. // Note that this will be < 2 * MaxTotalVotingPower in case high power validators are removed and
  374. // validators are added/ updated with high power values.
  375. //
  376. // err - non-nil if the maximum allowed total voting power would be exceeded
  377. func verifyUpdates(
  378. updates []*Validator,
  379. vals *ValidatorSet,
  380. removedPower int64,
  381. ) (tvpAfterUpdatesBeforeRemovals int64, err error) {
  382. delta := func(update *Validator, vals *ValidatorSet) int64 {
  383. _, val := vals.GetByAddress(update.Address)
  384. if val != nil {
  385. return update.VotingPower - val.VotingPower
  386. }
  387. return update.VotingPower
  388. }
  389. updatesCopy := validatorListCopy(updates)
  390. sort.Slice(updatesCopy, func(i, j int) bool {
  391. return delta(updatesCopy[i], vals) < delta(updatesCopy[j], vals)
  392. })
  393. tvpAfterRemovals := vals.TotalVotingPower() - removedPower
  394. for _, upd := range updatesCopy {
  395. tvpAfterRemovals += delta(upd, vals)
  396. if tvpAfterRemovals > MaxTotalVotingPower {
  397. return 0, ErrTotalVotingPowerOverflow
  398. }
  399. }
  400. return tvpAfterRemovals + removedPower, nil
  401. }
  402. func numNewValidators(updates []*Validator, vals *ValidatorSet) int {
  403. numNewValidators := 0
  404. for _, valUpdate := range updates {
  405. if !vals.HasAddress(valUpdate.Address) {
  406. numNewValidators++
  407. }
  408. }
  409. return numNewValidators
  410. }
  411. // computeNewPriorities computes the proposer priority for the validators not present in the set based on
  412. // 'updatedTotalVotingPower'.
  413. // Leaves unchanged the priorities of validators that are changed.
  414. //
  415. // 'updates' parameter must be a list of unique validators to be added or updated.
  416. //
  417. // 'updatedTotalVotingPower' is the total voting power of a set where all updates would be applied but
  418. // not the removals. It must be < 2*MaxTotalVotingPower and may be close to this limit if close to
  419. // MaxTotalVotingPower will be removed. This is still safe from overflow since MaxTotalVotingPower is maxInt64/8.
  420. //
  421. // No changes are made to the validator set 'vals'.
  422. func computeNewPriorities(updates []*Validator, vals *ValidatorSet, updatedTotalVotingPower int64) {
  423. for _, valUpdate := range updates {
  424. address := valUpdate.Address
  425. _, val := vals.GetByAddress(address)
  426. if val == nil {
  427. // add val
  428. // Set ProposerPriority to -C*totalVotingPower (with C ~= 1.125) to make sure validators can't
  429. // un-bond and then re-bond to reset their (potentially previously negative) ProposerPriority to zero.
  430. //
  431. // Contract: updatedVotingPower < 2 * MaxTotalVotingPower to ensure ProposerPriority does
  432. // not exceed the bounds of int64.
  433. //
  434. // Compute ProposerPriority = -1.125*totalVotingPower == -(updatedVotingPower + (updatedVotingPower >> 3)).
  435. valUpdate.ProposerPriority = -(updatedTotalVotingPower + (updatedTotalVotingPower >> 3))
  436. } else {
  437. valUpdate.ProposerPriority = val.ProposerPriority
  438. }
  439. }
  440. }
  441. // Merges the vals' validator list with the updates list.
  442. // When two elements with same address are seen, the one from updates is selected.
  443. // Expects updates to be a list of updates sorted by address with no duplicates or errors,
  444. // must have been validated with verifyUpdates() and priorities computed with computeNewPriorities().
  445. func (vals *ValidatorSet) applyUpdates(updates []*Validator) {
  446. existing := vals.Validators
  447. sort.Sort(ValidatorsByAddress(existing))
  448. merged := make([]*Validator, len(existing)+len(updates))
  449. i := 0
  450. for len(existing) > 0 && len(updates) > 0 {
  451. if bytes.Compare(existing[0].Address, updates[0].Address) < 0 { // unchanged validator
  452. merged[i] = existing[0]
  453. existing = existing[1:]
  454. } else {
  455. // Apply add or update.
  456. merged[i] = updates[0]
  457. if bytes.Equal(existing[0].Address, updates[0].Address) {
  458. // Validator is present in both, advance existing.
  459. existing = existing[1:]
  460. }
  461. updates = updates[1:]
  462. }
  463. i++
  464. }
  465. // Add the elements which are left.
  466. for j := 0; j < len(existing); j++ {
  467. merged[i] = existing[j]
  468. i++
  469. }
  470. // OR add updates which are left.
  471. for j := 0; j < len(updates); j++ {
  472. merged[i] = updates[j]
  473. i++
  474. }
  475. vals.Validators = merged[:i]
  476. }
  477. // Checks that the validators to be removed are part of the validator set.
  478. // No changes are made to the validator set 'vals'.
  479. func verifyRemovals(deletes []*Validator, vals *ValidatorSet) (votingPower int64, err error) {
  480. removedVotingPower := int64(0)
  481. for _, valUpdate := range deletes {
  482. address := valUpdate.Address
  483. _, val := vals.GetByAddress(address)
  484. if val == nil {
  485. return removedVotingPower, fmt.Errorf("failed to find validator %X to remove", address)
  486. }
  487. removedVotingPower += val.VotingPower
  488. }
  489. if len(deletes) > len(vals.Validators) {
  490. panic("more deletes than validators")
  491. }
  492. return removedVotingPower, nil
  493. }
  494. // Removes the validators specified in 'deletes' from validator set 'vals'.
  495. // Should not fail as verification has been done before.
  496. // Expects vals to be sorted by address (done by applyUpdates).
  497. func (vals *ValidatorSet) applyRemovals(deletes []*Validator) {
  498. existing := vals.Validators
  499. merged := make([]*Validator, len(existing)-len(deletes))
  500. i := 0
  501. // Loop over deletes until we removed all of them.
  502. for len(deletes) > 0 {
  503. if bytes.Equal(existing[0].Address, deletes[0].Address) {
  504. deletes = deletes[1:]
  505. } else { // Leave it in the resulting slice.
  506. merged[i] = existing[0]
  507. i++
  508. }
  509. existing = existing[1:]
  510. }
  511. // Add the elements which are left.
  512. for j := 0; j < len(existing); j++ {
  513. merged[i] = existing[j]
  514. i++
  515. }
  516. vals.Validators = merged[:i]
  517. }
  518. // Main function used by UpdateWithChangeSet() and NewValidatorSet().
  519. // If 'allowDeletes' is false then delete operations (identified by validators with voting power 0)
  520. // are not allowed and will trigger an error if present in 'changes'.
  521. // The 'allowDeletes' flag is set to false by NewValidatorSet() and to true by UpdateWithChangeSet().
  522. func (vals *ValidatorSet) updateWithChangeSet(changes []*Validator, allowDeletes bool) error {
  523. if len(changes) == 0 {
  524. return nil
  525. }
  526. // Check for duplicates within changes, split in 'updates' and 'deletes' lists (sorted).
  527. updates, deletes, err := processChanges(changes)
  528. if err != nil {
  529. return err
  530. }
  531. if !allowDeletes && len(deletes) != 0 {
  532. return fmt.Errorf("cannot process validators with voting power 0: %v", deletes)
  533. }
  534. // Check that the resulting set will not be empty.
  535. if numNewValidators(updates, vals) == 0 && len(vals.Validators) == len(deletes) {
  536. return errors.New("applying the validator changes would result in empty set")
  537. }
  538. // Verify that applying the 'deletes' against 'vals' will not result in error.
  539. // Get the voting power that is going to be removed.
  540. removedVotingPower, err := verifyRemovals(deletes, vals)
  541. if err != nil {
  542. return err
  543. }
  544. // Verify that applying the 'updates' against 'vals' will not result in error.
  545. // Get the updated total voting power before removal. Note that this is < 2 * MaxTotalVotingPower
  546. tvpAfterUpdatesBeforeRemovals, err := verifyUpdates(updates, vals, removedVotingPower)
  547. if err != nil {
  548. return err
  549. }
  550. // Compute the priorities for updates.
  551. computeNewPriorities(updates, vals, tvpAfterUpdatesBeforeRemovals)
  552. // Apply updates and removals.
  553. vals.applyUpdates(updates)
  554. vals.applyRemovals(deletes)
  555. vals.updateTotalVotingPower() // will panic if total voting power > MaxTotalVotingPower
  556. // Scale and center.
  557. vals.RescalePriorities(PriorityWindowSizeFactor * vals.TotalVotingPower())
  558. vals.shiftByAvgProposerPriority()
  559. sort.Sort(ValidatorsByVotingPower(vals.Validators))
  560. return nil
  561. }
  562. // UpdateWithChangeSet attempts to update the validator set with 'changes'.
  563. // It performs the following steps:
  564. // - validates the changes making sure there are no duplicates and splits them in updates and deletes
  565. // - verifies that applying the changes will not result in errors
  566. // - computes the total voting power BEFORE removals to ensure that in the next steps the priorities
  567. // across old and newly added validators are fair
  568. // - computes the priorities of new validators against the final set
  569. // - applies the updates against the validator set
  570. // - applies the removals against the validator set
  571. // - performs scaling and centering of priority values
  572. // If an error is detected during verification steps, it is returned and the validator set
  573. // is not changed.
  574. func (vals *ValidatorSet) UpdateWithChangeSet(changes []*Validator) error {
  575. return vals.updateWithChangeSet(changes, true)
  576. }
  577. // VerifyCommit verifies +2/3 of the set had signed the given commit and all
  578. // other signatures are valid
  579. func (vals *ValidatorSet) VerifyCommit(chainID string, blockID BlockID,
  580. height int64, commit *Commit) error {
  581. return VerifyCommit(chainID, vals, blockID, height, commit)
  582. }
  583. // LIGHT CLIENT VERIFICATION METHODS
  584. // VerifyCommitLight verifies +2/3 of the set had signed the given commit.
  585. func (vals *ValidatorSet) VerifyCommitLight(chainID string, blockID BlockID,
  586. height int64, commit *Commit) error {
  587. return VerifyCommitLight(chainID, vals, blockID, height, commit)
  588. }
  589. // VerifyCommitLightTrusting verifies that trustLevel of the validator set signed
  590. // this commit.
  591. func (vals *ValidatorSet) VerifyCommitLightTrusting(chainID string, commit *Commit, trustLevel tmmath.Fraction) error {
  592. return VerifyCommitLightTrusting(chainID, vals, commit, trustLevel)
  593. }
  594. // findPreviousProposer reverses the compare proposer priority function to find the validator
  595. // with the lowest proposer priority which would have been the previous proposer.
  596. //
  597. // Is used when recreating a validator set from an existing array of validators.
  598. func (vals *ValidatorSet) findPreviousProposer() *Validator {
  599. var previousProposer *Validator
  600. for _, val := range vals.Validators {
  601. if previousProposer == nil {
  602. previousProposer = val
  603. continue
  604. }
  605. if previousProposer == previousProposer.CompareProposerPriority(val) {
  606. previousProposer = val
  607. }
  608. }
  609. return previousProposer
  610. }
  611. //-----------------
  612. // IsErrNotEnoughVotingPowerSigned returns true if err is
  613. // ErrNotEnoughVotingPowerSigned.
  614. func IsErrNotEnoughVotingPowerSigned(err error) bool {
  615. return errors.As(err, &ErrNotEnoughVotingPowerSigned{})
  616. }
  617. // ErrNotEnoughVotingPowerSigned is returned when not enough validators signed
  618. // a commit.
  619. type ErrNotEnoughVotingPowerSigned struct {
  620. Got int64
  621. Needed int64
  622. }
  623. func (e ErrNotEnoughVotingPowerSigned) Error() string {
  624. return fmt.Sprintf("invalid commit -- insufficient voting power: got %d, needed more than %d", e.Got, e.Needed)
  625. }
  626. //----------------
  627. // String returns a string representation of ValidatorSet.
  628. //
  629. // See StringIndented.
  630. func (vals *ValidatorSet) String() string {
  631. return vals.StringIndented("")
  632. }
  633. // StringIndented returns an intended String.
  634. //
  635. // See Validator#String.
  636. func (vals *ValidatorSet) StringIndented(indent string) string {
  637. if vals == nil {
  638. return "nil-ValidatorSet"
  639. }
  640. var valStrings []string
  641. vals.Iterate(func(index int, val *Validator) bool {
  642. valStrings = append(valStrings, val.String())
  643. return false
  644. })
  645. return fmt.Sprintf(`ValidatorSet{
  646. %s Proposer: %v
  647. %s Validators:
  648. %s %v
  649. %s}`,
  650. indent, vals.GetProposer().String(),
  651. indent,
  652. indent, strings.Join(valStrings, "\n"+indent+" "),
  653. indent)
  654. }
  655. //-------------------------------------
  656. // ValidatorsByVotingPower implements sort.Interface for []*Validator based on
  657. // the VotingPower and Address fields.
  658. type ValidatorsByVotingPower []*Validator
  659. func (valz ValidatorsByVotingPower) Len() int { return len(valz) }
  660. func (valz ValidatorsByVotingPower) Less(i, j int) bool {
  661. if valz[i].VotingPower == valz[j].VotingPower {
  662. return bytes.Compare(valz[i].Address, valz[j].Address) == -1
  663. }
  664. return valz[i].VotingPower > valz[j].VotingPower
  665. }
  666. func (valz ValidatorsByVotingPower) Swap(i, j int) {
  667. valz[i], valz[j] = valz[j], valz[i]
  668. }
  669. // ValidatorsByAddress implements sort.Interface for []*Validator based on
  670. // the Address field.
  671. type ValidatorsByAddress []*Validator
  672. func (valz ValidatorsByAddress) Len() int { return len(valz) }
  673. func (valz ValidatorsByAddress) Less(i, j int) bool {
  674. return bytes.Compare(valz[i].Address, valz[j].Address) == -1
  675. }
  676. func (valz ValidatorsByAddress) Swap(i, j int) {
  677. valz[i], valz[j] = valz[j], valz[i]
  678. }
  679. // ToProto converts ValidatorSet to protobuf
  680. func (vals *ValidatorSet) ToProto() (*tmproto.ValidatorSet, error) {
  681. if vals.IsNilOrEmpty() {
  682. return &tmproto.ValidatorSet{}, nil // validator set should never be nil
  683. }
  684. vp := new(tmproto.ValidatorSet)
  685. valsProto := make([]*tmproto.Validator, len(vals.Validators))
  686. for i := 0; i < len(vals.Validators); i++ {
  687. valp, err := vals.Validators[i].ToProto()
  688. if err != nil {
  689. return nil, err
  690. }
  691. valsProto[i] = valp
  692. }
  693. vp.Validators = valsProto
  694. valProposer, err := vals.Proposer.ToProto()
  695. if err != nil {
  696. return nil, fmt.Errorf("toProto: validatorSet proposer error: %w", err)
  697. }
  698. vp.Proposer = valProposer
  699. // NOTE: Sometimes we use the bytes of the proto form as a hash. This means that we need to
  700. // be consistent with cached data
  701. vp.TotalVotingPower = 0
  702. return vp, nil
  703. }
  704. // ValidatorSetFromProto sets a protobuf ValidatorSet to the given pointer.
  705. // It returns an error if any of the validators from the set or the proposer
  706. // is invalid
  707. func ValidatorSetFromProto(vp *tmproto.ValidatorSet) (*ValidatorSet, error) {
  708. if vp == nil {
  709. return nil, errors.New("nil validator set") // validator set should never be nil, bigger issues are at play if empty
  710. }
  711. vals := new(ValidatorSet)
  712. valsProto := make([]*Validator, len(vp.Validators))
  713. for i := 0; i < len(vp.Validators); i++ {
  714. v, err := ValidatorFromProto(vp.Validators[i])
  715. if err != nil {
  716. return nil, err
  717. }
  718. valsProto[i] = v
  719. }
  720. vals.Validators = valsProto
  721. p, err := ValidatorFromProto(vp.GetProposer())
  722. if err != nil {
  723. return nil, fmt.Errorf("fromProto: validatorSet proposer error: %w", err)
  724. }
  725. vals.Proposer = p
  726. // NOTE: We can't trust the total voting power given to us by other peers. If someone were to
  727. // inject a non-zeo value that wasn't the correct voting power we could assume a wrong total
  728. // power hence we need to recompute it.
  729. // FIXME: We should look to remove TotalVotingPower from proto or add it in the validators hash
  730. // so we don't have to do this
  731. vals.TotalVotingPower()
  732. return vals, vals.ValidateBasic()
  733. }
  734. // ValidatorSetFromExistingValidators takes an existing array of validators and
  735. // rebuilds the exact same validator set that corresponds to it without
  736. // changing the proposer priority or power if any of the validators fail
  737. // validate basic then an empty set is returned.
  738. func ValidatorSetFromExistingValidators(valz []*Validator) (*ValidatorSet, error) {
  739. if len(valz) == 0 {
  740. return nil, errors.New("validator set is empty")
  741. }
  742. for _, val := range valz {
  743. err := val.ValidateBasic()
  744. if err != nil {
  745. return nil, fmt.Errorf("can't create validator set: %w", err)
  746. }
  747. }
  748. vals := &ValidatorSet{
  749. Validators: valz,
  750. }
  751. vals.Proposer = vals.findPreviousProposer()
  752. vals.updateTotalVotingPower()
  753. sort.Sort(ValidatorsByVotingPower(vals.Validators))
  754. return vals, nil
  755. }
  756. //----------------------------------------
  757. // safe addition/subtraction/multiplication
  758. func safeAdd(a, b int64) (int64, bool) {
  759. if b > 0 && a > math.MaxInt64-b {
  760. return -1, true
  761. } else if b < 0 && a < math.MinInt64-b {
  762. return -1, true
  763. }
  764. return a + b, false
  765. }
  766. func safeSub(a, b int64) (int64, bool) {
  767. if b > 0 && a < math.MinInt64+b {
  768. return -1, true
  769. } else if b < 0 && a > math.MaxInt64+b {
  770. return -1, true
  771. }
  772. return a - b, false
  773. }
  774. func safeAddClip(a, b int64) int64 {
  775. c, overflow := safeAdd(a, b)
  776. if overflow {
  777. if b < 0 {
  778. return math.MinInt64
  779. }
  780. return math.MaxInt64
  781. }
  782. return c
  783. }
  784. func safeSubClip(a, b int64) int64 {
  785. c, overflow := safeSub(a, b)
  786. if overflow {
  787. if b > 0 {
  788. return math.MinInt64
  789. }
  790. return math.MaxInt64
  791. }
  792. return c
  793. }
  794. func safeMul(a, b int64) (int64, bool) {
  795. if a == 0 || b == 0 {
  796. return 0, false
  797. }
  798. absOfB := b
  799. if b < 0 {
  800. absOfB = -b
  801. }
  802. absOfA := a
  803. if a < 0 {
  804. absOfA = -a
  805. }
  806. if absOfA > math.MaxInt64/absOfB {
  807. return 0, true
  808. }
  809. return a * b, false
  810. }