* Add processor prototype
* Change processor API
+ expose a simple `handle` function which mutates internal state
* schedule event handling
* rename schedule -> scheduler
* fill in handle function
* processor tests
* fix gofmt and ohter golangci issues
* scopelint var on range scope
* add check for short block received
* small test reorg
* ci fix changes
* go.mod revert
* some cleanup and review comments
* scheduler fixes and unit tests, also small processor changes.
changed scPeerPruned to include a list of pruned peers
touchPeer to check peer state and remove the blocks from blockStates if the peer removal causes the max peer height to be lower.
remove the block at sc.initHeight
changed peersInactiveSince, peersSlowerThan, getPeersAtHeight check peer state
prunablePeers to return a sorted list of peers
lastRate in markReceived() attempted to divide by 0, temp fix.
fixed allBlocksProcessed conditions
maxHeight() and minHeight() to return sc.initHeight if no ready peers present
make selectPeer() deterministic.
added handleBlockProcessError()
added termination cond. (sc.allBlocksProcessed()) to handleTryPrunePeer() and others.
changed pcBlockVerificationFailure to include peer of H+2 block along with the one for H+1
changed the processor to call purgePeer on block verification failure.
fixed processor tests
added scheduler tests.
* typo and ci fixes
* remove height from scBlockRequest, golangci fixes
* limit on blockState map, updated tests
* remove unused
* separate test for maxHeight(), used for sched. validation
* use Math.Min
* fix golangci
* Document the semantics of blockStates in the scheduler
* better docs
* distinguish between unknown and invalid blockstate
* Standardize peer filtering methods
* feedback
* s/getPeersAtHeight/getPeersAtHeightOrAbove
* small notes
* Update blockchain/v2/scheduler.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update comments based on feedback
* Add enum offset
* panic on nil block in processor
* remove unused max height calculation
* format shorter line
Fix for #4164
The general problem is that in certain conditions an overflow warning is issued when attempting to update a validator set even if the final set's total voting power is not over the maximum allowed.
Root cause is that in verifyUpdates(), updates are verified wrt to total voting power in the order of validator address. It is then possible that a low address validator may increase its power such that the temporary total voting power count goes over MaxTotalVotingPower.
Scenarios where removing and adding/ updating validators with high voting power, in the same update operation, cause the same false warning and the updates are not applied.
Main changes to fix this are in verifyUpdate() that now does the verification starting with the decreases in power. It also takes into account the removals that are part of the update.
## Commits:
* tests for overflow detection and prevention
* test fix
* more tests
* fix the false overflow warnings and golint
* scopelint warning fix
* review comments
* variant with using sort by amount of change in power
* compute separately number new validators in update
* types: use a switch in processChanges
* more review comments
* types: use HasAddress in numNewValidators
* types: refactor verifyUpdates
copy updates, sort them by delta and use resulting slice to calculate
tvpAfterUpdatesBeforeRemovals.
* remove unused structs
* review comments
* update changelog
* p2p/conn: simplify secret connection handshake malleability fix with merlin
Introduces new dependencies on github.com/gtank/merlin and sha3 as a cryptographic primitive
This also only uses the transcript hash as a MAC.
* p2p/conn: avoid string to byte conversion
https://github.com/uber-go/guide/blob/master/style.md#avoid-string-to-byte-conversion
* types: change `Commit` to consist of just signatures
These are final changes towards removing votes from commit and leaving
only signatures (see ADR-25)
Fixes#1648
* bring back TestCommitToVoteSetWithVotesForAnotherBlockOrNilBlock
+ add absent flag to Vote to indicate that it's for another block
* encode nil votes as CommitSig with BlockIDFlagAbsent
+ make Commit#Precommits array of non-pointers
because precommit will never be nil
* add NewCommitSigAbsent and Absent() funcs
* uncomment validation in CommitSig#ValidateBasic
* add comments to ValidatorSet funcs
* add a changelog entry
* break instead of continue
continue does not make sense in these cases
* types: rename Commit#Precommits to Signatures
* swagger: fix /commit response
* swagger: change block_id_flag type
* fix merge conflicts
Refs #1771
ADR: https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-044-lite-client-with-weak-subjectivity.md
## Commits:
* add Verifier and VerifyCommitTrusting
* add two more checks
make trustLevel an option
* float32 for trustLevel
* check newHeader time
* started writing lite Client
* unify Verify methods
* ensure h2.Header.bfttime < h1.Header.bfttime + tp
* move trust checks into Verify function
* add more comments
* more docs
* started writing tests
* unbonding period failures
* tests are green
* export ErrNewHeaderTooFarIntoFuture
* make golangci happy
* test for non-adjusted headers
* more precision
* providers and stores
* VerifyHeader and VerifyHeaderAtHeight funcs
* fix compile errors
* remove lastVerifiedHeight, persist new trusted header
* sequential verification
* remove TrustedStore option
* started writing tests for light client
* cover basic cases for linear verification
* bisection tests PASS
* rename BisectingVerification to SkippingVerification
* refactor the code
* add TrustedHeader method
* consolidate sequential verification tests
* consolidate skipping verification tests
* rename trustedVals to trustedNextVals
* start writing docs
* ValidateTrustLevel func and ErrOldHeaderExpired error
* AutoClient and example tests
* fix errors
* update doc
* remove ErrNewHeaderTooFarIntoFuture
This check is unnecessary given existing a) ErrOldHeaderExpired b)
h2.Time > now checks.
* return an error if we're at more recent height
* add comments
* add LastSignedHeaderHeight method to Store
I think it's fine if Store tracks last height
* copy over proxy from old lite package
* make TrustedHeader return latest if height=0
* modify LastSignedHeaderHeight to return an error if no headers exist
* copy over proxy impl
* refactor proxy and start http lite client
* Tx and BlockchainInfo methods
* Block method
* commit method
* code compiles again
* lite client compiles
* extract updateLiteClientIfNeededTo func
* move final parts
* add placeholder for tests
* force usage of lite http client in proxy
* comment out query tests for now
* explicitly mention tp: trusting period
* verify nextVals in VerifyHeader
* refactor bisection
* move the NextValidatorsHash check into updateTrustedHeaderAndVals
+ update the comment
* add ConsensusParams method to RPC client
* add ConsensusParams to rpc/mock/client
* change trustLevel type to a new cmn.Fraction type
+ update SkippingVerification comment
* stress out trustLevel is only used for non-adjusted headers
* fixes after Fede's review
Co-authored-by: Federico Kunze <31522760+fedekunze@users.noreply.github.com>
* compare newHeader with a header from an alternative provider
* save pivot header
Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349122824
* check header can still be trusted in TrustedHeader
Refs https://github.com/tendermint/tendermint/pull/3989#discussion_r349101424
* lite: update Validators and Block endpoints
- Block no longer contains BlockMeta
- Validators now accept two additional params: page and perPage
* make linter happy
* docs: remove specs, they live in spec repo (#4172)
* docs: remove specs, they live in spec repo
- moving specs to spec repo
- https://github.com/tendermint/spec/pull/62 PR for updating them
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add makefile command to copy in sepcs from specREPO
- move cloning of spec repo to pre and post scripts
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
Fixes#3986
This pull request is prefixing all the types in proto to avoid conflict.
When a go application is using Tendermint as a library and also define similar types in gogo proto some conflicts might occur (as types is a common package in go).
By prefixing the types with tendermint, this highly reduces the risk of conflicts.
BREAKING CHANGE.
This modification breaks the ABCI Application endpoint.
What was accessible before with `/types.ABCIApplication/Flush` is now accessible with `/tendermint.abci.types.ABCIApplication/Flush`.
- change link for bounties for different lang abci servers to interchainio funding repo
- link awesome repo ecosystem section in main docs readme
- closes#4110
- closes#4125
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
- tm-bench has a deprecation warning for 5 releases now, with the major release coming I removed the file and updated the docs to point to `tm-load-test` located in the interchainio repo
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
*libs/common/errors: remove package
- remove errors file from cmn pkg
- use errorf instead of wrap in async function
- add changelog entry
- closes#3862
https://www.jsonrpc.org/specification
What is done in this PR:
JSONRPCClient: validate that Response.ID matches Request.ID I wanted
to do the same for the WSClient, but since we're sending events as
responses, not notifications, checking IDs would require storing
them in memory indefinitely (and we won't be able to remove them
upon client unsubscribing because ID is different then).
Request.ID is now optional. Notification is a Request without an ID.
Previously "" or 0 were considered as notifications
Remove #event suffix from ID from an event response (partially fixes
#2949) ID must be either string, int or null AND must be equal to
request's ID. Now, because we've implemented events as responses, WS
clients are tripping when they see Response.ID("0#event") !=
Request.ID("0"). Implementing events as requests would require a lot
of time (~ 2 days to completely rewrite WS client and server)
generate unique ID for each request
switch to integer IDs instead of "json-client-XYZ"
id=0 method=/subscribe
id=0 result=...
id=1 method=/abci_query
id=1 result=...
> send events (resulting from /subscribe) as requests+notifications (not
responses)
this will require a lot of work. probably not worth it
* rpc: generate an unique ID for each request
in conformance with JSON-RPC spec
* WSClient: check for unsolicited responses
* fix golangci warnings
* save commit
* fix errors
* remove ID from responses from subscribe
Refs #2949
* clients are safe for concurrent access
* tm-bench: switch to int ID
* fixes after my own review
* comment out sentIDs in WSClient
see commit body for the reason
* remove body.Close
it will be closed automatically
* stop ws connection outside of write/read routines
also, use t.Rate in tm-bench indexer when calculating ID
fix gocritic issues
* update swagger.yaml
* Apply suggestions from code review
* fix stylecheck and golint linter warnings
* update changelog
* update changelog2
* Add pagination to /validators
- closes#3472
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add swagger params, default returns all
* address pr comments
* golint fix
* swagger default change, change to default in comment
* swagger.yaml: replace x-example with example
https://swagger.io/docs/specification/adding-examples/
* Revert "swagger.yaml: replace x-example with example"
This reverts commit 9df1b006de.
* update changelog and remove extra body close
## Issue:
Removed BlockMeta in ResultBlock in favor of BlockId for /block
Added block_size to BlockMeta this is reflected in /blockchain
fixes#3188
added breaking as some clients may be using header from blockmeta instead of block in /block
## Commits:
* cleanup block path
Remove duplication of data in `/block`
fixes#3188
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* Remove duplication of data in `/block`
- Created a secondary type to be used for /block
fixes#3188
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* remove commented out code
* fix ci
* add changelog_pending entry
* remove extra variable
* update swagger
* change int to uint for blocksize
* fix swagger
* remove extensive comments
* update changelog
* fix conflicts after merge
* use int for BlockSize and NumTxs in BlockMeta
- with 99.9% guarantee, the size of either will never reach int32
- most of the Go "Size" stdlib functions return int
* Removal of TotalTx & NumTx
- Removed totalTx and numTx
closes#2521
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* abci proto changes
* proto number fix
* txfilter_test fix
* comments on PR
* further changes
* bring back metrics
* fix indexer
* fix TestBlockMaxDataBytes and TestBlockMaxDataBytesUnknownEvidence
* indexer service back to header
* statistics.go fix
* fix ci
* listen for blocks, not headers
to be able to record txs throughput
* fix TestNetworkNewBlock
* fix tests
* fix tests in types package
* fixes after Anton's review
* fix tests
* bring back `consensus_total_txs` metric
I mistakenly thought it was removed.
* improve changelog
* remove LastBlockTotalTx from state
* docs: remove getNumTxs from BeginBlock Java example
## Issue:
This is an approach to fixing secret connection that is more noise-ish than actually noise.
but it essentially fixes the problem that #3315 is trying to solve by making the secret connection handshake non-malleable. It's easy to understand and I think will be acceptable to @jaekwon
.. the formal reasoning is basically, if the "view" of the transcript between diverges between the sender and the receiver at any point in the protocol, the handshake would terminate.
The base protocol of Station to Station mistakenly assumes that if the sender and receiver arrive at shared secret they have the same view. This is only true for a DH on prime order groups.
This robustly solves the problem by having each cryptographic operation commit to operators view of the protocol.
Another nice thing about a transcript is it provides the basis for "secure" (barring cryptographic breakages, horrible design flaws, or implementation bugs) downgrades, where a backwards compatible handshake can be used to offer newer protocol features/extensions, peers agree to the common subset of what they support, and both sides have to agree on what the other offered for the transcript MAC to verify.
With something like Protos/Amino you already get "extensions" for free (TLS uses a simple TLV format https://tools.ietf.org/html/rfc8446#section-4.2 for extensions not too far off from Protos/Amino), so as long as you cryptographically commit to what they contain in the transcript, it should be possible to extend the protocol in a backwards-compatible manner.
## Commits:
* Minimal changes to remove malleability of secret connection removes the need to check for lower order points.
Breaks compatibility. Secret connections that have no been updated will fail
* Remove the redundant blacklist
* remove remainders of blacklist in tests to make the code compile again
Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* Apply suggestions from code review
Apply Ismail's error handling
Co-Authored-By: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* fix error check for io.ReadFull
Signed-off-by: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* Update p2p/conn/secret_connection.go
Co-Authored-By: Ismail Khoffi <Ismail.Khoffi@gmail.com>
* Update p2p/conn/secret_connection.go
Co-Authored-By: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com>
* update changelog and format the code
* move hkdfInit closer to where it's used
BREAKING
Example response:
```json
{
"jsonrpc": "2.0",
"id": "",
"result": {
"height": "2109",
"txs_results": null,
"begin_block_events": null,
"end_block_events": null,
"validator_updates": null,
"consensus_param_updates": null
}
}
```
Old result consisted of ABCIResponses struct and height. Exposing
internal ABCI structures (which we store in state package) in RPC seems
bad to me for the following reasons:
1) high risk of breaking the API when somebody changes internal structs
(HAPPENED HERE!)
2) RPC is aware of ABCI, which I'm not sure we want
Fixes#4051
Function `parseRemoteAddr` is forcing protocol HTTP and protocol HTTPs to tcp. This causes the bug in the issue #4051.
I find that the tcp is only needed where `net.Dial`. So I moved the switch to makeHTTPDialer.
## Issue:
Hey, not sure if this is disallowed for any reason specifically, but it would be very beneficial to define additional types to decode tendermint key implementations from bytes, since it uses a static codec. If this is okay, let me know and I will add documentation.
Context: For Ethermint to switch to using Cosmos' keybase, decoding the keys requires this codec to be updated
Just to document, I did experiment with creating a mapping from string to objects to be able to keep track of the key types added to be able to be used in the RegisterAmino(..) call, but because of how go is compiled, cosmos would just use the base types. This may be a useful feature for someone just building on top of Tendermint and not going through Cosmos, but to not add confusion or unnecessary complexity, I left it out.
## Commits:
* Exposes amino codec to be able to decode pk bytes in application
* Change how codec is modified
* Remove unneeded comment
* Fix comment
* Fix comment
* Add registered type to nametable
* Add pending changelog entry
* Reorder change
* Added check if type is registered and added test
* Make test type private
* Remove unnecessary duplicate exists check
Sean Braithwaite
Black3HDF
Anca Zamfir
Zaki Manian
Anton Kaliaev
Marko
Leslie Cheung
Alex Kogon
Anthony
Greg Zaitsev
Gustavo Chaín
b-harvest
Aditya
Eric Mugo
Cyrus Goh
yk
Austin Abell
Denis Fadeev
Mircea Colonescu
dependabot-preview[bot]
Albert