The main change here is to use encoding/json to encode and decode RPC
parameters, rather than the custom tmjson package. This includes:
- Update the HTTP POST handler parameter handling.
- Add field tags to 64-bit integer types to get string encoding (to match amino/tmjson).
- Add marshalers to struct types that mention interfaces.
- Inject wrappers to decode interface arguments in RPC handlers.
* avoid re-signing vote when RHS and signbytes are equal
* avoid re-signing proposal when RHS and signbytes are equal
Co-authored-by: Callum Waters <cmwaters19@gmail.com>
Co-authored-by: William Banfield <4561443+williambanfield@users.noreply.github.com>
After #7592, @cmwaters noticed that the logic for re-using old timestamps for proposals may not work with proposer-based timestamps. This change removes the logic to re-use old proposal timestamps since it is no longer correct. Two proposals with different timestamps can no longer be treated as equivalent. Signing a proposal that only differs by timestamp in the new algorithm can be thought of as roughly equivalent to signing a proposal that only differs by `BlockID` in the old scheme.
I also investigated the codebase and checked for any place we updated a timestamp using the pattern `(Timestamp = |Timestamp: )` and saw no additional places where we are updating the timestamp of a proposal message.
Here is the output of that search:
```
privval/file.go:372: vote.Timestamp = timestamp
privval/file.go:453: lastVote.Timestamp = now
privval/file.go:454: newVote.Timestamp = now
internal/test/factory/commit.go:25: Timestamp: now,
internal/test/factory/vote.go:34: Timestamp: time,
internal/consensus/state.go:2261: Timestamp: cs.voteTime(),
internal/consensus/state.go:2286: vote.Timestamp = v.Timestamp
light/detector.go:414: ev.Timestamp = common.Time
light/detector.go:418: ev.Timestamp = trusted.Time
types/block.go:616: Timestamp: ts,
types/block.go:725: Timestamp: cs.Timestamp,
types/block.go:736: cs.Timestamp = csp.Timestamp
types/block.go:800: Timestamp: commitSig.Timestamp,
types/evidence.go:84: Timestamp: blockTime,
types/evidence.go:190: dve.Timestamp = evidenceTime
types/evidence.go:202: Timestamp: dve.Timestamp,
types/evidence.go:228: Timestamp: pb.Timestamp,
types/evidence.go:382: Timestamp: %v}#%X`,
types/evidence.go:491: l.Timestamp = evidenceTime
types/evidence.go:517: Timestamp: l.Timestamp,
types/evidence.go:546: Timestamp: lpb.Timestamp,
types/evidence.go:722: Timestamp: time,
types/vote.go:80: Timestamp: vote.Timestamp,
types/vote.go:216: Timestamp: vote.Timestamp,
types/vote.go:240: vote.Timestamp = pv.Timestamp
types/test_util.go:27: Timestamp: now,
types/proposal.go:44: Timestamp: tmtime.Now(),
types/proposal.go:132: pb.Timestamp = p.Timestamp
types/proposal.go:157: p.Timestamp = pp.Timestamp
types/canonical.go:49: Timestamp: proposal.Timestamp,
types/canonical.go:62: Timestamp: vote.Timestamp,
test/e2e/runner/evidence.go:186: Timestamp: evTime,
```
Add package jsontypes that implements a subset of the custom libs/json
package. Specifically it handles encoding and decoding of interface types
wrapped in "tagged" JSON objects. It omits the deep reflection on arbitrary
types, preserving only the handling of type tags wrapper encoding.
- Register interface types (Evidence, PubKey, PrivKey) for tagged encoding.
- Update the existing implementations to satisfy the type.
- Register those types with the jsontypes registry.
- Add string tags to 64-bit integer fields where needed.
- Add marshalers to structs that export interface-typed fields.
Where possible, replace uses of the custom JSON library with the standard
library. The custom library treats interface and unnamed lteral types
differently, so this change avoids those even where it would probably be safe
to switch them.
## Description
Internalize some libs. This reduces the amount ot public API tendermint is supporting. The moved libraries are mainly ones that are used within Tendermint-core.
## Description
- Add `context.Context` to Privval interface
This pr does not introduce context into our custom privval connection protocol because this will be removed in the next release. When this pr is released.
* test-vectors for backwards compatibility:
- copy & paste test-vectors from v0.33.5 to ensure
backwards compatibility for vote's SignBytes
* WIP: everything besides time seems to match :-/
* almost
* Found the culprit: field nums weren't consecutive ints ...
* fix order of partset header too
* this last votes-related test can easily be fixed
* some minor changes and fix last failing test
* move proto types back to stdtime, fix various linting
* use libs/protoio
* remvoe commented code
* add comments
* fix tests
* uncomment testscases
* dont ignore error panic
* fix signable test
* fix happy path testing
* fix comment
Co-authored-by: Marko Baricevic <marbar3778@yahoo.com>
* libs/common: Refactor libs/common 5
- move mathematical functions and types out of `libs/common` to math pkg
- move net functions out of `libs/common` to net pkg
- move string functions out of `libs/common` to strings pkg
- move async functions out of `libs/common` to async pkg
- move bit functions out of `libs/common` to bits pkg
- move cmap functions out of `libs/common` to cmap pkg
- move os functions out of `libs/common` to os pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix testing issues
* fix tests
closes#41417
woooooooooooooooooo kill the cmn pkg
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* fix goimport issues
* run gofmt
* libs/common: Refactor libs/common 4
- move byte function out of cmn to its own pkg
- move tempfile out of cmn to its own pkg
- move throttletimer to its own pkg
ref #4147
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* add changelog entry
* fix linting issues
* Fix long line errors in abci, crypto, and libs packages
* Fix long lines in p2p and rpc packages
* Fix long lines in abci, state, and tools packages
* Fix long lines in behaviour and blockchain packages
* Fix long lines in cmd and config packages
* Begin fixing long lines in consensus package
* Finish fixing long lines in consensus package
* Add lll exclusion for lines containing URLs
* Fix long lines in crypto package
* Fix long lines in evidence package
* Fix long lines in mempool and node packages
* Fix long lines in libs package
* Fix long lines in lite package
* Fix new long line in node package
* Fix long lines in p2p package
* Ignore gocritic warning
* Fix long lines in privval package
* Fix long lines in rpc package
* Fix long lines in scripts package
* Fix long lines in state package
* Fix long lines in tools package
* Fix long lines in types package
* Enable lll linter
This issue is related to #3107
This is a first renaming/refactoring step before reworking and removing heartbeats.
As discussed with @Liamsi , we preferred to go for a couple of independent and separate PRs to simplify review work.
The changes:
Help to clarify the relation between the validator and remote signer endpoints
Differentiate between timeouts and deadlines
Prepare to encapsulate networking related code behind RemoteSigner in the next PR
My intention is to separate and encapsulate the "network related" code from the actual signer.
SignerRemote ---(uses/contains)--> SignerValidatorEndpoint <--(connects to)--> SignerServiceEndpoint ---> SignerService (future.. not here yet but would like to decouple too)
All reconnection/heartbeat/whatever code goes in the endpoints. Signer[Remote/Service] do not need to know about that.
I agree Endpoint may not be the perfect name. I tried to find something "Go-ish" enough. It is a common name in go-kit, kubernetes, etc.
Right now:
SignerValidatorEndpoint:
handles the listener
contains SignerRemote
Implements the PrivValidator interface
connects and sets a connection object in a contained SignerRemote
delegates PrivValidator some calls to SignerRemote which in turn uses the conn object that was set externally
SignerRemote:
Implements the PrivValidator interface
read/writes from a connection object directly
handles heartbeats
SignerServiceEndpoint:
Does most things in a single place
delegates to a PrivValidator IIRC.
* cleanup
* Refactoring step 1
* Refactoring step 2
* move messages to another file
* mark for future work / next steps
* mark deprecated classes in docs
* Fix linter problems
* additional linter fixes
* Close and recreate a RemoteSigner on err
* Update changelog
* Address Anton's comments / suggestions:
- update changelog
- restart TCPVal
- shut down on `ErrUnexpectedResponse`
* re-init remote signer client with fresh connection if Ping fails
- add/update TODOs in secret connection
- rename tcp.go -> tcp_client.go, same with ipc to clarify their purpose
* account for `conn returned by waitConnection can be `nil`
- also add TODO about RemoteSigner conn field
* Tests for retrying: IPC / TCP
- shorter info log on success
- set conn and use it in tests to close conn
* Tests for retrying: IPC / TCP
- shorter info log on success
- set conn and use it in tests to close conn
- add rwmutex for conn field in IPC
* comments and doc.go
* fix ipc tests. fixes#2677
* use constants for tests
* cleanup some error statements
* fixes#2784, race in tests
* remove print statement
* minor fixes from review
* update comment on sts spec
* cosmetics
* p2p/conn: add failing tests
* p2p/conn: make SecretConnection thread safe
* changelog
* IPCVal signer refactor
- use a .reset() method
- don't use embedded RemoteSignerClient
- guard RemoteSignerClient with mutex
- drop the .conn
- expose Close() on RemoteSignerClient
* apply IPCVal refactor to TCPVal
* remove mtx from RemoteSignerClient
* consolidate IPCVal and TCPVal, fixes#3104
- done in tcp_client.go
- now called SocketVal
- takes a listener in the constructor
- make tcpListener and unixListener contain all the differences
* delete ipc files
* introduce unix and tcp dialer for RemoteSigner
* rename files
- drop tcp_ prefix
- rename priv_validator.go to file.go
* bring back listener options
* fix node
* fix priv_val_server
* fix node test
* minor cleanup and comments
* split immutable and mutable parts of priv_validator.json
* fix bugs
* minor changes
* retrig test
* delete scripts/wire2amino.go
* fix test
* fixes from review
* privval: remove mtx
* rearrange priv_validator.go
* upgrade path
* write tests for the upgrade
* fix for unsafe_reset_all
* add test
* add reset test
* WIP: switching to fixed offsets for SignBytes
* add version field to sign bytes and update order
* more comments on test-cases and add a tc with a chainID
* remove amino:"write_empty" tag
- it doesn't affect if default fixed size fields ((u)int64) are
written or not
- add comment about int->int64 casting
* update CHANGELOG_PENDING
* update documentation
* add back link to issue #1622 in documentation
* remove JSON tags and add (failing test-case)
* fix failing test
* update test-vectors due to added `Type` field
* change Type field from string to byte and add new type alias
- SignedMsgType replaces VoteTypePrevote, VoteTypePrecommit and adds new
ProposalType to separate votes from proposal when signed
- update test-vectors
* fix remains from rebasing
* use SignMessageType instead of byte everywhere
* fixes from review
* switch to amino for SignBytes and add Reply with error message
- currently only Vote is done
* switch Reply type in socket for other messages
- add error description on error
* add TODOs regarding error handling
* address comments from peer review session (thx @xla)
- contains all changes besides the test-coverage / error'ing branches
* increase test coverage:
- add tests for each newly introduced error'ing code path
* return error if received wrong response
* add test for wrong response branches (ErrUnexpectedResponse)
* update CHANGELOG_PENDING and related documentation (spec)
* fix typo: s/CanonicallockID/CanonicalBlockID
* fixes from review
Currently the top level directory contains basically all of the code
for the crypto package. This PR moves the crypto code into submodules
in a similar manner to what `golang/x/crypto` does. This improves code
organization.
Ref discussion: https://github.com/tendermint/tendermint/pull/1966Closes#1956
except now we calculate the max size using the maxPacketMsgSize()
function, which frees developers from having to know amino encoding
details.
plus, 10 additional bytes are added to leave the room for amino upgrades
(both making it more efficient / less efficient)
This is a maintenance change to move the private validator package out
of the types and to a top-level location. There is no good reason to
keep it under the types and it will more clearly coommunicate where
additions related to the privval belong. It leaves the interface and the
mock in types for now as it would introduce circular dependency between
privval and types, this should be resolved eventually.
* mv priv_validator to privval pkg
* use consistent `privval` as import
Follow-up to #1255
M. J. Fromberger
kmax.eth
William Banfield
Sam Kleinman
Sharad Chand
Marko
Erik Grinaker
Ismail Khoffi
Anton Kaliaev
Phil Salant
Juan Leni
Ethan Buchman
yutianwu
srmo
Zarko Milosevic
Dev Ojha
ValarDragon
Zach Ramsay
Liamsi
Alexander Simmerl