The main change here is to use encoding/json to encode and decode RPC
parameters, rather than the custom tmjson package. This includes:
- Update the HTTP POST handler parameter handling.
- Add field tags to 64-bit integer types to get string encoding (to match amino/tmjson).
- Add marshalers to struct types that mention interfaces.
- Inject wrappers to decode interface arguments in RPC handlers.
After #7592, @cmwaters noticed that the logic for re-using old timestamps for proposals may not work with proposer-based timestamps. This change removes the logic to re-use old proposal timestamps since it is no longer correct. Two proposals with different timestamps can no longer be treated as equivalent. Signing a proposal that only differs by timestamp in the new algorithm can be thought of as roughly equivalent to signing a proposal that only differs by `BlockID` in the old scheme.
I also investigated the codebase and checked for any place we updated a timestamp using the pattern `(Timestamp = |Timestamp: )` and saw no additional places where we are updating the timestamp of a proposal message.
Here is the output of that search:
```
privval/file.go:372: vote.Timestamp = timestamp
privval/file.go:453: lastVote.Timestamp = now
privval/file.go:454: newVote.Timestamp = now
internal/test/factory/commit.go:25: Timestamp: now,
internal/test/factory/vote.go:34: Timestamp: time,
internal/consensus/state.go:2261: Timestamp: cs.voteTime(),
internal/consensus/state.go:2286: vote.Timestamp = v.Timestamp
light/detector.go:414: ev.Timestamp = common.Time
light/detector.go:418: ev.Timestamp = trusted.Time
types/block.go:616: Timestamp: ts,
types/block.go:725: Timestamp: cs.Timestamp,
types/block.go:736: cs.Timestamp = csp.Timestamp
types/block.go:800: Timestamp: commitSig.Timestamp,
types/evidence.go:84: Timestamp: blockTime,
types/evidence.go:190: dve.Timestamp = evidenceTime
types/evidence.go:202: Timestamp: dve.Timestamp,
types/evidence.go:228: Timestamp: pb.Timestamp,
types/evidence.go:382: Timestamp: %v}#%X`,
types/evidence.go:491: l.Timestamp = evidenceTime
types/evidence.go:517: Timestamp: l.Timestamp,
types/evidence.go:546: Timestamp: lpb.Timestamp,
types/evidence.go:722: Timestamp: time,
types/vote.go:80: Timestamp: vote.Timestamp,
types/vote.go:216: Timestamp: vote.Timestamp,
types/vote.go:240: vote.Timestamp = pv.Timestamp
types/test_util.go:27: Timestamp: now,
types/proposal.go:44: Timestamp: tmtime.Now(),
types/proposal.go:132: pb.Timestamp = p.Timestamp
types/proposal.go:157: p.Timestamp = pp.Timestamp
types/canonical.go:49: Timestamp: proposal.Timestamp,
types/canonical.go:62: Timestamp: vote.Timestamp,
test/e2e/runner/evidence.go:186: Timestamp: evTime,
```
Where possible, replace uses of the custom JSON library with the standard
library. The custom library treats interface and unnamed lteral types
differently, so this change avoids those even where it would probably be safe
to switch them.
## Description
- Add `context.Context` to Privval interface
This pr does not introduce context into our custom privval connection protocol because this will be removed in the next release. When this pr is released.
* test-vectors for backwards compatibility:
- copy & paste test-vectors from v0.33.5 to ensure
backwards compatibility for vote's SignBytes
* WIP: everything besides time seems to match :-/
* almost
* Found the culprit: field nums weren't consecutive ints ...
* fix order of partset header too
* this last votes-related test can easily be fixed
* some minor changes and fix last failing test
* move proto types back to stdtime, fix various linting
* use libs/protoio
* remvoe commented code
* add comments
* fix tests
* uncomment testscases
* dont ignore error panic
* fix signable test
* fix happy path testing
* fix comment
Co-authored-by: Marko Baricevic <marbar3778@yahoo.com>
* format: add format cmd & goimport repo
- replaced format command
- added goimports to format command
- ran goimports
Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>
* fix outliers & undo proto file changes
This PR is related to #3107 and a continuation of #3351
It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction.
Given two hosts A and B:
Host A is listener/client
Host B is dialer/server (contains the secret key)
When A requires a signature, it needs to wait for B to dial in before it can issue a request.
A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect.
The original rationale behind this design was based on security.
Host B only allows outbound connections to a list of whitelisted hosts.
It is not possible to reach B unless B dials in. There are no listening/open ports in B.
This PR results in the following changes:
Refactors ping/heartbeat to avoid previously existing race conditions.
Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow.
Unifies and abstracts away the differences between unix and tcp sockets.
A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj)
The signer request handler (server side) is customizable to increase testability.
Updates and extends unit tests
A high level overview of the classes is as follows:
Transport (endpoints): The following classes take care of establishing a connection
SignerDialerEndpoint
SignerListeningEndpoint
SignerEndpoint groups common functionality (read/write/timeouts/etc.)
Signing (client/server): The following classes take care of exchanging request/responses
SignerClient
SignerServer
This PR also closes#3601
Commits:
* refactoring - work in progress
* reworking unit tests
* Encapsulating and fixing unit tests
* Improve tests
* Clean up
* Fix/improve unit tests
* clean up tests
* Improving service endpoint
* fixing unit test
* fix linter issues
* avoid invalid cache values (improve later?)
* complete implementation
* wip
* improved connection loop
* Improve reconnections + fixing unit tests
* addressing comments
* small formatting changes
* clean up
* Update node/node.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client_test.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* check during initialization
* dropping connecting when writing fails
* removing break
* use t.log instead
* unifying and using cmn.GetFreePort()
* review fixes
* reordering and unifying drop connection
* closing instead of signalling
* refactored service loop
* removed superfluous brackets
* GetPubKey can return errors
* Revert "GetPubKey can return errors"
This reverts commit 68c06f19b4.
* adding entry to changelog
* Update CHANGELOG_PENDING.md
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_listener_endpoint_test.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* updating node.go
* review fixes
* fixes linter
* fixing unit test
* small fixes in comments
* addressing review comments
* addressing review comments 2
* reverting suggestion
* Update privval/signer_client_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update privval/signer_client_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update privval/signer_listener_endpoint_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* do not expose brokenSignerDialerEndpoint
* clean up logging
* unifying methods
shorten test time
signer also drops
* reenabling pings
* improving testability + unit test
* fixing go fmt + unit test
* remove unused code
* Addressing review comments
* simplifying connection workflow
* fix linter/go import issue
* using base service quit
* updating comment
* Simplifying design + adjusting names
* fixing linter issues
* refactoring test harness + fixes
* Addressing review comments
* cleaning up
* adding additional error check
* Close and recreate a RemoteSigner on err
* Update changelog
* Address Anton's comments / suggestions:
- update changelog
- restart TCPVal
- shut down on `ErrUnexpectedResponse`
* re-init remote signer client with fresh connection if Ping fails
- add/update TODOs in secret connection
- rename tcp.go -> tcp_client.go, same with ipc to clarify their purpose
* account for `conn returned by waitConnection can be `nil`
- also add TODO about RemoteSigner conn field
* Tests for retrying: IPC / TCP
- shorter info log on success
- set conn and use it in tests to close conn
* Tests for retrying: IPC / TCP
- shorter info log on success
- set conn and use it in tests to close conn
- add rwmutex for conn field in IPC
* comments and doc.go
* fix ipc tests. fixes#2677
* use constants for tests
* cleanup some error statements
* fixes#2784, race in tests
* remove print statement
* minor fixes from review
* update comment on sts spec
* cosmetics
* p2p/conn: add failing tests
* p2p/conn: make SecretConnection thread safe
* changelog
* IPCVal signer refactor
- use a .reset() method
- don't use embedded RemoteSignerClient
- guard RemoteSignerClient with mutex
- drop the .conn
- expose Close() on RemoteSignerClient
* apply IPCVal refactor to TCPVal
* remove mtx from RemoteSignerClient
* consolidate IPCVal and TCPVal, fixes#3104
- done in tcp_client.go
- now called SocketVal
- takes a listener in the constructor
- make tcpListener and unixListener contain all the differences
* delete ipc files
* introduce unix and tcp dialer for RemoteSigner
* rename files
- drop tcp_ prefix
- rename priv_validator.go to file.go
* bring back listener options
* fix node
* fix priv_val_server
* fix node test
* minor cleanup and comments
* split immutable and mutable parts of priv_validator.json
* fix bugs
* minor changes
* retrig test
* delete scripts/wire2amino.go
* fix test
* fixes from review
* privval: remove mtx
* rearrange priv_validator.go
* upgrade path
* write tests for the upgrade
* fix for unsafe_reset_all
* add test
* add reset test
* WIP: switching to fixed offsets for SignBytes
* add version field to sign bytes and update order
* more comments on test-cases and add a tc with a chainID
* remove amino:"write_empty" tag
- it doesn't affect if default fixed size fields ((u)int64) are
written or not
- add comment about int->int64 casting
* update CHANGELOG_PENDING
* update documentation
* add back link to issue #1622 in documentation
* remove JSON tags and add (failing test-case)
* fix failing test
* update test-vectors due to added `Type` field
* change Type field from string to byte and add new type alias
- SignedMsgType replaces VoteTypePrevote, VoteTypePrecommit and adds new
ProposalType to separate votes from proposal when signed
- update test-vectors
* fix remains from rebasing
* use SignMessageType instead of byte everywhere
* fixes from review
Currently the top level directory contains basically all of the code
for the crypto package. This PR moves the crypto code into submodules
in a similar manner to what `golang/x/crypto` does. This improves code
organization.
Ref discussion: https://github.com/tendermint/tendermint/pull/1966Closes#1956
except now we calculate the max size using the maxPacketMsgSize()
function, which frees developers from having to know amino encoding
details.
plus, 10 additional bytes are added to leave the room for amino upgrades
(both making it more efficient / less efficient)
This is a maintenance change to move the private validator package out
of the types and to a top-level location. There is no good reason to
keep it under the types and it will more clearly coommunicate where
additions related to the privval belong. It leaves the interface and the
mock in types for now as it would introduce circular dependency between
privval and types, this should be resolved eventually.
* mv priv_validator to privval pkg
* use consistent `privval` as import
Follow-up to #1255
As calls to the private validator can involve side-effects like network
communication it is desirable for all methods returning an error to not
break the control flow of the caller.
* adjust PrivValidator interface
M. J. Fromberger
William Banfield
Sam Kleinman
Sharad Chand
Marko
Ismail Khoffi
Erik Grinaker
Juan Leni
yutianwu
Zarko Milosevic
ValarDragon
Anton Kaliaev
Zach Ramsay
Ethan Buchman
Jae Kwon
Liamsi
Alexander Simmerl