This PR is related to #3107 and a continuation of #3351
It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction.
Given two hosts A and B:
Host A is listener/client
Host B is dialer/server (contains the secret key)
When A requires a signature, it needs to wait for B to dial in before it can issue a request.
A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect.
The original rationale behind this design was based on security.
Host B only allows outbound connections to a list of whitelisted hosts.
It is not possible to reach B unless B dials in. There are no listening/open ports in B.
This PR results in the following changes:
Refactors ping/heartbeat to avoid previously existing race conditions.
Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow.
Unifies and abstracts away the differences between unix and tcp sockets.
A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj)
The signer request handler (server side) is customizable to increase testability.
Updates and extends unit tests
A high level overview of the classes is as follows:
Transport (endpoints): The following classes take care of establishing a connection
SignerDialerEndpoint
SignerListeningEndpoint
SignerEndpoint groups common functionality (read/write/timeouts/etc.)
Signing (client/server): The following classes take care of exchanging request/responses
SignerClient
SignerServer
This PR also closes#3601
Commits:
* refactoring - work in progress
* reworking unit tests
* Encapsulating and fixing unit tests
* Improve tests
* Clean up
* Fix/improve unit tests
* clean up tests
* Improving service endpoint
* fixing unit test
* fix linter issues
* avoid invalid cache values (improve later?)
* complete implementation
* wip
* improved connection loop
* Improve reconnections + fixing unit tests
* addressing comments
* small formatting changes
* clean up
* Update node/node.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client_test.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* check during initialization
* dropping connecting when writing fails
* removing break
* use t.log instead
* unifying and using cmn.GetFreePort()
* review fixes
* reordering and unifying drop connection
* closing instead of signalling
* refactored service loop
* removed superfluous brackets
* GetPubKey can return errors
* Revert "GetPubKey can return errors"
This reverts commit 68c06f19b4.
* adding entry to changelog
* Update CHANGELOG_PENDING.md
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_client.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_dialer_endpoint.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* Update privval/signer_listener_endpoint_test.go
Co-Authored-By: jleni <juan.leni@zondax.ch>
* updating node.go
* review fixes
* fixes linter
* fixing unit test
* small fixes in comments
* addressing review comments
* addressing review comments 2
* reverting suggestion
* Update privval/signer_client_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update privval/signer_client_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* Update privval/signer_listener_endpoint_test.go
Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>
* do not expose brokenSignerDialerEndpoint
* clean up logging
* unifying methods
shorten test time
signer also drops
* reenabling pings
* improving testability + unit test
* fixing go fmt + unit test
* remove unused code
* Addressing review comments
* simplifying connection workflow
* fix linter/go import issue
* using base service quit
* updating comment
* Simplifying design + adjusting names
* fixing linter issues
* refactoring test harness + fixes
* Addressing review comments
* cleaning up
* adding additional error check
* ungitignore
* add docs/.vuepress to enable local builds
* config.js needs to be here, one less step
* docs: make spec in sidebar nicer
* docs: local build instructions
* Init `\health` rpc endpoint
* remove additional info from `\health` rpc endpoint
* Cleanup imports
* Added time threshold for health check
* Update rpc doc
* Remove unnecessary checks for blocktime creation lag
* Clean up of unnecessary config usage
* Add CacheDB & SimpleMap
* Generic memBatch; Fix cLevelDB tests
* CacheWrap() for CacheDB and MemDB
* Change Iterator to match LeviGo Iterator
* Fixes from review
* cacheWrapWriteMutex and some race fixes
* Use tmlibs/common
* NewCWWMutex is exposed. DB can be CacheWrap'd
* Remove GetOK, not needed
* Fsdb (#72)
* Add FSDB
* Review fixes from Anton
* Review changes
* Fixes from review
query parser
use parser compiler to generate query parser
I used https://github.com/pointlander/peg which has a nice API and seems
to be the most popular Golang compiler parser using PEG on Github.
More about PEG:
- https://en.wikipedia.org/wiki/Parsing_expression_grammar
- https://github.com/PhilippeSigaud/Pegged/wiki/PEG-Basics
- https://github.com/PhilippeSigaud/Pegged/wiki/Grammar-Examples
rename
implement query match function
match function
uncomment test lines
add more test cases for query#Matches
fix int case
rename events to pubsub
add comment about cache
assertReceive helper to not block on receive in tests
fix bug with multiple conditions
uncomment benchmark
first results:
```
Benchmark10Clients-2 1000 1305493 ns/op 3957519 B/op 355 allocs/op
Benchmark100Clients-2 100 12278304 ns/op 39571751 B/op 3505 allocs/op
Benchmark1000Clients-2 10 124120909 ns/op 395714004 B/op 35005 allocs/op
```
124ms to publish message to 1000 clients. A lot.
use AST from query.peg.go
separate pubsub and query packages by using Query interface in pubsub
wrote docs and refactor code
updates from Frey's review
refactor type assertion to use type switch
cleanup during shutdown
subscriber should create output channel, not the server
overflow strategies, server buffer capacity
context as the first argument for Publish
log error
introduce Option type
update NewServer comment
move helpers into pubsub_test
increase assertReceive timeout
add query.MustParse
add more false tests for parser
add more false tests for query.Matches
parse numbers as int64 / float64
try our best to convert from other types
add number to panic output
add more comments
save commit
introduce client argument as first argument to Subscribe
> Why we do not specify buffer size on the output channel in Subscribe?
The choice of buffer size of N here depends on knowing the number of
messages server will receive and the number of messages downstream
subscribers will consume. This is fragile: if we publish an additional
message, or if one of the downstream subscribers reads any fewer
messages, we will again have blocked goroutines.
save commit
remove reference counting
fix test
test client resubscribe
test UnsubscribeAll
client options
[pubsub/query] fuzzy testing
do not print msg as it creates data race!