You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

151 lines
6.9 KiB

  1. # End-to-End Tests
  2. Spins up and tests Tendermint networks in Docker Compose based on a testnet manifest. To run the CI testnet:
  3. ```sh
  4. make
  5. ./build/runner -f networks/ci.toml
  6. ```
  7. This creates and runs a testnet named `ci` under `networks/ci/`.
  8. ## Conceptual Overview
  9. End-to-end testnets are used to test Tendermint functionality as a user would use it, by spinning up a set of nodes with various configurations and making sure the nodes and network behave correctly. The background for the E2E test suite is outlined in [RFC-001](https://github.com/tendermint/tendermint/blob/master/docs/rfc/rfc-001-end-to-end-testing.md).
  10. The end-to-end tests can be thought of in this manner:
  11. 1. Does a certain (valid!) testnet configuration result in a block-producing network where all nodes eventually reach the latest height?
  12. 2. If so, does each node in that network satisfy all invariants specified by the Go E2E tests?
  13. The above should hold for any arbitrary, valid network configuration, and that configuration space should be searched and tested by randomly generating testnets.
  14. A testnet configuration is specified as a TOML testnet manifest (see below). The testnet runner uses the manifest to configure a set of Docker containers and start them in some order. The manifests can be written manually (to test specific configurations) or generated randomly by the testnet generator (to test a wide range of configuration permutations).
  15. When running a testnet, the runner will first start the Docker nodes in some sequence, submit random transactions, and wait for the nodes to come online and the first blocks to be produced. This may involve e.g. waiting for nodes to block sync and/or state sync. If specified, it will then run any misbehaviors (e.g. double-signing) and perturbations (e.g. killing or disconnecting nodes). It then waits for the testnet to stabilize, with all nodes online and having reached the latest height.
  16. Once the testnet stabilizes, a set of Go end-to-end tests are run against the live testnet to verify network invariants (for example that blocks are identical across nodes). These use the RPC client to interact with the network, and should consider the entire network as a black box (i.e. it should not test any network or node internals, only externally visible behavior via RPC). The tests may use the `testNode()` helper to run parallel tests against each individual testnet node, and/or inspect the full blockchain history via `fetchBlockChain()`.
  17. The tests must take into account the network and/or node configuration, and tolerate that the network is still live and producing blocks. For example, validator tests should only run against nodes that are actually validators, and take into account the node's block retention and/or state sync configuration to not query blocks that don't exist.
  18. ## Testnet Manifests
  19. Testnets are specified as TOML manifests. For an example see [`networks/ci.toml`](networks/ci.toml), and for documentation see [`pkg/manifest.go`](pkg/manifest.go).
  20. ## Random Testnet Generation
  21. Random (but deterministic) combinations of testnets can be generated with `generator`:
  22. ```sh
  23. ./build/generator -d networks/generated/
  24. # Split networks into 8 groups (by filename)
  25. ./build/generator -g 8 -d networks/generated/
  26. ```
  27. Multiple testnets can be run with the `run-multiple.sh` script:
  28. ```sh
  29. ./run-multiple.sh networks/generated/gen-group3-*.toml
  30. ```
  31. ## Test Stages
  32. The test runner has the following stages, which can also be executed explicitly by running `./build/runner -f <manifest> <stage>`:
  33. * `setup`: generates configuration files.
  34. * `start`: starts Docker containers.
  35. * `load`: generates a transaction load against the testnet nodes.
  36. * `perturb`: runs any requested perturbations (e.g. node restarts or network disconnects).
  37. * `wait`: waits for a few blocks to be produced, and for all nodes to catch up to it.
  38. * `test`: runs test cases in `tests/` against all nodes in a running testnet.
  39. * `stop`: stops Docker containers.
  40. * `cleanup`: removes configuration files and Docker containers/networks.
  41. Auxiliary commands:
  42. * `logs`: outputs all node logs.
  43. * `tail`: tails (follows) node logs until canceled.
  44. ## Tests
  45. Test cases are written as normal Go tests in `tests/`. They use a `testNode()` helper which executes each test as a parallel subtest for each node in the network.
  46. ### Running Manual Tests
  47. To run tests manually, set the `E2E_MANIFEST` environment variable to the path of the testnet manifest (e.g. `networks/ci.toml`) and run them as normal, e.g.:
  48. ```sh
  49. ./build/runner -f networks/ci.toml start
  50. E2E_MANIFEST=networks/ci.toml go test -v ./tests/...
  51. ```
  52. Optionally, `E2E_NODE` specifies the name of a single testnet node to test.
  53. These environment variables can also be specified in `tests/e2e_test.go` to run tests from an editor or IDE:
  54. ```go
  55. func init() {
  56. // This can be used to manually specify a testnet manifest and/or node to
  57. // run tests against. The testnet must have been started by the runner first.
  58. os.Setenv("E2E_MANIFEST", "networks/ci.toml")
  59. os.Setenv("E2E_NODE", "validator01")
  60. }
  61. ```
  62. ### Debugging Failures
  63. If a command or test fails, the runner simply exits with an error message and
  64. non-zero status code. The testnet is left running with data in the testnet
  65. directory, and can be inspected with e.g. `docker ps`, `docker logs`, or
  66. `./build/runner -f <manifest> logs` or `tail`. To shut down and remove the
  67. testnet, run `./build/runner -f <manifest> cleanup`.
  68. If the standard `log_level` is not detailed enough (e.g. you want "debug" level
  69. logging for certain modules), you can change it in the manifest file.
  70. Each node exposes a [pprof](https://golang.org/pkg/runtime/pprof/) server. To
  71. find out the local port, run `docker port <NODENAME> 6060 | awk -F: '{print
  72. $2}'`. Then you may perform any queries supported by the pprof tool. Julia
  73. Evans has a [great
  74. post](https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/) on this
  75. subject.
  76. ```bash
  77. export PORT=$(docker port full01 6060 | awk -F: '{print $2}')
  78. go tool pprof http://localhost:$PORT/debug/pprof/goroutine
  79. go tool pprof http://localhost:$PORT/debug/pprof/heap
  80. go tool pprof http://localhost:$PORT/debug/pprof/threadcreate
  81. go tool pprof http://localhost:$PORT/debug/pprof/block
  82. go tool pprof http://localhost:$PORT/debug/pprof/mutex
  83. ```
  84. ## Enabling IPv6
  85. Docker does not enable IPv6 by default. To do so, enter the following in
  86. `daemon.json` (or in the Docker for Mac UI under Preferences → Docker Engine):
  87. ```json
  88. {
  89. "ipv6": true,
  90. "fixed-cidr-v6": "2001:db8:1::/64"
  91. }
  92. ```
  93. ## Benchmarking testnets
  94. It is also possible to run a simple benchmark on a testnet. This is done through the `benchmark` command. This manages the entire process: setting up the environment, starting the test net, waiting for a considerable amount of blocks to be used (currently 100), and then returning the following metrics from the sample of the blockchain:
  95. - Average time to produce a block
  96. - Standard deviation of producing a block
  97. - Minimum and maximum time to produce a block