zolfa
/
tendermint


								========================================

								RFC 004: E2E Test Framework Enhancements

								========================================


								Changelog

								---------


								- 2021-09-14: started initial draft (@tychoish)


								Abstract

								--------


								This document discusses a series of improvements to the e2e test framework

								that we can consider during the next few releases to help boost confidence in

								Tendermint releases, and improve developer efficiency.


								Background

								----------


								During the 0.35 release cycle, the E2E tests were a source of great

								value, helping to identify a number of bugs before release. At the same time,

								the tests were not consistently passing during this time, thereby reducing

								their value, and forcing the core development team to allocate time and energy

								to maintaining and chasing down issues with the e2e tests and the test

								harness. The experience of this release cycle calls to mind a series of

								improvements to the test framework, and this document attempts to capture

								these improvements, along with motivations, and potential for impact.


								Projects

								--------


								Flexible Workload Generation

								~~~~~~~~~~~~~~~~~~~~~~~~~~~~


								Presently the e2e suite contains a single workload generation pattern, which

								exists simply to ensure that the test networks have some work during their

								runs. However, the shape and volume of the work is very consistent and is very

								gentle to help ensure test reliability.


								We don't need a complex workload generation framework, but being able to have

								a few different workload shapes available for test networks, both generated and

								hand-crafted, would be useful.


								Workload patterns/configurations might include:


								- transaction targeting patterns (include light nodes, round robin, target

								  individual nodes)


								- variable transaction size over time.


								- transaction broadcast option (synchronously, checked, fire-and-forget,

								  mixed).


								- number of transactions to submit.


								- non-transaction workloads: (evidence submission, query, event subscription.)


								Configurable Generator

								~~~~~~~~~~~~~~~~~~~~~~


								The nightly e2e suite is defined by the `testnet generator

								<https://github.com/tendermint/tendermint/blob/master/test/e2e/generator/generate.go#L13-L65>`_,

								and it's difficult to add dimensions or change the focus of the test suite in

								any way without modifying the implementation of the generator. If the

								generator were more configurable, potentially via a file rather than in

								the Go implementation, we could modify the focus of the test suite on the

								fly.


								Features that we might want to configure:


								- number of test networks to generate of various topologies, to improve

								  coverage of different configurations.


								- test application configurations (to modify the latency of ABCI calls, etc.)


								- size of test networks.


								- workload shape and behavior.


								- initial sync and catch-up configurations.


								The workload generator currently provides runtime options for limiting the

								generator to specific types of P2P stacks, and for generating multiple groups

								of test cases to support parallelism. The goal is to extend this pattern and

								avoid hardcoding the matrix of test cases in the generator code.  Once the

								testnet configuration generation behavior is configurable at runtime,

								developers may be able to use the e2e framework to validate changes before

								landing changes that break e2e tests a day later.


								In addition to the autogenerated suite, it might make sense to maintain a

								small collection of hand-crafted cases that exercise configurations of

								concern, to run as part of the nightly (or less frequent) loop.


								Implementation Plan Structure

								~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


								As a development team, we should determine the features should impact the e2e

								testing early in the development cycle, and if we intend to modify the e2e

								tests to exercise a feature, we should identify this early and begin the

								integration process as early as possible.


								To facilitate this, we should adopt a practice whereby we exercise specific

								features that are currently under development more rigorously in the e2e

								suite, and then as development stabilizes we can reduce the number or weight

								of these features in the suite.


								As of 0.35 there are essentially two end to end tests: the suite of 64

								generated test networks, and the hand crafted `ci.toml` test case. The

								generated test cases help provide systemtic coverage, while the `ci` run

								provides coverage for a large number of features.


								Reduce Cycle Time

								~~~~~~~~~~~~~~~~~


								One of the barriers to leveraging the e2e framework, and one of the challenges

								in debugging failures, is the cycle time of running a single test iteration is

								quite high: 5 minutes to build the docker image, plus the time to run the test

								or tests.


								There are a number of improvements and enhancements that can reduce the cycle

								time in practice:


								- reduce the amount of time required to build the docker image used in these

								  tests. Without the dependency on CGo, the tendermint binaries could be

								  (cross) compiled outside of the docker container and then injected into

								  them, which would take better advantage of docker's native caching,

								  although, without the dependency on CGo there would be no hard requirement

								  for the e2e tests to use docker.


								- support test parallelism. Because of the way the testnets are orchestrated

								  a single system can really only run one network at a time. For executions

								  (local or remote) with more resources, there's no reason to run a few

								  networks in parallel to reduce the feedback time.


								- prune testnet configurations that are unlikely to provide good signal, to

								  shorten the time to feedback.


								- apply some kind of tiered approach to test execution, to improve the

								  legibility of the test result. For example order tests by the dependency of

								  their features, or run test networks without perturbations before running

								  that configuration with perturbations, to be able to isolate the impact of

								  specific features.


								- orchestrate the test harness directly from go test rather than via a special

								  harness and shell scripts so e2e tests may more naively fit into developers

								  existing workflows.


								Many of these improvements, particularly, reducing the build time will also

								reduce the time to get feedback during automated builds.


								Deeper Insights

								~~~~~~~~~~~~~~~


								When a test network fails, it's incredibly difficult to understand _why_ the

								network failed, as the current system provides very little insight into the

								system outside of the process logs. When a test network stalls or fails

								developers should be able to quickly and easily get a sense of the state of

								the network and all nodes.


								Improvements in persuit of this goal, include functionality that would help

								node operators in production environments by improving the quality and utility

								of the logging messages and other reported metrics, but also provide some

								tools to collect and aggregate this data for developers in the context of test

								networks.


								- Interleave messages from all nodes in the network to be able to correlate

								  events during the test run.


								- Collect structured metrics of the system operation (CPU/MEM/IO) during the

								  test run, as well as from each tendermint/application process.


								- Build (simple) tools to be able to render and summarize the data collected

								  during the test run to answer basic questions about test outcome.


								Flexible Assertions

								~~~~~~~~~~~~~~~~~~~


								Currently, all assertions run for every test network, which makes the

								assertions pretty bland, and the framework primarily useful as a smoke-test

								framework, but it might be useful to be able to write and run different

								tests for different configurations. This could allow us to test outside of the

								happy-path.


								In general our existing assertions occupy a fraction of the total test time,

								so the relative cost of adding a few extra test assertions would be of limited

								cost, and could help build confidence.


								Additional Kinds of Testing

								~~~~~~~~~~~~~~~~~~~~~~~~~~~


								The existing e2e suite, exercises networks of nodes that have homogeneous

								tendermint version, stable configuration, that are expected to make

								progress. There are many other possible test configurations that may be

								interesting to engage with. These could include dimensions, such as:


								- Multi-version testing to exercise our compatibility guarantees for networks

								  that might have different tendermint versions.


								- As a flavor or mult-version testing, include upgrade testing, to build

								  confidence in migration code and procedures.


								- Additional test applications, particularly practical-type applciations

								  including some that use gaiad and/or the cosmos-sdk. Test-only applications

								  that simulate other kinds of applications (e.g. variable application

								  operation latency.)


								- Tests of "non-viable" configurations that ensure that forbidden combinations

								  lead to halts.


								References

								----------


								- `ADR 66: End-to-End Testing <../architecture/adr-66-e2e-testing.md>`_