From df07f8b36a3e2c916d54204b8922166537e24652 Mon Sep 17 00:00:00 2001 From: Zach Ramsay Date: Wed, 6 Jun 2018 10:35:06 -0400 Subject: [PATCH] moar --- docs/subscribing-to-events-via-websocket.md | 26 ++++ docs/subscribing-to-events-via-websocket.rst | 28 ---- docs/terraform-and-ansible.md | 145 +++++++++++++++++++ docs/terraform-and-ansible.rst | 138 ------------------ docs/transactional-semantics.md | 25 ++++ docs/transactional-semantics.rst | 27 ---- 6 files changed, 196 insertions(+), 193 deletions(-) create mode 100644 docs/subscribing-to-events-via-websocket.md delete mode 100644 docs/subscribing-to-events-via-websocket.rst create mode 100644 docs/terraform-and-ansible.md delete mode 100644 docs/terraform-and-ansible.rst create mode 100644 docs/transactional-semantics.md delete mode 100644 docs/transactional-semantics.rst diff --git a/docs/subscribing-to-events-via-websocket.md b/docs/subscribing-to-events-via-websocket.md new file mode 100644 index 000000000..43d3f7767 --- /dev/null +++ b/docs/subscribing-to-events-via-websocket.md @@ -0,0 +1,26 @@ +# Subscribing to events via Websocket + +Tendermint emits different events, to which you can subscribe via +[Websocket](https://en.wikipedia.org/wiki/WebSocket). This can be useful +for third-party applications (for analysys) or inspecting state. + +[List of events](https://godoc.org/github.com/tendermint/tendermint/types#pkg-constants) + +You can subscribe to any of the events above by calling `subscribe` RPC +method via Websocket. + + { + "jsonrpc": "2.0", + "method": "subscribe", + "id": "0", + "params": { + "query": "tm.event='NewBlock'" + } + } + +Check out [API docs](https://tendermint.github.io/slate/#subscribe) for +more information on query syntax and other options. + +You can also use tags, given you had included them into DeliverTx +response, to query transaction results. See [Indexing +transactions](./indexing-transactions.md) for details. diff --git a/docs/subscribing-to-events-via-websocket.rst b/docs/subscribing-to-events-via-websocket.rst deleted file mode 100644 index f99d94b66..000000000 --- a/docs/subscribing-to-events-via-websocket.rst +++ /dev/null @@ -1,28 +0,0 @@ -Subscribing to events via Websocket -=================================== - -Tendermint emits different events, to which you can subscribe via `Websocket -`__. This can be useful for -third-party applications (for analysys) or inspecting state. - -`List of events `__ - -You can subscribe to any of the events above by calling ``subscribe`` RPC method via Websocket. - -:: - - { - "jsonrpc": "2.0", - "method": "subscribe", - "id": "0", - "params": { - "query": "tm.event='NewBlock'" - } - } - -Check out `API docs `__ for more -information on query syntax and other options. - -You can also use tags, given you had included them into DeliverTx response, to -query transaction results. See `Indexing transactions -<./indexing-transactions.html>`__ for details. diff --git a/docs/terraform-and-ansible.md b/docs/terraform-and-ansible.md new file mode 100644 index 000000000..32073a3f3 --- /dev/null +++ b/docs/terraform-and-ansible.md @@ -0,0 +1,145 @@ +# Terraform & Ansible + +Automated deployments are done using +[Terraform](https://www.terraform.io/) to create servers on Digital +Ocean then [Ansible](http://www.ansible.com/) to create and manage +testnets on those servers. + +## Install + +NOTE: see the [integration bash +script](https://github.com/tendermint/tendermint/blob/develop/networks/remote/integration.sh) +that can be run on a fresh DO droplet and will automatically spin up a 4 +node testnet. The script more or less does everything described below. + +- Install [Terraform](https://www.terraform.io/downloads.html) and + [Ansible](http://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) + on a Linux machine. +- Create a [DigitalOcean API + token](https://cloud.digitalocean.com/settings/api/tokens) with read + and write capability. +- Install the python dopy package (`pip install dopy`) +- Create SSH keys (`ssh-keygen`) +- Set environment variables: + + export DO_API_TOKEN="abcdef01234567890abcdef01234567890" + export SSH_KEY_FILE="$HOME/.ssh/id_rsa.pub" + +These will be used by both `terraform` and `ansible`. + +### Terraform + +This step will create four Digital Ocean droplets. First, go to the +correct directory: + + cd $GOPATH/src/github.com/tendermint/tendermint/networks/remote/terraform + +then: + + terraform init + terraform apply -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE" + +and you will get a list of IP addresses that belong to your droplets. + +With the droplets created and running, let's setup Ansible. + +### Ansible + +The playbooks in [the ansible +directory](https://github.com/tendermint/tendermint/tree/master/networks/remote/ansible) +run ansible roles to configure the sentry node architecture. You must +switch to this directory to run ansible +(`cd $GOPATH/src/github.com/tendermint/tendermint/networks/remote/ansible`). + +There are several roles that are self-explanatory: + +First, we configure our droplets by specifying the paths for tendermint +(`BINARY`) and the node files (`CONFIGDIR`). The latter expects any +number of directories named `node0, node1, ...` and so on (equal to the +number of droplets created). For this example, we use pre-created files +from [this +directory](https://github.com/tendermint/tendermint/tree/master/docs/examples). +To create your own files, use either the `tendermint testnet` command or +review [manual deployments](./deploy-testnets.md). + +Here's the command to run: + + ansible-playbook -i inventory/digital_ocean.py -l sentrynet config.yml -e BINARY=$GOPATH/src/github.com/tendermint/tendermint/build/tendermint -e CONFIGDIR=$GOPATH/src/github.com/tendermint/tendermint/docs/examples + +Voila! All your droplets now have the `tendermint` binary and required +configuration files to run a testnet. + +Next, we run the install role: + + ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml + +which as you'll see below, executes +`tendermint node --proxy_app=kvstore` on all droplets. Although we'll +soon be modifying this role and running it again, this first execution +allows us to get each `node_info.id` that corresponds to each +`node_info.listen_addr`. (This part will be automated in the future). In +your browser (or using `curl`), for every droplet, go to IP:46657/status +and note the two just mentioned `node_info` fields. Notice that blocks +aren't being created (`latest_block_height` should be zero and not +increasing). + +Next, open `roles/install/templates/systemd.service.j2` and look for the +line `ExecStart` which should look something like: + + ExecStart=/usr/bin/tendermint node --proxy_app=kvstore + +and add the `--p2p.persistent_peers` flag with the relevant information +for each node. The resulting file should look something like: + + [Unit] + Description={{service}} + Requires=network-online.target + After=network-online.target + + [Service] + Restart=on-failure + User={{service}} + Group={{service}} + PermissionsStartOnly=true + ExecStart=/usr/bin/tendermint node --proxy_app=kvstore --p2p.persistent_peers=167b80242c300bf0ccfb3ced3dec60dc2a81776e@165.227.41.206:46656,3c7a5920811550c04bf7a0b2f1e02ab52317b5e6@165.227.43.146:46656,303a1a4312c30525c99ba66522dd81cca56a361a@159.89.115.32:46656,b686c2a7f4b1b46dca96af3a0f31a6a7beae0be4@159.89.119.125:46656 + ExecReload=/bin/kill -HUP $MAINPID + KillSignal=SIGTERM + + [Install] + WantedBy=multi-user.target + +Then, stop the nodes: + + ansible-playbook -i inventory/digital_ocean.py -l sentrynet stop.yml + +Finally, we run the install role again: + + ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml + +to re-run `tendermint node` with the new flag, on all droplets. The +`latest_block_hash` should now be changing and `latest_block_height` +increasing. Your testnet is now up and running :) + +Peek at the logs with the status role: + + ansible-playbook -i inventory/digital_ocean.py -l sentrynet status.yml + +### Logging + +The crudest way is the status role described above. You can also ship +logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) +service provider. You can set up your nodes to log there automatically. +Create an account and get your API key from the notes on [this +page](https://app.logz.io/#/dashboard/data-sources/Filebeat), then: + + yum install systemd-devel || echo "This will only work on RHEL-based systems." + apt-get install libsystemd-dev || echo "This will only work on Debian-based systems." + + go get github.com/mheese/journalbeat + ansible-playbook -i inventory/digital_ocean.py -l sentrynet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345 + +### Cleanup + +To remove your droplets, run: + + terraform destroy -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE" diff --git a/docs/terraform-and-ansible.rst b/docs/terraform-and-ansible.rst deleted file mode 100644 index c47de9d43..000000000 --- a/docs/terraform-and-ansible.rst +++ /dev/null @@ -1,138 +0,0 @@ -Terraform & Ansible -=================== - -Automated deployments are done using `Terraform `__ to create servers on Digital Ocean then -`Ansible `__ to create and manage testnets on those servers. - -Install -------- - -NOTE: see the `integration bash script `__ that can be run on a fresh DO droplet and will automatically spin up a 4 node testnet. The script more or less does everything described below. - -- Install `Terraform `__ and `Ansible `__ on a Linux machine. -- Create a `DigitalOcean API token `__ with read and write capability. -- Install the python dopy package (``pip install dopy``) -- Create SSH keys (``ssh-keygen``) -- Set environment variables: - -:: - - export DO_API_TOKEN="abcdef01234567890abcdef01234567890" - export SSH_KEY_FILE="$HOME/.ssh/id_rsa.pub" - -These will be used by both ``terraform`` and ``ansible``. - -Terraform ---------- - -This step will create four Digital Ocean droplets. First, go to the correct directory: - -:: - - cd $GOPATH/src/github.com/tendermint/tendermint/networks/remote/terraform - -then: - -:: - - terraform init - terraform apply -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE" - -and you will get a list of IP addresses that belong to your droplets. - -With the droplets created and running, let's setup Ansible. - -Using Ansible -------------- - -The playbooks in `the ansible directory `__ -run ansible roles to configure the sentry node architecture. You must switch to this directory to run ansible (``cd $GOPATH/src/github.com/tendermint/tendermint/networks/remote/ansible``). - -There are several roles that are self-explanatory: - -First, we configure our droplets by specifying the paths for tendermint (``BINARY``) and the node files (``CONFIGDIR``). The latter expects any number of directories named ``node0, node1, ...`` and so on (equal to the number of droplets created). For this example, we use pre-created files from `this directory `__. To create your own files, use either the ``tendermint testnet`` command or review `manual deployments <./deploy-testnets.html>`__. - -Here's the command to run: - -:: - - ansible-playbook -i inventory/digital_ocean.py -l sentrynet config.yml -e BINARY=$GOPATH/src/github.com/tendermint/tendermint/build/tendermint -e CONFIGDIR=$GOPATH/src/github.com/tendermint/tendermint/docs/examples - -Voila! All your droplets now have the ``tendermint`` binary and required configuration files to run a testnet. - -Next, we run the install role: - -:: - - ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml - -which as you'll see below, executes ``tendermint node --proxy_app=kvstore`` on all droplets. Although we'll soon be modifying this role and running it again, this first execution allows us to get each ``node_info.id`` that corresponds to each ``node_info.listen_addr``. (This part will be automated in the future). In your browser (or using ``curl``), for every droplet, go to IP:46657/status and note the two just mentioned ``node_info`` fields. Notice that blocks aren't being created (``latest_block_height`` should be zero and not increasing). - -Next, open ``roles/install/templates/systemd.service.j2`` and look for the line ``ExecStart`` which should look something like: - -:: - - ExecStart=/usr/bin/tendermint node --proxy_app=kvstore - -and add the ``--p2p.persistent_peers`` flag with the relevant information for each node. The resulting file should look something like: - -:: - - [Unit] - Description={{service}} - Requires=network-online.target - After=network-online.target - - [Service] - Restart=on-failure - User={{service}} - Group={{service}} - PermissionsStartOnly=true - ExecStart=/usr/bin/tendermint node --proxy_app=kvstore --p2p.persistent_peers=167b80242c300bf0ccfb3ced3dec60dc2a81776e@165.227.41.206:46656,3c7a5920811550c04bf7a0b2f1e02ab52317b5e6@165.227.43.146:46656,303a1a4312c30525c99ba66522dd81cca56a361a@159.89.115.32:46656,b686c2a7f4b1b46dca96af3a0f31a6a7beae0be4@159.89.119.125:46656 - ExecReload=/bin/kill -HUP $MAINPID - KillSignal=SIGTERM - - [Install] - WantedBy=multi-user.target - -Then, stop the nodes: - -:: - - ansible-playbook -i inventory/digital_ocean.py -l sentrynet stop.yml - -Finally, we run the install role again: - -:: - - ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml - -to re-run ``tendermint node`` with the new flag, on all droplets. The ``latest_block_hash`` should now be changing and ``latest_block_height`` increasing. Your testnet is now up and running :) - -Peek at the logs with the status role: - -:: - - ansible-playbook -i inventory/digital_ocean.py -l sentrynet status.yml - -Logging -------- - -The crudest way is the status role described above. You can also ship logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) service provider. You can set up your nodes to log there automatically. Create an account and get your API key from the notes on `this page `__, then: - -:: - - yum install systemd-devel || echo "This will only work on RHEL-based systems." - apt-get install libsystemd-dev || echo "This will only work on Debian-based systems." - - go get github.com/mheese/journalbeat - ansible-playbook -i inventory/digital_ocean.py -l sentrynet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345 - -Cleanup -------- - -To remove your droplets, run: - -:: - - terraform destroy -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE" diff --git a/docs/transactional-semantics.md b/docs/transactional-semantics.md new file mode 100644 index 000000000..bab1864e8 --- /dev/null +++ b/docs/transactional-semantics.md @@ -0,0 +1,25 @@ +# Transactional Semantics + +In [Using Tendermint](./using-tendermint.md#broadcast-api) we +discussed different API endpoints for sending transactions and +differences between them. + +What we have not yet covered is transactional semantics. + +When you send a transaction using one of the available methods, it first +goes to the mempool. Currently, it does not provide strong guarantees +like "if the transaction were accepted, it would be eventually included +in a block (given CheckTx passes)." + +For instance a tx could enter the mempool, but before it can be sent to +peers the node crashes. + +We are planning to provide such guarantees by using a WAL and replaying +transactions (See +[this issue](https://github.com/tendermint/tendermint/issues/248)), but +it's non-trivial to do this all efficiently. + +The temporary solution is for clients to monitor the node and resubmit +transaction(s) and/or send them to more nodes at once, so the +probability of all of them crashing at the same time and losing the msg +decreases substantially. diff --git a/docs/transactional-semantics.rst b/docs/transactional-semantics.rst deleted file mode 100644 index 988ac682f..000000000 --- a/docs/transactional-semantics.rst +++ /dev/null @@ -1,27 +0,0 @@ -Transactional Semantics -======================= - -In `Using -Tendermint <./using-tendermint.html#broadcast-api>`__ we -discussed different API endpoints for sending transactions and -differences between them. - -What we have not yet covered is transactional semantics. - -When you send a transaction using one of the available methods, it -first goes to the mempool. Currently, it does not provide strong -guarantees like "if the transaction were accepted, it would be -eventually included in a block (given CheckTx passes)." - -For instance a tx could enter the mempool, but before it can be sent -to peers the node crashes. - -We are planning to provide such guarantees by using a WAL and -replaying transactions (See -`GH#248 `__), but -it's non-trivial to do this all efficiently. - -The temporary solution is for clients to monitor the node and resubmit -transaction(s) or/and send them to more nodes at once, so the -probability of all of them crashing at the same time and losing the -msg decreases substantially.