Terraform & Ansible

Automated deployments are done using Terraform to create servers on Digital Ocean then Ansible to create and manage testnets on those servers.

Install

NOTE: see the integration bash script that can be run on a fresh DO droplet and will automatically spin up a 4 node testnet. The script more or less does everything described below.

Install Terraform and Ansible on a Linux machine.
Create a DigitalOcean API token with read and write capability.
Install the python dopy package (pip install dopy)
Create SSH keys (ssh-keygen)
Set environment variables:

export DO_API_TOKEN="abcdef01234567890abcdef01234567890"
export SSH_KEY_FILE="$HOME/.ssh/id_rsa.pub"

These will be used by both terraform and ansible.

Terraform

This step will create four Digital Ocean droplets. First, go to the correct directory:

cd $GOPATH/src/github.com/tendermint/tendermint/networks/remote/terraform

then:

terraform init
terraform apply -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE"

and you will get a list of IP addresses that belong to your droplets.

With the droplets created and running, let's setup Ansible.

Ansible

The playbooks in the ansible directory run ansible roles to configure the sentry node architecture. You must switch to this directory to run ansible (cd $GOPATH/src/github.com/tendermint/tendermint/networks/remote/ansible).

There are several roles that are self-explanatory:

First, we configure our droplets by specifying the paths for tendermint (BINARY) and the node files (CONFIGDIR). The latter expects any number of directories named node0, node1, ... and so on (equal to the number of droplets created). For this example, we use pre-created files from this directory. To create your own files, use either the tendermint testnet command or review manual deployments.

Here's the command to run:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet config.yml -e BINARY=$GOPATH/src/github.com/tendermint/tendermint/build/tendermint -e CONFIGDIR=$GOPATH/src/github.com/tendermint/tendermint/docs/examples

Voila! All your droplets now have the tendermint binary and required configuration files to run a testnet.

Next, we run the install role:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml

which as you'll see below, executes tendermint node --proxy_app=kvstore on all droplets. Although we'll soon be modifying this role and running it again, this first execution allows us to get each node_info.id that corresponds to each node_info.listen_addr. (This part will be automated in the future). In your browser (or using curl), for every droplet, go to IP:26657/status and note the two just mentioned node_info fields. Notice that blocks aren't being created (latest_block_height should be zero and not increasing).

Next, open roles/install/templates/systemd.service.j2 and look for the line ExecStart which should look something like:

ExecStart=/usr/bin/tendermint node --proxy_app=kvstore

and add the --p2p.persistent_peers flag with the relevant information for each node. The resulting file should look something like:

[Unit]
Description={{service}}
Requires=network-online.target
After=network-online.target

[Service]
Restart=on-failure
User={{service}}
Group={{service}}
PermissionsStartOnly=true
ExecStart=/usr/bin/tendermint node --proxy_app=kvstore --p2p.persistent_peers=167b80242c300bf0ccfb3ced3dec60dc2a81776e@165.227.41.206:26656,3c7a5920811550c04bf7a0b2f1e02ab52317b5e6@165.227.43.146:26656,303a1a4312c30525c99ba66522dd81cca56a361a@159.89.115.32:26656,b686c2a7f4b1b46dca96af3a0f31a6a7beae0be4@159.89.119.125:26656
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

Then, stop the nodes:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet stop.yml

Finally, we run the install role again:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml

to re-run tendermint node with the new flag, on all droplets. The latest_block_hash should now be changing and latest_block_height increasing. Your testnet is now up and running :)

Peek at the logs with the status role:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet status.yml

Logging

The crudest way is the status role described above. You can also ship logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) service provider. You can set up your nodes to log there automatically. Create an account and get your API key from the notes on this page, then:

yum install systemd-devel || echo "This will only work on RHEL-based systems."
apt-get install libsystemd-dev || echo "This will only work on Debian-based systems."

go get github.com/mheese/journalbeat
ansible-playbook -i inventory/digital_ocean.py -l sentrynet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345

Cleanup

To remove your droplets, run:

terraform destroy -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE"

5.6 KiB Raw Blame History