diff --git a/mintnet-kubernetes/LICENSE b/mintnet-kubernetes/LICENSE new file mode 100644 index 000000000..64a33ddf1 --- /dev/null +++ b/mintnet-kubernetes/LICENSE @@ -0,0 +1,192 @@ +Copyright (C) 2017 Tendermint + + + + Apache License + Version 2.0, January 2004 + https://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/mintnet-kubernetes/README.md b/mintnet-kubernetes/README.md new file mode 100644 index 000000000..58b2ea11c --- /dev/null +++ b/mintnet-kubernetes/README.md @@ -0,0 +1,233 @@ +# Tendermint network powered by Kubernetes + +![Tendermint plus Kubernetes](img/t_plus_k.png) + +* [QuickStart (MacOS)](#quickstart-macos) +* [QuickStart (Linux)](#quickstart-linux) +* [Usage](#usage) +* [Security](#security) +* [Fault tolerance](#fault-tolerance) +* [Starting process](#starting-process) + +This should primarily be used for testing purposes or for tightly-defined +chains operated by a single stakeholder (see [the security +precautions](#security)). If your desire is to launch an application with many +stakeholders, consider using our set of Ansible scripts. + +## QuickStart (MacOS) + +[Requirements](https://github.com/kubernetes/minikube#requirements) + +``` +curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/kubectl +curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.17.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/ +minikube start + +git clone https://github.com/tendermint/tools.git +cd tools/mintnet-kubernetes/examples/basecoin +make create +``` + +## QuickStart (Linux) + +[Requirements](https://github.com/kubernetes/minikube#requirements) + +``` +curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/kubectl +curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.17.1/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/ +minikube start + +git clone https://github.com/tendermint/tools.git +cd tools/mintnet-kubernetes/examples/basecoin +make create +``` + +### Verify everything works + +**Using a shell:** + +1. wait until all the pods are `Running`. + + ``` + kubectl get pods -w -o wide -L tm + ``` + +2. query the Tendermint app logs from the first pod. + + ``` + kubectl logs -c tm -f tm-0 + ``` + +3. use [Rest API](https://tendermint.com/docs/internals/rpc) to fetch the + status of the second pod's Tendermint app. Note we are using `kubectl exec` + because pods are not exposed (and should not be) to the outer network. + + ``` + kubectl exec -c tm tm-0 -- curl -s http://tm-1.basecoin:46657/status | json_pp + ``` + +**Using the dashboard:** + +``` +minikube dashboard +``` + +### Clean up + +``` +make destroy +``` + +## Usage + +### (1/4) Setup a Kubernetes cluster + +- locally using [Minikube](https://github.com/kubernetes/minikube) +- on GCE with a single click in the web UI +- on AWS using [Kubernetes Operations](https://github.com/kubernetes/kops/blob/master/docs/aws.md) +- on Linux machines (Digital Ocean) using [kubeadm](https://kubernetes.io/docs/getting-started-guides/kubeadm/) +- on AWS, Azure, GCE or bare metal using [Kargo (Ansible)](https://kubernetes.io/docs/getting-started-guides/kargo/) + +Please refer to [the official +documentation](https://kubernetes.io/docs/getting-started-guides/) for overview +and comparison of different options. See our guides for [Google Cloud +Engine](docs/SETUP_K8S_ON_GCE.md) or [Digital Ocean](docs/SETUP_K8S_ON_DO.md). + +**Make sure you have Kubernetes >= 1.5, because you will be using StatefulSets, +which is a beta feature in 1.5.** + +### (2/4) Create a configuration file + +Download a template: + +``` +curl -Lo app.yaml https://github.com/tendermint/tools/raw/master/mintnet-kubernetes/app.template.yaml +``` + +Open `app.yaml` in your favorite editor and configure your app container +(navigate to `- name: app`). Kubernetes DSL (Domain Specific Language) is very +simple, so it should be easy. You will need to set Docker image, command and/or +run arguments. Replace variables prefixed with `YOUR_APP` with corresponding +values. Set genesis time to now and preferable chain ID in ConfigMap. + +Please note if you are changing `replicas` number, do not forget to update +`validators` set in ConfigMap. You will be able to scale the cluster up or down +later, but new pods (nodes) won't become validators automatically. + +### (3/4) Deploy your application + +``` +kubectl create -f ./app.yaml +``` + +### (4/4) Observe your cluster + +**web UI** <-> https://github.com/kubernetes/dashboard + +The easiest way to access Dashboard is to use kubectl. Run the following command in your desktop environment: + +``` +kubectl proxy +``` + +kubectl will handle authentication with apiserver and make Dashboard available at [http://localhost:8001/ui](http://localhost:8001/ui) + +**shell** + +List all the pods: + +``` +kubectl get pods -o wide -L tm +``` + +StatefulSet details: + +``` +kubectl describe statefulsets tm +``` + +First pod details: + +``` +kubectl describe pod tm-0 +``` + +Tendermint app logs from the first pod: + +``` +kubectl logs tm-0 -c tm -f +``` + +App logs from the first pod: + +``` +kubectl logs tm-0 -c app -f +``` + +Status of the second pod's Tendermint app: + +``` +kubectl exec -c tm tm-0 -- curl -s http://tm-1.:46657/status | json_pp +``` + +## Security + +Due to the nature of Kubernetes, where you typically have a single master, the +master could be a SPOF (Single Point Of Failure). Therefore, you need to make +sure only authorized people can access it. And these people themselves had +taken basic measures in order not to get hacked. + +These are the best practices: + +- all access to the master is over TLS +- access to the API Server is X.509 certificate or token based +- etcd is not exposed directly to the cluster +- ensure that images are free of vulnerabilities ([1](https://github.com/coreos/clair)) +- ensure that only authorized images are used in your environment +- disable direct access to Kubernetes nodes (no SSH) +- define resource quota + +Resources: + +- https://kubernetes.io/docs/admin/accessing-the-api/ +- http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html +- https://blog.openshift.com/securing-kubernetes/ + +## Fault tolerance + +Having a single master (API server) is a bad thing also because if something +happens to it, you risk being left without an access to the application. + +To avoid that you can [run Kubernetes in multiple +zones](https://kubernetes.io/docs/admin/multiple-zones/), each zone running an +[API server](https://kubernetes.io/docs/admin/high-availability/) and load +balance requests between them. Do not forget to make sure only one instance of +scheduler and controller-manager are running at once. + +Running in multiple zones is a lightweight version of a broader [Cluster +Federation feature](https://kubernetes.io/docs/admin/federation/). Federated +deployments could span across multiple regions (not zones). We haven't tried +this feature yet, so any feedback is highly appreciated! Especially, related to +additional latency and cost of exchanging data between the regions. + +Resources: + +- https://kubernetes.io/docs/admin/high-availability/ + +## Starting process + +![StatefulSet](img/statefulset.png) + +Init containers (`tm-gen-validator`) are run before all other containers, +creating public-private key pair for each pod. Every `tm` container then asks +other pods for their public keys, which are served with nginx (`pub-key` +container). When `tm` container have all the keys, it forms a genesis file and +starts Tendermint process. + +## TODO + +- [ ] run tendermint from tmuser + ``` + securityContext: + fsGroup: 999 + ``` diff --git a/mintnet-kubernetes/app.template.yaml b/mintnet-kubernetes/app.template.yaml new file mode 100644 index 000000000..1eff61a57 --- /dev/null +++ b/mintnet-kubernetes/app.template.yaml @@ -0,0 +1,265 @@ +--- +apiVersion: v1 +kind: Service +metadata: + annotations: + service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" + name: YOUR_APP_NAME + labels: + app: YOUR_APP_NAME +spec: + ports: + - port: 46656 + name: p2p + - port: 46657 + name: rpc + clusterIP: None + selector: + app: tm +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: tm-config +data: + seeds: "tm-0,tm-1,tm-2,tm-3" + validators: "tm-0,tm-1,tm-2,tm-3" + validator.power: "10" + genesis.json: |- + { + "genesis_time": "2017-01-02T10:10:10.164Z", + "chain_id": "chain-B5XXm5", + "validators": [], + "app_hash": "" + } + pub_key_nginx.conf: |- + server { + listen 80 default_server; + listen [::]:80 default_server ipv6only=on; + location /pub_key.json { root /usr/share/nginx/; } + } +--- +apiVersion: policy/v1beta1 +kind: PodDisruptionBudget +metadata: + name: tm-budget +spec: + selector: + matchLabels: + app: tm + minAvailable: 2 +--- +apiVersion: apps/v1beta1 +kind: StatefulSet +metadata: + name: tm +spec: + serviceName: YOUR_APP_NAME + replicas: 4 + template: + metadata: + labels: + app: tm + version: v1 + annotations: + pod.beta.kubernetes.io/init-containers: '[{ + "name": "tm-gen-validator", + "image": "tendermint/tendermint:0.9.0", + "imagePullPolicy": "IfNotPresent", + "command": ["bash", "-c", " + set -ex\n + if [ ! -f /tendermint/priv_validator.json ]; then\n + tendermint gen_validator > /tendermint/priv_validator.json\n + # pub_key.json will be served by pub-key container\n + cat /tendermint/priv_validator.json | jq \".pub_key\" > /tendermint/pub_key.json\n + fi\n + "], + "volumeMounts": [ + {"name": "tmdir", "mountPath": "/tendermint"} + ] + }]' + spec: + containers: + - name: tm + imagePullPolicy: IfNotPresent + image: tendermint/tendermint:0.9.0 + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + ports: + - containerPort: 46656 + name: p2p + - containerPort: 46657 + name: rpc + env: + - name: SEEDS + valueFrom: + configMapKeyRef: + name: tm-config + key: seeds + - name: VALIDATOR_POWER + valueFrom: + configMapKeyRef: + name: tm-config + key: validator.power + - name: VALIDATORS + valueFrom: + configMapKeyRef: + name: tm-config + key: validators + - name: TMROOT + value: /tendermint + command: + - bash + - "-c" + - | + set -ex + + # copy template + cp /etc/tendermint/genesis.json /tendermint/genesis.json + + # fill genesis file with validators + IFS=',' read -ra VALS_ARR <<< "$VALIDATORS" + fqdn_suffix=$(echo $(hostname -f) | sed 's#[^.]*\.\(\)#\1#') + for v in "${VALS_ARR[@]}"; do + # wait until validator generates priv/pub key pair + set +e + + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + while [ "$ERR" != 0 ]; do + sleep 5 + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + done + set -e + + # add validator to genesis file along with its pub_key + curl -s "http://$v.$fqdn_suffix/pub_key.json" | jq ". as \$k | {pub_key: \$k, amount: $VALIDATOR_POWER, name: \"$v\"}" > pub_validator.json + cat /tendermint/genesis.json | jq ".validators |= .+ [$(cat pub_validator.json)]" > /tendermint/genesis.json + rm pub_validator.json + done + + # construct seeds + IFS=',' read -ra SEEDS_ARR <<< "$SEEDS" + seeds=() + for s in "${SEEDS_ARR[@]}"; do + seeds+=("$s.$fqdn_suffix:46656") + done + seeds=$(IFS=','; echo "${seeds[*]}") + + tendermint node --seeds="$seeds" --moniker="`hostname`" --proxy_app="unix:///socks/app.sock" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - mountPath: /etc/tendermint/genesis.json + name: configdir + subPath: genesis.json + - name: socksdir + mountPath: /socks + + - name: app + imagePullPolicy: IfNotPresent + image: YOUR_APP_IMAGE + args: ["--addr=\"unix:///socks/app.sock\""] + volumeMounts: + - name: socksdir + mountPath: /socks + + ######## OR ######## + # + # - name: app + # imagePullPolicy: IfNotPresent + # image: golang:1.7.5 + # resources: + # requests: + # cpu: YOUR_APP_CPU_REQ + # memory: YOUR_APP_MEM_REQ + # limits: + # cpu: YOUR_APP_CPU_LIMIT + # memory: YOUR_APP_MEM_LIMIT + # command: + # - bash + # - "-c" + # - | + # set -ex + + # go get -d YOUR_APP_PACKAGE + # cd $GOPATH/YOUR_APP_PACKAGE + # make install + # + # rm -f /socks/app.sock # remove old socket + + # YOUR_APP_EXEC --addr="unix:///socks/app.sock" + # volumeMounts: + # - name: socksdir + # mountPath: /socks + + ######## OPTIONALLY ######## + # + # - name: data + # imagePullPolicy: IfNotPresent + # image: golang:1.7.5 + # command: + # - bash + # - "-c" + # - | + # set -ex + # go get github.com/tendermint/merkleeyes/cmd/merkleeyes + # rm -f /socks/data.sock # remove old socket + # merkleeyes server --address="unix:///socks/data.sock" + # volumeMounts: + # - name: socksdir + # mountPath: /socks + + - name: pub-key + imagePullPolicy: IfNotPresent + image: nginx:1.11.9 + resources: + requests: + cpu: 10m + memory: 12Mi + limits: + cpu: 20m + memory: 24Mi + ports: + - containerPort: 80 + name: pub-key + command: + - bash + - "-c" + - | + set -ex + # fixes 403 Permission Denied (open() "/tendermint/pub_key.json" failed (13: Permission denied)) + # => we cannot serve from /tendermint, so we copy the file + mkdir -p /usr/share/nginx + cp /tendermint/pub_key.json /usr/share/nginx/pub_key.json + nginx -g "daemon off;" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - mountPath: /etc/nginx/conf.d/pub_key.conf + name: configdir + subPath: pub_key_nginx.conf + + volumes: + - name: configdir + configMap: + name: tm-config + - name: socksdir + emptyDir: {} + + volumeClaimTemplates: + - metadata: + name: tmdir + annotations: + volume.alpha.kubernetes.io/storage-class: anything + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 2Gi diff --git a/mintnet-kubernetes/docs/SETUP_K8S_ON_DO.md b/mintnet-kubernetes/docs/SETUP_K8S_ON_DO.md new file mode 100644 index 000000000..7c1bd9d18 --- /dev/null +++ b/mintnet-kubernetes/docs/SETUP_K8S_ON_DO.md @@ -0,0 +1,13 @@ +# Setup a Kubernetes cluster on Digital Ocean (DO) + +Available options: + +1. [kubeadm (alpha)](https://kubernetes.io/docs/getting-started-guides/kubeadm/) +2. [kargo](https://kubernetes.io/docs/getting-started-guides/kargo/) +3. [terraform](https://github.com/hermanjunge/kubernetes-digitalocean-terraform) + +As you can see, there is no single tool for creating a cluster on DO. +Therefore, choose the one you know and comfortable working with. If you know +and used [terraform](https://www.terraform.io/) before, then choose it. If you +know Ansible, then pick kargo. If none of these seem familiar to you, go with +kubeadm. diff --git a/mintnet-kubernetes/docs/SETUP_K8S_ON_GCE.md b/mintnet-kubernetes/docs/SETUP_K8S_ON_GCE.md new file mode 100644 index 000000000..671d7ec7e --- /dev/null +++ b/mintnet-kubernetes/docs/SETUP_K8S_ON_GCE.md @@ -0,0 +1,31 @@ +# Setup a Kubernetes cluster on Google Cloud Engine (GCE) + +Main article: [Running Kubernetes on Google Compute +Engine](https://kubernetes.io/docs/getting-started-guides/gce/) + +## 1. Create a cluster + +The recommended way is to use [Google Container +Engine](https://cloud.google.com/container-engine/) (GKE). You should be able +to create a fully fledged cluster with just a few clicks. + +## 2. Connect to it + +Install `gcloud` as a part of [Google Cloud SDK](https://cloud.google.com/sdk/). + +Make sure you have credentials for GCloud by running `gcloud auth login`. + +In order to make API calls against GCE, you must also run `gcloud auth +application-default login` + +Press `Connect` button: + +![Connect button](../img/gce1.png) + +![Connect pop-up](../img/gce2.png) + +and execute the first command in your shell. Then start a proxy by +executing `kubectl proxy`. + +Now you should be able to run `kubectl` command to create resources, get +resource info, logs, etc. diff --git a/mintnet-kubernetes/examples/basecoin/Makefile b/mintnet-kubernetes/examples/basecoin/Makefile new file mode 100644 index 000000000..6d54d57d6 --- /dev/null +++ b/mintnet-kubernetes/examples/basecoin/Makefile @@ -0,0 +1,10 @@ +create: + @echo "==> Creating deployment" + @kubectl create -f app.yaml + +destroy: + @echo "==> Destroying deployment" + @kubectl delete -f app.yaml + @kubectl delete pvc -l app=tm + +.PHONY: create destroy diff --git a/mintnet-kubernetes/examples/basecoin/README.md b/mintnet-kubernetes/examples/basecoin/README.md new file mode 100644 index 000000000..58d3f39c2 --- /dev/null +++ b/mintnet-kubernetes/examples/basecoin/README.md @@ -0,0 +1,42 @@ +# Basecoin example + +This is an example of using [basecoin](https://github.com/tendermint/basecoin). + +## Usage + +``` +make create +``` + +### Check account balance and send a transaction + +1. wait until all the pods are `Running`. + + ``` + kubectl get pods -w -o wide -L tm + ``` + +2. wait until app starts. + + ``` + kubectl logs -c app -f tm-0 + ``` + +3. get account's address of the second pod + + ``` + kubectl exec -c app tm-1 -- cat /app/key.json | grep "address" + ``` + +4. send 5 coins to it from the first pod + + ``` + kubectl exec -c app tm-0 -- basecoin tx send --to 0x
--amount 5 + ``` + + +## Clean up + +``` +make destroy +``` diff --git a/mintnet-kubernetes/examples/basecoin/app.yaml b/mintnet-kubernetes/examples/basecoin/app.yaml new file mode 100644 index 000000000..d1874a254 --- /dev/null +++ b/mintnet-kubernetes/examples/basecoin/app.yaml @@ -0,0 +1,320 @@ +--- +apiVersion: v1 +kind: Service +metadata: + annotations: + service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" + name: basecoin + labels: + app: basecoin +spec: + ports: + - port: 46656 + name: p2p + - port: 46657 + name: rpc + clusterIP: None + selector: + app: tm +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: tm-config +data: + seeds: "tm-0,tm-1,tm-2,tm-3" + validators: "tm-0,tm-1,tm-2,tm-3" + validator.power: "10" + genesis.json: |- + { + "genesis_time": "2016-02-05T06:02:31.526Z", + "chain_id": "chain-tTH4mi", + "validators": [], + "app_hash": "" + } + pub_key_nginx.conf: |- + server { + listen 80 default_server; + listen [::]:80 default_server ipv6only=on; + location /pub_key.json { root /usr/share/nginx/; } + location /app_pub_key.json { root /usr/share/nginx/; } + } +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: app-config +data: + genesis.json: |- + [ + "base/chainID", + "test_chain_id", + "base/account", + { + "coins" : [ + {"denom": "blank", "amount": 1000} + ], + "pub_key" : ["tm-0"] + }, + "base/account", + { + "coins" : [ + {"denom": "blank", "amount": 1000} + ], + "pub_key" : ["tm-1"] + }, + "base/account", + { + "coins" : [ + {"denom": "blank", "amount": 1000} + ], + "pub_key" : ["tm-2"] + }, + "base/account", + { + "coins" : [ + {"denom": "blank", "amount": 1000} + ], + "pub_key" : ["tm-3"] + } + ] +--- +apiVersion: policy/v1beta1 +kind: PodDisruptionBudget +metadata: + name: tm-budget +spec: + selector: + matchLabels: + app: tm + minAvailable: 2 +--- +apiVersion: apps/v1beta1 +kind: StatefulSet +metadata: + name: tm +spec: + serviceName: basecoin + replicas: 4 + template: + metadata: + labels: + app: tm + annotations: + pod.beta.kubernetes.io/init-containers: '[{ + "name": "tm-gen-validator", + "image": "tendermint/tendermint:0.9.0", + "imagePullPolicy": "IfNotPresent", + "command": ["bash", "-c", " + set -ex\n + if [ ! -f /tendermint/priv_validator.json ]; then\n + tendermint gen_validator > /tendermint/priv_validator.json\n + # pub_key.json will be served by pub-key container\n + cat /tendermint/priv_validator.json | jq \".pub_key\" > /tendermint/pub_key.json\n + fi\n + "], + "volumeMounts": [ + {"name": "tmdir", "mountPath": "/tendermint"} + ] + }, + { + "name": "app-gen-key", + "image": "tendermint/basecoin:latest", + "imagePullPolicy": "IfNotPresent", + "command": ["bash", "-c", " + set -ex\n + if [ ! -f /app/key.json ]; then\n + basecoin key new > /app/key.json\n + # pub_key.json will be served by app-pub-key container\n + cat /app/key.json | jq \".pub_key\" > /app/pub_key.json\n + fi\n + "], + "volumeMounts": [ + {"name": "appdir", "mountPath": "/app"} + ] + }]' + spec: + containers: + - name: tm + imagePullPolicy: IfNotPresent + image: tendermint/tendermint:0.9.0 + ports: + - containerPort: 46656 + name: p2p + - containerPort: 46657 + name: rpc + env: + - name: SEEDS + valueFrom: + configMapKeyRef: + name: tm-config + key: seeds + - name: VALIDATOR_POWER + valueFrom: + configMapKeyRef: + name: tm-config + key: validator.power + - name: VALIDATORS + valueFrom: + configMapKeyRef: + name: tm-config + key: validators + - name: TMROOT + value: /tendermint + command: + - bash + - "-c" + - | + set -ex + + # copy template + cp /etc/tendermint/genesis.json /tendermint/genesis.json + + # fill genesis file with validators + IFS=',' read -ra VALS_ARR <<< "$VALIDATORS" + fqdn_suffix=$(echo $(hostname -f) | sed 's#[^.]*\.\(\)#\1#') + for v in "${VALS_ARR[@]}"; do + # wait until validator generates priv/pub key pair + set +e + + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + while [ "$ERR" != 0 ]; do + sleep 5 + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + done + set -e + + # add validator to genesis file along with its pub_key + curl -s "http://$v.$fqdn_suffix/pub_key.json" | jq ". as \$k | {pub_key: \$k, amount: $VALIDATOR_POWER, name: \"$v\"}" > pub_validator.json + cat /tendermint/genesis.json | jq ".validators |= .+ [$(cat pub_validator.json)]" > /tendermint/genesis.json + rm pub_validator.json + done + + # construct seeds + IFS=',' read -ra SEEDS_ARR <<< "$SEEDS" + seeds=() + for s in "${SEEDS_ARR[@]}"; do + seeds+=("$s.$fqdn_suffix:46656") + done + seeds=$(IFS=','; echo "${seeds[*]}") + + tendermint node --seeds="$seeds" --moniker="`hostname`" --proxy_app="unix:///socks/app.sock" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - mountPath: /etc/tendermint/genesis.json + name: tmconfigdir + subPath: genesis.json + - name: socksdir + mountPath: /socks + + - name: app + imagePullPolicy: IfNotPresent + image: tendermint/basecoin:latest + workingDir: /app + command: + - bash + - "-c" + - | + set -ex + + # replace "tm-N" with public keys in genesis file + cp /etc/app/genesis.json genesis.json + fqdn_suffix=$(echo $(hostname -f) | sed 's#[^.]*\.\(\)#\1#') + # for every "base/account" + i=3 + length=$(cat genesis.json | jq ". | length") + while [ $i -lt $length ]; do + # extract pod name ("tm-0") + pod=$(cat genesis.json | jq -r ".[$i].pub_key[0]") + + # wait until pod starts to serve its pub_key + set +e + + curl -s "http://$pod.$fqdn_suffix/app_pub_key.json" > /dev/null + ERR=$? + while [ "$ERR" != 0 ]; do + sleep 5 + curl -s "http://$pod.$fqdn_suffix/app_pub_key.json" > /dev/null + ERR=$? + done + set -e + + # get its pub_key + curl -s "http://$pod.$fqdn_suffix/app_pub_key.json" | jq "." > k.json + + # replace pod name with it (["tm-0"] => "[1, XXXXXXXXXXXXXXXXXXXX]") + cat genesis.json | jq ".[$i].pub_key = $(cat k.json | jq '.')" > genesis.json + rm -f k.json + + i=$((i+2)) # skip "base/account" field itself + done + + rm -f /socks/app.sock # remove old socket + + basecoin start --address="unix:///socks/app.sock" + volumeMounts: + - name: appdir + mountPath: /app + - mountPath: /etc/app/genesis.json + name: appconfigdir + subPath: genesis.json + - name: socksdir + mountPath: /socks + + - name: pub-key + imagePullPolicy: IfNotPresent + image: nginx:latest + ports: + - containerPort: 80 + command: + - bash + - "-c" + - | + set -ex + # fixes 403 Permission Denied (open() "/tendermint/pub_key.json" failed (13: Permission denied)) + # => we cannot serve from /tendermint, so we copy the file + mkdir -p /usr/share/nginx + cp /tendermint/pub_key.json /usr/share/nginx/pub_key.json + cp /app/pub_key.json /usr/share/nginx/app_pub_key.json + nginx -g "daemon off;" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - name: appdir + mountPath: /app + - mountPath: /etc/nginx/conf.d/pub_key.conf + name: tmconfigdir + subPath: pub_key_nginx.conf + + volumes: + - name: tmconfigdir + configMap: + name: tm-config + - name: appconfigdir + configMap: + name: app-config + - name: socksdir + emptyDir: {} + + volumeClaimTemplates: + - metadata: + name: tmdir + annotations: + volume.alpha.kubernetes.io/storage-class: anything + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 2Gi + - metadata: + name: appdir + annotations: + volume.alpha.kubernetes.io/storage-class: anything + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 12Mi diff --git a/mintnet-kubernetes/examples/basecoin/lightclient.md b/mintnet-kubernetes/examples/basecoin/lightclient.md new file mode 100644 index 000000000..67da8ef66 --- /dev/null +++ b/mintnet-kubernetes/examples/basecoin/lightclient.md @@ -0,0 +1,98 @@ +# Using with lightclient + +We have an awesome cluster running, let's try to test this out without +relying on executing commands on the cluster. Rather, we can connect to the +rpc interface with the `light-client` package and execute commands locally, +or even proxy our webapp to the kubernetes backend. + +## Setup + +In order to get this working, we need to know a few pieces of info, +the chain id of tendermint, the chain id of basecoin, and an account +with a bit of cash.... + +### Tendermint Chain ID + +`kubectl exec -c tm tm-0 -- curl -s http://tm-1.basecoin:46657/status | json_pp | grep network` + +set TM_CHAIN with the value there + +### Basecoin Chain ID + +`kubectl exec -c app tm-1 -- grep -A1 chainID /app/genesis.json` + +set BC_CHAIN with the value there + +### Expose tendermint rpc + +We need to be able to reach the tendermint rpc interface from our shell. + +`kubectl port-forward tm-0 46657:46657` + +### Start basecoin-proxy + +Using this info, let's connect our proxy and get going + +`proxy-basecoin -tmchain=$TM_CHAIN -chain=$BC_CHAIN -rpc=localhost:46657` + +## Basecoin accounts + +Well, we can connect, but we don't have a registered account yet... +Let's look around, then use the cli to send some money from one of +the validators to our client's address so we can play. + +**TODO** we can add some of our known accounts (from `/keys`) into +the genesis file, so we can skip all the kubectl money fiddling here. +We will want to start with money on some known non-validators. + +### Getting validator info (kubectl) + +The basecoin app deployment starts with 1000 "blank" coin in an account of +each validator. Let's get the address of the first validator + +`kubectl exec -c app tm-1 -- grep address /app/key.json` + +Store this info as VAL1_ADDR + +### Querying state (proxy) + +The proxy can read any public info via the tendermint rpc, so let's check +out this account. + +`curl localhost:8108/query/account/$VAL1_ADDR` + +Now, let's make out own account.... + +`curl -XPOST http://localhost:8108/keys/ -d '{"name": "k8demo", "passphrase": "1234567890"}'` + +(or pick your own user and password). Remember the address you get here. You can +always find it out later by calling: + +`curl http://localhost:8108/keys/k8demo` + +and store it in DEMO_ADDR, which is empty at first + +`curl localhost:8108/query/account/$DEMO_ADDR` + + +### "Stealing" validator cash (kubectl) + +Run one command, that will be signed, now we have money + +`kubectl exec -c app tm-0 -- basecoin tx send --to --amount 500` + +### Using our money + +Returning to our remote shell, we have a remote account with some money. +Let's see that. + +`curl localhost:8108/query/account/$DEMO_ADDR` + +Cool. Now we need to send it to a second account. + +`curl -XPOST http://localhost:8108/keys/ -d '{"name": "buddy", "passphrase": "1234567890"}'` + +and store the resulting address in BUDDY_ADDR + +**TODO** finish this + diff --git a/mintnet-kubernetes/examples/counter/Makefile b/mintnet-kubernetes/examples/counter/Makefile new file mode 100644 index 000000000..6d54d57d6 --- /dev/null +++ b/mintnet-kubernetes/examples/counter/Makefile @@ -0,0 +1,10 @@ +create: + @echo "==> Creating deployment" + @kubectl create -f app.yaml + +destroy: + @echo "==> Destroying deployment" + @kubectl delete -f app.yaml + @kubectl delete pvc -l app=tm + +.PHONY: create destroy diff --git a/mintnet-kubernetes/examples/counter/app.yaml b/mintnet-kubernetes/examples/counter/app.yaml new file mode 100644 index 000000000..b4a06af0e --- /dev/null +++ b/mintnet-kubernetes/examples/counter/app.yaml @@ -0,0 +1,217 @@ +--- +apiVersion: v1 +kind: Service +metadata: + annotations: + service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" + name: counter + labels: + app: counter +spec: + ports: + - port: 46656 + name: p2p + - port: 46657 + name: rpc + clusterIP: None + selector: + app: tm +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: tm-config +data: + seeds: "tm-0,tm-1,tm-2,tm-3" + validators: "tm-0,tm-1,tm-2,tm-3" + validator.power: "10" + genesis.json: |- + { + "genesis_time": "2016-02-05T23:17:31.164Z", + "chain_id": "chain-B5XXm5", + "validators": [], + "app_hash": "" + } + pub_key_nginx.conf: |- + server { + listen 80 default_server; + listen [::]:80 default_server ipv6only=on; + location /pub_key.json { root /usr/share/nginx/; } + } +--- +apiVersion: policy/v1beta1 +kind: PodDisruptionBudget +metadata: + name: tm-budget +spec: + selector: + matchLabels: + app: tm + minAvailable: 2 +--- +apiVersion: apps/v1beta1 +kind: StatefulSet +metadata: + name: tm +spec: + serviceName: counter + replicas: 4 + template: + metadata: + labels: + app: tm + annotations: + pod.beta.kubernetes.io/init-containers: '[{ + "name": "tm-gen-validator", + "image": "tendermint/tendermint:0.9.0", + "imagePullPolicy": "IfNotPresent", + "command": ["bash", "-c", " + set -ex\n + if [ ! -f /tendermint/priv_validator.json ]; then\n + tendermint gen_validator > /tendermint/priv_validator.json\n + # pub_key.json will be served by pub-key container\n + cat /tendermint/priv_validator.json | jq \".pub_key\" > /tendermint/pub_key.json\n + fi\n + "], + "volumeMounts": [ + {"name": "tmdir", "mountPath": "/tendermint"} + ] + }]' + spec: + containers: + - name: tm + imagePullPolicy: IfNotPresent + image: tendermint/tendermint:0.9.0 + ports: + - containerPort: 46656 + name: p2p + - containerPort: 46657 + name: rpc + env: + - name: SEEDS + valueFrom: + configMapKeyRef: + name: tm-config + key: seeds + - name: VALIDATOR_POWER + valueFrom: + configMapKeyRef: + name: tm-config + key: validator.power + - name: VALIDATORS + valueFrom: + configMapKeyRef: + name: tm-config + key: validators + - name: TMROOT + value: /tendermint + command: + - bash + - "-c" + - | + set -ex + + # copy template + cp /etc/tendermint/genesis.json /tendermint/genesis.json + + # fill genesis file with validators + IFS=',' read -ra VALS_ARR <<< "$VALIDATORS" + fqdn_suffix=$(echo $(hostname -f) | sed 's#[^.]*\.\(\)#\1#') + for v in "${VALS_ARR[@]}"; do + # wait until validator generates priv/pub key pair + set +e + + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + while [ "$ERR" != 0 ]; do + sleep 5 + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + done + set -e + + # add validator to genesis file along with its pub_key + curl -s "http://$v.$fqdn_suffix/pub_key.json" | jq ". as \$k | {pub_key: \$k, amount: $VALIDATOR_POWER, name: \"$v\"}" > pub_validator.json + cat /tendermint/genesis.json | jq ".validators |= .+ [$(cat pub_validator.json)]" > /tendermint/genesis.json + rm pub_validator.json + done + + # construct seeds + IFS=',' read -ra SEEDS_ARR <<< "$SEEDS" + seeds=() + for s in "${SEEDS_ARR[@]}"; do + seeds+=("$s.$fqdn_suffix:46656") + done + seeds=$(IFS=','; echo "${seeds[*]}") + + tendermint node --seeds="$seeds" --moniker="`hostname`" --proxy_app="unix:///socks/app.sock" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - mountPath: /etc/tendermint/genesis.json + name: tmconfigdir + subPath: genesis.json + - name: socksdir + mountPath: /socks + + - name: app + imagePullPolicy: IfNotPresent + image: golang:latest + command: + - bash + - "-c" + - | + set -ex + + go get -d github.com/tendermint/abci/cmd/counter + cd $GOPATH/src/github.com/tendermint/abci/ + make get_deps + make install + + rm -f /socks/app.sock # remove old socket + + counter --serial --addr="unix:///socks/app.sock" + volumeMounts: + - name: socksdir + mountPath: /socks + + - name: pub-key + imagePullPolicy: IfNotPresent + image: nginx:latest + ports: + - containerPort: 80 + name: pub-key + command: + - bash + - "-c" + - | + set -ex + # fixes 403 Permission Denied (open() "/tendermint/pub_key.json" failed (13: Permission denied)) + # => we cannot serve from /tendermint, so we copy the file + mkdir -p /usr/share/nginx + cp /tendermint/pub_key.json /usr/share/nginx/pub_key.json + nginx -g "daemon off;" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - mountPath: /etc/nginx/conf.d/pub_key.conf + name: tmconfigdir + subPath: pub_key_nginx.conf + + volumes: + - name: tmconfigdir + configMap: + name: tm-config + - name: socksdir + emptyDir: {} + + volumeClaimTemplates: + - metadata: + name: tmdir + annotations: + volume.alpha.kubernetes.io/storage-class: anything + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 2Gi diff --git a/mintnet-kubernetes/examples/dummy/Makefile b/mintnet-kubernetes/examples/dummy/Makefile new file mode 100644 index 000000000..825487fcd --- /dev/null +++ b/mintnet-kubernetes/examples/dummy/Makefile @@ -0,0 +1,17 @@ +create: + @echo "==> Creating deployment" + @kubectl create -f app.yaml + @echo "==> Waiting 10s until it is probably ready" + @sleep 10 + @echo "==> Creating monitor and transacter pods" + @kubectl create -f tm-monitor-pod.yaml + @kubectl create -f transacter-pod.yaml + +destroy: + @echo "==> Destroying deployment" + @kubectl delete -f transacter-pod.yaml + @kubectl delete -f tm-monitor-pod.yaml + @kubectl delete -f app.yaml + @kubectl delete pvc -l app=tm + +.PHONY: create destroy diff --git a/mintnet-kubernetes/examples/dummy/app.yaml b/mintnet-kubernetes/examples/dummy/app.yaml new file mode 100644 index 000000000..7efc48dc7 --- /dev/null +++ b/mintnet-kubernetes/examples/dummy/app.yaml @@ -0,0 +1,196 @@ +--- +apiVersion: v1 +kind: Service +metadata: + annotations: + service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" + name: dummy + labels: + app: dummy +spec: + ports: + - port: 46656 + name: p2p + - port: 46657 + name: rpc + clusterIP: None + selector: + app: tm +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: tm-config +data: + seeds: "tm-0,tm-1,tm-2,tm-3" + validators: "tm-0,tm-1,tm-2,tm-3" + validator.power: "10" + genesis.json: |- + { + "genesis_time": "2016-02-05T23:17:31.164Z", + "chain_id": "chain-B5XXm5", + "validators": [], + "app_hash": "" + } + pub_key_nginx.conf: |- + server { + listen 80 default_server; + listen [::]:80 default_server ipv6only=on; + location /pub_key.json { root /usr/share/nginx/; } + } +--- +apiVersion: policy/v1beta1 +kind: PodDisruptionBudget +metadata: + name: tm-budget +spec: + selector: + matchLabels: + app: tm + minAvailable: 2 +--- +apiVersion: apps/v1beta1 +kind: StatefulSet +metadata: + name: tm +spec: + serviceName: dummy + replicas: 4 + template: + metadata: + labels: + app: tm + annotations: + pod.beta.kubernetes.io/init-containers: '[{ + "name": "tm-gen-validator", + "image": "tendermint/tendermint:0.9.0", + "imagePullPolicy": "IfNotPresent", + "command": ["bash", "-c", " + set -ex\n + if [ ! -f /tendermint/priv_validator.json ]; then\n + tendermint gen_validator > /tendermint/priv_validator.json\n + # pub_key.json will be served by pub-key container\n + cat /tendermint/priv_validator.json | jq \".pub_key\" > /tendermint/pub_key.json\n + fi\n + "], + "volumeMounts": [ + {"name": "tmdir", "mountPath": "/tendermint"} + ] + }]' + spec: + containers: + - name: tm + imagePullPolicy: IfNotPresent + image: tendermint/tendermint:0.9.0 + ports: + - containerPort: 46656 + name: p2p + - containerPort: 46657 + name: rpc + env: + - name: SEEDS + valueFrom: + configMapKeyRef: + name: tm-config + key: seeds + - name: VALIDATOR_POWER + valueFrom: + configMapKeyRef: + name: tm-config + key: validator.power + - name: VALIDATORS + valueFrom: + configMapKeyRef: + name: tm-config + key: validators + - name: TMROOT + value: /tendermint + command: + - bash + - "-c" + - | + set -ex + + # copy template + cp /etc/tendermint/genesis.json /tendermint/genesis.json + + # fill genesis file with validators + IFS=',' read -ra VALS_ARR <<< "$VALIDATORS" + fqdn_suffix=$(echo $(hostname -f) | sed 's#[^.]*\.\(\)#\1#') + for v in "${VALS_ARR[@]}"; do + # wait until validator generates priv/pub key pair + set +e + + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + while [ "$ERR" != 0 ]; do + sleep 5 + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + done + set -e + + # add validator to genesis file along with its pub_key + curl -s "http://$v.$fqdn_suffix/pub_key.json" | jq ". as \$k | {pub_key: \$k, amount: $VALIDATOR_POWER, name: \"$v\"}" > pub_validator.json + cat /tendermint/genesis.json | jq ".validators |= .+ [$(cat pub_validator.json)]" > /tendermint/genesis.json + rm pub_validator.json + done + + # construct seeds + IFS=',' read -ra SEEDS_ARR <<< "$SEEDS" + seeds=() + for s in "${SEEDS_ARR[@]}"; do + seeds+=("$s.$fqdn_suffix:46656") + done + seeds=$(IFS=','; echo "${seeds[*]}") + + tendermint node --seeds="$seeds" --moniker="`hostname`" --proxy_app="dummy" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - mountPath: /etc/tendermint/genesis.json + name: tmconfigdir + subPath: genesis.json + - name: socksdir + mountPath: /socks + + - name: pub-key + imagePullPolicy: IfNotPresent + image: nginx:latest + ports: + - containerPort: 80 + name: pub-key + command: + - bash + - "-c" + - | + set -ex + # fixes 403 Permission Denied (open() "/tendermint/pub_key.json" failed (13: Permission denied)) + # => we cannot serve from /tendermint, so we copy the file + mkdir -p /usr/share/nginx + cp /tendermint/pub_key.json /usr/share/nginx/pub_key.json + nginx -g "daemon off;" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - mountPath: /etc/nginx/conf.d/pub_key.conf + name: tmconfigdir + subPath: pub_key_nginx.conf + + volumes: + - name: tmconfigdir + configMap: + name: tm-config + - name: socksdir + emptyDir: {} + + volumeClaimTemplates: + - metadata: + name: tmdir + annotations: + volume.alpha.kubernetes.io/storage-class: anything + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 2Gi diff --git a/mintnet-kubernetes/examples/dummy/tm-monitor-pod.yaml b/mintnet-kubernetes/examples/dummy/tm-monitor-pod.yaml new file mode 100644 index 000000000..59e110db3 --- /dev/null +++ b/mintnet-kubernetes/examples/dummy/tm-monitor-pod.yaml @@ -0,0 +1,13 @@ +--- +apiVersion: v1 +kind: Pod +metadata: + name: monitor +spec: + containers: + - name: monitor + image: tendermint/monitor + args: ["-listen-addr=tcp://0.0.0.0:46670", "tm-0.dummy:46657,tm-1.dummy:46657,tm-2.dummy:46657,tm-3.dummy:46657"] + ports: + - containerPort: 46670 + name: rpc diff --git a/mintnet-kubernetes/examples/dummy/transacter-pod.yaml b/mintnet-kubernetes/examples/dummy/transacter-pod.yaml new file mode 100644 index 000000000..7332631f0 --- /dev/null +++ b/mintnet-kubernetes/examples/dummy/transacter-pod.yaml @@ -0,0 +1,19 @@ +--- +apiVersion: v1 +kind: Pod +metadata: + name: transacter +spec: + containers: + - name: transacter + image: tendermint/transacter + command: + - bash + - "-c" + - | + set -ex + while true + do + ./transact 100 "tm-0.dummy:46657" + sleep 1 + done diff --git a/mintnet-kubernetes/examples/localchain/Makefile b/mintnet-kubernetes/examples/localchain/Makefile new file mode 100644 index 000000000..6d54d57d6 --- /dev/null +++ b/mintnet-kubernetes/examples/localchain/Makefile @@ -0,0 +1,10 @@ +create: + @echo "==> Creating deployment" + @kubectl create -f app.yaml + +destroy: + @echo "==> Destroying deployment" + @kubectl delete -f app.yaml + @kubectl delete pvc -l app=tm + +.PHONY: create destroy diff --git a/mintnet-kubernetes/examples/localchain/app.yaml b/mintnet-kubernetes/examples/localchain/app.yaml new file mode 100644 index 000000000..8d6a77754 --- /dev/null +++ b/mintnet-kubernetes/examples/localchain/app.yaml @@ -0,0 +1,288 @@ +--- +# Single pod installation (see basecoin for distributed setup) +apiVersion: v1 +kind: Service +metadata: + annotations: + service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" + name: localchain + labels: + app: localchain +spec: + ports: + - port: 46656 + name: p2p + - port: 46657 + name: rpc + clusterIP: None + selector: + app: tm +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: tm-config +data: + validators: "tm-0" + validator.power: "10" + genesis.json: |- + { + "genesis_time": "2016-03-24T23:29:20.457Z", + "chain_id": "chain-iqwZgb", + "validators": [], + "app_hash": "" + } + pub_key_nginx.conf: |- + server { + listen 80 default_server; + listen [::]:80 default_server ipv6only=on; + location /pub_key.json { root /usr/share/nginx/; } + location /app_pub_key.json { root /usr/share/nginx/; } + } +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: app-config +data: + genesis.json: |- + [ + "base/chainID", + "test_chain_id", + "base/account", + { + "pub_key": ["tm-0"], + "coins": [ + {"denom":"CITI/USD","amount":1000}, + {"denom":"UBS/EURO","amount":1000} + ] + } + ] +--- +apiVersion: policy/v1beta1 +kind: PodDisruptionBudget +metadata: + name: tm-budget +spec: + selector: + matchLabels: + app: tm + minAvailable: 2 +--- +apiVersion: apps/v1beta1 +kind: StatefulSet +metadata: + name: tm +spec: + serviceName: localchain + replicas: 1 + template: + metadata: + labels: + app: tm + annotations: + pod.beta.kubernetes.io/init-containers: '[{ + "name": "tm-gen-validator", + "image": "tendermint/tendermint:0.9.0", + "imagePullPolicy": "IfNotPresent", + "command": ["bash", "-c", " + set -ex\n + if [ ! -f /tendermint/priv_validator.json ]; then\n + tendermint gen_validator > /tendermint/priv_validator.json\n + # pub_key.json will be served by pub-key container\n + cat /tendermint/priv_validator.json | jq \".pub_key\" > /tendermint/pub_key.json\n + fi\n + "], + "volumeMounts": [ + {"name": "tmdir", "mountPath": "/tendermint"} + ] + }, + { + "name": "app-gen-key", + "image": "tendermint/basecoin:latest", + "imagePullPolicy": "IfNotPresent", + "command": ["bash", "-c", " + set -ex\n + if [ ! -f /app/key.json ]; then\n + basecoin key new > /app/key.json\n + # pub_key.json will be served by app-pub-key container\n + cat /app/key.json | jq \".pub_key\" > /app/pub_key.json\n + fi\n + "], + "volumeMounts": [ + {"name": "appdir", "mountPath": "/app"} + ] + }]' + spec: + containers: + - name: tm + imagePullPolicy: IfNotPresent + image: tendermint/tendermint:0.9.0 + ports: + - containerPort: 46656 + name: p2p + - containerPort: 46657 + name: rpc + env: + - name: VALIDATOR_POWER + valueFrom: + configMapKeyRef: + name: tm-config + key: validator.power + - name: VALIDATORS + valueFrom: + configMapKeyRef: + name: tm-config + key: validators + - name: TMROOT + value: /tendermint + command: + - bash + - "-c" + - | + set -ex + + # copy template + cp /etc/tendermint/genesis.json /tendermint/genesis.json + + # fill genesis file with validators + IFS=',' read -ra VALS_ARR <<< "$VALIDATORS" + fqdn_suffix=$(echo $(hostname -f) | sed 's#[^.]*\.\(\)#\1#') + for v in "${VALS_ARR[@]}"; do + # wait until validator generates priv/pub key pair + set +e + + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + while [ "$ERR" != 0 ]; do + sleep 5 + curl -s "http://$v.$fqdn_suffix/pub_key.json" > /dev/null + ERR=$? + done + set -e + + # add validator to genesis file along with its pub_key + curl -s "http://$v.$fqdn_suffix/pub_key.json" | jq ". as \$k | {pub_key: \$k, amount: $VALIDATOR_POWER, name: \"$v\"}" > pub_validator.json + cat /tendermint/genesis.json | jq ".validators |= .+ [$(cat pub_validator.json)]" > /tendermint/genesis.json + rm pub_validator.json + done + + tendermint node --moniker="`hostname`" --proxy_app="unix:///socks/app.sock" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - mountPath: /etc/tendermint/genesis.json + name: tmconfigdir + subPath: genesis.json + - name: socksdir + mountPath: /socks + + - name: app + imagePullPolicy: IfNotPresent + image: tendermint/basecoin:latest + workingDir: /app + command: + - bash + - "-c" + - | + set -ex + + # replace "tm-N" with public keys in genesis file + cp /etc/app/genesis.json genesis.json + fqdn_suffix=$(echo $(hostname -f) | sed 's#[^.]*\.\(\)#\1#') + # for every "base/account" + i=3 + length=$(cat genesis.json | jq ". | length") + while [ $i -lt $length ]; do + # extract pod name ("tm-0") + pod=$(cat genesis.json | jq -r ".[$i].pub_key[0]") + + # wait until pod starts to serve its pub_key + set +e + + curl -s "http://$pod.$fqdn_suffix/app_pub_key.json" > /dev/null + ERR=$? + while [ "$ERR" != 0 ]; do + sleep 5 + curl -s "http://$pod.$fqdn_suffix/app_pub_key.json" > /dev/null + ERR=$? + done + set -e + + # get its pub_key + curl -s "http://$pod.$fqdn_suffix/app_pub_key.json" | jq "." > k.json + + # replace pod name with it (["tm-0"] => "[1, XXXXXXXXXXXXXXXXXXXX]") + cat genesis.json | jq ".[$i].pub_key = $(cat k.json | jq '.')" > genesis.json + rm -f k.json + + i=$((i+2)) # skip "base/account" field itself + done + + rm -f /socks/app.sock # remove old socket + + basecoin start --address="unix:///socks/app.sock" + volumeMounts: + - name: appdir + mountPath: /app + - mountPath: /etc/app/genesis.json + name: appconfigdir + subPath: genesis.json + - name: socksdir + mountPath: /socks + + - name: pub-key + imagePullPolicy: IfNotPresent + image: nginx:latest + ports: + - containerPort: 80 + name: pub-key + command: + - bash + - "-c" + - | + set -ex + # fixes 403 Permission Denied (open() "/tendermint/pub_key.json" failed (13: Permission denied)) + # => we cannot serve from /tendermint, so we copy the file + mkdir -p /usr/share/nginx + cp /tendermint/pub_key.json /usr/share/nginx/pub_key.json + cp /app/pub_key.json /usr/share/nginx/app_pub_key.json + nginx -g "daemon off;" + volumeMounts: + - name: tmdir + mountPath: /tendermint + - name: appdir + mountPath: /app + - mountPath: /etc/nginx/conf.d/pub_key.conf + name: tmconfigdir + subPath: pub_key_nginx.conf + + volumes: + - name: tmconfigdir + configMap: + name: tm-config + - name: appconfigdir + configMap: + name: app-config + - name: socksdir + emptyDir: {} + + volumeClaimTemplates: + - metadata: + name: tmdir + annotations: + volume.alpha.kubernetes.io/storage-class: anything + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 2Gi + - metadata: + name: appdir + annotations: + volume.alpha.kubernetes.io/storage-class: anything + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 12Mi diff --git a/mintnet-kubernetes/img/gce1.png b/mintnet-kubernetes/img/gce1.png new file mode 100644 index 000000000..3bf3ad005 Binary files /dev/null and b/mintnet-kubernetes/img/gce1.png differ diff --git a/mintnet-kubernetes/img/gce2.png b/mintnet-kubernetes/img/gce2.png new file mode 100644 index 000000000..358dcc04b Binary files /dev/null and b/mintnet-kubernetes/img/gce2.png differ diff --git a/mintnet-kubernetes/img/statefulset.png b/mintnet-kubernetes/img/statefulset.png new file mode 100644 index 000000000..ac68d22b7 Binary files /dev/null and b/mintnet-kubernetes/img/statefulset.png differ diff --git a/mintnet-kubernetes/img/t_plus_k.png b/mintnet-kubernetes/img/t_plus_k.png new file mode 100644 index 000000000..bee9fe56e Binary files /dev/null and b/mintnet-kubernetes/img/t_plus_k.png differ