You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

290 lines
8.7 KiB

7 years ago
  1. Using Kubernetes
  2. ================
  3. .. figure:: assets/t_plus_k.png
  4. :alt: Tendermint plus Kubernetes
  5. Tendermint plus Kubernetes
  6. This should primarily be used for testing purposes or for
  7. tightly-defined chains operated by a single stakeholder (see `the
  8. security precautions <#security>`__). If your desire is to launch an
  9. application with many stakeholders, consider using our set of Ansible
  10. scripts.
  11. Quick Start
  12. -----------
  13. For either platform, see the `requirements <https://github.com/kubernetes/minikube#requirements>`__
  14. MacOS
  15. ^^^^^
  16. ::
  17. curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/kubectl
  18. curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.18.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
  19. minikube start
  20. git clone https://github.com/tendermint/tools.git && cd tools/mintnet-kubernetes/examples/basecoin && make create
  21. Linux
  22. ^^^^^
  23. ::
  24. curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/kubectl
  25. curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.18.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
  26. minikube start
  27. git clone https://github.com/tendermint/tools.git && cd tools/mintnet-kubernetes/examples/basecoin && make create
  28. Verify it worked
  29. ~~~~~~~~~~~~~~~~
  30. **Using a shell:**
  31. First wait until all the pods are ``Running``:
  32. ``kubectl get pods -w -o wide -L tm``
  33. then query the Tendermint app logs from the first pod:
  34. ``kubectl logs -c tm -f tm-0``
  35. finally, use our `Rest API <../specification/rpc.html>`__ to fetch the status of the second pod's Tendermint app.
  36. Note we are using ``kubectl exec`` because pods are not exposed (and should not be) to the
  37. outer network:
  38. ``kubectl exec -c tm tm-0 -- curl -s http://tm-1.basecoin:46657/status | json_pp``
  39. **Using the dashboard:**
  40. ::
  41. minikube dashboard
  42. Clean up
  43. ~~~~~~~~
  44. ::
  45. make destroy
  46. Usage
  47. -----
  48. Setup a Kubernetes cluster
  49. ^^^^^^^^^^^^^^^^^^^^^^^^^^
  50. - locally using `Minikube <https://github.com/kubernetes/minikube>`__
  51. - on GCE with a single click in the web UI
  52. - on AWS using `Kubernetes
  53. Operations <https://github.com/kubernetes/kops/blob/master/docs/aws.md>`__
  54. - on Linux machines (Digital Ocean) using
  55. `kubeadm <https://kubernetes.io/docs/getting-started-guides/kubeadm/>`__
  56. - on AWS, Azure, GCE or bare metal using `Kargo
  57. (Ansible) <https://kubernetes.io/docs/getting-started-guides/kargo/>`__
  58. Please refer to `the official
  59. documentation <https://kubernetes.io/docs/getting-started-guides/>`__
  60. for overview and comparison of different options.
  61. Kubernetes on Digital Ocean
  62. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  63. Available options:
  64. - `kubeadm (alpha) <https://kubernetes.io/docs/getting-started-guides/kubeadm/>`__
  65. - `kargo <https://kubernetes.io/docs/getting-started-guides/kargo/>`__
  66. - `rancher <http://rancher.com/>`__
  67. - `terraform <https://github.com/hermanjunge/kubernetes-digitalocean-terraform>`__
  68. As you can see, there is no single tool for creating a cluster on DO.
  69. Therefore, choose the one you know and comfortable working with. If you know
  70. and used `terraform <https://www.terraform.io/>`__ before, then choose it. If you
  71. know Ansible, then pick kargo. If none of these seem familiar to you, go with
  72. ``kubeadm``. Rancher is a beautiful UI for deploying and managing containers in
  73. production.
  74. Kubernetes on Google Cloud Engine
  75. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  76. Review the `Official Documentation <https://kubernetes.io/docs/getting-started-guides/gce/>`__ for Kubernetes on Google Compute
  77. Engine.
  78. **Create a cluster**
  79. The recommended way is to use `Google Container
  80. Engine <https://cloud.google.com/container-engine/>`__. You should be able
  81. to create a fully fledged cluster with just a few clicks.
  82. **Connect to it**
  83. Install ``gcloud`` as a part of `Google Cloud SDK <https://cloud.google.com/sdk/>`__.
  84. Make sure you have credentials for GCloud by running ``gcloud auth login``.
  85. In order to make API calls against GCE, you must also run ``gcloud auth
  86. application-default login``.
  87. Press ``Connect``:
  88. .. figure:: assets/gce1.png
  89. and execute the first command in your shell. Then start a proxy by
  90. executing ``kubectl` proxy``.
  91. .. figure:: assets/gce2.png
  92. Now you should be able to run ``kubectl`` command to create resources, get
  93. resource info, logs, etc.
  94. **Make sure you have Kubernetes >= 1.5, because you will be using
  95. StatefulSets, which is a beta feature in 1.5.**
  96. Create a configuration file
  97. ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  98. Download a template:
  99. ::
  100. curl -Lo app.yaml https://github.com/tendermint/tools/raw/master/mintnet-kubernetes/app.template.yaml
  101. Open ``app.yaml`` in your favorite editor and configure your app
  102. container (navigate to ``- name: app``). Kubernetes DSL (Domain Specific
  103. Language) is very simple, so it should be easy. You will need to set
  104. Docker image, command and/or run arguments. Replace variables prefixed
  105. with ``YOUR_APP`` with corresponding values. Set genesis time to now and
  106. preferable chain ID in ConfigMap.
  107. Please note if you are changing ``replicas`` number, do not forget to
  108. update ``validators`` set in ConfigMap. You will be able to scale the
  109. cluster up or down later, but new pods (nodes) won't become validators
  110. automatically.
  111. Deploy your application
  112. ^^^^^^^^^^^^^^^^^^^^^^^
  113. ::
  114. kubectl create -f ./app.yaml
  115. Observe your cluster
  116. ^^^^^^^^^^^^^^^^^^^^
  117. `web UI <https://github.com/kubernetes/dashboard>`__
  118. The easiest way to access Dashboard is to use ``kubectl``. Run the following
  119. command in your desktop environment:
  120. ::
  121. kubectl proxy
  122. ``kubectl`` will handle authentication with apiserver and make Dashboard
  123. available at http://localhost:8001/ui
  124. **shell**
  125. List all the pods:
  126. ::
  127. kubectl get pods -o wide -L tm
  128. StatefulSet details:
  129. ::
  130. kubectl describe statefulsets tm
  131. First pod details:
  132. ::
  133. kubectl describe pod tm-0
  134. Tendermint app logs from the first pod:
  135. ::
  136. kubectl logs tm-0 -c tm -f
  137. App logs from the first pod:
  138. ::
  139. kubectl logs tm-0 -c app -f
  140. Status of the second pod's Tendermint app:
  141. ::
  142. kubectl exec -c tm tm-0 -- curl -s http://tm-1.<YOUR_APP_NAME>:46657/status | json_pp
  143. Security
  144. --------
  145. Due to the nature of Kubernetes, where you typically have a single
  146. master, the master could be a SPOF (Single Point Of Failure). Therefore,
  147. you need to make sure only authorized people can access it. And these
  148. people themselves had taken basic measures in order not to get hacked.
  149. These are the best practices:
  150. - all access to the master is over TLS
  151. - access to the API Server is X.509 certificate or token based
  152. - etcd is not exposed directly to the cluster
  153. - ensure that images are free of vulnerabilities
  154. (`1 <https://github.com/coreos/clair>`__)
  155. - ensure that only authorized images are used in your environment
  156. - disable direct access to Kubernetes nodes (no SSH)
  157. - define resource quota
  158. Resources:
  159. - https://kubernetes.io/docs/admin/accessing-the-api/
  160. - http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html
  161. - https://blog.openshift.com/securing-kubernetes/
  162. Fault tolerance
  163. ---------------
  164. Having a single master (API server) is a bad thing also because if
  165. something happens to it, you risk being left without an access to the
  166. application.
  167. To avoid that you can `run Kubernetes in multiple
  168. zones <https://kubernetes.io/docs/admin/multiple-zones/>`__, each zone
  169. running an `API
  170. server <https://kubernetes.io/docs/admin/high-availability/>`__ and load
  171. balance requests between them. Do not forget to make sure only one
  172. instance of scheduler and controller-manager are running at once.
  173. Running in multiple zones is a lightweight version of a broader `Cluster
  174. Federation feature <https://kubernetes.io/docs/admin/federation/>`__.
  175. Federated deployments could span across multiple regions (not zones). We
  176. haven't tried this feature yet, so any feedback is highly appreciated!
  177. Especially, related to additional latency and cost of exchanging data
  178. between the regions.
  179. Resources:
  180. - https://kubernetes.io/docs/admin/high-availability/
  181. Starting process
  182. ----------------
  183. .. figure:: assets/statefulset.png
  184. :alt: StatefulSet
  185. StatefulSet
  186. Init containers (``tm-gen-validator``) are run before all other
  187. containers, creating public-private key pair for each pod. Every ``tm``
  188. container then asks other pods for their public keys, which are served
  189. with nginx (``pub-key`` container). When ``tm`` container have all the
  190. keys, it forms a genesis file and starts the Tendermint process.